Adjusting for recalled past vote in political polling · ↗ abacus-weighting.com
The founder of Abacus Data, a Canadian polling firm, dropped kind of an interesting URL yesterday: abacus-weighting.com. It’s a advertisement in the form of a case study on why Abacus weights their political polls on past vote. It fits perfectly with the theme of yesterday’s post on how pollster’s get different results from the same data (the answer is they weight the raw data differently).
If you follow Nate Silver (or American political polling in general), you probably know that pollsters undercounted Trump support in all three elections where he was on the ballot. What I learned from this post is that support for the Conservative Party of Canada has been underestimated in their firm’s polling data in every polling wave for every election since 2011:
In every single wave, across every single election cycle, Conservative voters are underrepresented in our demographically weighted sample relative to their actual share of the vote. Not in most waves. Not in some elections. In every case we can observe.
Weighting for recalled past vote improves the estimate in every case, sometimes dramatically so:
In every election, past vote weighting moved our Conservative estimates upward and our Liberal estimates downward — consistently in the direction of the actual result. The 2021 election shows the most dramatic correction: a 7-point improvement in our Conservative estimate.
…
How do pollsters get different results from the same data? · ↗ www.nytimes.com
Nate Silver linked to this throwback article from 2016 in The New York Times in his recent article on fake AI polls, which I also wrote about a few days ago. The article, entitled “We Gave Four Good Pollsters the Same Raw Data. They Had Four Different Results.” is a good reminder that modern polling diverges very far from the theoretical ideal of a simple random sample. Even after deciding on a methodology to sample participants and collecting the data, a lot of work goes into interpreting raw poll responses to give us top-line polling numbers. Every pollster needs to figure out how to weight the responses they get, since poll response rates are abysmal and variable across different demographic groups. As in the example given in this piece, these choices can result in large differences in those top-line numbers: from +4 Clinton to +1 Trump, all from the same raw data!
For an interesting follow-up: “Polling is becoming more of an art than a science”, also on Nate Silver’s Substack.
Scientists invent a fake disease, AI picks it up, other scientists cite it · ↗ www.nature.com
A somewhat disturbing bit of reporting from Nature tells the story of bixonimania, a fake eye disease invented by Swedish medical researcher Almira Osmanovic Thunström and her team. She seeded the idea for the fake disease in a series of ridiculous, joke-filled blog posts and preprints in mid-2024.
Because AI can be overly credulous with its sourcing (how often do Google’s AI answers confident cite random Reddit posts for the bulk of an answer?), the disease got picked up as an “emerging term” by the leading chatbots. The preprints even got cited a handful of times in real publications, which is further evidence that scientists don’t read the papers they cite (I guess the modern equivalent of copying citations from other papers is having AI dredge the literature for you).
I can see AI agents being exploited by those pushing dubious medical diagnoses to flood the Internet and preprint servers with articles aimed at convincing LLMs of the validity of their positions. That is if the agents aren’t too busy spinning of websites to defame those who incur their wrath.
A data point against the idea that AI will freeze/homogenize culture · ↗ arxiv.org
Here’s an interesting figure and accompanying passage from this 2023 preprint entitled “Machine Culture”:
The innovations generated by AlphaGo and AlphaGo Zero soon entered human culture, as shown by research comparing human gameplay before and after the algorithms’ introduction. The decision quality, as measured by an open-source variant of AlphaGo Zero, showed very little improvement in human gameplay from 1950 to 2016, followed by a sudden improvement after the introduction of AlphaGo in March 2016. However, this improvement wasn’t solely due to humans adopting strategies developed by AlphaGo. It also reflected an unexpected shift, wherein humans started developing moves that were qualitatively distinct both from previous human moves and from the novel moves introduced by AlphaGo. In summary, AlphaGo served as an early, quantifiable exemplar of machine culture, generating novel cultural variations through genuine, nonhuman innovation. This was followed by a major transition into an even broader range of traits as the result of humans building on the previous discoveries made by machines. As the methods underpinning AlphaGo have been generalized to other games and extended to scientific problems, we anticipate a continued infusion of machine-generated discoveries across diverse domains of human culture.
…
AI makes it easier to generate fake papers, too · ↗ tylervigen.com
Here’s a fun project from Tyler Vigen, creator of the famous Spurious Correlations page (which has been cited as a cautionary tale in many a science class). Using his database of real but spurious correlations (created by calculating the Pearson correlation coefficient r between a very large number of variables and picking out the hits), he used AI to create amusing fake manuscripts expounding on these statistical flukes as if they were real research questions.
These papers were generated in January 2024, and as previously discussed on this blog, the pipeline for end-to-end paper generation has come a long way in two years. I have no doubt Tyler could make these paper’s sound much more convincing using today’s models, though of course his goal here is to make you laugh (and think), not to trick you. But I have no doubt there will be many scholars adopting this data dredging strategy to generate “real” papers, contributing to a deluge of papers flooding the academic publishing system.
What is a public opinion poll without the public? · ↗ www.nytimes.com
A few days ago, two professors (Leif Weatherby and Benjamin Recht) published an opinion piece in the New York Times calling attention to Axios publishing a story on maternal health using invented polling results:
A recent Axios story on maternal health policy referred to “findings” that a majority of people trusted their doctors and nurses. On the surface, there’s nothing unusual about that. What wasn’t originally mentioned, however, was that these findings were made up.
Clicking through the links revealed (as did a subsequent editor’s note and clarification by Axios) that the public opinion poll was a computer simulation run by the artificial intelligence start-up Aaru. No people were involved in the creation of these opinions.
The piece goes on to argue that this so-called “silicon sampling” is seductive because good public opinion polling is expensive, hard to do, and still prone to bias. But this shortcut magnifies the the problem of bias rather than solving it.
I’ve read a little bit about this strategy of using LLM-generated survey participants in the context of social science research in a series of posts (mostly from Prof. Jessica Hullman) over on Andrew Gelman’s blog:
- Validating language models as study participants: How it’s being done, why it fails, and what works instead (2025-12-19)
- Survey Statistics: Thomas Lumley writes about Interviewing your Laptop (2025-08-26)
- When does it make sense to talk about LLMs having beliefs? (2025-08-15)
- Better and worse ways to mix human and LLM responses in behavioral research (but you still have to figure what you’re measuring) (2025-06-12)
- LLMs as behavioral study participants (2025-05-29)
Silicon sampling seems moderately interesting from a research perspective, but I can’t help but agree with the New York Times opinion piece authors that this will be ruinous for the already waning trust in public opinion polling. If you didn’t bother to ask the public, then why should the public care what you “find”? I think there is probably a lot of utility in using LLM samples to aid in designing and validating surveys, though.
Social media is a freak show · ↗ www.natesilver.net
I quite enjoyed Nate Silver’s recent Substack post “Social media has become a freak show” (curiously, the title element of the page is “Social media is turning into a freak show”—I think the transformation has already occurred).
Nate Silver is still a Twitter power user, and yet even he acknowledges the increasing uselessness of Twitter for driving traffic to his newsletter or even just providing a forum for thoughtful engagement. I myself abandoned the platform a few years ago, having seen the direction it was heading under Elon Musk. My impression is that the utility of Twitter in most domains is asymptotically approaching zero, with a handful of exceptions (I will occasionally lurk for AI news, as the discussion is still robust, if polluted with a ton of low-quality bot or bot-like replies).
The rest of the social media ecosystem isn’t much better. Bluesky has declining engagement, probably because it has replicated Twitter’s old schoolyard dynamics on steroids. Facebook hasn’t been relevant for years, and I have no idea what it’s even for anymore if not connecting with your friends (I haven’t had an account in many years). Instagram might still be fun, though I have no idea because I’ve never used it. But it’s certainly not a place where “the discourse” happens.
…How effective are Amber alerts? · ↗ www.mcgill.ca
A few weeks ago, I experienced a situation familiar to many Canadians, described in this article from Jonathan Jarry of McGill University’s Office for Science and Society:
On Sunday, March 22nd of this year, a large swath of the population in Quebec was woken up at 4:25 as cell phones lit up and screamed. An Amber alert had been broadcast. Less than four hours later, the two missing children were thankfully found, unharmed, and the alert was cancelled.
Thankfully, my iPhone respects silent mode and only vibrated forcefully, but apparently not all phone brands respect this setting. Unlike in the United States, Amber alerts to cell phones in Canada cannot be disabled.
The statistics regarding child abductions and Amber alerts discussed in this article are equal parts comforting and disconcerting. For example, most children who are the subject of an Amber alert are recovered unharmed:
a study published a decade ago and looking at 448 Amber alerts in the U.S. revealed that over 95% of the children had been recovered alive and nearly 90% recovered alive and without physical harm, sexual abuse, of withholding of needed medical care during the abduction. Even when Amber alerts don’t trigger a helpful tip, the child is usually found.
Other research from the United States indicates the Amber alert plays a part in the recovery about 25% of the time. However, they may be issued too late to prevent the worst outcomes:
…The definition of "agent"
An interesting exchange between Guido van Rossum and Andrej Karpathy a few days ago on Twitter:
Guido van Rossum: I think I finally understand what an agent is. It’s a prompt (or several), skills, and tools. Did I get this right?
Andrej Karpathy: LLM = CPU (data: tokens not bytes, dynamics: statistical and vague not deterministic and precise) Agent = operating system kernel
The triumph of the data raccoons
My PhD co-supervisor at the University of Toronto, Dr. David Fisman, liked to use the term “data raccoon” to describe the work of using messy, incomplete, hard-to-work-with data to do serious research. Or, as he described it in testimony to the Canadian House of Commons in May 2020 (emphasis mine):
I’ll tell you, my group at University of Toronto call ourselves “data raccoons”, because we’ve sort of managed to thrive for about 15 years on data that most people regard as garbage, so it’s sort of a bit of the normal state of affairs for us with public health data analysis.
It’s an unmistakably Toronto metaphor—the city isn’t called the raccoon capital of the world for nothing!
It occurred to me recently that data raccoons have basically taken over the world. The basis of the AI revolution is vast quantities of text dredged from the Internet, none of which was written for its final purpose of training the deus ex machina. Arguably the most important dataset for training LLMs has been Common Crawl, a mostly uncurated snapshot of the Internet that has been running since 2007. According to a Mozilla report from 2024, Common Crawl was used in two thirds of LLMs developed in the formative period between 2019 and 2023, and the archive also comprised 80% of tokens in OpenAI’s GPT-3. Unsurprisingly, the Common Crawl Foundation has received financial support from AI companies in recent years, all the while being accused of abetting these same companies to train their models on paywalled articles.
…Andrew Gelman's blog schedule
Andrew Gelman, professor of statistics at Columbia University, runs one of my favourite blogs on the Internet. He has been writing there for over 21 years, since October 2004. Many of his collaborators also contribute to the blog, but he is the primary author. In a 2024 post celebrating 20 years of blogging, Gelman mentions having over 12,000 posts. This is a cadence of over 1.6 posts/day sustained for two decades!
One of the more unusual things about Gelman’s blog is that most posts are not particularly topical. Sure, many posts are time-sensitive, posting about upcoming events or commenting on recent publications (like doing damage control on deeply flawed papers like to receive attention). But there is generally one non-topical post each day. A line in a recent post caught my eye:
As regular readers know, our posts are usually on a 6-month lag, but this one is so important I had to share it with you right away.
As a regular reader myself, I was aware of the delayed posting schedule, but out of curiosity, I wanted to see how far back this habit went. Here’s the rough timeline I came up with:
- In 2011, Gelman wrote that his “non-topical blog entries are on approximately one-month delay”.
- In 2012, he referred to “stacking up posts here with a roughly one-month delay”.
- In 2014, he said that “most of the posts here are on a 1 or 2 month delay.”
- In 2016, he casually mentioned “our 2-month delay”.
- Later that year (August 2016), in a post literally titled “My next 170 blog posts”, he said he had filled “the blog through mid-January” and had “170 blog posts in the queue.”
- By 2018, he mentioned the blog was “mostly on a six-month delay”.
- In 2019, he referred to “our 6-month blog delay.”
- In 2022, he wrote: “Usually I schedule these with a 6-month lag, but this time I’m posting right away”.
- In February 2026, he said the “current end of the blog queue is in early July”.
- Then, in April 2026, came the latest “usually on a 6-month lag” remark.
It seems the blog had about one month of content in the publishing pipeline by 2011, ramped up to one to two months by 2014, two months by early 2016, and finally jumped to six months by August 2016, where it been ever since. Quite the arsenal of scheduled content!
…Testing ZeroClaw, Part 2.5: ZeroClaw is alive!
Yesterday, I wrote about how the ZeroClaw GitHub repository had been down for two days with little explanation. Earlier today, the project provided a little more information on Twitter:
They flagged our org which is why we’re down. Code is safe and we’re still working, just waiting for @github
Since March 30 (the day after their repo started 404ing), they project has been promising a blog post to explain the situation. As of now, that post is now available:
Over the past few days, a maintainer used aggressive AI automation to review and merge PRs:
- Merges went through that shouldn’t have.
- In the process of trying to undo the damage, the maintainer’s GitHub account was flagged, which triggered enforcement actions on the ZeroClaw org itself.
- That maintainer has been removed from the project.
This sounds strikingly similar to the incident that occurred about a month ago, which I also mentioned in yesterday’s post:
Earlier today, during routine maintenance, the visibility of the `zeroclaw-labs/zeroclaw` repository was accidentally changed from public to private and was later restored to public.
After reviewing the GitHub API audit logs and collecting detailed feedback from our engineers, we confirmed that the incident was caused by improper use of an AI agent tool during maintenance.
Obviously, the use agentic workflows in open source projects is an emerging field where best practices have not yet been established. The case of ZeroClaw should be a warning to other projects to keep human review in the loop, or at least to limit the autonomy of agents when a project has numerous contributors. As they say in their blog post:
…Testing ZeroClaw, Part 2: ZeroClaw is dead?
Earlier this month, I wrote about setting up one of the many lightweight OpenClaw alternatives, namely ZeroClaw. I had some issues with initial setup, but I got to the point where I could talk with my bot over Telegram.
Some of my initial enthusiasm for ZeroClaw was dampened by the divergence between the docs and the features available in the release build. The release build was quite out of date due to the breakneck pace of development. In the week or two following my initial setup, the release build pipeline was broken, so even when they released a new tag, there were no new precompiled binaries available. Being forced to compile the Rust binary yourself kind of goes against the project’s philosophy of ultra-low resource consumption.
They eventually fixed the release pipeline and I started casually working on a system where I could send notes and ideas for blog posts to my bot through Telegram and have it turn them into structured Markdown files.
But two days ago (March 29), I noticed that the ZeroClaw GitHub repo was 404ing. On the same day, the project posted the following on Twitter:
Our GitHub repo is currently returning a 404 for some users. We’re aware and actively investigating. The repo is public and all code is safe.
…
One important fact about for-profit plasma donation
For-profit plasma donation is in the news today in Canada. Two people recently died after giving plasma at Grifols for-profit plasma clinics in Winnipeg, Manitoba, although Health Canada has yet to find a link between the plasma collections and the deaths. Today, it was reported that a Grifols clinic in Calgary, Alberta was found non-compliant during an inspection in December 2025:
The inspection found the Calgary centre didn’t accurately assess whether donors were suitable, didn’t collect blood according to its Health Canada authorization, didn’t thoroughly investigate errors and accidents, and didn’t carry out sufficient corrective and preventative actions.
This is obviously a problem for for-profit plasma collection in Canada, where the practice is already controversial. Paid plasma collection is illegal in Canada’s three largest provinces: Ontario, British Colombia, and Quebec, though Ontario allows a few for-profit clinics to operate through an agreement with Canadian Blood Services, Canada’s independent blood authority. British Colombia and Quebec together make up over 35% of Canada’s population; including Ontario, it’s nearly 80%. Besides Ontario, for-profit clinics exist in some other smaller provinces.
Vocal advocacy exists against paid plasma collection, leading to municipal resolutions against the practice in Ontario, even as clinics open. This advocacy is often premised on the fear that paid plasma will undermine voluntary donations. To my mind, the central fact in the for-profit plasma collection debate is that only a handful of countries are self-sufficient in plasma collection, and all of them allow for paid plasma collection. They are: the United States, Germany, Czechia, Austria, and Hungary (Egypt may have also recently joined the list). While other countries, like Canada, may have achieved self-sufficiency for plasma for direct infusion, no other country can meet its own needs for plasma-derived medical products. The world relies on a small number of self-sufficient countries, primarily the United States, to meet the demand for plasma products.
…How to avoid cognitive surrender to AI · ↗ alexpanetta.substack.com
I am sharing a thoughtful article today from Alex Panetta’s A.I. For You on avoiding over-reliance on AI: “cognitive debt”, “epistemic debt”, or “cognitive surrender”.
A particularly interesting nugget regarding the “Your Brain on ChatGPT” article from the MIT Media Lab (yes, that MIT Media Lab):
The paper is even written to get LLMs to read it carefully. The paper carries instructions telling LLMs which section to read first, which appears to be a clever way to force relevant context atop the context window, as LLMs tend to best remember the beginning and end of conversations — not the middle.
Opt out of very new Python package versions with uv
In light of several recent Python package compromises (litellm, telnyx), here is a useful tip from Hacker News commenter mil22:
For those using uv, you can at least partially protect yourself against such attacks by adding this to your pyproject.toml:
[tool.uv]
exclude-newer = "7 days"or this to your ~/.config/uv/uv.toml:
exclude-newer = "7 days"This will prevent uv picking up any package version released within the last 7 days, hopefully allowing enough time for the community to detect any malware and yank the package version before you install it.
Commenter notatallshaw follows up with how to achieve similar behaviour in *pip*:
Pip maintainer here, to do this in pip (26.0+) now you have to manually calculate the date, e.g. –uploaded-prior-to="$(date -u -d ‘3 days ago’ ‘+%Y-%m-%dT%H:%M:%SZ’)"
In pip 26.1 (release scheduled for April 2026), it will support the day ISO-8601 duration format, which uv also supports, so you will be able to do –uploaded-prior-to=P3D, or via env vars or config files, as all pip options can be set in either.
Colorado advances ban on algorothmic price and wage discrimination · ↗ coloradonewsline.com
The Colorado House voted today to ban the use of personal data to algorithmically set the price of a product or determine a wage. The legislation will now advance to the Colorado Senate for consideration. The summary of the bill, HB26-1210, reads:
Surveillance data is defined in the bill as data that is obtained through observation, inference, or surveillance of consumers or workers and that is related to personal characteristics, behaviors, or biometrics of an individual or group. The bill prohibits discrimination against a consumer or worker through the use of automated decision systems used to engage in:
- Individualized price setting based on surveillance data regarding a consumer; or
- Individualized wage setting based on surveillance data regarding a worker.
Obviously, the bill enumerates exceptions to the above rules, as it is not intended to ban, for example, charging a customer more to deliver an item a longer distance nor to prohibit schemes like discounts for students or seniors. One of the challenges of writing laws like this is to ensure they are written narrowly enough to target dystopian hyper-individualized pricing based on tracking of Internet and phone activity rather than normal business practices like pricing insurance policies according to demographic risk factors.
Colorado is one of at least a dozen American states considering similar bans. I don’t believe any of these proposed broad-based bans have been signed into law yet. I wrote about algorithmic price discrimination (surveillance pricing) last week in the context of proposed legislation in the Canadian province of Manitoba.
How SARS-CoV-2 variants get named on GitHub · ↗ github.com
Bioinformatics has long been an unusually collaborative and transparent field, with genomes, protein structures, and other complex biological data habitually deposited into open databases during the course of research. The situation was no different at the outset of the COVID-19 pandemic, when a small group of scientists developed the Pango nomenclature for classifying variants of the SARS-CoV-2 virus. Outside of a handful of Greek-letter “variants of concern” names assigned by the World Health Organization, the Pango nomenclature is the standard for tracking the evolution of the SARS-CoV-2 virus. You may recall names such as B.1.1.7 (Alpha or the UK variant), B.1.351 (Beta or the South African variant), and P.1 (Gamma or the Brazilian variant). You can see a complete list of active SARS-CoV-2 lineages using the Pango nomenclature here.
By August 2020, the work of defining new lineages of SARS-CoV-2 had moved to GitHub, where the scientific process could happen in transparent and collaborative way. The definition of new lineages happens on proposals submitted as GitHub issues. In May 2023, a second GitHub repository was opened to move discussions of smaller or less clear lineages out of the main repository. These discussions can be promoted to the main repository, as this issue tracking LP.8.1 sub-lineages was in May 2025.
The work of defining new lineages of SARS-CoV-2 continues to this day on the GitHub repository, as the virus continues to mutate and evolve. And bioinformatics continues to be a shining beacon for open science for the rest of us to learn from.
Prediction markets are coming to Canada · ↗ www.theglobeandmail.com
(Archive link to this story)
Wealthsimple is a fintech company at the forefront of a lot of innovation in Canada’s personal finance sector since the company’s founding in 2014. Notably, Wealthsimple was the first broker in Canada to offer zero-commission trades, back in 2019. In 2020, they started offering the ability to trade crypto. In 2025, they launched zero-commission options trading. This year, the company received regulatory approval to bring prediction trading to Canada.
Unlike in other parts of the world, prediction markets have not flourished in Canada and have been considered basically illegal since a 2017 ruling from Canada’s federal securities regulator. Wealthsimple has been able to get around this ruling by only offering contracts on a narrow set of questions:
Despite a 2017 ruling that largely banned these kinds of short-term, yes-or-no contracts, certain regulated firms that are CIRO members are able to offer certain types of “event contracts,” […] The approval for Ontario-based Wealthsimple permits it only to offer contracts tied to economic indicators, financial markets and climate trends, the company confirmed – not sports or elections, which are among the most popular uses of prediction markets in the United States.
Wealthsimple has driven innovation in the Canadian personal finance sector; however, their new product offerings over the last few years seem to be speedrunning the Robinhood trajectory toward high-risk, high-volatility trading and away from their traditional niche of broad, diversified funds/ETFs for ordinary people to set-and-forget. This pivot can be understood as part of a broader trend toward the casinofication of everything, which took off with crypto and the legalization of online sports betting.
Will AI help Canadian police counter a tsunami of fraud? · ↗ theijf.org
Zak Vescera, writing for the Investigative Journalism Foundation, observes that fraud cases reported to Canadian police has more than doubled between 2013 and 2024:
At the same time, the number of cases cleared by Canadian police has fallen. In 2013, the ratio between reported cases and cleared cases was about 3:1; by 2024, this ratio was over 9.5:1.
The vast majority of fraud cases go unsolved. This is unsurprising given that many are perpetrated over the Internet by individuals overseas and involve methods of sending money that are difficult to recover, such as crypto, gift cards, and physical transfers of cash.
In response, the National Cybercrime Coordination Centre (NC3) of the RCMP—Canada’s national police service—have built a case management system and data portal they hope will eventually be adopted by all Canadian police forces. According to the article, this system is aimed at improving coordination, data sharing, and analysis. The platform will also host a set of AI tools, though the RCMP is vague on details and which are currently implemented. The article gives a few examples: OCR allowing victims to scan gift cards used in fraud rather than typing numbers manually, a tool to classify reports to help police target their investigative resources, and a report generator to simply data sharing when investigations go international.
…Vandalism of OpenStreetMap · ↗ en.wikipedia.org
OpenStreetMap (OSM) is an open, community-driven map database powering countless apps and services and used by organizations including Amazon, Apple, Microsoft, Uber, Mapbox, and Wikimedia. In short, it is foundational infrastructure for the web. For regions with active communities (particularly in Europe), OSM is often noted for the superiority of its data on features such as cycling routes, hiking trails, and footpaths.
The Wikipedia article for OpenStreetMap documents several instances of data vandalism, which OSM is vulnerable to as a crowdsourced project. Three incidents stood out:
- In 2012, Google fired two “rogue contractors” for vandalizing the OSM database, intentionally adding false data such as reversing the direction of one-way streets.
- In 2018, a vandal made several viciously antisemitic edits to place names around New York City. While quickly reverted at the source, these changes nonetheless propagated into downstream applications pulling data from MapBox, such as Zillow, Snapchat, Citibike, and Wikipedia.
- Users of the mobile game Pokémon GO regularly vandalize the OSM database underlying the game to gain a gameplay advantage, although the authors of the research article on this subject note this vandalism tends to be transitory rather than sustained.
Side note: I was amused to note how strong Google’s regional results bias is for “OSM”—the entire first page is taken up by results related to the Orchestre symphonique de Montréal.
Properly the work of federal public health agencies · ↗ covidtracking.com
One of the reasons I started this blog was to have a place to put down posts and articles that have lodged themselves in my brain. The wind-down announcement of the COVID Tracking Project, a volunteer-led COVID-19 data tracking collaboration, is one such article.
But the work itself—compiling, cleaning, standardizing, and making sense of COVID-19 data from 56 individual states and territories—is properly the work of federal public health agencies. Not only because these efforts are a governmental responsibility—which they are—but because federal teams have access to far more comprehensive data than we do, and can mandate compliance with at least some standards and requirements.
After one year of work, the COVID Tracking Project decided to quite collecting data on COVID-19 in the United States, because they recognized that the work of collecting a comparable, national-level dataset was the responsibility of federal government agencies.
As someone who co-led the COVID-19 Canada Open Data Working Group, which curated COVID-19 data for Canada until the end of 2023, I think about this article a lot. It’s a good read, and it speaks to how essential open data was to filling in the gaps in the national and international understanding of the COVID-19 pandemic.
For map nerds only: An atlas of world history · ↗ www.oldmapsonline.org
I am sharing today TimeMap.org: an atlas of regions, rulers, people, and battles throughout history. Thoroughly enjoyable to swipe through, especially for connoisseurs of the map game genre.
Hat tip to agilek on Hacker News.
Fight club at the bird feeder · ↗ www.allaboutbirds.org
Alternate title: Blue Jay brutally feeder mogs Tufted Titmouse
From the Cornell Lab of Ornithology, a pretty neat article about dominance hierarchies at the bird feeder using over 7,600 observations collected by citizen scientists contributing to Project Feeder Watch. Essentially, bird watchers reported instances when one bird species successfully displaced another at the bird feeder, and used this network of comparisons to build a dominance hierarchy. By using information contained within the network, you can even compare birds that are rarely observed together. Not all dominance patterns are linear, however, as the article reports:
A separate analysis uncovered some dominance triangles in which three birds had one-to-one relationships independent of each other, like a game of birdy rock-paper-scissors. For example, the House Finch dominates the Purple Finch, and the Purple Finch dominates the Dark-eyed Junco, but the junco dominates House Finch.
The full paper is here: Fighting over food unites the birds of North America in a continental dominance hierarchy.
This work is reminiscent of network meta-analysis, in which three or more interventions (e.g., drugs) are compared using both direct and indirect evidence. For example, if there are studies comparing drug A versus drug B and drug B versus drug C, we can infer the comparison between drug A and drug C, even if no study has ever directly compared them.
Make buses faster and more reliable by having fewer stops · ↗ worksinprogress.co
This fascinating article by Nithin Vejendla in Works in Progress makes the case that bus networks would benefit from bus stop balancing: having fewer stops spaced further apart. This is especially true in the United States where stops tend to be only 700–800 feet (roughly 210–240 metres) apart. While having many bus stops theoretically improves access to the transit network, it also means that buses are slower (more time is spent accelerating, decelerating, and loading/unloading passengers) and less frequent, which reduces where you can actually go in a fixed amount of time, as well increasing the variability in the time it takes to get there.
The biggest problem holding back public transit in North America is that it is unreliable, and bus stop balancing is a rare policy solution that offers improved service without having to spend more. With fewer stops, the same number of buses can complete the same route faster and with greater frequency. This stops a single missed or delayed bus from ruining your plans or forcing you to build in extra time.
A research study from my city of Montreal even gets a shout out. As a big public transit user, I avoid buses where possible in favour of the metro and walking, because these modes of transportation tend to be much more reliable and less variable when it comes to the question of “how long will it take for me to get from point A to point B”. Stop balancing could go a long way toward addressing one of the main complaints about public transit: too many routes are not frequent or reliable enough to let riders stop worrying about the schedule.
Manitoba introduces bill to ban algorithmic price discrimination · ↗ www.cbc.ca
The Canadian province of Manitoba has introduced a bill to ban algorithmic price discrimination (also known as surveillance pricing), i.e., the use of personal data to set prices for individual consumers:
New Democrats announced in December they would begin cracking down on what’s known as differential or predatory pricing. That is when retailers charge different amounts for the same products based on the timing of customer purchases, where they live or other personal data. […] The proposed legislation would render the use of “personalized algorithmic pricing,” both online or in store, an unfair business practice.
Okay, I guess there’s a lot of different names for this particular practice. Whatever we call it, I believe bills cracking down on algorithmic price discrimination will be very popular, as it constitutes a very clear example of companies using our data against us to rip us off. The most famous recent exposé of this practice is Groundwork Collaborative’s report on how grocery delivery service Instacart charges users different prices depending on who they are.
Manitoba isn’t the only jurisdiction introducing bills targeting this practice, but I don’t believe anywhere in the US or Canada has actually managed to ban it yet. However, New York has made in mandatory for companies to disclose when they use personal data to set prices.
…Prediction markets incentivize bad behaviour · ↗ www.timesofisrael.com
The Times of Israel journalist Emanuel Fabian is claiming that Polymarket gamblers (sorry, “traders”) have threatened his life over a report he released about an Iranian missile attack on Israel on March 10. According to the rules, this bet resolves as true if Iran strikes Israel using a drone, a missile, or an air strike on this date. At issue here is this specific rule:
Missiles or drones that are intercepted and surface-to-air missile strikes will not be sufficient for a “Yes” resolution, regardless of whether they land on Israeli territory or cause damage.
On March 10, Fabian reported a single missile had hit an open area outside the Israeli city of Beit Shemesh; he included in the report a video of the strike. This would resolve the bet as “Yes”. Evidently, holders of “No” shares would very much like him to change his report to say that the missile was intercepted, which would resolve the bet as “No”, according referenced above. This bet has seen more than 23 million USD in trading volume.
If you look at the vitriol in the comments of the bet on Polymarket, I have no trouble believing people would send threats to a journalist demanding him to change his story, whether out of desperation to change their fortunes or just in an attempt to be edgy.
…Some insight into writing a book using Quarto · ↗ kieranhealy.org
Prof. Kieran Healy (Sociology, Yale University) shares some nice insight into the process of writing a book in Quarto using R in this post. The output screenshots he shares look beautiful, and the idea of deploying the same content as a clean PDF and a responsive website is awesome. A full draft of the book, Data Visualization: A Practical Introduction (Second Edition), is available as a website here.
I have grown increasingly tired of writing in any format other than a plain text file I can easily version control and move around, so the idea of writing a book in Quarto is appealing to me (as long as it has enough technical content to justify the format).
Using Claude Claude for cross-package statistical audits · ↗ causalinf.substack.com
Economist Scott Cunningham shared an important example of why we should always report the statistical package and version used in our analyses, as he used Claude Code to produce six versions of the exact same analysis using six different packages in R, Python, and Stata. In a difference-in-differences analysis of the mental health hospital closures on homicide using the standard Callaway and Sant’Anna estimator (for DiD with multiple time periods), he got very different results for some model specifications.
Since the specifications and the data were identical between packages, he discovered the divergences occurred due to how the packages handled problems with propensity score weights. Packages were not necessarily transparent about issues with these weights. If you were not running multiple analyses and comparing results across packages, or else carefully checking propensity score diagnostics, you might never have realized how precarious your results were.
Prof. Cunningham closes with the following advice:
The fifth point, and the broader point, is that this kind of cross-package, cross-language audit is exactly what Claude Code should be used for. Why? Because this is a task that is time-intensive, high-value, and brutally easy to get wrong. But just one mismatched diagnostic across languages invalidates the entire comparison, even something as simple as sample size values differing across specifications, would flag it. This is both easy and not easy — but it is not the work humans should be doing by hand given how easy it would be to even get that much wrong.
…
Getting citizenship just got a lot harder for those of Italian descent · ↗ www.cnn.com
Many people in the Americas would probably be surprised to learn that, in much of the rest of the world, being born in a country does not by itself make you a citizen. In most of the Americas, citizenship is automatically granted on the basis of jus soli (“right of soil”): birth on the territory. Elsewhere, citizenship is more often based on jus sanguinis (“right of blood”): descent. This is the case in most of the EU.
Citizenship in an EU country is considered unusually desirable because of the mobility rights and powerful passport it confers. However, the rules concerning exactly what kind of descent confers citizenship varies widely among member states. Italy used to be considered among the easiest, requiring only that an applicant prove they had an Italian ancestor alive after March 17, 1861, when the Kingdom of Italy was founded. That changed last year, when the country passed a new law significantly tightening the requirements for citizenship, which was recently upheld by the country’s Constitutional Court. The new law brings requirements more in line with norm among EU member states:
Now, only individuals with at least one parent or grandparent born in Italy will automatically qualify for citizenship by descent. The amended law does not affect the 60,000 applications currently pending review. Additionally, dual nationals risk losing their Italian citizenship if they “don’t engage” by paying taxes, voting or renewing their passports.
…