Guest Post: "Reproduceablity-enhancing" measures

Chill writes: Recently there's been a Chronicle of Higher Ed article circulating in my circles. The title says it all: "This study was hailed as a win for science reform. Now it's being retracted."

But the Chronicle article is paywalled and long and there is a lot of journalistic he-said-she-said, so while it's a good summary, I wanted to see what Andrew Gelman's (statistics, science, and open science) blog was going to say about it. Turns out various authors on that blog have been on the case for more than a year! It just took a long time for it to break through to broader consciousness. And you can hear the frustration by at least one author of a blog post that the flaws in this lauded article were not being talked about, probably partly because open science has its heroes and those heroes were publishing a "yay open science works" article that was full of problems. So maybe a lot of people in the open science movement were not that open when it was one of their own messing up.

Anyway, the story has all kinds of quirks - an origin story in an apparently well-funded effort to prove something that is frankly supernatural (that if you study something with science it gets smaller. Explicitly NOT that your first result might have been biased and you approach truth with further replications, but that the effect is really declining because you are studying it! That original goal got smothered out of the final article, only mentioned in supplemental materials). It's got skeptical open science person (a Data Colada guy, who was sued by an academic fraudster for exposing her fraud) as a coauthor on the misleading paper. Another coauthor is the Executive Director of the Center for Open Science. It's hubris and comeuppance. There's even a hero, Tal Yarkoni, who reviewed the manuscript for Nature and clearly laid out the problems. Kevin Munger, in a substack post I otherwise found kind of tough going, said "Yarkoni is a both a personal and intellectual hero of mine -- he bravely took his own advice outlined at the end of the article, to do something else after realizing the limitations of academic psychology research." Yarkoni's criticisms were mostly ignored or evaded by the final article.

As I mentioned, I found the blog posts the best entry to the story.

Gelman on the original article and how their "reproduceablity-enhancing" measures seemed misguided.

Gelman again on his top five ways of increasing reproduceabillity.

Jessica Hullman on the preregistration that wasn't. Hull has some quality snark here and in the comments in places where the Chronicle account held back.

Here's Hullman's post-retraction summary.

My take: the original sin for the reputable scientists was to hitch their wagon to a study trying to document an obviously wacko version of how the world works. Lay down with wackos, get retractions.

Heebie's take: Well, Chill walked right past "Lay down with wackos, get retrackos."

Also, Gelman's improved list of reproduceability-enhancing measures is hilarious, as it boils down to "Do higher quality work." But said more tactfully.

Hurricane Helene Here

Chat away.

Eric Adams

Is there energy for a thread?

Guest Post: Building Blocks of State Capacity

NickS writes: I was so excited to read Henry Farrell's most recent post:

He has written a few posts that have touched on questions I was already mulling over and helped clarify the nature of the problem.

In this case, I had been thinking about why it is that, right now, that there's particular interest and attention to the question of, "what can we do to help build a government that is better able to achieve the goals it sets out?" Farrell not only offers a good answer but looks ahead to spot some of the tensions in the different paths people are taking to solve the problem.

The post has a somewhat lengthy initial stage-setting but after that I found it a thrilling read.

The relationships between the different approaches are likely to be similarly complicated. Supply side progressivism and big fix liberalism have a lot more in common than their political valences might suggest. Democratic steering can be more easily reconciled with some kinds of adaptive state liberalism than with others. So too for their relations with cyborg bureaucracy. And so on.

That is why I think that these approaches and their internal variants are maybe best considered as building blocks from somewhat different toy boxes. Certain blocks won't fit easily together with certain others - the resulting structures will be awkward. Others can perhaps be assembled into new and unexpected structures that neither could support on its own. I can't say, and don't pretend to know, how the various pieces ought fit together, or what structures they ought be used to build.

For sure, I personally prefer some of these toys to others, and have my guesses as to where we should go. But the purpose of this post is not to set out my own, limited and specific position. Instead, I want to emphasize the variety of approaches to state problem solving that these different understandings of state capacity liberalism provide and the possibility of generating hybrid approaches, discovering unexpected combinations and new variants. That is what I think we need to focus on right now, rather than knocking the corners and odd angles off these various blocks so that they can be better shoved into the ideological containers of a pre-existing dispute.

Heebie's take: It is long but engaging! I'm only partway through, but throwing this up because it's almost time for Math Club.

Guest Post -- It Would Be Irresponsible Not To Speculate

Mossy Character writes: That this new Mozart piece

The newly discovered manuscript was not penned by Mozart himself but is believed to be a copy made in around 1780, the researchers said.

is a forgery composed by LLM and transcribed by hand.

Heebie's take: I kind of love forgeries. They raise interesting questions about what exactly we value about the the original Mozart music, and plus it's such an intricate, nerdy crime. BATTLE OF THE NERDS COMMENCE!

Learning vs re-learning

In the past few months, I've had two occasions to refresh things I originally learned 30-40 years ago. First I took a sewing class with Ace, and second I've been refreshing my Spanish with Language Transfer. Neither of these were totally dormant skills, but the classroom version of both had been mostly forgotten and replaced with a hacky day-to-day version.

Let me tell you: we all know that learning something new in middle age is much harder than it is when you're an absorbent little teenager. But re-learning something is totally different, and makes you feel like a brilliant mastermind. It comes so naturally! It is a delight and you get to feel like a goddamn genius to spring to life so readily. Just the merest nudge and I retain all these scholarly details! I am enjoying it a lot.

Ultra-processed.

I don't know why I found this comforting, but I did: it's the processed meats and sodas. (NYT gift link)

This year has had a two friends in their 40s die from cancer, and whenever life seems fragile, I get very susceptible to panicked thinking, and it's easy to find a lot of writers telling you to worry about ultra-processed foods. So I've been obedient. It's good for me to have some guardrails on how much to worry. It's not the yogurts and cereals.

I do have one question: when they say deli meats, is that just the pre-packaged stuff, or does it include the stuff sliced in-store at the deli counter? (I guess I can google. This says that it's the exact same stuff. Damn. I do eat a lot of deli turkey.)