Because this paper has been so influential, it is worthwhile to take a close look at the original study to try to understand why it did not replicate. We did that. This post – and the next one – is about what we found.

***

About 20 years ago, on April 20, 2006, one of the authors of the forthcoming replication, Kyle Hyndman, received the original data files in an email sent from ariely@mit.edu.1 And about 3 years ago, on August 9, 2023, a week after Francesca Gino sued us for $25 million, we received an out-of-the-blue email from Hyndman in which he sent those files to us. We performed quick analyses of the data, and then had a conversation with Hyndman and his co-author, Alberto Bisin. In that conversation, they told us they were going to conduct a replication, and, finding ourselves busy with the lawsuit, we left it at that.

We recently learned that their replication was forthcoming in Psychological Science. And upon reading Footnote 14 of their paper, we also learned this:

. . . In October 2024, at the request of the editors, we shared with Dan Ariely an analysis of the contents from the file purportedly for their Study 2 and asked for permission to include a summary of it in the paper. Dan Ariely denied our request, arguing, among other things, that the files we received may not be the actual data. He did not subsequently provide us with any additional data from the original paper. Consequently, we are unable to supplement our replication exercise with any additional analysis of the files we received in 2006 or any other data.

This motivated us to return to this paper and fully analyze the original data for the two main studies. We conclude that the data in Studies 1 and 2 were tampered with. In two posts, we present the evidence that led us to this conclusion. Today’s post focuses on the study that failed to replicate (Study 2), and our next post is about Study 1.

Our assessment that the data were tampered with are based entirely on the analyses presented in our posts. Readers can review the evidence and draw their own conclusions.

To the best of our knowledge, Klaus Wertenbroch has never had access to any version of the data for any of the studies. And, we believe it is thanks to him that we do. When Kyle Hyndman reached out to the authors back in 2006, Klaus replied with this email:2

Our ResearchBox contains the data and code to reproduce all of the results in this post.

Finally, it should be noted that when we shared these posts with the original authors a few weeks ago, they reached out to Psychological Science to request that the article be retracted. As of this writing, that process is ongoing.

The Study That Did Not Replicate: Study 2 of Ariely and Wertenbroch (2002)
As noted above, Ariely and Wertenbroch explored how deadlines influence performance. In a context in which people had multiple tasks to perform, the authors hypothesized that people would perform better in the face of evenly spaced deadlines for those tasks, rather than when they were all due at the end.

The experiment involved an incentivized proofreading task. Each participant received three 10-page documents, each containing 100 “grammatical and spelling errors” (p. 222). Participants were tasked with finding and correcting those errors.

Sixty participants were randomly assigned to one of three conditions, exactly 20 participants in each condition:

Condition 1. Evenly Spaced Deadlines. One document was due each week, so after 7, 14, and 21 days.
Condition 2. Set Your Own Deadlines. Participants chose their own deadlines (within 21 days).
Condition 3. Last Day Deadline. All three documents were due on the final (21^st) day.

The results perfectly and strongly supported the authors’ hypothesis. Participants given evenly spaced deadlines did much better, in terms of performance, delays, and earnings.3

Do We Have The Original Data?
As a reminder, in 2023 Hyndman sent us files he received from ariely@mit.edu in 2006. There were three Excel files – data for a pilot study, for Study 1, and for Study 2 – all with file properties indicating that the data were “Last saved by” “Dan Ariely”.

With these files we are able to reproduce all nine means and all nine standard errors shown in the figure above, as shown visually in this footnote4. We also successfully reproduce the six other means reported in the text.5

Red Flags
In our analyses we identified four major red flags. We discuss each in turn.

Red Flag #1: The Effect Is Too Big
As shown in the reprinted figure above, Ariely and Wertenbroch report a perfect pattern of results, for all three dependent variables, with a sample size of only 20 per condition. The effects are also large. Extremely, implausibly large.

Consider the proofreading performance results. Participants with Evenly Spaced Deadlines made an average of 136.1 corrections, whereas those with the Last Day Deadline made an average of only 71.1 corrections, about half as many. This effect has a Cohen’s d = 2.5, indicating that the condition means are 2.5 standard deviations apart. The correlation between experimental condition and number of corrections is r = .79.

To appreciate that this effect is just too big, consider it in the context of other effect sizes. An effect size of d = 2.5 is larger than obvious effects we notice in everyday life, effects that can easily be seen with the naked eye. For example, it is much larger than the effect of gender on height (men are taller: d ≈ 1.8) and on number of shoes owned (women own more shoes: d ≈ 1.2; see Colada[18]). It is also larger than some manipulation checks. For example, Petty and Cacioppo (1984) report that participants exposed to messages containing nine arguments said that they encountered more arguments than people exposed to messages containing three arguments. This has to be true. And it was true, but only to the tune of d = 1.49.6 It is not plausible that deadlines influence proofreading performance more strongly than the number of arguments influences the perceived number of arguments.

Effect sizes greater than or equal to 2.5 are not impossible – they are sometimes observed with manipulation checks – but they are extraordinarily rare for non-obvious psychological findings, particularly for a measure like proofreading error detection, which is likely to be noisy, and highly variable across people.

Another way to appreciate the enormousness of this effect is to look at the distribution of the dependent variable across conditions. The figure below shows that they barely overlap. For instance, whereas nobody in the Last Day Deadline condition made more than 100 corrections, 90% of the participants in the Evenly Spaced Deadlines condition did:

Red Flag #2: Duplicate Observations
If looking at Figure 2 you thought, “wait, why are there so many red bars with 2s?”, good catch. That is weird. The 2s represent people who found exactly the same total number of corrections made across three tasks. But it’s actually weirder than that. These participants found not just the same number of corrections in total, but also made the same number of corrections for each of the three separate proofreading tasks.

Here is a screenshot of the original data file, formatted and sorted to be easier to digest:

We see that 18 of the 20 participants in the Last Day Deadline condition had a “Corrections Twin”, another participant who found exactly the same number of errors for each of the three proofreading tasks. Interestingly, these twins have ID numbers that are exactly 10 positions apart (e.g., subject S1 and subject S11 are twins; so are S7 and S17; etc.). (There were no error twins in the other two conditions.)

The existence of so many of these twins – and all of them in only one condition – is inconsistent with these data being real.

Red Flag #3: Things That Should Be Very Highly Correlated Aren’t Correlated At All
At the end of their study, Ariely and Wertenbroch purportedly “asked participants to evaluate their overall experience [of the proofreading task] on five attributes: how much they liked the task, how interesting it was, how good the quality of the writing was, how good the grammatical quality was, and how effectively the text communicated the ideas contained in it” (p. 223). These questions were answered on scales ranging from 0 to 100. The replicators asked the same questions to their participants.

You might expect these judgments to be correlated. For example, if someone says they liked the task, you might also expect them to say that it was interesting.

In the replication, this was (super) true. Controlling for experimental condition, the partial correlation between liking and interest was, quite sensibly, close to perfect:7

But in the original data, this relationship was not only imperfect; it was not there at all. Participants who said they liked the task more did not say that they found the task to be more interesting:

In total, there are five subjective measures. In the replication, the (partial) correlations among these five measures range from +.63 to +.92. They are all large and very highly significant (ps < 0.0000024). In the original data, these correlations range from -.29 to +.18, and none of them are both positive and significant. This is very strange.

The problem is not limited to these subjective measures. Consider the fact that people did three very similar proofreading tasks, each with 100 mistakes. Surely, we’d expect people who do better on one task to also do better on another, nearly identical task. That simple fact should manifest in extremely large correlations between performance on one task and performance on another. And in the replication data it does, as the correlations range from +.74 to +.90. But in the original data it doesn’t, as the correlations range from +.03 to +.27.

Finally, consider that participants were asked to report how many minutes they spent on each of the three tasks. Again, we’d expect those who said they spent more time on one task to be more likely to say they spent more time on another, nearly identical task. And so we’d expect these variables to be very highly correlated. Once again, within the replication data they were – the correlations ranged from +.79 to +.95 – and within the original data they were not – the correlations ranged from +.05 to +.17.

The correlations we have reviewed in this section are essentially just sanity checks. Does liking correlate with interest? Does performance correlate with performance? Does reported time spent correlate with reported time spent? Sane data pass these checks. Insane data do not. The replication data are sane. The original data are not.

Red Flag #4: No Rounding In Self-Reported Minutes
As you’ll recall from a minute ago, Ariely and Wertenbroch (2002) purportedly asked participants to “estimate how much time they had spent on each of the three tasks” (p. 223). When people provide estimates like this, they tend to round. They usually say “20 minutes” or “30 minutes” instead of “17 minutes” or “32 minutes”. And, indeed, when the replicators asked people to report how many minutes they spent on each of the three tasks, 85% of them gave a round number:

This is what we’d expect humans to do.

But in the original data, they did not do that. Only 11.7% of estimated minutes were round, consistent with the 10% you’d expect by chance alone:

This is not what we’d expect humans to do.

Conclusion
We are unable to generate a benign explanation for all of the anomalies presented here. The original findings are too large and yet they do not replicate; there are duplicated observations; correlations that should be very strong are often non-existent; and values that should be rounded are not rounded. Based on this evidence, we believe the data for Study 2 of Ariely and Wertenbroch (2002) were severely tampered with or fabricated to produce the desired results.

In our next post, we will share analyses of the Study 1 data file that Hyndman received from ariely@mit.edu. That experiment is quite different. Our analyses are quite different. But our conclusions are quite similar.

4

We reproduce the published article’s figures with the spreadsheets we received. For example, here is Panel A of Figure 2:

Unfortunately, some of that is coming to an end. In an email today, the OSF Team notified users that, come November 2026, the OSF will no longer accept new file uploads.

The email links to a post (htm), where Brian Nosek explains that the OSF is making this change because it is being misused, both by spammers and researchers, leading to maintenance costs that are not sustainable. Regarding spammers, Nosek writes that they “set up thousands of accounts and tried to use OSF for sharing pirated movies, advertising health supplements, and anything and everything else.” Regarding researchers, Nosek writes that many use it for private storage instead of for open science practices.

Right now thousands of researchers are looking for a new home to upload their open-research files to. In this post we explain why ResearchBox is the home they are looking for.

Thank You For Your Service, OSF
We want to start by thanking Brian Nosek, a friend and supportive colleague, along with past and present Center for Open Science (COS) team members, for all the work they have done, and continue to do, to improve research credibility. To thousands of researchers around the world, the OSF is synonymous with open research practices, and many researchers use these practices today because they, or their advisors, learned about them through Brian and the COS team. We look forward to seeing the new initiatives COS will pursue as it shifts its resources and attention away from the OSF.

ResearchBox
ResearchBox is part of the Credibility Lab open-research ecosystem; it is fully integrated with AsPredicted (the most popular pre-registration platform in the social sciences) and AsCollected (the recently launched and sole platform for documenting results-provenance in all of science).
If you have ever done an AsPredicted, you have a ResearchBox account already.

ResearchBox was launched in October 2020 and currently hosts files from 8,970 researchers.
Since its launch, the total number of stored files has roughly doubled each year.

We built ResearchBox for a single purpose: to make it easy for researchers – and only researchers – to share data, code, materials, and pre-registrations with their readers. We did not design it as a collaboration tool, and nor as a place for non-researchers or non-research-related files. ResearchBox’s specialization and simplicity make it well equipped to avoid the issues the OSF has encountered.

On ResearchBox every detail is designed to make research content easy to share, navigate, and understand. Open Science loses much of its value when posted materials are difficult to find or interpret. ResearchBox actively guides researchers in making their shared content useful to others. Among the many unique ResearchBox features that enhance the value of posted content:

1. All boxes look the same, as all files are always organized in tables with the same columns.
   When visiting any box, you immediately know which files were, and were not, posted:

Fig 1. All boxes are presented in Bingo Tables like this one, always the same columns
Source of example: https://researchbox.org/446 by Overton, Evengelidis, & Vosgerau (2024)

2. Files can be previewed nearly instantaneously, including very large datasets [1].

3. Every dataset has a codebook.

4. To download files, you just check the ones you need, or click a single button to download all.

5. You can search the code (e.g., scripts using “lme4”) and data (e.g., variable descriptions including the word “depression”) of publicly available boxes.

6. Public boxes are automatically archived to the web archive and to Zenodo, where they obtain a DOI. If ResearchBox shut down today, all public boxes would remain available at those two file preservation platforms.

ResearchBox Works To Avoid The Problems Plaguing The OSF
Misuse and spam were key reasons why OSF is closing its doors to new files.

ResearchBox addresses them through several safeguards, including (but not limited to):

1. You can’t create a ResearchBox account with an anonymous email address (e.g., @gmail.com), making it very difficult for anonymous users and non-researchers to use the platform.
   Note: After the account is verified, any preferred email (e.g., Gmail) can be used.

2. The user interface convey explicitly and through design that it should only be used to share research files for publishable research (minimizing well-intentioned misuse).

3. Boxes are algorithmically screened for misuse, and content violating Terms of Use is eliminated.

4. Files are not allowed to stay private forever; authors get renewable 3-year clocks to make them public.

5. Default storage limits start low for accounts from users with email addresses not associated with research institutions (but upon request & verification, the limit is increased)

In Sum
When it comes to hosting research materials, ResearchBox is:

1. Fast: Instant file previews and fast downloads.

2. Searchable: Search by variable names, code content, or metadata.

3. Standardized: Every ResearchBox landing page follows the same structure.

4. Easy to use: No training or tutorials required.

5. Committed to file permanence: Public boxes are automatically backed up to both Zenodo, with a unique DOI, and the Internet Archive.

Footnotes.

1. ResearchBox creates “thumbnails” of datasets and code scripts, so preview files are tiny and thus quickly opened[↩]

Francesca Gino’s lawsuit against Harvard is still ongoing. (The trial, if it gets to that, is scheduled for December 2026) While the case drags on, both sides keep filing documents with the court, and most of those documents are publicly available (htm). This post is about a recent filing: Document #222 (pdf).

The Cover-Up File On October 1^st, 2023, Gino claimed on her website francesca-v-harvard.org that she had found an Excel file that Harvard should have used in its investigation but didn’t. As she put it, “our unequivocal conclusion is that [Harvard Business School] used the wrong dataset in its investigation . . . The correct dataset . . . is 100% consistent with the [published results]“ (htm) ¹. Note: in this post, text appearing in [brackets] within quotations indicates that the quote has been edited for clarity.

Harvard countered that the file “was fabricated by Professor Gino and placed on her laptop“ ². Harvard refers to this file as the “Cover-Up File”, and I will follow suit.

Gino took her post down, but did not recant her claims about the Cover-Up File ³.

Document #222 is key in Harvard’s claim of a Gino cover-up. The document includes a 42-page PDF containing a data-forensic report by Julian Ackert (htm), an expert hired by Harvard (pdf). The report lays out extraordinarily detailed information about computer files moving in and out of Francesca Gino’s laptop over the years.

**

To get started, imagine you were the analyst tasked with evaluating the legitimacy of the Cover-Up File. What’s the first step you need to take? Finding the file.

Step 1: Finding the Cover-Up File Harvard had two copies of Gino’s work laptop. One from when they started the investigation in the Fall of 2021, the second from the tenure revocation proceedings in 2024. The forensic analyst found the Cover-Up File only in the 2024 copy. In the 2021 copy, there is a file with the same filename, in the exact same folder, but with different contents ⁴. This isn’t great for Gino’s story.

Step 2: When was the Cover-Up File added to Gino’s laptop? Because the file was found in the 2024 but not the 2021 copy of Gino’s laptop, we know it was added between those years. That’s a wide window. Exactly when it was added seems forensically relevant.

Helpfully, computers keep track of these things. Most relevant to us, they record in the “last-modified” field of each file, when they were last saved. For Excel files, last-modified times are saved in two places:

• In the Excel file itself (henceforth, the Excel date)

• In the operating-system’s file catalog (henceforth, the macOS date)

The Excel date for the Cover-Up File is “September 23^rd, 2023” ⁵. That’s just eight days before Gino wrote on her website about finding the file. This sounds bad, but it need not be. If Gino found the file it would make sense for her to open it. If she then saved it (possibly with auto-save), the Excel date would update. All we learn from this timestamp is that someone using Gino’s laptop accessed the Cover-Up File on September 23^rd, 2023.

The macOS date for the Cover-Up File, in contrast, is older, much older. It is July 17^th,2010⁶. So:

Excel said: “this file was last modified in 2023”. macOS said: “this file was last modified in 2010”.

It turns out this pattern of dates, a macOS date 13 years before an Excel date, is “highly irregular” ⁷ and it led the forensic expert to conclude that the “metadata in the 2023 Cover-Up File was manually backdated” ⁸. This means the macOS date was tampered with, it was manually moved back to 2010.

The underlying issue is this: When you save an Excel file, both the the Excel date and the macOS date update (they match) When you copy an Excel file, only the macOS date updates (so macOS is newer)

Because you can copy a file after you save it, but you cannot copy a file before you save it,the macOS date can be newer but notolderthan the Excel date.OK. This leads to the next question: where was thisbackdatedfile before it was put on Gino’s work laptop?

Step 3: A Black USB Drive (Serial #: 21013403D7FFA005) ⁹ . Mac computers have something called the “Unified Log”. It’s a record of actions taken on the computer. The log for Gino’s work laptop shows that on September 23^rd, 2023, the day the Excel file was last saved, Gino’s work laptop was opened for the first time at 5:31 PM ¹⁰. Just one minute later a black USB drive was inserted into the laptop, and 18 seconds later the Cover-Up File was copied onto the laptop ¹¹.

The Cover-Up File had the same name as a file already in the folder it was copied into, so this means that the Cover-Up File literally covered up (i.e. replaced) an existing file ¹².

The analyst notes that on a Mac computer, when copying a file that would overwrite an existing one, a warning pops up ¹³. That warning would have looked something like this:

Figure 1. Illustration of a confirmation popup on a Mac computer

The user of Gino’s laptop must have clicked on “Replace”.

Step 4: Where did the file in the USB come from? We can again rely on metadata to answer that. Metadata often don’t just tell you when something was saved, but also who saved it.

The username for Gino’s work laptop is “Francesca Gino” (with 2 spaces). The Cover-Up File in the USB drive was last saved on a computer with a different username: “Gino, Francesca”. The forensic analyst believes that “Gino, Francesca” is the username for a personal laptop Gino bought in 2023, suggesting that that’s where the Cover-Up File was created. ¹⁴.

Step 5: Benjamin Edelman The analyst also had access to a (redacted) copy of Gino’s Google Drive account. He states that Gino uploaded the Cover-Up File to Google Drive and shared a copy with Benjamin Edelman ¹⁵. Edelman is a former HBS professor who got denied tenure, sued Harvard for it, and lost (but is now appealing, see his website: edelman-v-harvard.org). Harvard has been trying to get Edelman to testify under penalty of perjury on his dealings with the Cover-Up File, but he refused to do so during his deposition by asserting attorney-client privilege. (Edelman, who has a law degree from Harvard, has not formally represented Gino ¹⁶, but has advised her). As background: this post exists because Document #222 was made public, and it was made public because Harvard filed it to try to persuade the judge that Edelman should be compelled to answer questions about the Cover-Up File. The request got a surprising reject-and-resubmit a few days ago, see footnote for details ¹⁷.

The covered-up evidence Document #222 also provides new (to me) evidence that Gino was responsible for the data anomalies in the sign-at-the-top experiment in the retracted PNAS paper. Specifically, it shows that there is simply too little time for anybody but Gino to have generated those anomalies. I summarize the key relevant events in Table 1 below ¹⁸.

Table 1. The sequence of events for fabricated Study 1 in retracted PNAS paper.

The story begins on Friday July 16^th, when at 4:57 PM Gino’s inbox receives an email from the Research Assistant with the subject line “That’s a wrap: tax study” for Study 1. Gino saves that file on her computer the next day, Saturday at 6:43 PM. Less than 22 hours later, on Sunday at 4:30 PM, she has already emailed her co-authors results based on a copy of that file that has anomalies: 3 extra observations, N=101 instead of N=98, and many cell value changes. This short time span between the clean and the anomalous versions of the same dataset being saved on Gino’s computer is what a backdated copy of the anomalous dataset, the Cover-Up File, would cover up.

One last thing. Nine days after Gino sends the anomalous results for Study 1 to her coauthors, the RA fortuitously resends Gino the Study 1 data. The RA was sending Gino partial data for Study 2, and she happened to put those new results in the same spreadsheet she had used for Study 1. Those Study 1 data that were resent by the RA still did not have any of the documented anomalies.

Julian Ackert, again, Harvard’s forensic analyst, writes “If [the RA] had participated in a modification of the raw data [that Gino sent coauthors] . . . it is implausible that she would continue using the raw data“ ¹⁹.

In a visual nutshell, this is Harvard’s story:

Further reading
Colada[109] “Clusterfake”: Original post documenting evidence of fraud in this PNAS study.
Colada[114] “Exhibits 3,4 an 5”: Follow-up post showing exhibits from Gino’s own lawsuit against us supported our allegations.

Subscribe now

Author Feedback Our policy is to reach out to authors whose work we discuss and solicit feedback. In lieu of reaching out to Gino we quote from her affidavit filed with her opposition to Harvard’s motion to compel Edelman to answer questions about the Cover-Up File: ²⁰ “[Harvard’s] allegations of data fabrication and evidence destruction are false and unfounded. Neither I, nor anyone acting on my behalf, created a backdated file to fabricate evidence. Nor did I, or anyone acting on my behalf, overwrite any file data in an effort to destroy evidence . . . The so-called [Cover-Up File] is a genuine file that I found in or about the summer of 2023 when searching through materials that had been stored in my former office at Harvard . . . “

Footnotes.

1. Gino’s removed blog post: The original URL for the now removed post was https://www.francesca-v-harvard.org/hbs-data-reconciliation-1, but the link no longer works. If you visit it now, June 2026, it redirects to a generic landing page for a redesigned webpage (htm) where Gino claims that “For transparency, I am leaving the original website live. It launched on September 29 2023, but it is now not up to date“. But that is not entirely true. The removed blogpost was in the original website and it is not live. It is, however, on the web archive (htm). ↩

2. See Document #154, page 162, paragraph 90 (pdf) ↩

3. See Document #216, an affidavit by Gino (pdf). ↩

4. See Document #222, paragraph 41. ↩

5. See Document #222, Figure 4. ↩

6. See Document #222, Figure 4 (same as previous footnote). ↩

7. See Document #222, paragraph 45 ↩

8. See Document #222, page 19, section header. ↩

9. See Document #222, paragraph 57. ↩

10. See Document #222, paragraph 56. ↩

11. See Document #222, paragraph 58. ↩

12. See Document #222, paragraph 59. ↩

13. See Document #222, paragraph 59. ↩

14. See Document #222, paragraph 55. ↩

15. See Document #222, paragraph 68. ↩

16. See Document #207, page 10, footnote 6 where Harvard states that “Edelman has never made an appearance on behalf of [Gino] in this litigation or in her tenure proceeding”. (pdf) ↩

17. Harvard’s request for compelling Edelman’s testimony got a Reject and Resubmit. It turns out that Harvard filed the request to compel Edelman’s testimony in the wrong court (See Document 229 page 2 (pdf) ). Because Edelman lives in the state of Washington, and thus would be deposed in Washington, Harvard should have submitted the request to compel his testimony to a Washington court, but it filed it instead in the court where all other proceedings are taking place, and all other documents were filed; a Massachusetts court. Harvard could resubmit to the Washington court if they wanted. In that same court document (See Document 229 pages 3 & 4 (pdf) ) the court orders Gino’s husband, Mr. Burd, to answer a question about conversations he had with Edelman about the Cover-Up File. Mr. Burd had also declined to provide that information in his deposition alleging attorney-client privilege, but he lives in Massachusetts, so Harvard’s request that he be compelled to answer was submitted to the right court, and the judge agreed he should be compelled. ↩

18. See Document #222, Figure 3 ↩

19. See Document #222, Paragraph 33. ↩

20. See Document #216, Paragraphs 2 & 3 (pdf). ↩

Before I show you the figure that perplexed me, some background is necessary.

The paper makes many points, one of which is that our standard statistical procedures – like running basic OLS regressions on our full datasets – often convey an extremely optimistic impression of how good our models are at predicting new observations.

To help make that case, the authors analyzed data from Many Labs, an effort by many different labs around the world to try to replicate some published findings in psychology (.htm). The figure that sparked this post focused on replications of a sunk cost effect, in which participants were randomly assigned to one of two conditions and then submitted a rating on a 9-point scale (the details aren’t critical, but if you want them they are in this footnote).2 Specifically, it plots the results from 15 labs that found a statistically significant effect of the manipulation on the dependent variable.3 The models producing these results could not be simpler: regressing ratings on experimental condition (0 = control; 1 = treatment). That’s it. No additional variables.

The figure compares the R² estimates that those labs obtained using OLS regression (orange dots) to the R² estimates that these authors obtained using 10-fold cross validation (purple dots). It lists each of the 15 Many Labs sites on the y-axis and represents the R²s on the x-axis. Here it is, along with my annotations; the figure note is theirs:

The takeaway is that the cross-validation estimates are always lower than the OLS estimates, and in many cases much lower. For example, the median OLS R² is .051, and the median cross-validation R² is -.128.

The authors interpret this as evidence that the “R²s differ widely because of overfitting” and that the OLS estimates are “overly optimistic.” But that is not obvious to me. I had at least two questions.

1. Most of the cross-validation R²s are negative. Indeed, one of them – for the “ku” sample – is very negative. What??

2. Overfitting happens when a model fits (in-sample) data in a way that does not generalize to other (out-of-sample) data. This problem is much more likely when models are complex, contorted to fit the data that it is analyzing. But this model is built on a single binary predictor, a situation in which overfitting seems almost impossible. What??

I am no expert on cross-validation, and I wanted to understand these things. To do so, I dug into the authors’ posted (and helpfully immaculate) code. But before I tell you what I found, there are two things you need to understand first: (1) What is 10-fold cross-validation? and (2) How can R² be negative?

What is 10-fold cross-validation?

To perform cross-validation, you build a model on one subset of data – called the training sample – and then use it to predict data in a different subset – called the test sample.

To perform 10-fold cross-validation, you divide the sample into 10 equally sized subsamples called folds. So if your dataset has a total of 100 observations, you divide the sample into 10 folds of 10 observations each.

You then use Fold 1 as the test sample and build a model using the remaining 90% of the data (Folds 2-10). You use that model to predict the Fold 1 observations and measure how well it performs. You repeat this process nine more times, each time holding out a different fold as the test set.

The assessment of model performance can take many forms, but what the authors did was calculate an R² for each fold – yielding 10 R² values – and then average them. Those across-fold averages are what appear in Figure 1.4

How can R² be negative?

Inconveniently, R² means different things in different contexts.

For everyday OLS regressions, R² represents the percentage of variance explained. And since you can’t explain negative percent of the variance, this R² cannot be negative.5 The authors refer to this as in-sample R². I will refer to it as OLS R² since it’s coming from our OLS regressions.

In the context of cross-validation, R² is defined this way:

This cross-validation R² is inherently comparative. It compares how well you’d do if you used your model to predict a set of values vs. how well you’d do if you simply predicted the mean every time. It is positive if your model does better than the mean, and negative if it does worse than the mean. In general, these R² values can range from -∞ to +1.67

Figuring Out Fold #9

In trying to understand Figure 1, I decided to focus on the outlier at the far left: the ‘ku’ sample’s (n = 113) very negative cross-validation R² value of -.88. (’Ku’ stands for Koç University in Istanbul. Go Rams.) I wanted to see exactly where that came from. So I ran the authors’ code, and I looked at the R² values generated in each of the 10 folds. Here they are:

Check out Fold #9 and its gigantically negative R² value of -8.077. Let’s see what’s going on there.

My first thought was that the model used to estimate the Fold #9 values must be a lot worse than the model used to estimate, say, the Fold #1 values. Let’s take a look at those models:

Fold #1 model: y = 6.66 + 1.15*treatment

Fold #9 model: y = 6.39 + 1.19*treatment

Um, those aren’t very different.

Fold #1’s model estimates a mean difference of +1.15, while Fold #9’s model estimates a nearly identical mean difference of +1.19. And yet the former has an R² of .019 while the latter has an R² of -8.077.

What??

To make sense of this, remember that these R² values are comparative. They are comparing how well the model is doing against how well the mean is doing. This R² value of -8.077 is not telling us that the model is doing terribly in an absolute sense. It is telling us is that it is doing terribly in a relative sense. The mean is doing a lot better at predicting the Fold #9 values than the model is.

To understand why, let’s take a look at the training sample data (n = 102) and test sample data (n = 11) used in Fold #9:

You can see that in the training sample, the values range from 1-9. But in the Fold #9 test sample, there is almost no variation. No values are below 7, most are equal to 9, and the mean is very close to 9.

This led me to ask a naïve question: When we compare the model’s performance to “the mean”, which mean are we using: the training sample mean (6.91) or the test sample mean (8.73)?

I was surprised to learn that in the land of cross-validation, the benchmark is the test-sample mean. That’s kinda brutal for the model, because the test-sample mean is a cheat: it is calculated from the very observations we are trying to predict. It effectively gets to peek at the answers.

By analogy, imagine I flip 10 coins, you predict 5 heads, but it turns out to be 7 heads. Your R² will be negative, because although you (sensibly) predicted 50% heads, you had no chance against the observed mean of 70% heads. You’ll be punished for not foreseeing the unusual outcome.

You can’t beat the mean when every observation is equal to the mean.

That cross-validation R² behaves this way is not a mistake. It’s simply how it is defined. It isn’t telling you that the model is generally bad or that the OLS R² is extremely optimistic. It is telling you that, in this particular sample, a model built on 90% of the observations does worse than the observed mean at predicting the remaining 10% of observations. It is telling you something super specific, not something general about model performance.

Once you understand the machinery of all of this, you can see that, by the metric of R², models stand very little chance when samples are really small. This is because small samples can be very unusual, containing observations that are unusually clustered, and thus very close to that sample’s mean. When I flip 10 coins, I might observe 70% heads. But when I flip 1,000 coins, I won’t. Similarly, when I observe 11-12 observations in the ‘ku’ sample, I might not see any values below 7, even though they range from 1-9 in the whole sample. But when I task the model with predicting a larger sample of observations, that happenstance clustering of values is less likely, the mean’s advantage shrinks, and the model has more of a fighting chance.

To illustrate this, what if instead of doing 10-fold cross-validation on the “ku” sample – which has us predicting folds containing 11-12 observations – we do 3-fold cross-validation – which has us predicting folds containing 33-34 observations? Note this is still an extremely small sample, and our model can still look bad. But we sure see a lot of improvement, with an R² of .027 instead of -.881:

So making a small change to how cross-validation is done drastically changes the results.

Though I have focused on the small “ku” sample, it is worth noting that all but two of the 15 samples are small, containing fewer than 300 observations. Not coincidentally, the two large samples – mturk and pi – had cross-validation R²s that were pretty similar to the OLS R²s. But, in general, you can see that if we rebuild Figure 1 using 3-fold cross-validation, the R²s are better, and in fact mostly positive:

There is something peculiar about this. In 10-fold cross validation the models are built on 90% of the data, but in 3-fold cross-validation they are built on only 67% of the data. And yet, the models look better under 3-fold cross validation than under 10-fold cross-validation. That is, the models look better when you build them on smaller samples. This strange fact emerges entirely because R² is comparative and the thing it compares against – the test-sample mean – has a bigger edge when the test samples are smaller (as in 10-fold cross-validation) than when they are larger (as in 3-fold cross-validation).

So the negative cross-validation R²s in Figure 1 don’t show that those OLS R² estimates are wildly optimistic or that these simple models are somehow severely overfitting. Rather, they largely reflect how difficult it is for those models to outperform a structurally advantaged sample mean when test folds are very small.

So Are The Models Actually Overfitting?

R² can behave strangely in small samples. So instead of asking “how much variance is explained,” let’s ask a simpler question: how wrong are the predictions?

If a model is overfitting, it will be a lot worse at predicting new observations than at predicting the data it was trained on. For each of the 15 samples, we can use cross-validation to compare how far off the model’s predictions are when predicting new observations (out-of-sample error) vs. how far off they are when predicting the data it was trained on (in-sample error).

We can do this using two common performance metrics: Root Mean Squared Error – which gives greater weight to larger errors – and Mean Absolute Error – which simply represents the average mistake. The results are in the figure below. Note that the diagonal line marks the points at which the in-sample and out-of-sample errors are identical. Points above that line are consistent with overfitting (more error when predicting new observations than in-sample observations) and points below are consistent with underfitting (less error when predicting new observations):

Across all 15 samples, those errors are nearly identical. In fact, the left panel shows that in 13 out of the 15 samples, the model made slightly smaller squared errors when predicting the new data than when predicting the data it was built on, more consistent with underfitting than with overfitting. The right panel shows slightly larger absolute errors in the out-of-sample than in the in-sample predictions, but the largest gap is only .06 points on this 9-point scale, hardly something to care about.

This evidence is much more consistent with the view that these simple models are not meaningfully overfitting than with the claim that they are.

Conclusion

In their paper, the authors make some important points. For example, in-sample R² is generally optimistic about how well a model will perform on new data, behavioral scientists should care about how well our models forecast new observations, and we should adopt methods that allow us to evaluate that performance.

But that paper also included Figure 1, which, to a naïve reader like me, makes it look like even these simple models suffer from extreme overfitting; and that, if anything, those models are performing so poorly as to have no predictive value at all. That impression, I think, is misleading. Figure 1’s results look so dire because the authors are using R² to assess cross-validated predictions in a context in which samples are very small. When you rely on different performance metrics – ones that don’t structurally disadvantage the model in very small test samples – you can see that these models are performing about as well as you’d expect given the modest effect sizes that the traditional analyses reveal.

From this post, you might take away some specific lessons about cross-validation or R². But for me this exercise reinforced a broader lesson: Everything that scientists say hinges on the specifics of what they actually did. If you can’t (or don’t) evaluate the details, you can’t evaluate the science. I’m grateful to the authors for providing access to those details.

Author feedback
I shared an earlier draft of this post with the authors last summer, and they provided me with extremely helpful feedback that led me to significantly revise the post. I am very grateful for their feedback. I shared the updated draft with them more recently and they did not offer any additional feedback. If they do decide to comment or reply, I will post it here.

Psychologists basically never use robust standard errors.
But they always should. If you have never heard of robust SEs, this post is especially (rather than not) relevant for you.

Economists basically always use robust standard errors.
But I suspect intuitive understanding lags behind adoption. If you are robust SE user but don’t have an intuition for why robust errors can be (much) smaller than classical ones, this post is still relevant for you.

R Code to reproduce all figures https://researchbox.org/6145

Heteroskedasticity = “uneven noise”
Robust standard errors solve the problem of heteroskedasticity. This anglicized Greek compound word is an unnecessary mouthful. The underlying concept is straightforward: some data points are noisier than others. So, the noise in the data is uneven1.

Noise, or sampling error, is simply how results would change if you had collected a different set of random observations (e.g., ask the same question of one random set of people vs. another).

Suppose we ask participants to guess the weight of a newborn vs. of this man on TikTok who uses the handle @thefatbutcher.

Disagreements about the newborn may be around 2 pounds, while those about the @fatbutcher could easily reach 20 pounds. This obvious fact produces uneven noise across data points: for the butcher vs. for the newborn. Which survey-respondent happens to answer our question about the butcher moves the data around by 20 pounds, while which survey-respondent happens to answer the question about the newborn by only 2 pounds.

The noise for the butcher is different from the noise for the newborn. Noise is uneven: heteroskedasticity.

More generally, observations with bigger values of the dependent variable will often have more noise than observations with smaller values. An extreme example is outliers: an observation that deviates from the rest by much more than expected is noisier than most observations.2

Interestingly, ceiling and floor effects can produce the opposite pattern, more noise in the middle than the extremes. If we ask people to rate how wealthy someone is (1 = not at all; 10 = extremely), ratings of Bill Gates would get all 10s – no noise – while ratings of a truck driver’s wealth would get a wide variety of responses.

So this is what uneven noise (heteroskedasticity) is. Let’s now see why you should care.

From noise in the data to noise in the beta
Let’s imagine a regression predicting adult happiness with number of children.
The regression table could look something like this:

Source: entirely made-up data

The regression estimate says that happiness increases by 1.22 points per child.
We also have the standard error of that estimate: .31.

What does that standard error represent? It represents how much we expect the regression coefficient – the beta – to move around across new samples. Specifically, the standard error is the standard deviation of estimates across counterfactual samples. It is effectively a summary of how much noise is in the data, expressed in terms of how that noise in the data manifests as noise in the beta.

Let’s try to get an intuitive sense of how the regression computes the SE.
Let’s make super simple data:
x=1,2,3,4
y=1,2,3,5

Resulting in this simple plot:

The noise in the data is how far the dots are from the regression line, the residuals: .2, -.1, -.4, and .3.
A regression combines those four measures of noise in the data into the SE, a single number for the noise in the beta.

When you run a regression, lm(y~x), R uses math to go from 4 values to a single value.

The math does the equivalent of something like:

• Let’s imagine alternative datasets “like this one“, but with different noise

• Let’s run a regression in each of them and keep track of the beta

• The SE of our beta is the standard deviation of those other betas

One could do this with simulations, but the standard error is typically estimated with math (the math for standard errors is older than computer chips). The math makes assumptions about what datasets “like this one“ may look like.

Most relevant to us today is the assumption of whether those datasets have

• Even Noise: all observations exhibit the same level of noise.

• Uneven Noise: some observations exhibit more noise than others.

If we assume all observations exhibit the same level of noise, all new imagined dots move by similar random amounts away from the regression line.

In our example above, the biggest data noise, 0.4, appeared when x = 3, but if we assume even noise, then we would assume that’s just a fluke. In the imagined datasets the noise = 0.4 would appear anywhere, in x=1, 2, 3, or 4. If instead we assume noise is uneven, new imagined datasets will tend to have the level of noise they had in the data we saw, so that the biggest error of 0.4 will once again tend to happen around x = 3, not around x=1, 2, or 4.

So why does this matter?

It matters because when going from noise in the data to noise in the beta it matters not just how much noise is in the data, but where the noise is.

Noise in the beta is more sensitive to noise in some datapoints than others.

When going from noise in the data to noise in the beta we don’t have a “one datapoint, one vote situation”.
The noise in some datapoints matters a lot, the noise in some datapoints is nearly irrelevant.
This is probably very far from obvious or intuitive, and I hope it is about to become both of those things.

Conference location
Say we have data on a bunch of researcher ratings of three candidate cities for a future conference: Caracas, Chicago, and Paris. We have the official tourism score for the three cities, 1, 50 and 100 respectively, and decide to run a regression to predict academic preferences with those tourism scores.

Source: totally made up tourism scores; the Caracas Lonely Planet cover is also made up.

Let’s say that on average academics’ preference for location is the same as the tourism score y = x, so 1, 50 and 100 for Caracas, Chicago & Paris respectively.

But academics don’t perfectly agree on anything, let alone conference location.
So not everyone gives those three scores; they only tend to do that on average.

Of interest to us today is noise, whether it is even or uneven across cities.

The figure below illustrates four stylized scenarios that will help you understand why where the noise occurs in the data matters for the noise in the beta.

Fig 1. Even vs Uneven noise in data used for regressions

We start with the first box. Here we imagine that academics’ disagreement about locations has the same level across all three cities. The dots appear about as far away from the tourism score for all three cities.

Some academics think Caracas is much better than its tourism score, some much worse, and the same is true of all three cities. Even noise. Homoskedastic errors.

The next three boxes exhibit uneven noise. In box 2, the Chicago observations are the only highly variable option. Academics largely agree Caracas and Paris are as good as their tourism score, but disagree on Chicago. In the next two boxes, it is Paris and Caracas that academics disagree about.

Let’s consider how this noise will impact the other dataset we imagine. Let’s start with the first box, where the errors are even. Because there is noise all around, in one imaginary new dataset we could end up, just by chance, with more academics who like Caracas and dislike Paris, thereby flattening the line. Alternatively, we could also end up with academics who especially dislike Caracas and love Paris, steepening the line. So the line moves a fair amount across imagined datasets.

In the second box, in contrast, in new datasets Caracas is always around 1 and Paris always around 100. While Chicago jumps up and down, say between 65 and 35, it does not matter. Chicago noise is inconsequential. The straight line is always extremely similar. Moving on to the blue boxes, when the noise is in the extremes, the lines has much more noise than in the brown box, and also than in the green box.

An intuition for why the blue lines move more than the green line is that we take noise from the middle, where it does not matter, it does not impact the line, and we move it to the extremes, where it does matter.

The figure below illustrates this idea of the noise impacting alternative regression lines.

Fig 2. How even vs uneven noise in data impacts regression lines across alternative datasets

Figure 2 is illustrative, showing regression lines I drew “by hand” to convey the intuition.

I did run actual simulations also, keeping track of the regression coefficient. The results are in Figure 3 below. We see that if the noise is even across the cities, the regression coefficient moves around less than if it is uneven and higher in the extreme, but moves more than if noise is uneven and concentrated in Chicago.

Fig 3. Variability of regression coefficients, beta noise, when data noise is even vs uneven

Robust errors
Alright. So hopefully by now the following three things are intuitive:

1. The standard error is a measure of noise in the beta

2. Noise in the beta comes from noise in the data

3. Which observations are noisy impacts how noisy is beta

Let’s now wrap it up with robust vs classical standard errors.

Classical standard errors, the ones taught in the first regression course, the ones that R prints when you run lm(y~x), assume noise is even, assuming every dataset is like the green box, whether they are, or whether they look like the brown or blue datasets. So even if all the noise is in one portion of the data, they assume that’s just a fluke, that the noise in other datasets could be anywhere.

Robust standard errors, in contrast, assume that other dataset will have the noise in roughly the same places as the observed data has it. If all the noise is in Caracas, it assumes other datasets will also have the noise in Caracas. Robust standard errors, therefore, give you different standard errors when the data are green, brown, or blue.

A stylized description that is very close to being literally true is that:

• Classical standard errors are the simple average noise, all residuals get equal weight

• Robust standard errors are a weighted average, residuals impacting beta more get more weight.

That weighted average, the robust SE, does a pretty good job.

In this last figure I plot the true variability of the regression coefficient across the 5000 simulations (the SD of the beta across simulations) and the different estimates next to it.

Fig 4. True and estimated standard error of regression coefficients with uneven noise

While it is obvious that classical errors can be quite terrible estimates, let’s point out the less obvious thing, which is that even if the data are homoskedastic, green bars, the robust errors do just as well as the classical ones. So you don’t need to worry about whether to choose classical or robust.3

Robust errors is always the right call.

How to run robust standard errors?

A t-test involves computing the difference between two means. And yet, t.test(), does not report... ...said difference of means. It reports the p-value for the difference of means, it reports the confidence interval for the difference of means, but not the difference of means itself. This is particularly annoying when teaching, as I try to focus students’ attention away from the p-value, and that’s harder without a point estimate to focus their attention into instead. The t.test() moreover, often does not even tell you whether it computed A - B or B - A, so you have a confidence interval where you cannot interpret the sign easily. Statisticians: think of the children!

Preparing for my teaching this term, I thought I would try to write a better t-test function for R. An expensive branding consultant landed on calling it t.test2().
It has the same back-end, but the front-end is hopefully more useful for the humans.

I used Cursor, coding it with AI assistance, and the experience was pleasant enough that I then thought, hm, table() is another pretty bad function, i could improve that as well. So I wrote table2(). One thing led to another, I wrote lm2(), text2(), list2()... I put them in a new package I called statuser because it is written with users of statistical tools in mind. That package is now on CRAN, so you can install it with install.packages('statuser'). If you try it, I welcome all suggestions.

I get the sense that many packages are written assuming users will be as familiar with the tools in the package as those writing the package, and this is obviously not true. Many packages are also written to produce results without paying much attention to readability, interpretability, or formatting of those results. statuser is different. Functions produce self-contained output, often with explanations, and produce publication-ready figures in one line of code.

This post highlights a few of the functions in the starting lineup in statuser1

Function 1: t.test2()
Let’s look at the output of t.test() vs. t.test2() side by side

In terms of saving the output. If you save t-test results:
t1=t.test(x,y), you get a list with some obscure naming conventions (e.g., the degrees of freedom are saved under ‘parameter’).
t2=t.test2(x,y), you get a data row, just like the one shown on the console.

Function 2. table2()
Another super basic base R function I tried to improve is table(); it tabulates and cross-tabulates frequencies.
It has a few issues. First, when crossing two variables, table() does not show which variable is rows and which columns (Figure 2),

Fig 2. table() vs table2() for variable names

The second big shortcoming of table() is that to show proportions instead of frequencies you need a second clunky function, prop.table(), and to run a chi-square test you need a third function chisq.test(). With table2() you achieve all of that in one brief intuitive call: table2(x,y,prop='column', chi=TRUE)

Fig 3. Tabulating proportions in table() vs table2()

Function 3. desc_var()
It can be surprisingly clunky to obtain summary stats by group in R.
There is aggregate(), a function that is not impossible to use.
The statuser alternative is desc_var():

It shows the usual stuff, in an orderly fashion, but also number of unique and missing values, which is quite useful too. See footnote for contrast with psych::describeBy()2

Functions 4-6: Descriptive plots
I wished all papers showed descriptive plots before showing data analysis, so I included in statuser functions that create publication-ready descriptive-plots with a single line of code. Three functions: one for CDFs, one for frequencies, and one for densities.

Function 4: plot_cdf()
I think CDFs plots can be especially useful when considering two groups that do not differ from each other on average. This one line of code produces the entire figure. In that figure I put two groups that do differ, to more easily appreciate all the output that’s generated.
plot_cdf(widgetness~gender)

The p-values at 25^th, 50^th, and 75^th percentile come from quantile regressions.

Function 5: plot_freq()
When the DV takes a few values, a frequency plot may be more informative.
Again, publication ready figure with one line of code.
plot_freq(y~gender)

Function 6: plot_density()
When one does find an effect, but the variable is continuous, plotting the density can be better.
plot_density(value~condition)

Function 7. twolines()
In Colada[131] I re-analyzed data behind a recently published claim that the impact of AI use on creativity was U-Shaped, by running the two-lines test on the posted data. I did the analysis on a stand-alone online app.

But now that test is also part of statuser. To reproduce the two-lines figure in Colada 131:
twolines(Creativity ~ HumanAI +Intrinsic+Creative_Requirement+ Consc+Open,data=df)

Other functions

1. lm2():linear regression with informative output (Colada[133], the next post, will explain robust standard errors, and propose getting them with lm2())

2. message2(): like message() but with color, and can stop execution

3. text2(): like text() but you can do text(’hello’, align=’left’, bg=’red’)

4. scatter.gam(x,y): makes scatterplot, with a best fitting GAM curve

5. list2(x,y) a list where you don’t need to name objects, list2(x,y) = list(x=x,y=y)

Github: https://github.com/statuser

How to install in R
groundhog.library('statuser', date)
or
install.packages('statuser')

If you check it out, let me know what’s wrong with it, or what you wished it included.

2

Comparing statuser::desc_var() with psych::describeBy()

The syntax for describeBy is a bit clunkier, as you need to enter multiple groups as a list, and the output feels cluttered to me. I haven’t used skew or kurtosis before (which are reported by describeBy, but I always want to know about missing and unique values (which are not reported by describeBy):

The paper has three studies. Studies 1 & 2 are experiments. This post is about Study 3, which is described as a “field study”. I will argue the U shape in that study is spurious.

Note: this post is crosslisted in our blog DataColada

Study 3.
Study 3 involves a two-wave survey run on CloudResearch (a pool of online participants). In the first wave, a sample of “creative workers” rate on 7-point scales how much they use AI for work. In the second stage, conducted a week later, each worker’s supervisor (!) evaluated the employee’s creativity with a nine-item scale.

In an analysis described as ‘pre-registered’ 1, the paper reports results for a quadratic regression where the dependent variable is the supervisor’s creativity rating and the key predictors are the employee self-reported AI usage, and its square term. Both are significant. The paper says this “indicated a nonsymmetrical inverted-U-shaped relationship“ (p.9), then shows Figure 6.

Fig 1. Reprint of Figure 6 in the JEP:G paper

The paper says the graph implies that “the maximum value of creativity reaching 4.5. After this point, creativity should decline.” This inference, however, is not justified by the data. At issue are two different types of error: sampling error and specification error.

Sampling error
Just because the quadratic term is significant, it does not follow that the U-shape is significant. The most succinct way to make this point is with a figure, I simply added the confidence band to Figure 6 (I analyzed the posted data to compute that band).

Fig 2. Adding confidence band
R Code & data to reproduce this and other figures: https://researchbox.org/5159

There is a LOT of uncertainty in the estimated function, and the possibility that the association is monotonic, that more AI use is associated with more creativity, for all levels of AI use, is very far from rejected with the data.

Specification Error
Figure 6 in the paper, and my airbrushed-in confidence band, assume that the true association between AI Involvement and Creativity is perfectly captured by a quadratic function. That is, it estimates a quadratic regression and takes the resulting estimates at face value.

In two blogposts and a published paper I have explained why this approach is invalid, and have proposed to instead test for U-shapes relying on a “two-lines” procedure.

1. Colada[27] Thirty-somethings are Shrinking and Other U-Shaped Challenges
2. Colada[62] Two-lines: The First Valid Test of U-Shaped Relationships
3. Simonsohn (2018). Two lines: A valid alternative to the invalid testing of U-shaped relationships with quadratic regressions. AMPPS, 1(4), 538-555

With that two-lines test, this U-shape is not significant (p = .81). See figure below, generated with the (open source) online app

Fig. 3. Results from two-lines test.
(dashed curve is based on a GAM; breakpoint set with ‘Robin Hood’ procedure)

This call to use two-lines tests instead of quadratic regressions has been sufficiently successful; I will not rehash the two-lines arguments here [2].

I will instead give a new intuition for what the quadratic does wrong.

New intuition for problematic quadratic: effect here based on data over there
Let’s start where one should always start, plotting the distribution of the underlying data.

Fig 4. Distribution of key predictor: self-reported AI usage

We see that 38% of the data has AI values between [1 & 2).
In contrast, only 7% of the data is between [6 & 7].

This means that the (quadratic) regression, when fitting these data, will pay A LOT more attention to values [1 & 2), than [6 & 7]. OK. That’s intuitive enough. The next thing is less intuitive.

When the quadratic regression produces that inverted U, a negative slope for values [6 & 7], it is not because it is detecting a negative slope in that range, instead, it is because it is detecting a very positive slope in a different range, [1 & 2).

The regression is literally bending over backwards to fit a steep curve between 1&2 that then flattens out between 2 & 5. It is bending over in the range between [5 & 7], but only to accommodate the flattening of the curve at lower values.

More generally and precisely, the shape in any part of the curve need not be an adequate or interpretable summary of the local data.

A heartfelt statement by a quadratic regression
In an exclusive conversation with a quadratic regression, it shared with me this statement:

Statement by a Quadratic Regression
Look man, my job is to get close to the data overall. If you give me data where lots of observations have a steep slope, I cannot afford to not pay attention to that, so I do; I will give you a steep slope back, no problem. I can then even deliver a flattening of the curve for nearby data. But I cannot deliver a plateau, cannot deliver curves that stay flat. Eventually I must go negative as I get further from this flattening section; will I produce a U? Yes. Do I care? No. I am not paid to get shapes right. I am paid to get close to data. A spurious U will get a few datapoints wrong, but I am getting a lot more datapoints right in return, so I go for it, I cut my losses. The U is a ‘you problem’; I don’t see shapes. If you care about shapes, that’s a deal-breaker. I should see other people, and you should see other tools.

I will show two more figures just to provide some intuition for this point.

Without low values, no reversal for high values.
First, let’s rerun the quadratic regression in the paper, but excluding those observations with low values, between [1 & 2). If the spurious U-Shape is caused by them, we may expect the U-shape to go away, and it does. This highlights that when the quadratic reports a negative slope between 5 & 7 it is not doing that based on data between 5 & 7, that data is still there, it is doing it based on data between 1 and 2.

Fig 5. Fitted values and confidence band from quadratic regression on observed data, dropping steep section

Producing a U-shape there by change data here
For a final figure I took the real AI values but I produced a fake dependent variable, so that I would have control over the true functional form. I generated three functions. All three are monotonic, more AI always produces more creativity, but I varied how disproportionately steep was the function between [1 & 2]. Going from steep, to steeper, to steepest.

The steepest function in the range between [1 &2] produces a spurious U-shape for high-AI values.
The merely steep function does not.

Fig 6. The negative slope in the quadratic can result from a steep positive slope far away in the data
R Code & data to reproduce this and other figures: https://researchbox.org/5159

Feedback policy
Our policy is to contact authors whose work we cover to receive feedback before posting. I first contacted the author a bit over two weeks ago. We had a cordial and constructive exchange, where he clarified some of the details I thought peer-reviewers at JEP:G should have made sure were included in the paper (namely, how the sample was constructed and what the response rates were). The author provided feedback on wording that I tried to incorporate. He noted that his analysis, a quadratic regression to test for U-shapes, is mainstream in his field and mentioned a 2013 paper by an influential researcher, published in Psych Science, that he had used as a benchmark (htm).

Share

Footnotes.

1

The analysis in the pre-registration (https://aspredicted.org/DC2_WKN ) does not indicate which covariates will be included in the regression. It also doesn’t specify how the quadratic regression will be interpreted. It reads:

The paper does report results for alternative specifications in the supplement, and the main text notes that the quadratic effect of AI is not significant when all covariates are excluded (p = .14, page 3302). There is also something a bit odd with sample size. The sample size reported in the paper, N=188 pairs, is larger than that in the AsPredicted pre-registration (N=150), but the paper apologizes for it being too small.

Neither expected nor observed return/completion rates are reported. I asked the author about them who indicated that 250 workers provided information, and he later obtained responses from 188 supervisors “who could be matched to those workers”. He also explained that “although the sample size reached the pre-registered target for practical considerations, it still did not reach the larger target specified by the power analysis requested during the review process”.

ResearchBox is hosted by the Wharton Credibility Lab, which I co-direct. We also host the pre-registration platform AsPredicted, and a new platform for results provenance we just launched, AsCollected. I will write about AsCollected in another blog post soon.

It’s not just me: ResearchBox is easier to use and more effective than the OSF
I find ResearchBox very easy, even pleasant to use, fast, and effective. But I was involved in designing ResearchBox so I can’t really judge. You may be unsurprised to learn that I also think my kids are remarkable, and the food I ate as a kid is some of the best in the world1.

My situation, then, is that I have an intuition —ResearchBox is better— but don’t know if it is actually true for other people. It turns out that as a behavioral scientist I have just the right tool to figure out whether my intuitions are universally true: design a Qualtrics survey and give it to MTurkers. I took the OSF page (htm) of a JEP:G paper (htm) we had just read in journal club, downloaded all files, and created its twin repository on ResearchBox (htm).

They look like this:

I then randomly assigned N=30 Cloudresearch participants to get the OSF or the ResearchBox link and asked them questions about the files. I ran a ridiculously small sample because I expected ridiculously large effects. The results supported my intuitions.

Question 1. Is the data for Study 4 available?

Question 2. Can you download the “Study Materials” for Study 3?

Question 3. How easy to use is the website? See figure

Naturally, there are important limitations to this exercise, including that I only used one repository and that these differences may attenuate with experience. Maybe PhDs using the OSF for years are as fast navigating the site as MTurkers are navigating their first ResearchBox.

ResearchBox is more transparently permanent than before
Every data repository needs to think about how files will remain available after the platform ceases to exist. Like the OSF, ResearchBox relies on the Web Archive (archive.org) for this purpose. When a box becomes public, a zip file with its entire contents is saved to a single archived folder containing every public ResearchBox (see that folder here).

In addition, ResearchBox now also backs up all public boxes to Zenodo, a European Union funded open science repository. Zenodo also generates DOIs. ResearchBox then generates the following recommended reference for each public box, it shows both the current URL (underlined in red) and the backup DOI for Zenodo (underlinen in blue)

Codebook requirements took a chill pill
From the very beginning, every dataset on ResearchBox has had a codebook (you cannot post data without a codebook). While it generally makes sense to describe all variables in a traditional tabular format, sometimes there are too many variables and sometimes it makes more sense for authors to do their own thing describing the data (with free text). So we have relaxed the requirement that all variables be described.

Now authors can describe as few as 10 variables in a dataset, and can provide a free text codebook if they prefer. We still encourage people to describe all variables.

Upload Facelift
Among many updates to the page where authors upload and organize files, there are now automatic dividers to visually group of file, like folders. By default, files are sorted by section, and look like this:

To group files by type, we just click that header and:

More integrated with AsPredicted
We have merged accounts with AsPredicted. Users logged in into one are logged in into the other. The platforms feel like a single website, navigating across them is seamless.

That middle-tab, “AsCollected”, is the topic of the next post.

Before I go
I emphasized here new things about ResearchBox, but some old things are valuable too.
The platform has many features no other open research platform has, including:

• The platform alerts authors if their posted code includes their name or email.

• The platform alerts authors if their posted data has participants identifying information (phone numbers, prolific IDs, IP addresses).

• Posted datasets can be searched by variable name or description

  • e.g., I just checked and there are 160 variables described or named with WTP (willingness-to-pay) and 24 with “Trump“

• Posted analysis scripts are searchable too.

  • e.g., I just checked and there are 349 scripts using the lmer() function for mixed models in R, and 150 scripts using the groundhog package.

• Every file can be instantaneously previewed by clicking on the name, even large datasets take about 2 seconds to be previewed.

Note: Colada is experimenting with Substack, this is our first post here, we will continue posting in DataColada.org

Subscribe now