As we head into the world’s largest edtech gathering this week, I find myself thinking about this again and the stories people are telling about our work.

Last year at this time, the anti-edtech movement was a small ember. I wrote then that we needed to do a better job of engaging with it and grappling with dialectical thinking: Intentionally considering ideas contrary to our own. I wrote that our community needed to get outside of our bubble.

We’re starting to do that. I’m seeing more conversations about pedagogy and efficacy in edtech spaces than I ever have before. Several featured sessions this year (see here, here, and here, among many others) are wrestling with the difficult questions being raised about technology. ISTE President Richard Culatta is grappling with it. Influential edtech funders are too. Holly Clark and I will be co-presenting on it.

I’m heartened to see this willingness to engage with ideas we might disagree with.

How might we get the anti-edtech crowd to do the same?

We can’t do it with empirical evidence alone, try as we might. Quibbling over evidence and validity does little to assuage the emotions of those whose views have been hardened by years of provocative media coverage.

As Mr. Mo would say, others are telling our story for us. And they’re telling it well.

Edtech Insiders spoke recently about why the anti-edtech story is such a compelling one:

• It’s scary: It’s about distraction, addiction, isolation, and harm to our most vulnerable population.

• It has villains: Sal Khan, iReady, Youtube. “Big Tech.”

• It has heroes: parent groups, contrarian researchers, the lone teachers out there “standing their ground” against technology.

• It has a happy ending looming in the distance: Banning it all for good.

• It’s easy to understand and validating.

To paraphrase a great writer: It’s not that this story is untrue, it’s that it’s incomplete.

There are many other stories about technology in classrooms. There’s a reason thousands of educators, during their summer break, are going out of their way to attend ISTE this week. These teachers have created moments of connection, belonging, and deep learning in their classrooms that wouldn’t have been possible without the affordances of technology. Their stories are real, too. I’ve been excited to see more and more of them of late.

In that spirit and the spirit of Mr. Mo’s advice, I’d like to retell one of ours. I think it speaks to what an amazing educator can do when empowered with technology. I’m excited to hear more of these stories this week:

The following was originally posted in February of 2025.

She waited until no one else was there before walking up to our booth.

“I’m going to try to tell this without crying…”

We suddenly had welled-up eyes, too.

“I teach at a very small school here in Texas. We’ve had several students join our school recently who can’t speak any English. Not a word. And so it’s been very challenging. But we used your tool, and I had these students write their responses in Spanish. And we did it in Chrome, so when they read each other’s writing, their Spanish was translated to English and the English responses were translated to Spanish for them.

I’m glad I didn’t look before starting the feedback part because I’m not sure I would have had the courage to include hers.

One of my new students wrote about how, back in Mexico, she was one of the smartest kids in school. And how she loved school…

…and how, since coming here and not being able to speak the language, people look at her like she’s stupid. And she’s never been treated like that before…

…and her writing was selected by her classmates as the top response…

I think it was the first time she ever really communicated with her classmates. And it started such a meaningful conversation as a class. And ever since that moment, we’ve noticed such a change in her and how she interacts with our school.

I just wanted you to know that story. Because this truly made a difference for that child.”

We got out from behind our table and hugged.

The edtech vendor hall is not typically a place for these moments. It’s too often a place for the opposite: Transactional sales gimmicks and bad fluorescent lighting. Forced conversation. Avoiding eye contact.

It’s funny, then, that maybe one of the most meaningful moments of my career happened in a vendor hall. I’ll never forget that story. Both because of the impact on that child and teacher, and because of the impact it’s already had on my wife, myself, and our work moving forward.

A great educator recently suggested that, in edtech, you either bet on teachers or you bet on software. When you bet on teachers, and design things primarily interested in empowering human-to-human connection in the classroom, special things can happen.

It can be easy to forget that when you spend too much time in the vendor hall.

Beyond the commerce, real human stories are unfolding. Stories of marginalized students rediscovering their identity and teachers bridging impossible social divides. Small moments, “on fire with the same force that lit the stars.”

Her story was a reminder that technology is most powerful when it gets out of the way and makes human connection possible where it wasn’t before. More human-to-human connection is critical for our educators and students. I leave Texas with that on my mind and in my heart.

The Chromebook reseller next to us probably summarized it best.

He leaned over the black metal booth divider after she walked away and whispered, “That was powerful.”

• A 4th-grade teacher in LAUSD recently wrote about how his district’s proposed screen-time bans won’t serve teachers and students.

• A family in California was featured by NPR over a fear of losing some of the positive affordances of technology in education: Voice-to-text and text-to-speech.

• School leaders in Massachusetts measured how often screens are actually being used and for what purposes. Kindergartners were found to be on devices for about 20 minutes a day, almost 99% of that time being instructional. High schoolers averaged a little over an hour a day during a 6½-hour school day, with about 66% of that being instructional. Rather than a ban, school leaders recommended no devices at lunch and recess and no use of screens as a reward. The Superintendent concluded that reform “requires a scalpel more so than a chainsaw.”

• Securly, a web filtering and security service used by schools, analyzed YouTube use across 380 districts and 600,000 students. The average elementary student spent just 5 minutes per day on YouTube. Less than 1% of students used YouTube in the ways espoused by the anti-edtech movement and in click-bait anecdotes.

• Dr. Chris Ferguson, a psychology professor at Stetson University, reviewed Jared Cooney Horvath’s book and found significant timeline mismatches in the evidence provided, manipulative data visualizations, an overstating of declines in test scores, and (as I wrote about recently) contradictions by recent meta-analyses demonstrating educational technology generally improves learning.

• Dr. Natalie Wexler, a literacy expert and author of The Writing Revolution (we led a great webinar with this team!), wrote about how the arguments made by Horvath are papering over issues that existed long before technology became ubiquitous.

• Dr. Peter Bergman of the University of Texas criticized Horvath for his unscientific presentation of effect sizes in his Senate testimony. This is in line with a private conversation I had recently with a researcher who described the testimony as “appalling.”

• Dr. Matthew Kraft, an influential education researcher at Brown University, pushed back on sweeping negative statements about educational technology.

• Dr. Justin Reich, a media professor at MIT and world leading expert on educational technology, spoke to a parent group that advocates for the removal of screens from schools. Reich was equivocal about technology, but when asked if screens caused declining test scores, he said: “It’s very unlikely that it is the school adoption of technology that is related to decreasing scores.” Asked if he felt the immediate removal of screens would improve outcomes: “Almost assuredly no.”

This is heartening.

I fear none of it will matter.

This has never been about evidence and reason, but rather emotion and confirmation bias. Horvath knows this. He tacitly acknowledged it in an interview with NBC News:

“There’s no way in hell my book has this big of an impact… my thought is it (“it” referring to anti-edtech sentiment) was there, it was fomenting, it was always about to happen. It’s just people needed the arguments, and I think that’s probably where the book kind of slid in and just said, ‘Here’s the word you’ve been looking for.’”

To translate what I think Dr. Horvath actually said there: “Parents were already primed to be anxious about technology, so I presented research in a way that validated their anxieties. And because I have a background in neuroscience and “Dr.” in front of my name, it reads as authoritative, so it took off.”

A parent quoted in the story reinforces this interpretation:

“As parents, we feel a lot of imposter syndrome sometimes when we’re talking about this…so having a book written by someone with a background in education as well as neuroscience added a lot of credibility to what parents were experiencing.”

Parents are genuinely exhausted and anxious about the digital world their kids are navigating. I empathize with this. What I can’t abide is a credentialed expert coming along and essentially saying, “Your anxiety is 100% correct, and here is a singular villain to blame.” As is becoming clearer by the day, existing educational research contradicts this narrative. We’ve seen this before:

Subscribe now

The Haidt Precedent

Two years ago, Jonathan Haidt’s The Anxious Generation exploded in popularity despite a lack of alignment with mental health research.

In a review for Nature when the book was released, Dr. Candice Odgers, Associate Dean for Research & Professor of Psychology at the University of California, Irvine, found that Haidt’s conclusions were “not supported by science.” That conclusion is supported by research from Dr. Amy Orben, Dr. Andrew Przybylski, Dr. Matti Vuorre, Dr. Patti Valkenburg, Dr. Sonia Livingstone, and (the aforementioned) Dr. Christopher Ferguson. Their work was recently compiled by Dr. Sara Grimes, a children’s media researcher at McGill University, who published an explainer in an effort to help push back against Haidt’s work.

What strikes me is that Dr. Odgers’ critique maps almost perfectly onto Horvath’s anti-edtech crusade:

Giving credence to this corollary is Horvath himself. He said he was inspired to write his book because of the success of The Anxious Generation: “The next apple up the tree was always going to be ed tech,” he said in the recent NBC news story, “because once parents realize, ‘Wait a second, we can push back against cellphone use in schools, then that means we can push back against Chromebook use in schools too.’”

As I’ve said before, using cellphones and social media is completely different from using school managed devices under the guidance of a licensed educator. That hasn’t stopped folks like Horvath from equating them, validating parental anxiety in the process.

Validation-by-Virality

Will the truth win out with edtech, or will virality?

Recent precedent set by Haidt makes me pessimistic.

After being rejected by most researchers, his book went to #1, spent 52 weeks straight on the bestseller list, sold two million copies, and became the basis for major legislation around the world. As a result, major media outlets continue to lift up his work as authoritative. Worse, policy-makers continue to listen. Randi Weingarten, leader of the largest teachers’ union in the country, recently cited both Haidt and Horvath to justify her pivot towards supporting sweeping “screen-time” regulations in schools.

Evidence proved to be no match for viral opinion validation.

Meanwhile, a recent Stanford-led study of the cell phone bans in schools championed by Haidt showed almost no positive impact on test scores, attendance, attention in class, or cyberbullying. Little impact has been seen for social media bans.

This isn’t surprising. Many of these findings are in line with previous research. We just ignored it.

The evidence-based pushback against Horvath gives me hope that we can avoid making this mistake again. Hopefully it can gain enough traction before viral narrative once again hardens into misguided policy.

Hopefully.

Subscribe now

On one level, this was all heartening. I’m glad so many folks connected with the research. Many educators reached out with messages of support for the articles and educational technology in general. It was a hell of an ego boost to have national news outlets asking for interviews.

On another level, it’s depressing that the most-read thing I’ve ever written is a tirade that I was uncomfortable posting to begin with. I didn’t fully appreciate how much controversy drives attention. No wonder Penguin Random House has decided to publish Horvath’s manifesto. Controversy sells.

What controversy doesn’t do is serve the best interests of students and teachers. To that end, in this and future posts, I want to share some ideas about what healthy edtech reform might look like.

Right now, it seems that policy is the most pressing domain for reform, as 17 states have now introduced or passed laws limiting or banning “screen time” in classrooms. These proposals are misguided for a simple reason: They ignore quality of instruction. Whether a teacher is using research-backed, high-quality resources and practices is deemed irrelevant. The medium of instruction is, somehow, more important than the quality. ISTE president Richard Culatta pointed this absurdity out recently.

So, under some of these laws, a PreK teacher who wants to use, say, Khan Academy’s free (!!), research-backed digital reading curricula and practice activities would be forced to replace them with a paper-and-pencil alternative. This is true even if the paper-and-pencil alternative is low quality. Coloring in letters of the alphabet with crayons? It’s somehow better simply because it’s analog.

This is nonsense. It’s what happens when we misunderstand a device as an intervention rather than a medium.

A more helpful question for lawmakers: How do we bend edtech towards high-quality instruction? Here are three policy ideas:

1. Require ESSA Level 4 efficacy.

Every edtech product must have at least a research-based rationale as to why its product will improve outcomes for students. In order to become a registered vendor with a district, we could be required to share our ESSA certification along with our W9 form.

The problem here is that certification is not possible without a credible certification body. The current political climate and gutting of the Department of Education make federal oversight difficult, but the need for an objective arbiter has never been more urgent. To move beyond claims, we need…

2. A Food and Drug Administration for EdTech

We need a government body responsible for verifying the educational efficacy and privacy practices of educational technology (and, for that matter, all education interventions). Edtech companies should be responsible for supporting and funding their own efficacy research, both through their own revenue and grants, but there are also misaligned incentives that need to be addressed with this. It’s in our interests for research to come back favorably. It can therefore be tempting to skew, misrepresent, or hide findings.

This happens all too often.

TurnItIn recently misrepresented the efficacy of their AI writing detection product. An independent study showed that TurnItIn “should not be used in academic settings,” because of how wildly inaccurate it is (wrong over 20% of the time). TurnItIn took that research, cherry-picked quotes, and used it to conclude in a white paper that their product is, in fact, “reliable and effective.”

I experienced something similar at a conference recently. A competitor had an adjoining booth in the exhibit hall. Over the course of two days, I heard about how their product “improved writing scores by 20%.” I searched their website and found a case study in which one teacher’s state test scores rose 20% from one year to the next. So the 20% rise was across two entirely different sets of students. There was no control group, statistical analysis, attempt to isolate the software from other factors, etc.

This is all it takes to manufacture “research” in edtech.

These sorts of claims need a “these-statements-have-not-been-reviewed-by-the-Food-and-Drug-Administration” stamp equivalent.

Because, here’s the real rub: I’d be lying if I said, at the end of those long days in the booth, I wasn’t tempted to put together our own case study showing something similar. When a teacher comes up to our booth and says, “Their product improves writing by 20%. Does yours?”... it leaves you in a tough spot.

Do I tell the truth, or do I survive?

It’s easy to say “tell the truth.” But who would you rather work with?

1. A tool that “improves writing instruction by 20%” or

2. A tool that “.. is grounded in a research-backed theory of change, as it was my master’s project, but we’re a young company and still in the process of working to understand our efficacy, so we’re still working on causal evidence, but early survey data shows very strong self-reported results from thousands of educators across widely different contexts.”

You can see the race to the bottom forming. Without a uniformly recognized external validation body, claims become equivalent to results.

This is not a new idea. ISTE has a product index. EdSurge has theirs. Common Sense Media had one, but couldn’t keep up with the volume of applications and updates. Johns Hopkins has attempted to pick up the slack after the gutting of the Department of Education and IES, but given that their current program doesn’t even measure instructional interventions related to something as foundational as writing, I have questions about their ability to keep up with the speed and size of the market.

More importantly, I’ve never been asked about any of these indexes during procurement conversations. A government body is needed to create incentives so that I am not just asked, but required, to show that my product actualizes practices that are at least supported by a research-backed logic model.

3. Experiment with outcomes-based contracting

Here’s the problem with everything I just said.

Edtech is ever-changing. We could run a massive, randomized control trial on Short Answer as it exists today and establish causal evidence of improved writing outcomes as a result of using it. Do those findings matter if over the course of the year(s) it took to run the study, we launched 15 new features that completely change our tool? We end up measuring tools that no longer exist.

This is the conundrum of tech and research. Tech moves fast. Research moves slow(wwwww).

A way around this might be outcomes-based contracting (OBC). I became aware of this system through the work of The Center for Outcomes-Based Contracting.

In OBC, schools pay a portion of the full price for edtech products (say 60%). They pay the other 40% when the product demonstrates improved outcomes. Vendors and schools define what expected outcomes look like: “Writing scores on the state test will improve by 15%.” The necessary level of implementation is established: “All teachers will receive at least an hour of training on how to use Short Answer by X date. They will use it at least 3 times per week and complete all content for their assigned grade level by X date.” How each step is documented is determined in advance.

If the implementation plan is followed, but results don’t accrue, companies don’t receive the remainder of their payment.

This aligns incentives.

Schools are incentivized to train staff and use the products they purchase, which has been a problem for a while. If they don’t, they’ll have to pay. Early results show that schools that use outcomes-based contracting have 10X better implementation than the national average. For their part, edtech companies would now also have to focus more on improving outcomes rather than just juicing engagement.

This system is intriguing, but also flawed. Many schools don’t have the bandwidth to manage it. There are just too many stakeholders involved in edtech procurement (the school board, instructional leadership, business office, tech department, building principals, instructional coaches, teachers, etc) to easily align this work. Early research echoes these concerns.

As someone in the trenches of edtech procurement, I can tell you from first hand experience that many districts struggle to even manage existing purchase order processes. The idea that they could handle conditional constraints on contracts, measured based on a series of implementation checkpoints and assessment measures, seems like a recipe for a mess.

It’s not just schools though. Many companies would struggle to manage this process. The edtech market has always been a challenging one to build a successful business in. It’s given rise to more micro-companies like ours at Short Answer: Intentionally small teams that don’t scale as the business scales. This structure allows us to better focus on impact and sustainability, but the idea that our small team would be able to handle the logistics of outcomes-based contracts at scale seems far fetched. We’re already stretched to the limits of our bandwidth.

It’s why I’ve gone back and forth on this. In my interview with Edutopia, I actually talked about why I think outcomes-based contracting is a bad idea. But as I’ve read more about it, I think it could be a good option to help deal with the asymmetrical nature of educational research and edtech development. We’ll need more experimentation.

In sum

The “BAN IT ALL!” laws sweeping the nation are misguided and lazy. They ignore the quality of instruction in favor of the medium. It’s nonsense, but it’s easy to legislate, so it’s what we’re doing.

Building an “FDA for Edtech,” enforcing real efficacy standards, and rethinking contracts is not easy. It requires bureaucracy to move faster and school districts to build capacity they currently don’t have. But the alternative is what we have now: a race to the bottom where the loudest claims win, a market in which operating with integrity starts to almost feel self-righteous, and a seemingly never ending cycle of controversy and debate.

Controversy generates article views, gets you interviews with the press, and sells books. It doesn’t improve outcomes for kids.

We don’t need fewer screens. We need higher-quality instruction and verified results.

Like Ms. Cherkin, Dr. Horvath makes some valid points that we in the Edtech industry should heed. Unfortunately, they’re buried beneath cherry-picked statistics, correlational evidence, logical fallacies, straw men, and moral panic.

In his book, Dr. Horvath addresses “5 Lies” told by “Edtech.” In the spirit of that rhetoric, I’ll address 5 lies told by Dr. Horvath.

1. Student-facing-device-as-instructional- intervention

Within the first 5 pages, the central flaw of The Digital Delusion becomes clear: Dr. Horvath views ‘EdTech’ as a singular instructional intervention.

This is wrong. A laptop is an instructional medium, not an instructional intervention.

Dr. Horvath should know this, given some of the research he cites.

In his landmark Visible Learning: The Sequel, Dr. John Hattie dedicates a chapter to instructional technology. After recounting generally positive effect sizes for the different ways in which it can be used to actualize pedagogy (“The message throughout this chapter is not that technology has not made a difference. It has, and that is what an effect of 0.3 to 0.5 signifies” (p.408, Visible Learning), Dr. Hattie says the following. Bolding is mine:

“If there is to be a third edition of Visual Learning, there would not be a chapter on technology influences, as it is time to think of technology like curricula, books, and other resources, and focus more on the implementation effects when including technologies and teaching methods, the when and how within the teaching cycle, the dosage and the fidelity of various apps, and so on….”

The UN Report previously cited by Ms. Cherkin makes a simular argument. To quote that report: “This confusion between the tool and the solution, between the means and the end, is what this report invites us to address…Technology should not be viewed as the solution, but as a supportive tool.”

“Edtech” isn’t a single treatment, it’s a blanket term for thousands of different digital curricula and applications. You can’t aggregate the impact of all of them simply because they all have a screen and connect to the internet. To do so is bad science, as is argued by Dr. John Hattie, the United Nations, and many other researchers.

To me, this undercuts the entire thesis of the book. It also leads directly into some of the logical fallacies that follow.

2. Straw Men

Dr. Horvath argues that “Edtech” perpetuates 5 lies:

1. Education is broken

2. Multimedia enhances learning

3. Free choice leads to better learning

4. Kids learn best on their own

5. Intelligent tutors make kids more intelligent

Ummmm. Okay. First, who is “Edtech”? As someone building an instructional technology tool, I disagree with points 1, 3, and 4. I think points 2 and 5 can be true, depending on the tool, the problem it’s being used to solve, and the context of the problem. Given that I consider myself a part of the Edtech community, I was curious to learn which individuals or organizations are perpetuating these lies.

Dr. Horvath doesn’t provide sources.

For example, while explaining the first “lie” that Edtech perpetuates, that education is broken, he says: “This narrative has been repeated so many times in op-eds, on TED stages, and in keynote speeches that it almost feels self-evident.” He then fails to cite a single op-ed, TED talk, or keynote speech.

It seems he’s referencing Sir Ken Robinson’s famous Do Schools Kill Creativity? If so, what does that have to do with technology?

Somewhere in the middle of slaying each of these 5 straw men, I started to wonder: Who published this? How could they let him say all this without citing specific companies, people, or organizations perpetuating these lies? Then I looked at the copyright page and realized it was self-published.

3 & 4. Correlative, cherry-picked evidence

Chapter 3 of Dr. Horvath’s book is its most important chapter, as it presents the corpus of evidence he feels justifies “pulling the plug” on Edtech.

Dr. Horvath cites declining PISA scores (the world’s largest standardized test) from 2012, 2015, and 2018, which correlate with the widespread adoption of educational technology in developed countries: “The more time students spent on screens at school, the further their scores fell. Those who used a computer for more than 6 hours per day scored 66 points lower than their peers who didn’t use them at all.”

This argument is a logical fallacy: Post-hoc-ergo-propter-hoc. After, therefore, because of. Correlation is not causation. Just becuase EdTech use increased, and scores dropped, does not prove that EdTech caused the drop. People who carry umbrellas are more likely to wear rain boots, etc., etc.

None of the research on test scores presented is causal. Let’s look at it anyway, as we’ll find that even the correlational evidence was cherry-picked to fit a narrative.

Here is the OECD analysis correlating screen time and PISA scores (see charts above and below). The study showed a correlation in which students who spent 5–7 hours per day on “digital devices” for leisure at school scored 10–12 points lower than students who spent 3–5 hours per day.

First, Dr. Horvath has left the “for leisure” qualifier. Second, he’s left out that this study lumps all digital devices under one label. No distinction is made between personal cellphones vs. school-managed laptops. These are both critical distinctions if we’re going to use this study to justify school laptop policies.

So, I guess in a sense, he’s right that spending the majority of your school day using a device for leisure correlates with negative learning outcomes. That said, I don’t know of a single technology integration specialist or Edtech developer arguing that kids should be spending 7 hours per day using a digital device for leisure.

Regardless, other important data have also been left out (see chart above):

• Students who spend one to five hours per day on digital devices for learning at school achieve twenty PISA score points higher in mathematics than those who spend no time on such devices.

• 15-year-old students who use digital devices moderately for learning at school tend to perform better and report a greater sense of belonging at school.

• (again, because it bears repeating) Most of the negative correlations were related to student device use for leisure, not student device use for learning.

Why were these points left out? Dr. Horvath even acknowledges them later in the book (see p.78-79), but argues “learning isn’t unique to computers,” so these researchers are wrong. If the researchers are wrong, why are you citing them?

This becomes common when you engage with anti-edtech folks. The conversation goes:

Them: Show me the evidence that technology can help learning.

Me: Okay, here’s some evidence.

Them: …that evidence doesn’t count.

Another critical point to note: Many great technology tools aren’t primarily designed to improve standardized test scores! If a tool’s theory of change isn’t focused on raising PISA scores, then declines in those scores don’t falsify their value.

I’m building a digital writing tool. Writing ability is not directly measured on PISA. How do declining PISA scores falsify the value of the writing tool I’m building?

These illogical conclusions happen when you ignore the advice of researchers and erroneously lump all digitally mediated interventions under one misleading “Edtech” banner.

Dr. Horvath continues with the correlational evidence between declining test scores and technology usage using Trends in International Mathematics and Science Study (TIMSS) and Progress in International Reading Literacy Study (PIRLS) data. He simply references scores on the respective websites of these tests. If he were to seek out studies of these scores, here’s what he’d find:

Researchers warn against his exact approach. Simply correlating aggregate tech use and test scores is not useful. Researchers rightfully argue, similar to Hattie and the U.N. previously, that we should instead study effective tech use & scores vs. ineffective tech use. They argue for this precisely because there are so many experimental and small-scale studies showing positive effects of technological applications on student performance in various subject areas such as literacy, mathematics, and science (Chauhan, 2017; Cheung & Slavin, 2012, 2013; Sung et al., 2016). I would also add this recent meta-analysis by Dr. Rebecca Silverman at Stanford, which, among other things, showed a positive effect size for digitally mediated early childhood literacy tools. Because of this, Dr. Silverman concluded that “investment in educational technology to support literacy is warranted, and the prevalence of educational technology in schools is justified.”

Not only does Dr. Horvath not include this mountain of research, he does the opposite of what the researchers in these studies advocate for. Why?

Dr. Horvath does acknowledges Hattie’s meta-analysis showing positive effect sizes for Edtech. He then does mental gymnastics to deny them, essentially arguing “yes, they are positive, but other interventions have higher effect sizes, so we should use those instead.” To quote him: “So yes, students can learn with EdTech: the effect sizes aren’t negative. But this learning is almost always slower, shallower, and less durable than what’s achieved through human-centered methods like explicit instruction (+.64), guided discussions (+.82), and jigsaw teaching (+1.16).”

The misunderstanding of technology as a intervention rather than a medium has again led into a logical fallacy: False dichotomy: You either use “Edtech” OR you use explicit instruction, guided discussion, etc. This fails to acknowledge that…YOU CAN USE EDTECH TO AUGMENT EXPLICIT INSTRUCTION, GUIDED DISCUSSIONS, AND JIGSAW TEACHING!

I’ll share two examples that I have personal experience with.

The Modern Classrooms Project uses technology exceptionally well to improve explicit instruction. Because lecture and modeling are recorded, students can pause, rewind, speed up, and consume content asynchronously (that last one is critical in high-poverty districts with high absence rates). This allows students to work through guided and independent practice at different paces in ways that would otherwise be impossible with only live, teacher-led instruction. Research done by Johns Hopkins University found this approach significantly improved teachers’ ability to differentiate instruction, reduced stress, boosted student self-efficacy, and improved self-directed learning skills. I used this framework while teaching in Chicago and experienced all of these benefits. They wouldn’t have been possible without access to technology.

For guided discussions, I’ll submit the writing tool I’m building. Our theory of change at Short Answer is, in part, to improve guided discussions precisely because they are so valuable for learning. As an example, we recently launched an activity in which AI-characters argue about students’ writing, then the class discusses and votes for which character they agree with. Teachers tell us that kids will come up to the front of the room and read aloud the feedback in the voice of the characters (i.e. Surly Sam, Cool Cali, etc.) and that debates are so lively, they have to cut things off to make sure the activity doesn’t run too long.

These are two of hundreds of examples we could share as evidence of the false dichotomy Dr. Horvath presents. Given they’re somewhat anecdotal (and in the case of Short Answer, could be viewed as self-serving) I would encourage you to check out the previously listed meta-analyses showing causal evidence of improved learning outcomes associated with different digital learning applications.

That causal evidence is ignored by Dr. Horvath. It doesn’t fit his narrative.

5. Neuro-garbage

Dr. Horvath concludes the evidence section of the book by arguing that chemicals and structures in our brains make technology an inherently flawed medium for learning. He leans heavily into his background as a neuroscientist here (“..rulesets must be loaded into a small area called the lateral prefrontal cortex (LatPFC)” etc.).

Some of what he argues is true. Split attention is bad for learning. Technology can be distracting. Empathy and belonging are built around human connection, which screens cannot offer. Technology can sometimes make things too easy.

This is all overstated, as it ignores context.

Most of the studies cited in this chapter were done at the neurological level or in the context of large, undergraduate courses on personal devices with no content filters. Most importantly, neuroscience ≠ classroom practice, because teaching is inherently social and complex.

Dr. Daniel Willingham has referred to these sorts of arguments as nuero-garbage: using neuroscience language and concepts in education that overstate what brain findings can tell teachers about how to teach. To quote Dr. Willingham:

“We can’t take lab findings and pop them right into the classroom… lab findings consistently show that repetition is good for memory. But you can’t mindlessly implement that in schools--”keep repeating this til you’ve got it, kids!” Repetition is good for memory, but terrible for motivation.”

Neurons are not classrooms. They describe what is happening inside a brain; classrooms describe what is happening between people. The behavioral outcomes teachers care about (e.g., reading fluently, solving equations, writing, etc.) are shaped by instruction, motivation, and social context. The technology tools teachers use operate at that same level.

Pointing to a brain region that lights up during a grammar lesson doesn’t tell a teacher what to do on a cold Thursday morning in February when Bart keeps making fart noises and Lisa is mad at Jessica because Steven told her something he wasn’t supposed to, so now Lisa is acting out and distracting Jackie, who is now behind and confused about her grammar lesson.

There’s a gap between knowing that a neural process exists and knowing how to design instruction around it. It’s never one-to-one in the way that Dr. Horvath presents. There are too many ever-changing variables in teaching to reduce it to laboratory science and neurological chemicals. It’s why “good teaching”, and even intervention effect size, are deeply contextual.

The best I’ve seen this explained was by Dr. Dylan Wiliam in his Why Teaching Will Never Be a Research Based Profession lecture.

Brain science doesn’t show that technology is incompatible with learning. It shows that attention, retrieval, feedback, and cognitive effort drive learning. Technology can both support and undermine those processes, depending on how it’s designed and implemented.

For those interested in what good instructional design looks like in digital applications, check out Dr. Karen Forsell’s work. Most of what I know about it comes from her. For those interested in what good technology integration looks like, I’d encourage you to speak with one of the amazing instructional technology coaches at your local school district.

Conclusions

I became aware of Dr. Horvath’s work when I presented at a ResearchED conference in 2024. He gave an inflammatory keynote.

I never published my post about the experience, as I felt it was unproductively inflammatory, too much in the vein of Dr. Horvath’s rhetoric. Well, you’ve just read much of what was going to be in it, so I want to close by explaining why I’m sharing this.

If Dr. Horvath and Ms. Cherkin were simply writing books and selling their consulting services, we could ignore them. They’re not. They’re testifying in front of the United States Senate encouraging lawmakers to strip educational resources from the hands of educators.

Twelve states, including California, Florida, and Missouri, are considering moving beyond cellphone bans and towards laws that will limit “instructional screen time” in the vein advocated for by Dr.Horvath and Ms. Cherkin. Some of these are as blunt as “teachers can only use technology for 45 minutes per day,” regardless of subject, grade level, or instructional purpose.

These blanket screen bans would be a profound mistake. They ignore the difference between poorly designed technology and thoughtful design and implementation. They would remove tools that tens of thousands of educators rely on to differentiate instruction, provide feedback, reduce administrative burden, and respond to the very real assessment challenges posed by AI. They would do so based on cherry-picked data, correlational evidence, logical fallacies, straw men, and moral panic.

We should improve how technology is used in K12 classrooms.

We should push back on arguments that treat technology as inherently incompatible with learning. Basing policy on this narrative would be a mistake.

Put another way: Educational technology is worth fighting for.

In testimony before the United States Senate last month, Dr. Jared Cooney Horvath and Ms. Emily Cherkin testified that educational technology should be removed from K12 classrooms. When asked how we might improve edtech, rather than ban it, Dr. Horvath responded:

“Man, I could ask you, “How do I make anthrax better?” Maybe the answer is you don’t.”

Pretty radical stuff.

It felt even more radical juxtaposed against the testimony of Dr. Jean Twenge, a researcher from San Diego State University, and Dr. Jenny Radesky, a pediatrician at the University of Michigan. Dr. Twenge and Dr. Radesky’s testimony focused on cellphones, social media, and their impact on children. This was the primary focus of the hearing. They made an even-handed, evidence-based, and compelling case for passing the Kids Off Social Media Act (KOSMA). KOSMA would ban social media for children under 13, bar companies from using personalization algorithms for users under age 17, and require schools to block access to social media on school networks.

It was jarring to contrast their testimony with the polemic, at times sarcastic, testimony of Dr. Horvath and, to a lesser extent, Ms. Cherkin. Both repeatedly attempted to push the committee beyond the scope of the current bill and strip almost all educational technology from K12 classrooms.

As someone who’s experienced the power of educational technology and who saw a measurable positive impact on learning outcomes in my classroom as a direct result of it, the testimony was baffling. As someone currently spending about 60 hours a week building an edtech tool that I believe in, it was infuriating. (I also want to admit, up front, that I recognize that this undercuts my objectivity to some extent.)

But.

Respectfully engaging with those who hold different opinions is important. I wrote about this recently, ironically enough, in response to an experience with Dr. Horvath.

In the spirit of the dialectical thinking I argued for in that post, I read up on the evidence that Dr. Horvath and Ms. Cherkin presented, both in their testimony and through their consulting companies.

I learned a lot. In some cases, they make great points that we in the edtech industry would do well to heed. In many others, they’re lying. Sometimes overtly, but more often in the style common in Senate subcommittee hearings: By omission and manipulation.

There is too much to cover here in one post, so we’ll do it in two. My core argument, for the TL/DR crowd, is this: Technology is a medium, not an intervention. Outcomes depend on implementation, and blanket bans ignore thoughtful instructional design and implementation. The right unit of analysis for edtech is use case, not presence.

With that, we’ll start with Ms. Cherkin, who presents “4 key harms” that she believes justify the banning of edtech in K12 classrooms: Student privacy, skill displacement, harm to learning, and harm to teaching.

On Student Privacy

1. “96% of applications used in schools sell children’s data to third parties.” A 2022 Internet Safety Labs report is used as evidence.

This is not true. The Internet Safety Labs report states that “Nearly all apps (96%) share children’s personal information with third parties.” Sharing personal information is completely different from selling it.

Internet Safety Labs uses a broad definition of “third-party data sharing” that counts any instance where student data is transmitted outside the app to another company’s servers. In this broad definition, any app that is cloud hosted qualifies as “data sharing.” It doesn’t matter whether it be crash reporting, hosting a database, or simply authenticating that someone has an email with Google so they can sign in. All is considered third-party sharing. Collapsing these routine, service-related data flows into the claim that apps are “selling children’s data” is a category error.

It’s illegal for edtech companies that sell their products into public schools to sell children’s data. COPPA and FERPA laws help protect against it, as do basic data privacy agreements (DPAs). Here’s a template of one we recently signed that is used by 16 US States. To quote section 4 of that DPA: “Provider will not Sell Student Data to any third party.”

Direct-to-consumer products in app stores downloaded on personal devices outside of school might be different, but as someone who spends an inordinate amount of time reading DPAs, I can tell you with confidence that 96% of edtech applications used in schools do not sell students’ data. It is against the law.

2. “If a company collects data, it is vulnerable to cyberattacks.”

Standard DPAs require that edtech companies conduct regular penetration testing and security auditing to guard against cyberattacks. Many also require that edtech companies carry multi-million dollar cybersecurity insurance coverage so that, in case of a cyberattack, the school district can be reimbursed.

Ms. Cherkin is correct that edtech services are prone to cyberattacks.… as are banks, insurance providers, hospitals, public transit records, the IRS, Social Security, and almost every other aspect of modern life that is digital and connected to the internet.

3. “Schools ask children to sign user agreements before utilizing any of the school-issued devices or required EdTech platforms. Parents are not fully aware of what their children are consenting to or how their children’s data is being used.”

This is true, but overstated. Parents often sign off on acceptable use policies without considering what they’re signing off on. Luckily, most are benign legal documents, akin to student handbooks, that exist to give districts avenues for recourse when students damage devices or intentionally use technology inappropriately. If you want to read one, here’s one from the district tech office I worked in.

Aside from this, COPPA protects families from both edtech and school district overreach. That law is in the process of being strengthened.

On Skill Displacement

4. “Time spent on screens is time not spent in other activities. Children 8-18 years old now average 7.5 hours per day on screens, outside of school hours…it doesn’t matter where “screen time” occurs: time spent on screens is time not spent doing other things.”

It is true that kids spend 7+ hours on screens daily, not counting school. This is deeply concerning, and what made Dr. Twenge and Dr. Radesky’s testimony so powerful. I generally support KOSMA because of it.

That said…what does this have to do with edtech? Why is there no distinction being made between a student scrolling through TikTok at 1 a.m. on a school night vs. a student logging into Google Classroom to complete a check for understanding quiz under the direct guidance of a licensed professional? Because they both take place on screens, we’re supposed to believe they’re equally bad. That is nonsense.

Kids spend too much time on their cell phones and social media, the negative effects of which are well-documented. We should hold social media companies accountable and work as parents, schools, and a society to address this. Restricting classroom technology will do nothing to help. It will penalize educators by removing important teaching tools from their hands while doing nothing to solve the root cause of digital addiction: Cell phones and social media.

On Learning

5. “EdTech tools do not improve learning outcomes. In the United States, student math and reading scores have dropped in the past decade.”

This is a logical fallacy: Post-hoc-ergo-propter-hoc. After, therefore, because of. EdTech use increased, yet scores dropped, so EdTech caused the drop.

Correlation is not causation. Correlation means two things are happening at the same time. Causation means one thing makes the other thing happen. Just because two trends are rising together doesn’t mean they are connected. People who carry umbrellas are also more likely to wear rain boots. One didn’t cause the other, the rain did.

In the case of test score declines, the “rain” might be edtech. It might also be chronic absenteeism. Or teacher shortages and turnover. Or constant curriculum shifts (e.g., math wars, balanced literacy debates). Or rising childhood poverty. Or, mental health challenges associated with phones and social media use. Or a decline in reading for pleasure. Or, in recent years, a historic global pandemic. All of these also correlate with declining test scores. It doesn’t mean they caused the decline.

Maybe more importantly, “edtech” is not a single treatment. There are thousands of digital learning tools and curricula. We cannot lump them all together into one “effect size” because they happen to have a screen and be connected to the internet. It’s like saying medicine doesn’t improve health. Which medicine, and for which ailments?

A medium of instruction is different from an instructional method.

Technology is a medium, not an intervention.

Also, many great edtech tools are not primarily designed to improve standardized test scores. If an edtech tool’s theory of change isn’t focused on raising PISA or NAEP scores, then declines in those scores don’t falsify their value.

As an example, I would sometimes use a digital discussion board called Flipgrid when I taught middle school social studies. I loved it. During the State of the Union address, I had students post their reactions live while watching at home. The goal was for my students to connect in-class content to real-world contexts and engage with their peers in civic dialogue. This was during Trump’s first year in office, so it was…um…an important time to encourage civil dialogue in a social studies classroom. Dismissing this sort of relevant, human experience because it doesn’t move the needle on NAEP scores is a rejection of some of the primary reasons we educate kids.

6. A report in Science Direct found that when students use a laptop in class, 38 minutes of every 60 minutes are spent off task. Expecting children to stay “on task” when provided with an internet-connected device that has access to YouTube is absurd from a brain development perspective.”

This is true, but it lies by omission: This study was done in the context of an unmonitored, large lecture-based (3000 enrolled students) undergraduate course with students using personal devices. That is a completely different context than a classroom of twenty 7th graders using school-managed devices, with school-managed wifi networks and content filters, under direct guidance from a professional educator. (Dr. Horvath similarly misrepresents this study within the first 5 pages of his book.)

The larger point is valid. Laptops and internet browsers can be very distracting for kids. Edtech tools need to do a better job of addressing this through more intentional instructional design and stronger training and implementation support. Just as importantly, educators can help by using technology as the targeted tool it should be, used for specific purposes that it is uniquely good for, then put it away. Technology should not be used as something you simply park a student in front of and let them use, unmonitored (more on this in an upcoming post).

7. A recent United Nations Global Education Monitoring Report found: “There is little robust evidence on digital technology’s added value in education…and [the evidence that does exist] comes from those trying to sell it.”

This report is as legitimate as it gets, but it’s nuanced. It’s almost 550-pages, so I can’t recount all of it here. The criticism of edtech publishing its own efficacy research is legitimate, and something I plan to write about more soon.

That said, this report expressly does not support the ban of edtech in K12 classrooms. It argues against both blanket bans and blanket adoption. To quote the report:

“This report’s basic premise is that technology should serve people and that technology in education should put learners and teachers at the centre….Technology should not be viewed as the solution, but as a supportive tool in overcoming certain barriers to education access.”

Its core position, as it should be, is that good edtech use is conditional. It depends on context, implementation, and matching the problem to the tool rather than the other way around.

You wouldn’t walk away with this understanding if you were to only read Ms. Cherkin’s cherry-picked quote. That quote comes from page 3 of the report. Here are some other quotes from that page:

• “Technology offers an education lifeline for millions but excludes many more.”

• “Accessible technology and universal design have opened up opportunities for learners with disabilities.”

• “Some education technology can improve some types of learning in some contexts.”

• “Digital technology has dramatically increased access to teaching and learning resources.”

• “Digital technology has brought small to medium-sized positive effects to some types of learning.”

On Teaching

8. “In an attempt to address large class sizes…schools choose to install surveillance software on student computers to “help” the teachers. Such platforms enable teachers to view each student’s individual screens simultaneously, pulling teachers away from direct instruction or engaging with students.”

It’s hard to address this claim, as Ms. Cherkin doesn’t specify what “surveillance software” she’s referencing. Regardless, it’s a strange argument, because technology enables the opposite.

As an example, one of the great affordances of technology in classrooms is its ability to be used as an all-student response system during direct instruction. An oldie but a goodie here is Socrative. It has a feature that lets students submit short constructed responses, then vote on who has the strongest response. Simple, but a great way to check for understanding during direct instruction, because it challenges each student to construct understanding and analyze worked examples of peers. In the age of AI, we can augment this further, giving each student a proficiency score and feedback on constructed understanding to move beyond simple right/wrong binaries. Critically, those trends are built on the submissions of all students, and those trends can be used to adjust direct instruction in the moment to better address student misunderstanding.

In this way, technology does not pull teachers away from direct instruction. If used correctly, it makes direct instruction more effective. To give the specific examples that Ms. Cherkin fails to provide, see Socrative, Peardeck, Nearpod, Formative, Wayground, and many more. These are massively popular instruction and assessment platforms that teachers have used to make direct instruction more effective for decades. They are not “surveillance software.”

Conclusion

Ms. Cherkin’s case for limiting edtech in K12 just doesn’t hold up to scrutiny. Her concerns around privacy, cybersecurity, and distraction are legitimate, but her evidence repeatedly relies on category errors, missing context, and what seems to be a growing policy trick: weaponizing “screen time” into a singular, simplistic negative. In this worldview, TikTok scrolling at 2 am on a school night is somehow the same as logging in to Google Classroom to do a check for understanding quiz on Chapter 6 of the Great Gatsby under the guidance of a trained professional.

We have to be more thoughtful about these distinctions.

We should not be using these oversimplified conclusions to guide national education policy in the United States.

In the next post, we’ll look at the arguments and data presented by Dr. Horvath. We’ll see many of the same flaws.

I think we should also be aware that not everyone is celebrating. An anti-edtech movement is growing. We should consider this at ISTE this week.

This has been on my mind at every conference we’ve attended this year: Edtech needs more dialectical thinking.

To explain what I mean, I need to go back to a much smaller conference earlier this year that first got me thinking about this: ResearchED.

ResearchED Delaware

The most memorable conference we presented at this year was ResearchED, Delaware. ResearchED has shaped how I understand cognition and learning. Dylan Wiliam, an important voice in the community, is an advisor of ours at Short Answer. I’ve long looked to their work for guidance, so I was excited to share ours.

I was caught off guard, then, to arrive and find out that our work is, in fact, evil.

Turns out digital technology in the K12 classroom is a big no-no for ResearchED. This was expressed in different ways, but most clearly in the closing keynote, which was recorded and shared as a podcast.

In the keynoters’ view, edtech is a distraction. It prioritizes engagement over best practices in teaching and learning. In doing so, it too often violates tenets of cognitive load theory, especially the split attention effect. Most dubiously, “research shows that it doesn’t improve learning outcomes.” The primary research used to support that statement seemed to be a 2015 OECD report

Because of their talk, I read that OECD report. I also read John Hattie’s meta-analysis on edtech in Visible Learning. I re-listened to their podcast to better understand their arguments. I reflected. I wrote a blog post (that I came to feel was unproductively inflammatory, like their keynote, so I never posted it).

Here’s what I learned:

The 2015 OECD report does not support their claims. Where it does show “no appreciable improvements in student achievement in reading, mathematics, or science in the countries that had invested heavily” in edtech, it doesn’t argue against the use of edtech. It generally argues that edtech isn’t bad, our implementation of it is. We need to use technology more strategically to support good pedagogy and invest in better training.

Now, my point here is not actually to debate whether or not research was misrepresented. My point, oddly enough, is to celebrate how much I learned. ResearchED changed some of my beliefs, especially around the value and meaning of “engagement” in K12 classrooms (more in future post). And in that way, it deepened my understanding of edtech in ways that few edtech conferences have.

It pushed me to engage with an opposing viewpoint…which made me realize how little this happens in edtech. I think that needs to change.

Dialectical thinking and its absence

Dialectical thinking means deliberately seeking out ideas that are contrary to the ones you hold. It requires one to consider contradictory ideas in their mind at the same time and explore the tension between them to arrive at a deeper understanding. A recent Cal Newport podcast got me thinking about this.

The podcast was (fittingly) focused on whether smartphones have made us dumber, along with strategies to avoid being lulled into stupidity. An argument made is that today’s algorithmic platforms trap us in echo chambers, feeding us content that affirms what we already believe, which discourages us from fully wrestling with challenging ideas.

Edtech conferences too often do the same thing.

You start to notice it when you attend dozens of them. They all feel so similar, which is strange given how contextual teaching and learning is. Edtech conferences don’t feel contextual, they feel homogenous: The same tool focused presentations that are light on pedagogy and heavy on “fun!” using the same Canva slide templates with the same politely nodding audience partly paying attention near the same vendor hall promising the same “game changing” transformations and the same lingering feeling of deja vu. The glue that binds it all together is the singular, uncritical belief that technology is almost always the answer in education. We might want to investigate that belief this week at ISTE.

Bursting the edtech bubble

It’s not just ResearchED. Influential voices are starting to challenge the value of edtech. We need to address this as a community. We live in a bubble that is too often championing tech use instead of effective tech use. We should be having more frank conversations about this, and why increased technology access in schools hasn’t had a big impact on outcomes.

At edtech conferences like ISTE, this might look like:

• Inviting edtech skeptics, and researchers, to be featured presenters. I was SO EXCITED to find that ISTE includes a research-track of sessions (check out my wife’s research on AI & Feedback at Stanford with Mei Tan!). I’d love to see more conferences follow suite. To that end…

• Prioritize presenters who can speak to the results of their work. One of the most powerful sessions I attended this year, again at ResearchED, was “Effective First-Grade Writing Instruction: Lessons from a District-University Partnership” with David Cocker (University of Delaware) and Kelly Harkins (Red Clay School District). They detailed an approach that drastically improved writing outcomes for first graders in Red Clay Schools. In the most powerful moment, they shared writing samples of 1st graders before and after their initiative. The growth was stunning. We need similar “before” and “after” results being shared in edtech sessions. To that end…

• Pedagogy first, technology second. Too many sessions are overly tool focused. I say that as someone building an edtech tool. The Eduprotocols community is particularly good at avoiding this in their presentations. It’s not “10 ways to use Short Answer!”, it’s “Effective approaches to teaching writing”.. and oh by the way, here are some tools that can help actualize those approaches. Demoing tools is okay, but much more meaningful in the context of the pedagogy they actualize.

• Panel discussions and actual debate on important topics in education and technology. One of the best sessions I attended this year was at ASU/GSV in San Diego (see image above), and it was so memorable because the panel disagreed (thank you Dan Meyer for continuing to be the heal that edtech needs right now)! Enough patting each other on the back, let’s get into it!

• Session tracks focused on ineffective technology use: What it looks like and how to avoid it. Technology’s affordances, especially efficiency and accessibility features, make it well suited to solve certain problems in classrooms. Those same affordances can make it poorly suited to solve others. Let’s discuss this.

• Vendor presentation standards. I say this as a vendor: No more 60-minute sales pitches. What’s your theory of change? What research supports it? How does your product actualize that research? What results have you seen? And be clear: What level of rigor supports those results? Enough overselling anecdotes and pretending that your case study is a randomized control trial. Conferences should treat vendors like fellow educators and expect them to rise to the occasion.

Outside of the confines of edtech conferences like ISTE, if you believe in edtech and want to challenge your thinking, I would recommend:

• Watching this speech from Sophie Winkleman (skip to the 8 minute mark for edtech specific discussion).

• Listening to the closing keynote from ResearchED Delaware.

• Reading Michael Bloomberg’s recent op-ed against edtech.

A worry here might be that exposure to opposing viewpoints will undermine edtech integration. My experience this year is that it’s the opposite. Engaging with the best counterarguments from an opposing side deepens understanding and makes one’s advocacy more effective. As the anti-edtech movement grows, we should encourage more of this kind of engagement.

I’d love to see this at ISTE this week.

“I’m going to try to tell this without crying…”

We suddenly had welled-up eyes, too.

“I teach at a very small school here in Texas. We’ve had several students join our school recently who can’t speak any English. Not a word. And so it’s been very challenging. But we used your tool, and I had these students write their responses in Spanish. And we did it in Chrome, so when they read each other’s writing, their Spanish was translated to English and the English responses were translated to Spanish for them.

I’m glad I didn’t look before starting the feedback part because I’m not sure I would have had the courage to include hers.

One of my new students wrote about how, back in Mexico, she was one of the smartest kids in school. And how she loved school…

…and how, since coming here and not being able to speak the language, people look at her like she’s stupid. And she’s never been treated like that before…

…and her writing was selected by her classmates as the top response…

I think it was the first time she ever really communicated with her classmates. And it started such a meaningful conversation as a class. And ever since that moment, we’ve noticed such a change in her and how she interacts with our school.

I just wanted you to know that story. Because this truly made a difference for that child.”

We got out from behind our table and hugged.

The edtech vendor hall is not typically a place for these moments. It’s too often a place for the opposite: Transactional sales gimmicks and bad fluorescent lighting. Forced conversation. Avoiding eye contact.

It’s funny, then, that maybe one of the most meaningful moments of my career happened in a vendor hall. I’ll never forget that story. Both because of the impact on that child and teacher, and because of the impact it’s already had on my wife, myself, and our work moving forward.

A great educator recently suggested that, in edtech, you either bet on teachers or you bet on software. When you bet on teachers, and design things primarily interested in empowering human-to-human connection in the classroom, special things can happen.

It can be easy to forget that when you spend too much time in the vendor hall.

Beyond the commerce, real human stories are unfolding. Stories of marginalized students rediscovering their identity and teachers bridging impossible social divides. Small moments, “on fire with the same force that lit the stars.”

Her story was a reminder that technology is most powerful when it gets out of the way and makes human connection possible where it wasn’t before. More human-to-human connection is critical for our educators and students. I leave Texas with that on my mind and in my heart.

The Chromebook reseller next to us probably summarized it best.

He leaned over the black metal booth divider after she walked away and whispered, “that was powerful.”

When we lead our writing workshop with schools, the room gets quiet when I say “AI detection software doesn’t work and shouldn’t be used by K12 schools”. In this post, I want to unpack the research that supports that claim, the companies that are lying by omission about that research, and what we should do with K12 writing instruction moving forward instead of AI detection.

I. Independent Research

A growing body of research shows that AI detection is unreliable and easily fooled. Dr. Laura Weber-Wulff at the University of Applied Sciences of Berlin, alongside researchers across the world, published a landmark 2023 study showing that leading AI detection software is largely inaccurate. Figure 1 below, taken from the study, shows that even the best AI detection software is only correct about 80% of the time. Put another way, it will incorrectly label writing more than 1/5th (!!) of the time.

If being wrong 1/5th of the time wasn’t bad enough, other studies have shown that this number drops dramatically with simple paraphrasing. AI detection software works by looking for distinct probabilistic patterns that emerge from pieces of words (i.e. tokens) that have been strung together by large language models. To break up those patterns, all you need to do is change some words around. In doing so, the University of Marlyand’s Computer Science department was able to drop AI detection in passages of 300 tokens in length from 99.3% to 9.7%. So students need simply to paraphrase a few different chunks of a larger AI output to drop AI detection chances by 90%. This study joins a body of research (see here, here, here, and here) that shares the opinion offered by Dr. Weber-Wulff and her team: “Our conclusion is that the systems we tested should not be used in academic settings.”

Stanford researchers also recently came to the same conclusion. A study out of the Stanford Computer Science Department found that AI detection tools disproportionately flag non-native English speakers’ writing. “Our findings reveal that these detectors consistently misclassify non-native English writing samples as AI-generated, whereas native writing samples are accurately identified. Furthermore, we demonstrate that simple prompting strategies can not only mitigate this bias but also effectively bypass GPT detectors, suggesting that GPT detectors may unintentionally penalize writers with constrained linguistic expressions.”

This finding, along with all the others, have caused many leading universities to effectively ban AI detection. Alabama, Yale, MIT, Syracuse, UC Berkeley, The University of Central Florida, Colorado State and countless others have all either banned AI detection tools or advised their faculty against using them.

If you don’t trust peer-reviewed 3rd party research teams or dozens of universities that span the political spectrum, then take it directly from the developers of world-leading AI tools. OpenAI, the company behind ChatGPT and arguably the world’s leading collection of AI researchers and developers, shut down its effort to develop an AI detection tool in 2023. Why? “As of July 20, 2023, the AI classifier is no longer available due to its low rate of accuracy.” Open AI directly addresses educators in the post: “We recognize that identifying AI-written text has been an important point of discussion among educators,” they say, “and equally important is recognizing the limits and impacts of AI-generated text classifiers in the classroom.” Put another way: This stuff doesn’t work, and the impact of using a tool that doesn’t work might be just as bad as not being able to identify AI in student writing.

II. The Dishonest AI Detection Industry

It seems the only people claiming that AI detection software works are the people selling it. TurnItIn, the largest and most influential of the AI detection companies, recently published a whitepaper in which they claim that, “Turnitin’s AI writing detection system has been independently shown to have high effectiveness in correctly identifying AI-generated content.” When I dug into this claim, I found it was partly supported by the following research: “Weber-Wulff (2023) showed that Turnitin’s AI writing detection system outperformed all other AI writing detection solutions on the market in accurately detecting AI writing across a diverse set of evaluations.”

The Weber-Wulff study above showing that TurnItIn was wrong over 20% of the time? The same one that concludes “these systems should not be used in academic settings”? Yeah, TurnItIn cited that study as supporting their product.

One conclusion I come to here is that TurnItIn assumed that no one would read the research they cited. Another conclusion I come to is that TurnItIn is knowingly lying, by omission, to teachers.

I’m not a confrontational person. At all. I grew up in a small town in the Midwest, I apologize to other people when they bump into me on the sidewalk. But when I see a big company lying to schools and teachers for profit? That makes me more confrontational.

So, a few months ago, I tweeted about this. And it got picked up. Journalists reached out. I did an interview with Edutopia. All the sudden, I was becoming “the anti-AI detection guy.” To be honest, I don’t want to be the anti-AI detection guy. And as much as it upsets me that a large company is fooling schools into wasting resources, I also recognize that it does no good for educators to just rant and rave about it.

So I reached out to TurnItIn.

I was at the AI AIR Show in San Diego in April, where TurnItIn had a booth and keynote focused on their AI detection software. I (nervously) went up to their booth and asked if they would put me in touch with the folks that wrote their recent white paper (dubiously, the authors were not listed).

To TurnItIn’s credit, they were open to talking. They put me in contact with Patti Smith-West, their customer engagement lead. I thought it was interesting that, when asked to clarify which white-paper I was referring to, Patti’s response was, “I suspected this might be the one you meant, and my team does not own this one. You would need to reach out to Eric Wang, our VP of AI.” So I reached out to Eric and, again to his credit, he was open to meeting with me.

Over the course of about a 30-minute Zoom, we actually had an amazing conversation. I appreciated Eric’s candor and willingness to engage with me, especially given we started the call with a terse, “I saw your Twitter thread.” 

“I think the skepticism that exists is healthy, but false positives cannot go to zero. That’s not how statistics work.” Eric argued that failures in using TurnItIn are often failures of school policy. “TurnItIn can’t be a policy and you shouldn't buy us as that… You should never give a kid a zero based on TurnItIn results alone.” He also argued, as do many teachers who continue to insist on using AI detection, that TurnItIn should instead be used as a “healthy conversation starter”. If you suspect students of using AI, run it through TurnItIn, then use those results to start a conversation with a student about how or if they violated academic integrity expectations.

In addressing the Weber-Wulff study, Eric made analogies to breast cancer detection and weather forecasts. “When doctors use cancer screenings, they have an X% false positive rate. But that doesn’t prevent them from using the screening and it doesn’t make the screening a bad thing.” Eric continued by arguing that the weather forecast isn’t always right either, but there is still utility in forecast apps.

I deeply appreciate Eric’s willingness to talk, especially given he knew I was a skeptic. At a time when it feels like education is becoming more polarized (see Jo Boler and the Algebra wars in California, Dan Meyer & the AI ChatBot wars, or the ongoing science of reading wars), I think it’s vital that we be open to this sort of dialogue with people we disagree with. Dialogue and empathy will lead to solutions that better serve teachers. That said, here’s why I still think Eric’s arguments are misguided.

III. Why the AI detection juice ain’t worth the squeeze

The point Eric made repeatedly in our conversation is the same point I’ve seen teachers make (see comments here) in defense of AI detection: It’s a conversation starter for when teachers suspect AI. The problem with this is that, given how largely unreliable these tools are, those conversations with students will start with a baseless accusation. 

I don’t know any relationship experts that advise baseless accusations as a meaningful avenue for relationship building. In fact, therapists advise omitting blame entirely and starting tough conversations from a place of empathy. This is particularly important in K12 education because the relationship between a teacher and student is one of the most important factors in determining student learning outcomes. Given how important relationships are in K12 classrooms, using a tool that has a >20% chance of causing distrust or harm in a teacher/student relationship just isn’t worth it.

This leads to the second issue: The chance for misuse is too high.

Some teachers may use AI detection as a conversation starter. Many others (as you will see in the comments on my recent Facebook post) will simply use it to hand out zeros, even though the people creating this software agree it should “never be used to give a kid a zero.” The risk of teachers reaching for AI detection as the “easy button” that they can use when they suspect improper AI use is too high, especially given the aforementioned research showing that these tools will disproportionately flag non-native English speakers’ writing.

If the risk of damaging relationships with students isn’t enough, think about the opportunity cost. K12 funding is always tight, but with ESSER funding running out this year, schools are making mission-critical decisions about what to cut or keep. It pains me to think that broken AI detection software might be prioritized over the other amazing uses we could allocate that funding towards.

It’s not listed on their site (at least not that I could find), but I believe TurnItIn costs anywhere from $3 to $5 per student (that link is dated, this price may have gone up). The top 100 largest US school districts all have at least 40,000 students. So district-wide access to TurnItIn might cost anywhere from $120K to $200K for our largest districts (again, it’s probably higher than this).

Think of the infrastructure, training, support services, extracurriculars, safety measures, library resources, early childhood programs, arts and music opportunities, STEM boosts, etc. that this money might be used for instead of software that has (at best!!) a 20% chance of damaging a relationship with a student.

What makes this all particularly galling is that there are free alternatives that are better at monitoring AI influence on student work.

Simply set an expectation with your students up front that they need to construct their paper in the same Google Doc or Microsoft Word document for the entirety of their writing and have them use a free extension like Draftback or Revision History. These are much better avenues for conversation starters because they playback videos of students working through a document. You could use these to build metacognitive reflection into the writing process even when you don’t suspect AI usage. We discuss this in-depth in our writing workshop.

This speaks to the larger issue here: AI detection leads us down the wrong path in adjusting instruction and assessment post-AI. It represents the easy-button option - “I don’t have to change my instruction and assessment, I’ll just tell kids not to use AI and use detection software”. 

As we head into a world in which AI can increasingly do almost any task we throw at a student and in which it will be almost impossible to monitor AI’s influence if and when work leaves the classroom, we must instead make much larger, systemic changes to our instruction and assessment. Part of this means accepting that students can, will, and (in specific instances) should use AI to help them write.

The trick becomes teaching students how to use AI to write with them rather than for them. This is an incredibly difficult balance to pull off. Students often lack the motivation, metacognition, and self-regulated skills necessary for this balance. We lead entire workshops and are building a platform focused on finding this balance, but we’ll address at least a few points on the path forward in the next section.

Before doing so, I want to reiterate: Please do not use AI detection. The risk of damaging relationships and opportunity costs associated with this software just aren’t worth it. Your spidey senses should be tingling when the only folks arguing that this software works are the people selling it.

IV. The path forward

So how do we move forward in a world without AI detection software? Again, that’s a huge question that we lead entire workshops on, but here are a few entry points.

1. Set clear Classroom Policies for Academic Integrity that allow for some AI use

The first step in finding balance in AI’s influence on student writing lies in setting clear classroom policies and expectations. Given that AI detection software doesn’t work and that AI will more than likely influence student work once it leaves the classroom, I agree with those advocating for thinking about AI through the lens of a tiered level of influence. Assignments should be labeled in advance as to whether or not AI can be used to complete them. A simple “red, yellow, green” framework is a great place to start:

• Red = No AI 

• Yellow = Some AI (with specific constraints)

• Green = Full AI.

 Leon Furze and his AI assessment scale and Adrian Cotterell are doing great work on this. Dr. Sonja Bjelobaba’s work on reframing academic integrity is also fantastic.

2. Bring more writing instruction and assessment into the classroom and make it cross-curricular.

In-class writing allows for much more control over how much influence AI has in the writing process. Not through any technical process, simply through classroom management. It’s partly why we’re building Short Answer. If we’re going to bring writing into the classroom, it will need to be bite-sized, engaging, and motivating for students. For us, that means making writing a social experience.

I also believe that we’re going to need to find ways to make up for the lost practice repetitions that will result from AI being embedded in almost every piece of technology our students use. That means writing instruction can no longer just fall on the English teachers. It needs to be, as has been advocated for decades, embedded in every subject. And if we’re going to ask the math, science, social studies, and electives teachers to embed writing in their classes, it better be easy. Because teachers already have WAY too much on their plates. To us, that again means making writing bite-sized and used as a means to teach content. We’re building a platform for that 😊.

3. Teach students to use AI effectively in their writing

This is a huge topic and one that is particularly scary given: 1. All the risks inherent in AI use and 2. Peer-reviewed research is limited on how best to go about this yet. This doesn’t mean we should sit on our hands. 

The dynamic of any effective AI use in the classroom needs to be AI in the loop, teacher in charge. If and when we don’t set clear classroom policies, we turn kids loose on AI we haven’t trained them on, and/or we enable too much influence over student thinking, the dynamic can too quickly become teacher in the loop, AI in charge.  While we wait for a more robust research base, try using well-established instructional frameworks on how best to introduce students to new content. For example, we’ve repurposed Dr. Nancy Grey & Dr. Douglas Fischer’s gradual release of responsibility framework and paired it with a variety of new AI tools in our workshops to show teachers how they might go about effectively introducing AI use in their content area to students.

If you teach in the core subjects, AI literacy doesn’t need to be your central focus. That said, I do think you need to coach kids through, at a basic level, how it can be used effectively in your content area. In the context of ELA, that might mean showing students how to write effective prompts, reorganizing and paraphrasing the generic writing produced by AI, comparing and contrasting human vs. AI written content, using AI as a brainstorming partner, debating and AI chatbot for persuasive writing, etc. This is a huge topic that many people are doing amazing work on.

I really love Anna Mills’ work on this if you’re looking for inspiration. English teachers Meg Lamont and Kristina Zarlengo also have an inspiring post on how they’re going about this at Stanford Online High School. We also discuss this in-depth in our workshops.

4. Center more personal reflection and/or defense in writing practice

Building in a reflection or defense element to writing assessments can be an important step towards developing students’ writing ability while also ensuring we don’t lose the benefits of going through the writing process. Reflections and defenses aid in personal growth and encourage critical thinking. They also require students to explore personal feelings, experiences, and perceptions in ways that are difficult for AI to replicate easily (i.e.  it’s hard to use ChatGPT to “cheat” on this).

Include a video reflection requirement using Google Forms to have students submit a video reflection they can make on their phones or Padlet to have class discussions about these reflections (I would have said Flip, but…). I also recently began exploring Mirror Talk and like what I’ve experienced, but haven’t used it enough to fully vouch for it yet. We also embed reflection in every writing activity in Short Answer.

5.  Be intentional about the “why” 

“Why should I learn how to write when AI can do it for me?” is an incredibly complicated question that deserves to be addressed directly, both as a staff in professional development and in conversations with our students when assigning writing tasks.

The simple answer here is that literacy is one of the most important predictors of health and well-being as students go through their lives. It is the means through which they will (or will not) be able to communicate their thoughts and ideas with the world. Going through the process of writing literally transforms thought. Being explicit about this and having an open dialogue with students about it is a start. More importantly, embedding writing practices that allow students to experience the vital skill that is writing should be a focus moving forward.

Closing

Notice that none of these steps involve AI detection software. Rather than wasting time and money on these tools, we should focus on more effective strategies that promote academic integrity, enhance in-class writing, and teach students to use AI responsibly. 

Ultimately, we need to stop chasing easy solutions in an attempt to maintain the status quo and start accepting that AI requires fundamental shifts in the way we approach K12 writing, instruction, and assessment. Our resources should be invested towards those ends rather than the dead end of AI detection software.

After all, it’s been a big month for “free” K12 edtech. Khan Academy announced that Khanmigo, their flagship AI tool, will now be free for educators through a new tranche of funding and compute offered by (wouldn’t you know it!) Microsoft. The same Microsoft that just gutted Flip. So all those administrative tasks you’ve been able to dramatically speed up through amazing 3rd party tools like MagicSchool, Eduaide, Diffit, etc? Those are now fully free. And, oh by the way, because Microsoft/OpenAI is giving funding AND compute, Khan Academy is able to offer all this tooling on GPT4. So not only is it free, it will likely offer better outputs for educators when compared to 3rd party alternatives

The teacher side of me was excited by this news (as was reflected in my Twitter feed). An amazing learning tool was being made available to schools for free! What’s not to like?! The founder side of me was (and is) nervous. But maybe not for the reason you might expect: “Free” comes with a cost in K12.

I’m not talking about the maxim that “when it’s free, you’re the product.” That is mostly true in consumer and B2B markets, but it’s somewhat less true in K12 edtech where strict data privacy laws and agreements ensure that schools own their data. I’m also not talking about the administrative annoyance and opportunity costs of having to manage hundreds of free platforms in a district. Rather, I’m talking about a concept that Ben Kornell and the team at Edtech Insiders wrote about last month: Market distortions:

“Free access to some level of AI will be an ongoing element of BigTech education offerings. This should worry entrepreneurs, investors, and even educators. We have seen this movie before with Learning Management Systems: the entire LMS market of the 2000s sagged under the weight of Google Classroom’s $0 price tag. The result was one-size-fits-all, relatively low-quality LMS offerings all selling for $4-8/student. At this low customer lifetime value, all growth strategies skewed to high volume and limited features….the pattern we have often seen is that a free B- rated product often beats a paid A+ rated product in the education space.  Even if buyers are willing to pay a premium for a “better” offering, how much more than free?”

Put another way, “Free” products propped up by large corporations to serve strategic business interests can put smaller, sometimes better products out of business. With competition stifled, mediocrity follows. Notice that, since Microsoft made Flip free, few other K12 video discussion tools have stood up. This was true even as Flip skyrocketed to become one of the most used K12 edtech tools. Usually, rapidly growing products invite fast followers. As an example, I LOVE my Hokas….and now almost every other major shoe brand has their own version of them. This same dynamic hasn’t played out in the K12 video discussion platform market because competitors can’t compete with free. And, as an executive at a major K12 edtech company responded when I suggested someone should build a new Flip: “After 12 years of giving it away, I'd be afraid that MSFT has trained the market not to value this kind of product.”

You may be thinking to yourself: “That’s fine, we’re getting great products for free, who cares if there’s competition! I’m a teacher, I’m resource-strapped!” That’s a fair reaction. I still have that reaction. But that laissez-faire attitude can come back to bite us when the people actually funding the tools decide it’s no longer in their interest to do so. They’re offering the service to serve their interests, not yours. And when they pull the plug, there may not be many great alternatives for a while because they were all put out of business by “free.”

It seems we may be headed in this direction with K12 Administrative AI tools. The offerings of Eduaide, MagicSchool, Diffit, etc are incredible. They’re made by current and former educators dedicated to building incredible learning products to serve teachers. But in the short term, their intentions of serving teachers may be no match for the market share, data collection, social good marketing spin, or myriad of other intentions of BigTech like Microsoft that can casually throw millions of dollars and compute at a free version of all of these tools. At the risk of throwing stones in a glass house, I don’t see how the new startups (who actually are dedicated to serving teachers) compete with a completely free version of their product. In the meantime, I find bravery in their bravado.

Again, teachers and schools may not care. Those tools are now free and, arguably, better because they’re running on more advanced foundation models. But in 10 years, if and when Microsoft or the foundations funding Khan Academy decide that offering this free product no longer serves their purposes, they might decide to stop funding Khamigo. And all the sudden, educators may be left out to dry with few suitable alternatives because they were all put out of business by "free.” Good luck getting schools to care about that, though, when that might be 10 years off (if ever) and budgets are tight now. 

Maybe when free products are discontinued, the free market will ride in to save the day. This happened when the Gates Foundation spent $100M to build InBloom in 2013, a free single sign-on and edtech administrative hub for districts. Like many free, foundation-driven offerings, it failed. And in its place swooped Clever, one of the most successful and beloved platforms in K12. It’s telling that the day Flip announced it was being sunsetted, Padlet announced they would be adding new Flip-like features soon. Maybe this is just the ebb-and-flow of the completely bonkers K12 edtech market. Maybe it’s proof that free products suppress competition and lead to lower-quality offerings for teachers (why didn’t Padlet have these features before?) Who knows.

The insight here, at least for me, is that products serve the people that fund them. If schools are the primary folks funding a product, that's who the product will serve. If venture capitalists or philanthropic foundations or large corporations and shareholders are the primary funders…well, then that's who the products will serve. Sometimes their interests might align with teachers and schools. And when they do, magic can happen. But as we saw with Flip this week and other products before it, they often don’t. And when they don’t, educators are the ones left out to dry.

As K12 edtech creators, I think the best way for us to align with schools is to be primarily funded by them through a paid product. But that’s incredibly difficult to pull off when “free” is taken as a matter of course.

Subscribe now