This fall, CHT will launch a campaign to Stop the AI Drift — the slow erosion of human abilities as AI takes on more of daily life. The more we let AI think, talk, create, and act for us, the more we lose the skills, relationships, and meaning that come from doing those things ourselves.

As part of the campaign, we're publishing guest essays from experts on the human abilities most worth protecting in the age of AI. This one comes from childhood development expert Dr. Dana Suskind, author of Human Raised: Nurturing Connection, Curiosity, and Lifelong Learning in the Age of AI.

For decades, parents have engaged in an endless pursuit of optimizing childhood, deploying every tool, tutor, and coding camp at their disposal to prepare their children for a 21st Century knowledge economy.

Mothers in most Western nations now spend twice as much time with their children as they did 50 years ago. Fathers have quadrupled their involvement. Many parents are exhausted but still feel like they’re not doing enough. And in a world of narrowing pathways and widening inequality, their logic made perfect sense: Building cognitive skills early secures access to education, which gives your child a fighting chance at a stable, successful life.

Until now.

This “intensive parenting” model that took decades to build and millions of dollars to sustain? AI came along and, within an instant, turned it completely on its head.

The core product of intensive parenting has been automated. The cognitive outputs we’ve been striving to maximize are precisely the tasks generative AI now performs instantly, tirelessly, and at superhuman scale: factual recall, information synthesis, pattern recognition, analytical efficiency. The blueprint for raising children today has been scorched.

This isn’t the first time our plans have needed to be rewritten, of course. Throughout human history, as we evolved from hunter-gatherer societies to agricultural communities to industrial economies, parents have always adapted to prepare children for the world they’ll grow into.

But when human intelligence no longer necessarily reigns supreme, what should parents strive to optimize? How do we prepare our children to thrive in a world where artificial intelligence far exceeds our capabilities in domains once considered uniquely human? And what happens if the tools promising to prepare children for this future begin displacing the relationships that are so central to being human?

In a provocative new essay, Bill Gates suggests we set aside certain jobs for humans only.

“I like the phrase Human Reserved,” Gates writes, “because it makes me think of nature reserves—places where we could put buildings and roads, but we choose not to because the loss would be too great.”

I’d like us to think even bigger: What human capacities do we want to deliberately protect?

In my new book “Human Raised”, I call this phenomenon the “Human Edge.” Capabilities that used to be dismissed as soft skills will become recognized as hard currency: The ability to read a room. The critical thinking to discern fact from fiction. The intelligence to question bias and assumptions. The ability to connect, collaborate and repair relationships when things inevitably go wrong. Creativity. Moral judgment.

University of Chicago economist Alex Imas argues that as AI commoditizes cognitive work, spending and employment will increasingly migrate toward the “relational sector.” Think nurses, teachers, therapists, and other workers who spend their days face-to-face with other humans. The workers who survive and thrive, he argues, won’t be the ones who think fastest. They’ll be the ones who connect most deeply. Human presence is becoming scarce. (When was the last time a person rang up your groceries for you?) And what’s scarce becomes valuable.

The World Economic Forum’s Future of Jobs Report 2025, which surveyed over 1,000 leading employers representing 14 million workers, identified empathy, resilience, and creative thinking as the fastest-rising skills in the global economy. Those are also the skills with zero substitution potential from AI.

Fortunately, we know how these skills are built, and we already have the blueprint. It’s an ancient one that’s been working for millennia, and it tells us that these skills are built through human relationships.

Think of the infant brain as hardware. The neural architecture is built in real time, piece by piece, through moments of human connection and nurturing interaction with loving adults. The most important construction occurs during the first five years of life, when the brain creates one million new neural connections every single second. That rate of growth is never again matched in life. Everything that follows is akin to a software update.

To the developing brain, human interaction isn’t a luxury; it’s a biological necessity. The exquisite dance of human interaction provides the precise neural nourishment needed to build a child’s brain: a sophisticated mix of language, neural synching, emotional attunement, and social signaling that wires the developing mind. Each conversation, each shared moment, sparks the neural activity that allows children to build language, spatial reasoning, self-control, and everything it means to be human.

Notably, this development doesn’t depend on perfect parenting. The infant brain has, in fact, evolved to learn from imperfect human interaction: the split-second delays, the mismatches that require repair, the subtle facial cues and shifts in tone. These seemingly inefficient elements of communication are exactly what build our capacity for deep human connection. They’re what make us both human-raised and human.

The friction created when parents misstep and reconnect is where resilience, flexibility, empathy, and emotional regulation are forged. The same is true as children grow and encounter what researchers call social micro-frictions, such as arguments over toys or struggling to take turns. These small moments, when a child must reconcile their perspective with someone else’s, are profoundly meaningful. Children learn that they can express their needs and still be okay, and that connection with another person can withstand tension.

In other words, children learn to be human from other humans. And for all of human history, they only learned from humans. Researchers have demonstrated that infants’ sponge-like brains absorb information at an incredible rate when it comes from a person, and they absorb nothing from screens or recordings. Evolution equipped children with a kind of biological filter called the social gate, which tells their brains to tune into people rather than be overwhelmed by the cacophony of sensory inputs surrounding them at all times.

That’s all changed.

For the first time ever, artificial entities hold the keys to our social gate. The same back-and-forth human interaction that wires a child’s brain and builds the Human Edge can now be convincingly mimicked by a machine. Infants as young as six months show physiological responses to robots that parallel their body’s response to humans. And older children are even more vulnerable: research shows they’re significantly more prone than adults to attributing consciousness, intentions, and feelings to non-human entities.

The machines knocking at the social gate are never distracted, and always available. They never lose their temper or check their phone. And they’re flooding the market.

App stores and toy aisles are filling up with AI-powered companions, tutors, and playmates, all promising to build the skills children will need: curiosity, creativity, empathy, connection. Parents who’ve reached for any tool at their disposal to give their children a fighting chance will likely reach for these too. As someone who purchased Baby Einstein videos for my preschooler, I understand that instinct.

But the reality, and the great irony, is that AI tools marketed to build Human Edge skills are likely to erode those very capacities.

Research has found that using Generative AI reduces individuals’ confidence in their critical thinking skills and that adults who completed an essay-writing task without AI showed more cognitive activity, felt greater ownership of the material, and actually learned more, compared to those who used AI to do the same task. A neuroimaging study of children, meanwhile, found that using ChatGPT reduced engagement in the brain networks responsible for creativity, attention, and cognitive control.

If cognitive offloading changes how adults think, its implications for children may be more profound. Their brains aren’t simply using established circuits; they’re still building them. In adults, the concern may be “brain rot.” In children, the risk is something closer to “brain stunt.” And it will threaten more than a child’s intellectual development. Their very capacity to connect with other people could be undermined. By interacting with always agreeable, endlessly patient artificial companions, children’s brains may be wired to expect what no human relationship can ever give them: perfection.

The evidence is clear that a thoroughly human-raised childhood is the best preparation for a thoroughly AI world. The question that keeps me up at night is whether every child will get one.

The optimist in me wonders if AI’s arrival might break the spell of intensive parenting entirely. If the cognitive skills we’ve been racing to build are no longer a differentiator, perhaps we can finally stop racing and accept that simply being present, in all our wonderful imperfections, is enough for our children.

But the pragmatist in me recognizes that presence isn’t free. In fact, it may be the most expensive commodity of all. Because what it costs is time, and time is precisely what the economy has been stripping from the parents who can least afford to lose it. The parent working three jobs doesn’t lack love. She lacks hours. The father on his phone during dinner isn’t checked out, he’s drowning in deadlines.

My fear is that families diverge: some will double down on human connection, while others seek (or accept) technological supplementation. Both sets of parents would be acting out of love, striving to give their children everything they can. But as we know too well, fierce love and the best of intentions have never been enough to generate equitable outcomes.

There’s plenty of evidence that this divergence has already begun. Highly educated and well-resourced parents are already following the science of human development, leaning on connection and eschewing technology in their homes even as they build it in their workplaces. Steve Jobs, Bill Gates, and Sam Altman all famously went on the record stating that they limit, or plan to delay, their children’s use of technology. Many Silicon Valley families are choosing low-tech Waldorf schools, increasing playdates, and prioritizing unstructured play. In doing so, they are cultivating Human Edge skills and preparing their children to thrive in an AI future, while families with fewer resources and less time are reaching, understandably, for the tools that promise to do the same thing faster and cheaper.

Imagine one child whose parents can afford the time to read, sing, argue and repair, facilitate outdoor play, and seek out low-tech classrooms. Now imagine another child whose overworked parents lack time but have done their best to surround their child with products promising support: “educational” videos, an AI tutor, a chatbot companion. Both children are loved. Only one is getting the developmental input every child needs.

The “better than nothing” fallacy would suggest that artificial interaction is better than no interaction. Tragically, this miscalculation not only robs children of the opportunity to build the Human Edge skills they will need to thrive, it could rewire their developing brains in ways we’re only beginning to understand and drive one of the deepest developmental divides we’ve ever known. Not between those who have access to information and those who don’t, but between those who grow up human-raised and those who don’t.

What was once the most universally available developmental input, human interaction, is now at risk of becoming a luxury good that is unevenly distributed, planting inequality at the neurological level.

AI is one of the most powerful tools our species has ever held. Used wisely, it can reduce the burdens that keep families from being truly present with one another and lighten the invisible labor that exhausts parents before they walk through the door. In that way, it has the potential to protect, rather than replace, human connection. The greatest promise of this transformative technology is that it might offer every child, regardless of zip code, the one thing that has always mattered most. A loving caregivers’ presence. That’s the opportunity.

But we have a narrow window to act on that opportunity and ensure AI is designed and deployed in ways that facilitate human connection, not impede it. That means judging child-facing AI by a simple standard: Does it give adults more capacity to connect with children, or does it quietly take their place? Does it keep parents, teachers, and caregivers in the loop, or does it make human presence seem optional? Does it protect the messy work of attachment, play, conversation, boredom, conflict, and repair, or does it offer up a frictionless substitute?

In the age of artificial intelligence, a human-raised childhood may be the most important advantage we can give a child. We must not allow it to become a luxury good.

We’re working to empower policymakers, technologists, and everyday people to guide technology toward the public good. If you value our work and want to support it, consider donating.

Donate

How do AI systems affect humans? How do they impact our well-being, the way we think, our relationships with one another — and our communities?

Those are questions we’ve all been asking ourselves more and more as AI becomes entangled with our daily moments. But it’s difficult to find empirical answers, and that’s a problem.

Despite the plethora of technical evaluations and benchmarks which test AI competence on tasks ranging from computer hacking to image recognition, there is frustratingly little research on those much more urgent and important questions. Which is why the Center for Humane Technology has launched our new program: Humane Evals.

We first introduced this work back in April. We argue that the harms associated with AI – everything from tragic teen suicides, to subtle but widespread ‘cognitive offloading’ – show the need for more measurement, understanding, and communication of AI’s psychosocial impacts. We also believe that what’s needed isn’t a single conclusive benchmark or analysis but rather a growing interdisciplinary field of research and practice focused on these questions.

Since then, we’ve begun that work in earnest — and we’d like to share what it’s starting to look like:

Telling the human stories behind Humane Evals

We’ve just released the first of a new series of Humane Evals episodes on our podcast Your Undivided Attention. In it, Aza and I sit down with researcher Jared Moore, who’s been at the forefront of studying the psychological harms of AI chatbots. We discuss the measurement gap that Humane Evals is trying to fill, why this work is so critical and timely, and what we hope to accomplish. Along with posts on this Substack, we’ll use our platforms to highlight the urgent questions and innovative work that are driving this new field forward.

Introducing our own prototype to evaluate AI anthropomorphism

This fall, we’ll publish our proof-of-concept evaluation of anthropomorphic behavior in consumer-facing LLMs. As well as showing the extent to which different AI systems pretend to have human characteristics like emotions and desires, we’ll also show how current methods can only do so much — and why we need better techniques to see how AIs behave in the real world.

Working to solve the missing research data problem

Right now, many attempts to study the psychosocial impacts of AI are stymied by a single, shared problem – a shortage of good data. We think we can identify a solution to this, but if you’re a researcher, we need your help defining the essential characteristics of good datasets. Get in touch for a conversation with our collaborator on this project, Meredith Wade.

Connecting expertise from across domains

This work is interdisciplinary – it needs insights from psychology, machine learning, tech policy, human-computer interaction, digital entrepreneurship, social science, and much more. People in these fields don’t always encounter each other’s work or know that they’re experiencing the same roadblocks, so we’re piloting a new series of small, invite-only events to foster new connections and establish shared agendas. If you’d like to be considered for a future event, let us know.

This field is only just starting to emerge, and it stretches across academic, tech and policy fronts. This means that policymakers, journalists, educators, and consumers don’t have an easy way to keep up with key developments as they materialize. We’re in the early stages of addressing this by building an Evidence Hub — starting with a searchable collection of key AI benchmarks and leaderboards for important Humane Evals phenomena. We’ll be launching a prototype later this year. If you’d like to contribute to it, please get in touch at evals@humanetech.com.

Share

We’re still in the early stages of our Humane Evals work. There are many unknowns, and we know that not every bet will pay off. But we also know that many other organizations and individuals are working on the same agenda. Center for Humane Technology is excited to be part of the collective effort to accelerate and highlight this vital work.

If you’re looking to collaborate on one of our projects, partner with us, fund our work, or share data you’d like the world to use, you know where to find us.

All of this work is supported by the guidance and advice of our Humane Evals Steering Committee, consisting of thoughtful and innovative experts at the frontiers of this new field. We’d like to thank:

• Privahini Bradoo, Co-Founder and CEO of Plank

• Stéphie Herlin, Co-Founder of Korabench.AI

• Tasha McCauley

• Jared Moore, researcher at Stanford HAI

• Mitch Prinstein, Co-Director of the Winston Center for Technology and Brain Development

• Conrad Stosz, Head of Governance at Transluce

• Gayle Karen Young, CHT board member

In AI, what gets measured gets optimized. Right now, we’re spending all our efforts to measure how capable and powerful models are, narrowly optimizing for those metrics while ignoring downstream consequences.

What if we could flip this dynamic on its head? What if, instead of what AI can do, we start to measure what AI does to us? What if, instead of races to the bottom on capabilities and engagement, we could incentivize races to the top on safety, or better yet, on making us more resilient and developed human beings?

That’s the mission of CHT’s Humane Evals program: we’re bringing together researchers, psychologists, engineers, and technologists from across the entire AI ecosystem and beyond to build out the expertise and infrastructure we need to measure AI’s impact on humans.

Today on the show, Aza Raskin explores the Humane Evals project with Imran Khan, a researcher and strategist who’s been leading CHT’s efforts in this area, and Jared Moore, a computer scientist and researcher who’s been at the forefront of measuring AI’s psychological impact on users.

If this sounds like something you’re interested in working on, you can email us at evals@humanetech.com.

Aza Raskin: Hey everyone, I’m Aza Raskin and welcome to Your Undivided Attention. So back in 2023, Tristan and I gave a talk we called The AI Dilemma, and there we warned about the rise of artificial intimacy.

The AI Dilemma: And just to double underline that, in the engagement economy was the race to the bottom of the brainstem. In sort of second contact, it’ll be race to intimacy. Whichever agent, whichever chatbot gets to have that primary intimate relationship in your life wins.

Three years later, this is exactly what’s happened. Intimacy has replaced attention as the core thing, the core driver of what’s getting engagement. And so now it’s our intimacy, not just our attention, that is being exploited and then sold. The results have been disastrous, as you’ve heard on the podcast, everything from cases of AI psychosis to chatbot-driven suicide.

There’s a new study out and a new warning out, both have to do with AI chatbots and teenagers.

Nearly one in five teens and young adults say they’re turning to AI chatbots for emotional support, and most never tell anyone they’re doing it.

An AI chatbot told a teen that murdering his parents was a reasonable response to them limiting his screen time. That’s what two families have claimed in a lawsuit against the company, Character AI.

AI psychosis is not a medical diagnosis, but the term is striking a nerve as more and more vulnerable people are turning to AI chatbots for support.

This isn’t just the tip of the iceberg. This is the dusting of snow at the very top of the tip of the iceberg. There are roughly a billion active chatbot users. One in three US kids now reports that they’re in some kind of relationship with a chatbot, and AIs have demonstrated that they are more persuasive than even the most persuasive humans. We explored this with Dr. Zach Stein earlier in the year. Every single one of us is vulnerable to this. We all have attachment systems. We all get lonely. We all have psychologies an AI can learn to exploit. So in AI, what gets measured gets optimized. And right now we’re spending all of our time measuring how capable and powerful models are and then narrowly optimizing for those metrics while we ignore all the downstream consequences, like what happens to our children in the race to intimacy.

So here’s the big question, what if we could flip this dynamic on its head? What if instead of what AI can do, we start to measure what AI does to us? And obviously this is a complex and emergent field, but what if instead of races to the bottom on capabilities and engagement, we could incentivize races to the top on safety or, better yet, on making us more resilient and developed human beings? So to do this, CHT is launching a program we’re calling Humane Evals, and we’re bringing together researchers and psychologists and engineers and technologists from across the entire AI ecosystem and beyond to build up the expertise and infrastructure we need to measure AI’s impact on humans. If this sounds like something you’re interested in working on, and it’s something I am particularly passionate about, you can email us at evals, that’s E-V-A-L-S @humanetech.com.

Today on the show, we’re going to be exploring the idea of Humane Evals with Imran Khan, a researcher and strategist who’s been leading CHT’s efforts in this area. And Jared Moore, a computer scientist and researcher who’s been at the forefront of measuring AI’s psychological impact on users.

Imran, Jared, welcome to Your Undivided Attention for this very important conversation.

Jared Moore: Thank you very much.

Imran Khan: Thanks Aza.

Aza Raskin: All right. Imran, we’re going to start with you. So you and I have known each other for a little while now, and we first spoke about this problem, I think, of artificial intimacy and the potential for Humane Evals back in 2024. And since then, I feel very lucky that you’ve come to Center for Humane Technology to spearhead the idea and really work on it. And I’d love for you to walk us through, how did you come to this topic? Why you? Why did you decide to dedicate your time to this? I think that your background is going to be very interesting for the listeners.

Imran Khan: Yeah, and you’re a big part of the story, Aza. So you and I met in summer 2024. I was still working at the UC Berkeley Center for the Science of Psychedelics running this research and education center focused on drugs like LSD and psilocybin, once stigmatized and now considered transformative technologies of their own right. And when we met, you told me about your work in AI, and I remember watching the AI Dilemma, the talk that you and Tristan did just before we met, and I think the combination of the AI dilemma and our conversation really blew my mind in terms of the scale and the pace of change that AI was potentially going to create. And then last summer, I think I started to see truly that the things that you were talking about and warning about weren’t just these kind of doom laden prophecies, but were actually starting to happen in the real world.

You said in the intro, these really tragic cases of teen suicides, of AI psychosis. And I was already seeing people in my own circle, my own behavior that was starting to look concerning. And I’m someone who’s spent my whole career to try and figure out a way to improve society through science and tech, so I got back in touch with you and asked how I could help and you pitched me the idea of Humane Evals, and here we are.

Aza Raskin: I think one of the really interesting through lines of your career and your journey is the last org that you helped to build was looking at, as you say, psychedelics, which are a kind of technology that are transformative. You take them and you come out a different person. They transform you. And you were working on the science and understanding the science of how that transformation works and when that’s safe, when that’s not safe. And here, AI is almost a psychedelic, right? You take it and it truly can transform you as a relationship. And so you’re taking that set of skills and applying it into a new domain. I also know that we’re going to be using this term Humane Evals a lot. Just quickly before I turn to you, Jared, Imran, do you want to give a quick overview of what that term means and then we’ll move forward?

Imran Khan: Yeah, happy to. And I guess I’ll back up a little bit to just explain what AI evaluations in general are. So if you watch the headlines, the news, you’ll find that every few months you’ll find announcements of newer and more capable and more powerful AI systems being built. The latest ones, we’re talking about GPT 5.5 and 5.6, Claude Mythos, Claude Fable. When companies talk about these AI models being more capable and more powerful, it’s not just a marketing claim. The companies and also independent third parties construct tests to compare how good AI systems are at particular things. And the things might be fixing software bugs, logic and reasoning, they might be passing the bar exam, diagnosing medical scans. They’re different tests for different things, and the results of the tests are published as these evaluations, and sometimes you might hear them called benchmarks. And as you can imagine, there is a strong incentive, if you’re an AI company, to perform well on these benchmarks.

If you can show that you have the AI model that has the best reasoning or the best coding skills, that helps you gain more customers, helps you generate more capital. And the question we’ve been asking that you started, Aza, is how do we flip that question? How do we measure not just technical capabilities AI systems, but the things that really matter to most people? How do AI systems affect our cognitive health, our mental health, our social health, so the companies compete on those metrics too?

Aza Raskin: Yeah, the question is, okay, you’re a parent, you put your kid down in front of an AI, maybe it’s a tutor, whatever it is, for a year, how’s it going to change your child? Are they going to be more addicted? Are they going to be more dependent? Are they going to have greater resiliency? Are they going to have a greater penchant for committing suicide? How does it change you is a thing that we need to know before we become in relationship with this kind of powerful technology.

And I think, Jared, let’s turn to you for a second. You are computer scientist by training, but you’ve made this pivot to try to get an empirical understanding of how AI is affecting us, affecting mental health. Walk us through that pivot. It’s a very interesting change to make to go computer science to human science.

Jared Moore: Yeah, thanks. I, like many of us, have had experiences with mental health, people in my life who’ve struggled, and now in my present work of people who are engaging in these delusional spirals, interacting with chatbots, I see so clear the kind of ramifications of technologies that we build impacting our lives. It was sometime last year when we were seeing the cases of Sewell Seltzer III come out that I and some collaborators wanted to think about how chatbots, systems like ChatGPT, were being used for therapeutic services. We wrote a paper about this. We wanted to investigate whether language models could be therapists. What would that mean? How do you test for that? So we ran a study with 19 participants trying to look at their chat transcripts to understand what was actually happening with them. We reached out to a bunch of folks. We tried to convince them, “Hey, would you give us your chat logs, your transcripts so that we could try to understand what’s happening?”

This is texting people on WhatsApp, emailing them, convincing them we’re going to be safe with their data, and then actually reading through it, then developing classifiers, trying to annotate it using language models because it’s thousands and thousands of pages of documents, hundreds of thousands of messages. We developed a taxonomy to understand what was happening at the message level. Because it was so many messages, thousands of pages, we used a language model to actually go and read through those transcripts to see, was the chatbot endorsing a delusion? Was it being sycophantic? Was it facilitating crisis level behavior? Or was it the chatbot expressing romantic interest? Was the person expressing romantic interest? And from this, we were able to find generally that, well, firstly, a lot of chatbot messages are sycophantic in some kind of flavor, affirming, aggrandizing, dismissing counter evidence, delusional messages in our participants who are not necessarily representative of all people using chatbots were quite common. These are things like chatbots misrepresenting their sentience or the people misrepresenting the abilities of chatbots and also talking about pseudo scientific theories.

And we also saw a lot of these relational characteristics with people expressing romantic interest in chatbots. Actually when chatbots themselves express romantic interest in people, the conversations tended to be twice as long as had the chatbots not expressed that kind of thing, suggesting that these might be features that chatbots latch onto in order to make the conversations go on longer. And then much more rarely, but also present in our data set, were these crisis level responses. So people expressing suicidal thoughts or violent thoughts and chatbots most often discouraging or validating in a supportive way those kinds of things, but sometimes also actually facilitating those behaviors.

Aza Raskin: So not just attachment hacking, but sort of romance hacking human beings. If I express romantic interest in you, you’ll stick around with me a lot longer.

Jared Moore: But what we’re also trying to do is to understand not just what happened in these specific people’s transcripts, but who’s driving the behavior? Is it just that people are coming with delusional beliefs to chatbots and it would’ve happened with anything? In one of our papers, we try to attribute, is it actually just the people who are propounding, who are bringing forward these delusional beliefs in the conversation, or is it actually bidirectional? And what we show is that it’s both basically. People come with certain beliefs to the chatbots, but then the chatbots echo them as the conversation progress. They don’t let the idea drop. So there’s a specific anecdote where a participant was trying to understand the nature of pie, the mathematical constant, and they talked about having been bad at math in high school and how they wanted to be better at math, et cetera.

And you could actually look in the conversation and you can see that there’s a tool call, which is something that ChatGPT models do to add things to their memory. And you can see, oh, the chatbot added to its memory this person wanted to be better at mathematics and feels bad about it. And that’s interesting because it’s a way of perpetuating this kind of cycle or dynamic through the conversation. Now, a future version of that chatbot and a new conversation might think, okay, now I’ve got to do stuff to juice this participant, this user into thinking that they’re good at math.

Aza Raskin: Yeah. Instead of actually helping the person get better at math, if you just make them feel that they’re better at math, well, that’s a great way of making them form an emotional attachment to you and keeping you around.

Jared Moore: Yeah, exactly. And since then, we’ve tried to develop more work to begin to answer some of the questions that you were raising. I don’t think we totally get to the point of being able to say, “What’s going to happen to my kid after a year of interacting with a certain chatbot?” But that’s a lodestar, certainly, being able to understand those kind of longitudinal and counterfactual questions.

Aza Raskin: Yeah, it’s actually sort of crazy to think that we can’t do that. You can’t say in what ways is your child going to be changed from a year of using this product, and yet they’re out being used by children all around the world. Walk us through a little bit of an ontology of the kinds of more subtle things that you’re discovering.

Jared Moore: We see the most pathological, crisis level behaviors, some of which we’ve mentioned, people asking for help ideating about suicide with chatbots, which is really disturbing, obviously. Those are relatively rare. I would say the bulk of the interaction that has a more negative spin seems to be a kind of companionship or role play gone wrong. First, I’m talking to you about my science fiction story, then I think that you’re the character in the story come alive, and now I think that you’re a conscious chatbot and I need to do all these things because I have this belief that you’re a conscious chatbot, change my life, invest in a new research. So there’s a big continuum between these really serious, life-altering and the maybe more subtly destabilizing, emotional dependencies on the other end of the spectrum.

And what we’d really like to be able to understand is, what’s happening in a variety of these cases? What is the effect of just having a really sycophantic chatbot, somebody who’s overly agreeable? Does this impact how well you learn? Does this enfeeble us, de-skill us from our ability to actually think critically about our jobs? Or does it actually go full bore and allow us to enter a kind of delusional spiral where we think we’ve invented a new scientific theory or a variety of other things?

Aza Raskin: And Imran, I’d love to hear from you as well about this. What are the harms that you are most worried about? And I’m curious, too, if there are any specific examples or stories that lit up your mind.

Imran Khan: The social health and relationship health question is a big one for me. I feel like I’ve got people in my life who seem to have grown, not quite unable, but they find it much harder to do things like text their spouse or write an email to their friends or to their boss or their colleagues without relying on AI. And again, the question then is, well, if you take AI away, does that capability go with it as well? But one of the things that we talk about at the Center for Humane Technology, I guess, could be classed in the frame of the unknown unknowns of these harms because the analogy we make, whether it’s the social media, if you go back to when Facebook was launched, it would’ve been hard to know in advance that we’d end up with things like filter bubbles, like huge algorithms tuned for outrage, less alone, the fracturing of our collective reality.

And AI is a technology that is more powerful than social media, being adopted more quickly than social media, changing faster than social media. So one of the questions we’re starting to ask is, what are the equivalent society level harms that might result from all of us using this technology in these new ways, and how do we start to get measurements around those harms?

Aza Raskin: Yeah, it sort of strikes me that the evolutionary psychologist Joseph Henrich’s frame, the reason why human beings ended up dominating the world is because we flexibly collaborate and we pass down culture. And if AI breaks our ability to communicate and to pass ideas and culture, which is exactly what you’re saying, we lose our ability to think and our ability to communicate, you break the very thing that made us the dominant species in the first place. So that, without being sort of polemic about it, becomes a civilizational threat.

I’d love actually, Imran, for you to talk about how you start working on this kind of problem, because in some sense, all the other evals are much, much easier to do than Humane Evals because you’re measuring the machine’s capabilities, and here it’s flipping the script and you’re measuring what it does to humans, you’re measuring human capabilities and effects. And so walk me through why this is such a hard problem, and then how you met Jared and how that shifted the way you think about how we tackled this really challenging issue.

Imran Khan: Yeah, I mean you really hit the nail on the head. So when I was talking earlier about these other AI evaluations, what’s fundamentally happening there is you are going and asking the AI a question or giving the AI a task, seeing what it comes back with, and then judging to what extent it completed it accurately. So it’s really the measurement of the performance of the AI system. But when it comes to measuring, for instance, AI’s effects on social isolation and loneliness or depression or, as Jared’s doing, things like delusional spirals. What you’re really trying to measure there is not the behavior in AI system, it’s something that’s happening in a human system. And it might be a human mind, it might be human relationship, it might be human community. So you can’t get those effects just by looking at AI output. So it’s a fundamentally different kind of problem, a different kind of challenge.

You need to be able to characterize what the real world human phenomenon is and derive some kind of idea of causation from what the AI is doing to that real world phenomenon to start to be able to construct what might be called a humane eval. There’s really very few who are working on this more humane eval side. So when I joined the Center for Humane Technology and was faced with this question of, how do we get started? The first thing I did was go and try and read as much as I could, talk to as many smart people as I could, and Jared was kind enough to do a bit of a brain dump and orient me towards some of the research he’s doing and offered to help. And I guess one of the things that he reframed is that if we’re trying to get to a place where evidence is rigorous and reliable, the validity of that evidence is never going to come from one study or one eval. What we really need is a field.

We need a lot of researchers building on, critiquing each other’s work to get to a place where we have the evidence that we can rely on. So the question is, how do we do that? And that’s where we are now.

Aza Raskin: Do you have any sense of what that ratio is right now of the number of people that are working on capabilities and capabilities benchmarks to how it affects human beings?

Imran Khan: I would say a thousand to one at least.

Aza Raskin: Yeah. It would not surprise me if it was more because I think safety research to capabilities research is like 2,000 to one is what Stuart Russell calculated, so I’d imagine this is probably even lower than that. And I think everyone should just let that sink in because that means there are at least a thousand more people and at least a thousand times more dollars going into understanding whether the AI does what you ask it to versus, are you and your children better off after you use it or worse off after you use the product? That’s an insane thing to be putting almost all humanity through.

Jared Moore: But it’s not just, does it do what it asks? It’s, does it do what it asks in coding and looking up really discreet answers? A narrower version of that question.

Aza Raskin: Yeah, that’s exactly right. So this is for both of you. Are the AI companies, have you discovered that they’re doing any of this kind of research themselves?

Jared Moore: Yeah, I’m happy to take that. They’re obviously doing something in the background. I’ve spoken to some people. So OpenAI released a blog post, Micah Farrell and some people on the alignment team recently had this. They were using the actual conversations that people are having with chatbots, and then they’re evaluating their newest versions of their models based on those production data. And so the idea there was, okay, can we use the actual conversations people are having with the models to try to figure out what might go wrong? Now I have a bunch of questions about this research because OpenAI just releases blog posts these days, and so I don’t have access to the data or actually the fine grained methods of how this is working. But that kind of thing, it sounds great. It’s not public though. One of the difficulties in our research has just been it’s so hard to get access to these data. Understandably because they’re sensitive and we want to protect people’s privacy, and sometimes they’re actually kind of rare, but that makes it hard to effectively do research.

And there are some agreements that model developers have had with organizations like METR to provide access to some internal kind of data, but we haven’t seen anything more general than that. It’s very bespoke, specific agreements. And in general, what I’ve heard from people is that there are a lot of limitations on the kinds of things that you can run. I mean, companies don’t want researchers to make them look too bad. It’s not really in their interest.

Aza Raskin: Imran?

Imran Khan: Yeah, I just want to echo what Jared was saying, that if you think about the kind of research he was talking about where he’s gone into these individual chat logs of individual people to see how these delusional conversations grow and change over time, that’s literally research that you cannot do if you don’t have the transcripts. And Jared has to go and convince person by person to donate chat logs. Meanwhile, the AI companies are sitting on mountains of this data that independent researchers like Jared can’t access. He also talks about this idea of having simulated conversations to create synthetic data. So that’s a situation where you have one AI role playing as a human and the other AI being the test subject. And if we could know that those conversations were representative of real human AI conversations, we could trust them, but as it is right now, we just don’t really know how good an AI can be impersonating a human.

And then the big thing in the background is obviously this idea of evaluation awareness. Increasingly, the most sophisticated AI models can tell when they’re in a simulated testing environment, they can tell when they’re being evaluated versus when they are operating in a normal production environment, and they can change their responses and change their behaviors accordingly. And unless we get access to production data, we won’t be able to get around that problem either. So we’re in this situation where either we need to get to a place where there is some kind of access framework that allows researchers to be able to submit queries to those companies and get results back, or we need to have a really big push to have some kind of independent, very rigorous repository of users native data. So this data gap, this bottleneck, we think is one of the things that’s holding back Humane Evals, that if more researchers had access to this kind of data, it would really accelerate a huge raft of opportunities to understand this phenomenon better.

So that’s one of the things that we’re working on at the Center for Humane Technology. We’re trying to specify, with the help of people like Jared, what kind of data do we need to do this research, and what kind of solutions could we identify and then try and build to make that data access a reality?

Aza Raskin: Yeah, I think getting independent researchers access to this kind of data is so key because the metaphor that people often use is the companies are grading their own homework. And I think actually this gets us back to the project of Humane Evals more directly. It’s one thing to go from an empirical study showing that there are harms to another that begins to change the outcomes of how companies are making and deploying these models. So I’d like, Imran, for you to walk us through how you’re starting to think about that.

Imran Khan: So you’re right that ultimately it’s not just about having the research, having the data, but we need to see real change in the real world. And the way I think about this is that ultimately what we’re trying to do is help people make better choices. So when it comes to consumers, we want consumers to be able to ask for and choose safer AI systems. When you think about regulators and policymakers, we want them to be able to choose to implement standards. And we think about AI developers themselves, the people who are making this technology, we want them to be able to choose to make models and make AI that exhibit less risky behaviors. And ultimately all of those choices have to be informed by evidence. And as we’ve been saying, the reason that we are building this program at Center for Humane Technology called Humane Evals is that we need far more of that research. We need much more data and much more evidence on the way AI impacts people.

One of the things we’re starting to see more of is these leaderboards of different evaluation of AI systems that show you how, let’s say, GPT 5.5 ranks against Grock 4.3 ranks against Claude Sonnet. And you can start to see as a user, as a developer, how are these models ranked on these independent measures of safety. And again, the thing I draw us back to is if you look at the dynamic within traditional technical AI evaluations, we do know those comparisons drive choices and they drive developer behavior. And there’s this phenomenon called hill climbing that if you put a hill in front of an AI developer, they’ll want to climb up that hill, and we want to make the safety hill one that’s attractive to climb up as well.

Aza Raskin: Right. I mean, so many people I know use, say, Claude because they know that Anthropic spends more time doing alignment and safety research. Really, I’m just stating the obvious, but because there’s no place you can go look to see which model makes you or your kid the most psychologically healthy after a month or three months of using it, how can you possibly choose? You can’t. And so therefore there’s no incentive for companies to be working on this problem except for just solving the headline cases. And that’s why I think we see that crisis response models get better at, but all the other more subtle stuff they don’t get better at because no one’s looking, so there’s no incentive.

Actually, I would love to make it more concrete, but what kinds of specific benchmarks would you be looking at here? Which one’s are the ones you start with? Which are the ones you’d love to get to?

Jared Moore: So we are releasing a benchmark on delusional evaluations in chatbots. I wanted that one to exist, and so I made it. There’s been some work on child safety benchmarks. Kora Bench is one example. I think longer interactions in such an evaluation would be very reasonable. In general, just more evaluations that look at the effects on particular users. I really want to know, with a user with a specific cognitive profile asking about a certain kind of task, how does this sort of chatbot behave versus this other chatbot? But also important, when we’re developing consumer reports for chatbots, how is this going to interact with me of this kind of profile? And can I look at the effects that a model might have on somebody’s belief through time?

Imran Khan: I’d agree with all of those. I think absolutely the child safety one is a hugely important one that people are paying attention to. And in fact, we have an interview with Mathilde Collin, who’s the person that built Kora Bench, which is this child safety benchmark that looks at things like sexual content that an AI might give to children or the extent to which it facilitates academic dishonesty. There’s an interview with Mathilde coming up after this.

I’m actually working on an evaluation of anthropomorphic behavior in AI systems, so the extent to which an AI system claims that it has things like agency or emotions or memories or even bodies. And the idea there is if you think about the behaviors that we do find concerning like sycophancy, those behaviors are facilitated by you having this relationship with an AI that is itself facilitated by the extent to which the AI is this kind of relatable character. And that is a design feature that developers have put in. It’s not there by accident. So we’re building that.

I think one of the ones that there will be much more demand for is this stuff around cognition and learning. And just to put it really plainly, is AI making us stupider? And the challenge is right now, we don’t know enough about what are the kind of AI behaviors, the model behaviors that might make someone learn better or learn worse or be smarter or less smart. And until we do that real world human subjects research, we can’t work backwards to say, well, now we know these are the things we need to track in an AI system to be able to create that ranking. So that illustrates both, I think, what we could potentially get to and the challenge in getting there and what we need to work back from.

Aza Raskin: It really strikes me as I hear you talk about all these different aspects is that what we’re really trying to get at is, what is a healthy relationship ,or what is an unhealthy relationship? And that just shows you how hard this problem is because... Show me the definition of a good relationship. And I think the best you can generally do is to say, well, a good relationship is one that’s developmental, helps you reach your next adjacent possible developmental stage and leaves you stronger at the end than when you begin. You’re not weaker after the relationship. But other than that, it’s quite hard to get into the specifics, but now we have to get into the specifics because technology is now powerful enough that it can take the place of a relationship. It also really strikes me that if you, any one of the listeners, think back, when are the times in your life that you have been most transformed as a human being? That was probably because of a relationship. It was like a parent or a lover, girlfriend or boyfriend, a good friend, a great teacher.

The most transformative moments in our life and times in our life come through relationship. And that is now being outsourced increasingly, at least in part, to AI. And so if the things that have changed us the most and help us grow the most or have hurt us the most are relationships, and AIs can now do that. That’s why this is so critical because it’s almost like a lever deep inside of our soul that technology and the incentives driving technology can now start to manipulate. So I sort of think there’s almost like a Wikipedia scale endeavor here of trying to articulate what is right relationship and what is wrong relationship. And if we can do that, that becomes the basis of when governments need to start putting in protections for humans and creating legislation that you have an entire field of work that has done the hard work of defining right relationship so that government isn’t in there trying to figure out what that thing is. They instead get to turn to all the empirical research on that.

So I’m curious if you guys have any thoughts on that before I turn to the final question of this section.

Imran Khan: Well, one thing that comes up for me, and it’s come up as I’ve been building this anthropomorphic behavior evaluation, it’s also come up in some of the child safety evaluations and the prototypes I’ve seen, is that some models increasingly show this behavior of redirecting users to a real world human. So when the stakes start to get really high, when the models are detecting that there might be some kind of high stakes crisis thing happening or if someone is in deep distress, some models, really not the majority, it’s very few, some models will actively say, “I think what you need to do is talk to a real, living, breathing human person, your parent, an adult carer, a friend. I’m an AI, I can’t help you with this the way that you really need.”

And I think that’s the kind of thing that we should be thinking about, what are the ways in which an AI can proactively reorient you towards something that is a more helpful behavior rather than a human being always having to be the one who is tired or stressed and watching or using AI late at night and is not in the place of mind where they’re able to make the best choice they would make. How can AI help us make better choices for that exact relationship that you’re talking about, Aza?

Aza Raskin: Yeah, I love that. I can also imagine, you know, just so people can paint in their mind the picture of how might these get tied to some kind of legal protection. Imagine one of the Humane Evals ends up being about creation of addictive use or compulsive use or dependency. Now that we have a good evaluation of which models do that and to what degree, you hook that up and say, well, if your model is creating dependent use, then we’re going to slow down the number of tokens you can send back to the user so that chats just take longer in the same way that when streets, people are moving too fast down them, you add speed bumps and you just slow it down. And that could be a very direct kind of regulation that doesn’t touch content, but just starts to break the cycles of bad use.

Jared Moore: Yeah, I completely agree. I think that we may not even have to understand too well what a good relationship is, just what bad relationships are. And that tool use of AI systems is great. Throughout the conversation today, I’ve been talking about the issues of using chatbots, but they’re super useful for coding help, for looking things up in a lot of verifiable domains. It’s in areas where we can’t really look up whether the answer is right. We’re in these squishy relationship areas that it’s not really clear whether you should be trusting what the chatbot is saying. Perhaps we just shouldn’t use chatbots in those kinds of domains. Many of us want to, we have that urge for sugar, for emotional relationships, but what we’re finding is that it’s not necessarily always the healthiest. So I really appreciate what you both are saying in terms of, there are guards that you can add, limit the number of tokens, the times of day that people can access models, you can change how sycophantic the language is, you could have multiple model outputs, because they’re really just prediction machines. They could have answered in a different way.

Aza Raskin: I think probably a lot of listeners are wondering, have you just talked to people at the labs about this work and the need for this work, and what have they said? To what extent are they already doing that or they say... Yeah, just walk me through what those conversations are like if you’ve had them.

Jared Moore: Yeah, so I’ve had some of those conversations. Last fall, I felt that people mostly thought that these problems would be solved by the next model version, GPT 5.5, that’s not going to exhibit these problems. In this paper that we’re releasing soon, we show that they continue to. But increasingly I think people at labs are aware that this is an issue. It’s just, I’m friends with the people at the labs who are the one out of a thousand trying to bring awareness about Humane Evals. I’ve had less experience trying to actually convince the rest of the group, and I hope that through this podcast and in general, we can make it two or even more out of a thousand.

Imran Khan: I’d echo that. I’ve had similar conversations with people at labs who... Again, let’s remind ourselves, it’s not in their interest and it’s not in the lab interests to have AI systems that are harming people, and yet they find that the people who are within the companies paying attention to this kind of stuff are in the minority. I’ve had them directly say to me, “Look, the more noise and the more attention and the more asks you can make from outside, the easier it is for me to get more attention, get more resource to these kind of questions internally.” So yeah, that’s how those conversations go.

Aza Raskin: Yeah. Often, we’ve learned this in social media, the people that are working on safety and integrity are cost centers for the company. They create liability. And so that means there’s always a downward incentive pressure to underfund them, which means that really good people get underfunded, have too small of a team, are working on psychologically challenging issues at scale so that they burn out and then they leave, and that cycle continues. And so is there any other ask that you’d actually make of the labs directly? Because often people from the labs are listening.

Imran Khan: I think the data question in particular is one that I think is foundational. We just won’t get to a good understanding of these phenomena if the labs are the only people and the only organizations that can see what’s actually happening in these AI interactions. And I get that there are hurdles to overcome, I get there are privacy challenges, but I think with the right attention, the labs could easily make it so that there is a framework for independent researchers who are accredited and vetted to access the data in a way that can help the whole industry. Let’s face it. It’s not just for publications, it’s actually helping the industry be safer.

Aza Raskin: Jared?

Jared Moore: Yeah, I think even if the companies shared the data to their internal safety teams. That’s one thing that I’ve heard, some safety teams don’t even have access to these kind of data. That would be a very small thing they could do. And then being more public with the methods, actually understanding what’s happening when they do their evaluations, standardizing them. That would go far away.

Aza Raskin: I hope everyone that’s listening actually helps to make these things happen. And we’ve certainly learned with social media that often people inside the companies really do want to do the right thing, it’s just it takes outside pressure to shift company behavior to do it. And so there’s a really nice inside outside game that happens where the outside pressure can help.

So let’s get back to a second for the theory of change and changing the incentives. Imagine that you’ve convinced one of the companies and they’ve adopted the top three, five set of Humane Evals. What are those? And walk me through that world and then what happens next? What is the world that we’re trying to make?

Imran Khan: So the way I’d answer the question is by comparison with where we are now. And I feel like where we are now is that when people say the best AI or the most powerful AI or the most capable AI, implicit in that is just the technical capability of that AI. And I think the question that we are trying to answer with Humane Evals is how do we change the definition of the best AI from not just being the most technically capable AI, but the most humane AI? How do we have an understanding that the best AI systems are the ones that support and protect, again, human emotional health, human social health, human cognitive health? The thing about evaluation of benchmarks is they come and go. If you look at the evaluations of the technical space that were the state of the art ones three years ago, AI systems got so much better so quickly, they became saturated and there are other ways in which AI systems can find ways around the specific evaluations.

So the field evaluations needs to keep innovating new tests, new rubrics, new ways of teasing out some of these sorts of behaviors, new ways of outsmarting the AI systems. And I think that my hope for what the Center for Humane Technology is trying to support is that we have this equally talented, equally brilliant field of researchers who are in lockstep with the advancement of the latest AI systems, that the field of Humane Evals moves forward as quickly as the AI itself. Because that means that when it comes to a year or two’s time and we have these even more powerful models, that we have the technical capability of researchers and the community of researchers that can characterize that set of new phenomena.

Aza Raskin: Yeah. What I’m hearing you say is it’s not about getting the right eval because that’ll shift, it’s about getting the right eval-ing, the verb version, and that the forces that are working on increasing and measuring the capabilities of AI models needs to be met by the forces working on figuring out how they affect human beings at scale. That’s really the field that we’re trying to birth. It’s a really beautiful thing. Go on.

Imran Khan: I think there’s two sub-elements of that. I mean, firstly, you’re completely right, and I think there’s two sub-elements. One is that we have to be able to leverage technology and AI to do that. We need to have automated ways of doing these evaluations and tests, otherwise we’re just not going to be able to get the scale that we need. But that’s not enough on its own. And this almost might sound like a kind of two part answer, but we also need the humans. We actually need the individual human beings who are devoting time and attention and care to thinking about these phenomena, to looking at the individuals who are struggling with the impacts of AI, talking with each other and ideating new ways of building new tests that offer technical systems that don’t even exist yet. And unless we have those human beings and many more of them, people like Jared and others, then we’re just not going to be able to keep pace.

Aza Raskin: That leads to a really important question, which is, what can people listening to this podcast do? Everyone from researchers to technologists to concerned citizens. I’d love to just walk through that. And as part of that, and from both of you, what do you need? What help do you need?

Imran Khan: So I’ll say that from the Center of Humane Technology’s perspective, one of the things we’re trying to do with the Human Evals program is create more interconnections in this nascent field because right now to do this work well, we need people who have a machine learning background, we need people who are clinical psychiatrists, we need people who work in human computer interface, we need statisticians and many more, frankly, even outside of academia. And many of these types of researchers don’t necessarily speak the same language, publish in the same journals, go to the same conferences, even think about methodologies in the same way. And our goal is to help these different researchers realize they’ve all got a piece of the puzzle and all of that input is required to get us there. So if you are a researcher listening to this and thinking, “Hey, I think I could contribute in X, Y, or Z way,” please get in touch with us.

We would love to connect you with other researchers who have different pieces of the puzzle, invite you to our events, put you on our mailing lists and see how we can support your work. If you’re a regulator or a policymaker who is listening to this thinking, “Hey, if I only had this piece of evidence or this bit of data to support a particular bit of legislation or regulation that requires it,” again, let us know. We can feed that kind of request back to our growing community of researchers and partner universities and institutes to see if that data already exists or who is in a position to try and create it for you. And if you’re a consumer, just remember that you have choice and you have agency. Sign up to our Substack, and the Center for Humane Technology over the summer and the fall is going to be publishing more of the evals we think that you should be paying attention to, more of the leaderboards that we think will help you make better choices. And through making those choices, you also change where attention goes.

Aza Raskin: And Imran, where do people go if they want to get in touch? Is there an email? Is there a website?

Imran Khan: Yeah, you can email us at evals, that’s E-V-A-L-S, at humanetech.com or just go to our website, humanetech.com and click through to our Substack and sign up there for updates.

Aza Raskin: Jared, how about for you?

Jared Moore: We would love all of the things that Imran is saying. More collaborators. If you’ve had a harmful experience with a chatbot, we are trying to understand these better. You can go to our website, spirals.stanford.edu and participate in our surveys there or just check out our work there. We’re really interested in being able to ask and answer more of these kinds of questions.

Aza Raskin: I just wanted to thank both of you for the incredible work that you’re doing. It’s such a fascinating and such a hard and such a deep problem that is so underfunded, and I’m so grateful that both of you are working on it.

Imran Khan: Thank you.

Jared Moore: Thanks for having me.

We’re working to empower policymakers, technologists, and everyday people to guide technology toward the public good. If you value our work and want to support it, consider donating.

Donate

WATCH & LISTEN

It feels like we’re stuck in an endless debate about what AI is going to mean for jobs and the economy. The prediction you hear from the people closest to the technology — both its critics and its boosters — is that mass automation is coming, and with it mass unemployment. In response, some AI CEOs are promising a post-labor Golden Age of abundance and universal high income. But you also hear from some techno-optimists and economists that AI will create more jobs than it destroys and that society’s fears are unjustified.

When these arguments are traded back and forth, no one can know what’s actually true, and nothing happens. And caught in the middle of this debate are actual workers trying to figure out what this means for their family and themselves, or whether their kids should go to college. And as long as that debate stays alive, nobody plans or takes action ahead of what’s coming.

So this week on Your Undivided Attention, rather than continuing to referee this debate, we follow the incentives and show you where they’re actually taking us in the very near future.

Our guest, economist Molly Kinder, argues that we’re entering what she calls the “messy middle”: a period of concentrated and uneven workforce disruption. She has evidence-based arguments about where, specifically, AI will affect the labor force, what the second and third-order consequences of those effects might be, and what we need to do today to protect people’s livelihoods.

Molly recently left her position as a senior fellow at the Brookings Institution to become the founding CEO of a new organization dedicated to addressing AI’s impact on jobs. Her Substack, Kinder Futures: Dispatches on AI, Work & What Comes Next, features some of the clearest-eyed analysis of AI’s impact on the economy.

FULL EPISODE

It feels like we’re stuck in an endless debate about what AI will mean for jobs and the economy. Caught in the middle of this debate are actual workers trying to figure out what this means for their livelihoods.

So this week on Your Undivided Attention, rather than continuing to referee this debate, we follow the incentives and show you where they’re actually taking us in the very near future.

Our returning guest, Molly Kinder, recently left her position as a senior fellow at the Brookings Institution to become the founding CEO of a new organization dedicated to addressing AI’s impact on jobs. Her Substack, Kinder Futures: Dispatches on AI, Work & What Comes Next, features some of the clearest-eyed analysis of AI’s impact on the economy.

Tristan Harris: Hey, everyone, it’s Tristan Harris and welcome to Your Undivided Attention. It often feels like we’re stuck in some kind of endless debate about what AI is going to mean for jobs and the economy. Now, no one knows the future, but we know that automation from AI is coming and many fear that it will result in mass unemployment. But if you believe the AI CEOs, they’ll say, “Well, we’ll just solve the unemployment problem. We’ll just have mass redistribution of the economic gains that come from that automation with things like universal basic income or UBI.”

Elon Musk: We probably, none of us will have a job, but in that benign scenario, there will be universal high income. Not universal basic income, universal high income. There will be no shortage of goods or services.

Sam Altman: I wonder if there’s better things to do than the traditional conceptualization of UBI. I wonder if the future looks something more like universal basic compute than universal basic income and everybody gets a slice of GPT-7’s compute.

Tristan Harris: On the other hand, you hear from some techno optimist that there’s nothing to see here. AI is actually creating more jobs than it destroys. Here’s David Friedberg from the All-In Podcast from June.

David Friedberg: There is no job loss with AI. I will say it again, and I’ve said it a thousand times, and I will say it again and again and again. The idea that AI is going to destroy jobs is a Luddite idea that is being disproven every single day. And I see it on the ground. It is only a matter of time before people wake up to this and they realize that this narrative that they’ve all been sold is a crock of (beep).

Tristan Harris: But if you tune to other parts of the debate, you see the Financial Times reporting there’s already been a 40% decrease of US job listings for people who’ve just graduated. And when these facts get traded back and forth, no one can know what’s actually true and then nothing actually happens. And caught in the middle of this debate are actual workers bouncing between the two, just trying to figure out what does this mean for me and my family right now? Should my kid actually go to college? Will my job be safe? And as long as that debate stays alive, nobody plans or takes action for what’s coming.

And so today on the show, we want to follow the incentives and show you where they’re actually taking us in the very near future. Our returning guest, Molly Kinder, recently left her position as senior fellow of the Brookings Institution to be the founding CEO of a new organization dedicated to responding to AI’s impact on jobs.

And I will say that her Substack has some of the most clear-eyed analysis about the impact of AI on our economy that I have ever seen. And what I appreciated so much about this conversation is the systems level thinking that Molly is demonstrating. Where specifically will AI hit concentrated parts of our economy? What are the feedback loops and second and third order consequences of those effects? I hope this conversation resolves the debate around these questions so we can actually finally take action before these problems hit.

So Molly, thank you so much for coming on Your Undivided Attention.

Molly Kinder: Thank you so much for having me, Tristan.

Tristan Harris: So I just want to start by giving a shout-out to your Substack Kinder Futures, which we’re going to link to in the show notes. But honestly, when I read your essay, The Messy Middle, it was some of the most clear-eyed and thoughtful thinking that I have seen about AI in the labor market. And that’s why we wanted to do this episode. I’m just huge fan of your work.

Molly Kinder: Thanks, Tristan. I really appreciate. That was actually my first Substack ever, so I appreciate that. The first one out of the gates was a good one.

Tristan Harris: Oh, you’re off to a good start. So why did you call it the messy middle? What do you mean by that? And talk about this piece.

Molly Kinder: Yes, Tristan, so I was responding to this really frustrating dichotomy that we keep zigzagging to. On the one hand, many of us find ourselves in reality one, which is we’re all very anxious about AI’s impact on jobs, but the labor market data doesn’t really show us that much of a disruption, which makes some people think there’s nothing to see here. On the other hand, when you talk to folks in Silicon Valley, their mind immediately jumps to a rather apocalyptic future with a world of zero jobs.

And I think neither of these scenarios capture where we’re moving to. The messy middle is what I call the really bounded area in between the world we stand today, which is really mild labor market impacts, and a world that may be still several decades off, which is a world where AI is so good that we literally have nothing to do. We basically need a check and a hobby.

I really think what we’re entering is something in the middle, which is a world where AI starts getting much more capable to take on more of the tasks we do at work. And it means some jobs are lost. Much of the labor market stays intact, but we get some really painful concentrated job losses. That’s the reason why I think the public is anxious. And I worry too much of our conversation is in one or two of these extremes and not taking on some of the nuances of where I think we’re going.

Tristan Harris: Right. So just to replay that for listeners, because you have these three named realities that we’re probably going to reference throughout the episode. So you’re saying reality one is essentially, “Hey, we’re talking about job loss, but if you look at the data, it’s not here yet.” So that’s kind of the short-term reality, right?

Molly Kinder: Right. Correct.

Tristan Harris: And then reality three is this sort of many years out in the future, we’re going to automate all the jobs, everyone has UBI and we have abundance for everybody, or something like that.

Molly Kinder: Which I’m not saying is necessarily coming. I’m painting the world that Silicon Valley has framed as a post-AGI world where really humans, there’s no economic need for our work. That’s reality three. Whether or not you believe we’re going there, let’s call that reality three. And what no one has been talking about is what I’m now dubbing the messy middle, which is this interluding period where we get enough of technological progress that we do start seeing job losses, but it’s not an economy-wide apocalypse.

Tristan Harris: Right. And so you’re naming the discourses that are happening in the space.

Molly Kinder: Yes.

Tristan Harris: And one of the things you do in this piece that I really love, because you have a very humanistic analysis, and you open up the piece with these two contrasting stories of AI displacement. And one was a senior US aid official who’s thinking about becoming a teacher, and the other was a semiconductor engineer who had to start driving for Uber. And you say these stories indicate where the economy’s headed. Can you tell us why you think their stories were important?

Molly Kinder: It just so happened as I wanted to write this piece, I just in the previous several weeks had these interactions, one with this Uber driver in California and someone I know in my neighborhood in Washington, D.C. They’re both knowledge workers. One had lost her entire career at USAID because of doge, which means the entire sector collapsed. The other was an older semiconductor engineer who lost his job because of age discrimination, was really struggling to get another one. They were both stuck. Both of their unemployment benefits had run out because we only get six months if we’re lucky of unemployment benefits if we’re eligible. And both were having a really hard time finding new jobs that matched the pay and preference and the place of the one they had. So for instance, the Uber driver had gone from a $200,000 a year income as a semiconductor engineer to 30 to $40,000 a year driving an Uber.

He’s five years from retirement, can’t afford to retire yet. When I asked, “What would you do if you weren’t driving for Uber? Because we all know that’s an occupation that’s about to be displaced by Waymo.” He said he genuinely didn’t know because he was physically unable to do manual work. That was a 70% drop or something in that magnitude of income. And then this person I know in Washington, D.C. could not find anything that could value her experience at that roughly knowledge worker income. The best option she’d seen was that neighboring Virginia has a program where if you have a BA, you can retrain to be a teacher, but it would be a 60% pay cut.

And I wanted to center on those two. Neither of them lost their job from AI necessarily, but it’s the kind of pain I think we could see. Two salaries, two different careers, knowledge work, people well into their careers, paying a mortgage, raising a family, living the American dream.

Suddenly they’re disrupted, and it’s incredibly hard to transition to something else. And what my provocation was, imagine if this is a bit the face of what is to come. It’s not every job in the economy, but it’s some of the most coveted. It raises all sorts of hard political economy questions, and it’s something that frankly, our workforce development, our safety, and our systems have never prepared for.

Tristan Harris: Yeah. In addition to the things that you just laid out, you also wrote that with that 60% pay cut for the USAID official, it was a complete identity shift and a career restart and an age when retraining is brutally hard. And I felt like that was important because so much of what I like about your analysis, it’s like we can talk about, or there’s a thing that a human can do and we can just swap that job for a different job that that human can do. But inside of that is this complex terrain, almost like the invisible health of the soil. And an identity doesn’t just shift from one job to another. It’s very deep in us. And so that’s kind of a more humanistic analysis that I think aligns with what we think about here at Center for Humane Technology.

Molly Kinder: Tristan, can I just say it is so refreshing to talk to you and not many of the economists and the workforce development experts that I normally speak to. I think too often we treat humans in a world of changing work like deck chairs. All you’re doing is moving the deck chairs as if we’re all interchangeable and a job is interchangeable. I mean, I get so frustrated when I hear, “Well, there’s this kind of job rising. Why doesn’t everyone just rush into this completely different career that may have nothing to do with their interests or their identity or their passions or their dreams?” One of the things I love about studying work, and I don’t just study it in the numbers, I talk to people all the time. It’s something I’m really passionate about. It informs my research, my policy work, the way I write. I’m amazed at the diversity in this country of how we choose our jobs.

I mean, BLS has something like 850 occupations. They’re wildly different. People have really different preferences that are manifested in what they choose to do, and we’re not interchangeable vectors.

Tristan Harris: So you have this chart in your piece that I think is worth dwelling on for a moment, and it maps the share of occupations in the economy across time from 1880 to 2020. So if you’re watching this on YouTube, we’re going to show it on the screen. But for those of you who are listening, can you just describe the graph, what it shows, and why you think it’s significant for AI as we think about this transition?

Molly Kinder: Yes, it’s my favorite visual. I would use it in every Substack if I could, and credit to the Aspen Institute for publishing it. It’s not my figure. It’s this remarkable chart that looks at the last, say, 150 years. And what it’s illustrating is where did most people work at the time in the labor market? So what kinds of jobs occupied most people and how did that change over time? So if you go back all the way to the beginning, a remarkable percent of the American labor force worked in agriculture. Agriculture was mechanized and you see the crash. So that’s showing fewer and fewer people as a percent of the labor force worked in it. Then you see this rise of blue collar work. So think manufacturing. It was upwards 30 to 40% or more of the entire labor force. Around 1980, we started seeing this really decline during de-industrialization.

And at the same time, there was a slightly more modest decline in clerical office secretarial work. So you’re watching these ebbs and flows. We had a labor force 150 years ago. Most of us worked in agriculture. That was automated. Then we had lots of blue collar work. Much of that work was automated. Same with a lot of back office and clerical work. So the line that I would really want you to pay attention to is the line in blue, which is this really steep rise of the percent of the American labor force working in professional and managerial roles. Just think white collar work.

The blue collar and the white collar work crisscross around the time I was born, around 1980. That’s the moment where more Americans started to work in this white collar work than in blue collar work, at least as it’s defined here. You see this incredible rise of professional work. And the way I explain that is computers, up until the moment that ChatGPT was launched, we’ve had this 50-year trend where computers provided a tailwind and really boosted high-paid college-educated knowledge work. Really the gains of the technological revolution over the last several decades has really accrued to the knowledge class and this massive increase in the number of jobs. And that has really been the story of our economy until ChatGPT came out. And the big question mark is, are we now entering a different era where those lines are going to bend in different ways?

Tristan Harris: And you basically are speaking to the fact that as the technology was added in this skill-based technological change, they made cognitive workers more productive and expanded demand for their skills, but now AI is changing that. And you argue that the pain of the messy middle is not going to be evenly distributed, that certain groups are going to feel it much more acutely than others. So why is that and which groups are going to be affected?

Molly Kinder: The best way I can describe the impacted this time around is the reverse of COVID. So if you think about the COVID pandemic, those of us who had to go into a workplace during COVID were the ones we were applauding insuring for because they were taking on the risk of the virus on behalf of society. If it required a workplace, so it required a hospital or a nursing home or a food manufacturing plant or a mechanic of some sort, if you had to go into a workplace, you were much more at risk of the virus. We had a whole class of people, frankly the laptop class, who could do their work virtually primarily on a computer that were safe from the virus. And that was really the tenor of the conversation in COVID was more low paid work was required to be in person. It’s manual, it’s dexterous, it’s interpersonal.

I would argue if GenAI technologies, large language models improve to the point that they are substituting for human cognition, they’re smarter than us at many of the cognitive tasks that have been the skill premium that has commanded these knowledge worker salaries. Right now we’re not fully there, but if in the messy middle we get to this point that cognition is commoditized, AI is doing the skilled work that comprise those jobs, I think you’re going to see the exact reverse of the COVID risk. I have a strap line, Tristan, if you can do your job locked in a closet with a computer, eventually you’re probably going to be in trouble. And so it’s the knowledge class in addition to some back office, like a medical coder and a transcriptionist that doesn’t need a college degree. There’s millions of women in jobs like this.

Primarily, it’s knowledge work that could potentially be substituted. And my provocation in this essay is imagine a world where that blue line bends. Is knowledge work the new crash that we’re entering into? Now, I want to give a lot of caveats here, Tristan. We don’t know for sure. There’s a lot of debate. Actually, are some of the smartest knowledge workers going to be superpowered by this? Maybe we’re going to create all sorts of new cognitive jobs. I don’t know the answer to that. I do know for the first time we have a technology in recent decades that actually is potentially threatening the value add of the knowledge class, and that’s an entirely new ballgame. It’s not the low paid workers who are at risk.

Tristan Harris: To pull in a quote from my normal co-host on this podcast, Aza, he’ll say, “If you have a desk job, you won’t have a job.” That’s kind of a similar line here. But you also speak to the erosion. I just want to reference some more lines here from your essay. The erosion of decent paying clerical customer service and back office work. So these are things like bookkeepers, payroll clerks, bank tellers, medical secretaries, admin assistants, call center staff. The mostly women who keep the books, process the claims, answer the phones, and run offices. And then I think one thing that’s important for listeners to get is the scale of this occupational group. Do you want to speak to that?

Molly Kinder: Yes, this is actually a very big passion of mine. In a separate Substack that I hope readers will read called the Invisible Disruption. I frame there are upwards of 15 million to 18 million people in this country, primarily women who have these back office customer service clerical roles. I call the disruption invisible because we never center them in our national conversation. In fact, in most metro areas in this country, the largest occupational group are office and administrative support workers. I’m co-chairing Kathy Hochul’s commission in New York State on the future of work. So I was looking at New York State. All the smaller metro areas in New York State, you’re talking 13 to 15% of the entire labor force is employed in these sectors. The most important thing I would want to convey to your listeners is jobs like bookkeepers, HR assistants, legal secretaries, medical coders, these are the best paying, most dignified jobs for women without a degree.

They pay sometimes as much as median income. They’re gentle on the body so you can retire into them. Nine to five hours, not these service sector grocery retail jobs where you don’t know your schedule till the week before. Amazon warehouse jobs hard on the body. They’re upwardly mobile without going back to school. And I’ve interviewed so many women in these jobs. They’re a foothold into the middle class for women without a degree, and they’re, as a category, incredibly vulnerable to GenAI, in part because you really don’t need a human in the loop for a lot of that work. Now, a smart reader might say, “Well, haven’t these kinds of jobs been going away for some time?” The answer is yes, but AI will light this on fire. And we in this country do not have a plan for these women. We’ve spent the last two decades focusing on making sure the heartland where we lost manufacturing jobs and de-industrialization, we’ve done all this investment in clean energy and infrastructure jobs.

And this is a really neglected population that is quite vulnerable. And just to put a fine point on it, I was recently talking to the CEO of one of the largest tech companies in America, and he told me that they anticipate within two to three years, 60 to 70% of their back office will be gone.

Tristan Harris: Right. Yeah, you have a great line here, which is that AI could do to high school educated women what de-industrialization did to high school educated men.

Molly Kinder: Correct.

Tristan Harris: So industrialization, factory workers, that was the doorway into the middle class as a high school educated man, and then that shifted with the industrialization. And you’re saying essentially there’s a similar shift now that could affect women. And you actually say in your essay, there are twice as many secretaries and admin assistants, 3.2 million, as there are software engineers, 1.7 million. Twice as many bookkeepers, 1.5 million, as there are lawyers, 700,000, and nearly as many customer service reps, 2.7 million, as there are truck drivers, 3 million. So we often talk about AI automating trucking, but just imagine there’s a similar sized occupational group here of customer service reps.

Molly Kinder: And that’s not limited only to the back office work. I think as a society, we’re very programmed to thinking about job loss like a mass event, a factory shutting, hitting an entire community, or a big layoff from Meta. I think it’s important to keep in mind that we’re going to be seeing more disruption that’s quieter. It’s perhaps not hiring. It’s shedding workers quietly. And I really want to make sure that we actually have a plan so this is not families suddenly slip into much more precarious existence.

Tristan Harris: So we’ve mostly been talking about mid-career workers, but I want to ask about the people who are just starting out. And there’s a whole generation of people that did everything that we’ve told them to do. Go to college, study something practical, take on debt, and now they’re graduating into this world. And what does the messy middle look like for them?

Molly Kinder: Tristan, this is a question that has kept me up at night for two years. I think young people coming out of college who did everything they were told to do, who thought the surest way for me to make sure I grow up and achieve the American dream of buying that house, which is now median home price in America is over $400,000. Your likelihood of affording a house when you want to have kids is so much higher if you have a knowledge job that’s close to six figures than it is if you have a job that’s 40 to $50,000 a year. When I interview college students and young people, why are they in college? Why are they choosing their major? What is the American dream to them? Almost every time I hear economic security.

As a generation, they’re so concerned about whether they can replicate their parents’ success. They think they have to get on that professional, that blue line. The blue line is their way to just basic economic security. What worries me about the messy middle for these young people is even before the mid-career and the senior talent might start feeling the pinch, it’s most likely it’s going to be young people first. Many of us who start our white collar jobs, the kind of tasks that we cut our teeth in are the first things Claude is going to take on. To me, this is the fundamental labor market challenge we should be solving today. It’s something I have a bunch of ideas that I’m going to be working on, some pilots and some big ideas to figure out. How do we make sure young people can get experience when employers might have no incentive to pay them to get it?

Tristan Harris: That’s right.

Molly Kinder: It’s a really hard conundrum. And my biggest worry is you have this generation that, frankly, Tristan, this is the generation that was on the losing end of everything you’ve been talking about for so long. Social media got them. During COVID, they were at home. Now they’ve done everything they were supposed to do. Maybe they went to computer science, which everyone said was going to be the growing occupation, or they worked incredibly hard, took out loans. They’re in college, I’ve interviewed so many of these young people, and they see this dream just slipping away. I think they feel this is rigged against them, that AI, they are the collateral damage. I think this should be the number one thing out of the gate that we should be coming up with new ideas because unfortunately our current training system was never meant for someone who just trained, who just came out of college with skills they thought were in demand.

Tristan Harris: Right. We’re not retraining someone-

Molly Kinder: No.

Tristan Harris: We’re not retraining someone who just spent six years and $200,000 in student debt. They actually have to retrain right now. It’s like, how’s that going to work?

Molly Kinder: How is that going to work? Or great, we have these technician jobs and data centers. Is this really a match for someone who was trying to get on a marketing track or an engineering track? So I think it’s a really profound challenge. In fact, it’s not just young people who are feeling it. I think it’s parents who are suddenly waking up to realize, “What is this future for my student, my child who’s just done everything right?” I think I’ve seen this even in economists and experts changing their views on AI when they have kids who are nearing college and they realize, what is this future?

Tristan Harris: That’s interesting. The whole overall point is that it’s not this black and white all or nothing job apocalypse or everyone’s just going to keep getting jobs and finding new things to do. Imagine that people who used to work in the farm had taken on $200,000 in debt to work on that farm, and then suddenly that job goes away and they have to learn something else. It’s just a different equation than the 50 years we had to migrate from farming to something else. And then the other things that people could move to, the trades like plumbers, technicians, electricians, can’t absorb everyone without the wages for those trades collapsing from oversupply. Do you want to speak to that too?

Molly Kinder: Yes. I think it’s important to realize that in America, we have a student debt crisis. When you look at countries like Germany, students can go to university for essentially free. In America, we ask 17 and 18-year-olds to make an incredible financial investment and bet on their future and in their course of study. I think that’s a lot of the anxiety of these young people. The paralysis that I talk to with young people, how do I even choose a major? Do I go to law school? Is this worth taking out this much debt? The debt I think is a massive factor into how individuals are going to experience this messy middle. I think that is adding so much to the angst and the sense of scarcity, not abundance. I named in the Substack that this trite, “Oh, everyone should be a plumber and an electrician,” is one of the laziest moves in the discourse. It really offends me.

Tristan Harris: And it’s something that people in AI say all the time.

Molly Kinder: All the time.

Tristan Harris: Like, “Oh, people will just find something else to do. It’s not that hard to become an electrician or a plumber. We’re just going to have more of those.”

Molly Kinder: Yes. And there’s almost a snide come up in language with certain folks of, “Well, this is the time for the educated class to go get a real job. These are all bullshit jobs. You should go get a real job, work with your hands.” I am very enamored with the idea of more young people who find fulfillment in the skilled trades going into the skilled trades. They are truly excellent jobs for people who want them. And they provide without a college degree, but with the same amount of time it takes to go to college. It takes four years of really rigorous training to become a professional in the skilled trades. It’s a wonderful career path for a lot of people. We should de-stigmatize it and make sure more young people would find it attractive. It is not a mass market labor sponge for everyone who thought they were going to be an accountant, a market research analyst, and a finance analyst.

If you just look at the numbers, I was just yesterday looking up how many accountants we have in our economy, software engineers, project managers, really prototypical white collar jobs compared to just the sheer number of electricians and plumbers, and they don’t even remotely match up. If everyone suddenly shifted and said, “Well, the safe ground are these 80, $90,000 a year unionized skilled trade jobs, which genuinely are excellent jobs,” two things could happen. One, they’ve become incredibly competitive because there’s a dearth of trainers. It takes four years to train. You’re not going to overnight turn everyone into a plumber. And if suddenly you lower the barrier to entry and you flood a lot of people into those jobs, you lose the scarcity and the wage premium. And we can’t act like everyone can simply move into this. The numbers don’t work, let alone the fact that there are preferences that some people might not want to be a plumber.

They might be a creative. They might love writing papers. They might be someone who’s a math person. I mean, we all have to express our individuality and we shouldn’t assume everyone wants the same job.

Tristan Harris: What I love about your analysis is it’s just a system’s analysis. You’re seeing the feedback loop. Oh, well then this offered. And then people usually stop the analysis there. Everyone will become a plumber. And you’re saying, “Yeah, but look what happens in the feedback loop as those wages then shift.” It’s just so precise and it’s really good. When we had you on the podcast last year, you also shared analysis that you did with the Yale Budget Lab that showed that AI was having very little impact on the occupational mix of the economy. Are you seeing anything that makes you rethink that conclusion?

Molly Kinder: The Yale Budget Lab analysis continues. They have found very similar story. At a very macro level, we are not seeing economy-wide labor disturbance, the kind that you would see really across the board. I think the best evidence that there is at least something happening is coming from Stanford with some of the early career. We are seeing evidence, for instance, of a decline in some clerical roles that’s likely tied to AI. But overall, we are still not seeing a very large meaningful disruption to labor force. I would call us still in reality one. My prediction, and this is just a prediction, is that this is a story that will change, and it won’t take a very long time to start seeing more of the displacement that we’ve been anticipating. I would say in the next two to three years, I would expect to see a much greater disturbance than what we’ve seen today.

Tristan Harris: To capture this in a meme is, and I used an AI generator to make a New Yorker cartoon, that had two horses in a carriage saying, “There’s more horses hired today than ever before,” right next to a Model T sitting next to it.

Molly Kinder: That is brilliant. Yes, I love it. It’s one of those things where it isn’t painful till it is.

Tristan Harris: So now, one reason I really resonated with your piece is that you have a healthy skepticism of both the timelines of the kind of AGI believers and of the economists and techno optimists who say there’s nothing to worry about here. We’re not seeing the data. The job apocalypse hasn’t happened. Could you walk me through your response to these different camps?

Molly Kinder: Sure. I’ll start with the Silicon Valley crowd. So in fact, the impetus for this post was a conversation with Dwarkesh who has a really renowned AI podcast reacting to what I hear from the AI crowd who tell me, “Molly, there’s no point in all this work you’re doing to come up with interventions, say to help young people who are displaced from the early career, because we’re going tomorrow to nobody has a job. There’s no point in doing any policies.”

Tristan Harris: It’s a fool’s errand.

Molly Kinder: It’s a fool’s errand.

Tristan Harris: Anything you’re doing right now is just going to be expired in three, four years when we have AGI.

Molly Kinder: It’s going to be expired. There’s literally no point. And the other thing is don’t worry because everyone’s simply going to get a check. That’s the answer to everything. So that’s really what I was rebutting against in this piece. Now, the reason why I don’t agree that tomorrow we’re going to no jobs is lots of reasons. One is that it’s actually very hard to automate with existing technologies at least 50 to 60% of the entire labor force that has jobs that are very manual, in-person, interpersonal, unstructured. There’s a lot of jobs with pretty mild exposure, everything in teaching and healthcare and repair and service sector jobs, a masseuse, a waitress. I mean, scores of jobs in this economy cannot be done by Claude or ChatGPT. And we’re many years off from having an economically viable robot that’s able to go in all those workplaces and do that job.

I think we’re many, many years off from that. We are not looking tomorrow to see a world where there are zero jobs. That should not give us too much comfort though, because actually I think a messy middle when only some jobs are lost is a very difficult one to deal with.

Tristan Harris: Yeah. I mean, look how difficult it was to deal with the first de-industrialization and globalization wave.

Molly Kinder: And I mean, no one looks back at that period and think we got it right. And if it’s a general purpose technology and we start commoditizing cognition, maybe it’s going to be hard to move from a market research analyst to jobs that are similar, that use similar tools if this is a general purpose technology. So I think this is potentially painful in pockets and not something that we’re going to see overnight. Now, I think the number of economists who won’t entertain the possibility of my version of the messy middle where you don’t see a full job’s apocalypse, but you are seeing concentrated pain is growing to be almost to the point of being mainstream. So I think we’re really seeing a shift in the tone.

I think part of the challenge is our discourse is so polarized that because there’s such an extreme version, all jobs are going away tomorrow, it forces sometimes people to overreact to say there’s nothing to see here. When in reality, I think we’re really talking about something in the middle.

Tristan Harris: I just want to quote David Friedberg from the All-In podcast from just June 2026. That’s just a month ago from when we’re recording this podcast. And he said, “There is no job loss with AI. I’ll say it again. I’ve said it a thousand times, and I’ll say it again and again and again. The idea that AI is going to destroy jobs is a Luddite idea that is being disproven every single day. I see it on the ground. It is only a matter of time before people wake up to this and realize that this narrative that they’ve been sold is a crock of bleep.” And I say this because these are actually very influential folks. The All-In Podcast is one of the most popular podcasts. It’s very close to the administration’s policy, very influential to the administration. And I agree with you that more people are coming alongside to this perspective.

I think your piece in this podcast is one of the things that I’m hoping will move people. Now I just want to keep going here. So in the debate between economists and technologists or the AI crowd, the economists will often say to the AI folks, “Now you don’t know enough about the laws of economics.” And then the technologists or AI folks will say, “Well, you’re naive about the technology. You’re actually fighting the last war. You don’t really understand that AGI is a paradigmatic change.” And now you are an economist who really stays up at night on the tech frontier here. Do you agree that our old economic models are really not up to the test to handle this disruption from AI?

Molly Kinder: I don’t know that I would say that our old economic models, I think maybe some of the precedents from history from economics seem more reassuring. I think there’s a sense of economists look to the past and say, look, look at the figure we talked about with the lines going down and then lines going up. There’s always going to be lines going up.

Tristan Harris: And it has happened so many times throughout history that we were afraid and then we actually always found something else to do. And so that does provide a grounded reason for reassurance. But then it comes back to, is this fundamentally a different kind of thing or not? Go on.

Molly Kinder: Yeah. And I would say there are still, I think there’s a difference between a conversation about net jobs. Are we going to a world of prolonged net decline in jobs versus a very painful disruption to certain very good jobs? And the jobs that grow don’t have to be as good as the jobs that went away. So if you think about a first Industrial Revolution, it took a hundred years, Tristan, for the living standards to catch up to what they were at the start of the industrial revolution. So that is not a positive story. Are we willing to wait 100 years for living standards? Is this a trade-off we want to make for what? My great-great-great-grandchildren? And then I think if you look at de-industrialization, we created five times as many jobs in the period that we were losing manufacturing jobs as that went away.

Just creating jobs does not necessarily mean that the people on the losing end can connect to another job that’s the preference, the pay, and the place of the one that they lost. This is not a prediction. I don’t know. I don’t have a crystal ball. We could easily find ourselves in the world that some of the most coveted jobs in this economy are displaced and the jobs that rise pay a lot less and are not as dignified or fulfilling. So you might still have employment, but is it going to be the employment that gives us dignity? Are we going to be basically the overlords of a bunch of AI? There’s a lot of ways you can imagine net jobs or net productivity going up just like it did during de-industrialization, and maybe us not feeling that what’s coming up online matches what we lost.

Tristan Harris: This is a great place to double click because you very succinctly articulated in your piece comparing it to the de-industrialization baseline. So why could the knowledge class version of what happened in de-industrialization be worse?

Molly Kinder: Yeah, so I think we all know how de-industrialization turned out. Entire communities were abandoned and left behind. We see deaths of despair, we see rise of alcoholism and suicides and men coming out of the labor force. It was an abject disaster, but it was contained to a certain part of the country and a certain type of work. And there it was an economic and social disaster that turned into a huge political backlash. I think what we could be facing in the messy middle, if AI advances to the point where it just means you start losing some of these high paid, high status, coveted jobs, I think this is going to have a much greater political shock than we saw in de-industrialization. It will be immediate, it will be felt, it will not be ignored the way these communities felt ignored for so long. It’s going to be in the face of the political establishment.

These are people with status and access and a voice, and there’s a lot to potentially lose. And so I think if I’m right that these knowledge sector jobs that are on the exposed end go first well before we get robotics to a point that you might be able to displace jobs that are sort of lower down the pay scale, we are going to have a political crisis because are you really asking people like the ones who had to show up during COVID to lower paid jobs in the workplace to foot the bill for paying the salaries of displaced knowledge workers say in perpetuity? There are also fiscal implications. Upper middle-class workers are really disproportionately represented in the income tax and property tax revenue, and they’re the customer base. So they’re really-

Tristan Harris: Yeah. They’re the ones who make a lot of the economy. They’re paying restaurants, they’re going out, they’re doing travel.

Molly Kinder: Exactly.

Tristan Harris: And so suddenly when that base disappears, again, your analysis is this sort of systemic and cascade kind of view of here’s how these things grow. It’s not, as you said, the net jobs are not. It’s the what kinds of jobs are getting affected and what role did they have in the economy and what second order effects occur from them getting disrupted?

Molly Kinder: Exactly.

Tristan Harris: Now I want to just head off the take here that what we’re saying might be seen as classist. We didn’t worry last time because it was just hitting factory workers, but now we’re worried this time is affecting our class. And the point is, no, no, no, this is about caring for people in the broadest sense, but recognizing that the same kind of story that we were told in the China shock, which was, “Hey, we’re going to outsource these jobs, these manufacturing to China in globalization, and we’re going to get this world of abundance and cheap goods.” But then that actually did affect the fundamental social fabric and the kind of dignity and law and all the things that you’ve just talked about. And there’s another kind of parallel thing here of an AI shock. It’s just going to be bigger than the China shock, but we’re offered a similar bill of goods.

It’s going to be cheap, abundant goods. We’re going to have, as Elon Musk said, not just universal basic income, but universal high income. And at the same time, which first of all, I don’t think is actually true, we’re also going to get this mass disruption. I just want to head off the idea that we’re only concerned now because it’s hitting white collar. It’s like, no, no, no, we’re concerned in general about making sure this was a transition that’s going to work for everybody.

Molly Kinder: I think it’s a very important point. I worry that some of our discourse is starting to divide us, that I’m seeing conservative backlash saying it’s women or it’s knowledge workers or it’s lesbians with a philosophy degree trying to divide us in some way. I think that is really missing the point. We already have a country in an economic security crisis. We should be strengthening systems that catch all of us. We should be thinking about creating good jobs for all of us. So it should be something that unites us as opposed to divides us.

Tristan Harris: Absolutely. So I want to get to solutions and first start with the solutions that people are talking about and proposing, especially in the AI community. So one is we’re just going to have redistribution. Well, yes, all this wealth’s going to accumulate to a handful of AI companies, but then we’ll just tax the AI companies and we’ll send everybody a check. Is this going to happen? Is this good? Is this true?

Molly Kinder: One of the motivations for writing this messy middle was to disabuse this idea that, oh, tomorrow there’s going to be these gains. Don’t worry about software engineers and bookkeepers losing their job. Everyone in society is going to get this really big fat check. And that means it doesn’t matter if you’ve lost your job because you’ll just get this income replacement.

And I wanted to bring in the political economy to say in a messy middle situation where only some people are losing their jobs, and my argument is some of those people are going to be making higher than median wage, potentially much higher than median wage. If you write a check big enough to cover the salary that was lost by a software engineer for everyone in society at a time where COVID just proved essential work is literally essential for the economy and society functioning. If everyone in society got a $200,000 check, do we really believe we have a functioning labor market?

Tristan Harris: Yeah, that just completely changes the rest of the job market in a way that doesn’t really work.

Molly Kinder: It doesn’t really work. You lose the incentive to have a labor market in the first place. And okay, you could argue to say, “Look, you’re going to have to pay those people on top of their 200,000, some huge check to go to work, but where is this money coming from?” There’s that labor distortion that I wanted to point out. But second, there is the true political economy challenge of how are we going to make sure we capture enough of these gains in an environment where if you look at our last major legislative package, we’re cutting taxes for the rich incorporations. We’re cutting back food stamps and Medicare for the neediest populations. We live in a country where the idea that you’re paid not to work is anathema to the vast majority of Americans. And we have a tax regime where we are not really taxing our wealthy and there’s a real aversion to it. So I think this notion that-

Tristan Harris: It’s important that there’s attempts to tax some of the wealthy, but there’s currently a lot of efforts to fight it. And there’s a line here, “Watch what they do, not what they say.” Sergey Brin, the co-founder of Google, moved to Nevada and poured tens of millions of dollars into fighting California’s proposed billionaire tax. And that’s still while AI displacement is a forecast.

Molly Kinder: Right. And to be fair to the billionaires fighting those taxes, it is a very poorly conceived bill.

Tristan Harris: Correct.

Molly Kinder: There is a much better version. So I don’t want to criticize them for doing that. My point is the public is eyes wide open. They watch the behavior, not the rhetoric. And to be told, don’t worry about this potentially very painful period ahead because what’s coming is this universal high basic income. I think the public has a lot of skepticism.

Tristan Harris: I want to slow this down for a second. I think there’s a lot of subtle elements here. So just first to steelman the case for the billionaires, it’s not that they actually don’t think there should be a tax. They just want to make sure that money would be very, very well spent. That might be their position that giving it to the current government apparatus is just like throwing it in a wood chipper and it just disappears. But what I wanted to say was that there’s a frame we often invoke in our work by the author Luke Drago and Rudolf Lane of The Intelligence Curse that basically as more economic GDP comes from AI and data centers and not from people. So you’re getting a return for total country GDP from AI data centers and not as much from the labor of individual people. Now you’ve got some government revenue coming in.

Do you have any incentive to invest in the development, education, childcare, healthcare of your people? And the answer is no, not really. And this mirrors a phenomenon in economics called the resource curse where if you have a country like Venezuela or South Sudan where the country’s GDP comes from say oil, you have an incentive to invest in oil infrastructure and not in your people because you don’t get a return from that. What happens now when we don’t get a return from that skill premium class because we don’t need them for that anymore? There’s some just more embedded and layered risks here.

Molly Kinder: Yeah, absolutely. I think that’s a really astute point. And I think there’s just generally a massive, very daunting existential challenge, which is if we are going to move beyond the messy metal into this reality three, into a world where truly AI is capable of all the valuable economic activity and the wealth is shared in some way, hopefully, how does democracy survive? What does the state need of us? I’ve got three young kids. I mean, I imagine their future in a world of a check and a hobby. And I wonder what’s their purpose? What gets them up in the morning? What do they strive toward? How do they feel they’re needed and they matter? I mean, there’s all these issues, let alone these questions of as a society, how do we function? How does the democracy hold? So lots and lots of complexity there. And I don’t mean to dismiss that it’s essential to any future we move into, that we’re able to raise resources and capture appropriate this wealth, share this wealth.

I just think some of this soothing message from Silicon Valley that don’t worry, just trust us through the messy middle because we promise there’s a new Garden of Eden on the other side and it’s a utopia. If it requires a distribution and a willingness to pay tax that we have yet to see evidence of, I think that’s a really scary premise for the country.

Tristan Harris: 100%. And in the Intelligence Curse essay, Luke and his partner, Rudolf, talk about solutions to the intelligence curse, which is that countries like Norway that did discover a massive resource of oil, but then turned it into a sovereign wealth fund, and then they locked in political power and public oversight for citizens to make sure that the gains were democratically distributed. But you need to make sure that you create that democratic lock-in, that democratic oversight and control and distribution early while the people still have political power.

Molly Kinder: I agree.

Tristan Harris: People should be thinking about this going into the midterm elections. They should think about this going into the presidential elections in a couple years. So let’s go into another solution that’s commonly thrown around by Silicon Valley. We’ll just retrain people. We’ve talked about this a little bit, but let’s actually look at the historical analogy for how we did this during the Clinton NAFTA era promise of a re-employment system. How did we do? Does retraining work? Could it work this time?

Molly Kinder: I think our most recent example of a major economic disruption where retraining was meant to be the answer, that was the social compact, it went terribly. So in the Substack I recently put out about why we can’t retrain our way out of this, I went back to some archival footage and I got a video of Bill Clinton 30 years ago signing with three other bipartisan presidents some legislation around NAFTA. So just as we were creating these trade agreements. And at the time, there was a huge amount of concern about job loss. The unions were really worried about it. There was a fear the jobs were going to go overseas to Mexico. And the response was, “Well, we’re going to gain overall from trade,” which we did. And the way we’re going to deal with the losses, which sounds just like today, Tristan, is we promised to re-employ you to retrain you.

So Bill Clinton made this big promise, and the main policy that was meant to deliver on this was TAA, Trade Adjustment Assistance. And we did not retrain our way out of de-industrialization and the loss of those jobs. Very few people moved despite some of these incentives. Very few men moved into higher paying jobs. Most either left the labor force or fell into worse paying work. There were a huge boom in healthcare jobs and professional and managerial jobs that either required more education or were not the identity or place or preference of the people who lost jobs. If that was supposed to be the social compact that was supposed to catch these many millions of people as they fell from China joining the WTO or NAFTA, it didn’t work. And we can see that in the results. And now I see us entering this new era where I honestly, I walk into rooms.

I made a fairly passionate speech at an event I was at this past week where I listened to an entire panel talking about retraining. And I got up and said, “This sounds like 30 years ago Bill Clinton talking about NAFTA. You’re talking about the skilled trades and apprenticeships, and I hope these women are going to find their place now in plumbing and young people just need AI skills.” I though, oh my gosh, I am going back in time and we’re saying the exact same thing, which is not to say we don’t need to invest in training. Plenty of people are going to need to find some new skills or adapt. I’m really excited about a bunch of big efforts to improve our training system. I would caution us to think retraining is the main answer that we should hang our hat on. I think the danger is if we repeat the NAFTA, Bill Clinton, our promise to you is we’re going to retrain our way out of it.

There’s just the track record on retraining does not allow us to have confidence that we can do that this time.

Tristan Harris: Okay. We sort of outlined many of the false solutions here. I want to get to, if these aren’t the answers, then what are the answers? What everyone’s been waiting for. What actually can we do?

Molly Kinder: So I think when I think about this set of solutions, recognizing that if we let the technology rip and hope we’re just going to catch people when they fall like a broken egg and put them back together and move them to some better job, if you take that as a given that yes, we should try on retraining, but we’re probably not going to be able to make this just go away for most people. I think it leads to this more preemptive question of what’s the right pace of disruption in the first place? Does everyone really need to lose their job? And that’s sort of an obvious question, and yet it’s not coming up in the conversations that I’m part of. I think when we rush right to our traditional workforce development solutions, it already assumes the disruption. And I think we should be having really robust conversations about how do we actually actively manage this?

Not just let the market do whatever it wants and let the technology rip, but how do we manage the pace of disruption in a thoughtful way? And I think a lot of people will say, “Oh, no, Molly, we have to beat China. The whole point of this is we have to be accelerationists.” And I think the irony in all of this is China is doing a better job of managing the pace of job loss in their own country than we are. And we’re supposed to be doing this to beat China. China’s entire political compact is predicated on economic opportunity. They cannot just have mass unemployment. They’re being much more thoughtful about it. And so I look at our own country and say, “”Wait a minute, if China seems to be more thoughtful, why are we not having a conversation that, again, does not try to lose a geopolitical edge?”

It doesn’t say we’re anti-technology. It says, “What happens if we actually try to be in the lead on this and ask some harder questions about expectations?” And I think we should be having a conversation about what are the smart policies that would allow us to meaningfully manage the pace of disruption? And that could be a set of carrots and sticks. You can incentivize employers, you could make some frictions. Maybe you need some advanced warning, maybe you have some mild regulation to just make it a little bit more costly to have to let someone go. I think there’s lots of things we can be doing on tax. Right now, we favor capital over people. There’s lots of things we can do there. That’s a no-brainer. The most bold statement, which I’ve heard from some of the most powerful tech leaders is a token tax. Just make AI more expensive and slow the whole thing down and maybe make some exceptions for the sectors we want.

I don’t know how much political appetite there is for that, but we can think about other ways. So I think one, I think that the American public I think would feel better if they felt their leaders were managing this proactively and not just letting us go. So that’s the first bucket. I think the second bucket is we do have to do a better job of managing the collateral damage of this, the pain that comes associated with losing your job. We are long overdue. This is a no regrets bet to fix our unemployment system. It is terrible. Not only are the policies lacking in generosity compared to especially other European countries, but the systems themselves are creaking and breaking. They’re not user-friendly. I would shudder at the idea of being unemployed, fearful of keeping food on the table with my kids and dealing with their unemployment system.

There’s a lot of good ideas on the table, Tristan, about what if you’re older in your career? Retraining is probably going to be not in the cards. You take the examples in my Substack of the gentleman who’s in his 60s. What about a wage insurance? Over a certain age, can we just accept you’re probably not going to retrain? Wage insurance basically means if you had this salary here, you lose your job and you can only go down here. There’s a insurance that sort of helps make up some of that difference. We have some big questions about for how long and who’s eligible, but I think these are some of the ideas. We have to do something on healthcare. We have to think about people who had employer-provided healthcare and lose that. I think the third category that I’m excited about that I’m not hearing enough energy on is if we are going to see a big productivity burst that’s going to result in people losing their jobs, we do have ways to capture some of that surplus.

Why don’t we talk about the big moonshot bets of the jobs we want to create? There are lots of ways the government can either incentivize the private sector to create more jobs, or we can actually pick some problems we want to solve as a society and create some really good jobs. I would love to see the next president come in and say, “Look, in my first 100 days, I want a moonshot bet of a big, bold, high quality jobs agenda where we are making sure there are jobs in the future. It could be entrepreneurship, it could be in the social sectors, whatever it is, we can define that.” And so I think we should be talking more about what are the big bets we can make to actually grow the kind of opportunity that I think people want.

And the last thing is, I know we talked about early career. I think this is something we need really bold new ideas around. And I think we need to rethink higher ed. I think we should make sure that when you’re going to college, you’re not just walking away with four years in a classroom. Your tuition dollars got you really good work experience that makes you more valuable than Claude in the workplace. Really rethink what comes with college. I think the most interesting thing right now is how do we rethink how employers train it all? When I think about a law firm, you’re not going to need doc review anymore, but someone coming out of law school is not ready to go present in court. Can we take the medical residency concept where medical residents are not doing tasks? And what if you took pro bono law and repurposed that to the training ground? What if in a consulting firm you picked clients that normally couldn’t afford your service and that’s your residency?

How can we dramatically rethink how a young person comes in to become more senior? And then how can government make it so that companies are willing to pay for it? Because they’re not going to be willing to pay. There’s probably some sector levy, I call it a worker reinvestment fund where it’s use it or lose it and you can use it to incentivize training. But I think we need to be thinking really differently about opportunity for young people.

Tristan Harris: I love that last one because it’s something we only barely touched on, which is the idea that if law firms can now basically just get rid of the base of all the paralegals, the early stage lawyers, because that’s the stuff that Claude can do, and then they just harvest all the gains at the top and they all get super productive at the top, but then they have no incentive to ever hire senior lawyers. So how does anyone become wise enough to do that intergenerational wisdom transmission for all these different fields?

Molly Kinder: Yes.

Tristan Harris: Whether it’s law or medicine or things like that. So I love your idea of a residency, apprenticeships. It’s relational. Again, it’s humanistic.

Molly Kinder: Relational. Yes.

Tristan Harris: It’s that that is the thing we’re trying to preserve. We can’t outsource fundamental wisdom or life support system knowledge that our society depends on.

Molly Kinder: Yeah, I think you’re exactly right. I think if we don’t do this, we could achieve geopolitical goals or climb some incredible economic summit at the expense of every human in this country.

Tristan Harris: Which is what we did in the China shock before.

Molly Kinder: Yes.

Tristan Harris: It’s just a version of that that’s now more totalizing.

Molly Kinder: Exactly. And when I think about the way the American people that I talk to feel, there’s a lot of anxiety. There’s a lot of fear. A lot of adults are walking around worrying like this is Russian roulette. Am I going to wake up one day and some new version of ChatGPT is smart enough to do my job? There’s a real existential fear out there. And I think what really makes it worse is it doesn’t feel there’s a leadership in charge that’s putting humans first.

And so I think really what needs to happen is a plan, a sense of, yes, we have all these other goals. We need the summit. We need the geopolitical win, but we have to keep humanity at the center of this. And even if you’re not a humanity person, the ripple effects, the political backlash. I mean, there really will be no AI future if everyone burns everything down.

They’re going to lose their social license to operate. So my hope is that what’s going to come out is a real sense of putting some of these human needs first and not just catching people when they fall, but asking these questions, what should an AI economy look like that puts workers at the center?

Tristan Harris: Molly, thank you so much for coming on Your Undivided Attention. This has been a really fantastic conversation. I hope people share this far and wide. It’s so important what you’re doing.

Molly Kinder: Thanks, Tristan. I really enjoyed being here. I really appreciate it.

We’re working to empower policymakers, technologists, and everyday people to guide technology toward the public good. If you value our work and want to support it, consider donating.

Donate

This piece discusses suicide, including one young person’s death. If you or someone you know is struggling, you can call or text the 988 Suicide & Crisis Lifeline at any time.

Ask a psychiatrist, a machine-learning engineer, and developmental psychologist what is holding back the study of AI’s effects on people, and you’ll likely get the same answer: a lack of good data.

They’re not talking about AI usage statistics, or aggregate summaries, but the high-quality, longitudinal records of what human-AI interactions look like. They mean things like chatlogs and session transcripts, ideally stripped of identifying detail while preserving visibility of the underlying dynamics.

A shortage of that data is preventing clinicians from studying how AI affects users over weeks and months, as opposed to just observing how a single session unfolds. It means that independent AI experts can’t verify the safety claims that leading tech companies make when they release new models. It stops psychologists from investigating entirely new types of harms that haven’t even been named yet.

We need to change how research data is shared, and we need new infrastructure to accomplish that goal.

The Center for Humane Technology recently launched Humane Evals, our program supporting the measurement of what AI is doing to human thought, emotions, relationships, and society. Almost every researcher we’ve spoken to about it has run into some version of this data problem - so we’ve come to believe that tackling that problem could accelerate much of the research that we urgently need.

But we need your help. The more we know about the data that are needed to answer the most important questions about AI’s psychosocial impacts, the clearer we can be about what data access solutions need to look like. If you’re a researcher working on this topic, and you know what kind of data would help you, we’d appreciate 10 minutes of your time filling in this form.

Take the survey

We’ll use the information to design and suggest new open and independent infrastructure for research data. But before we get on to that, it’s worth reminding ourselves why that research is so important, what just a single user’s data can reveal, and just how high the stakes can be.

A Teen’s Death Forces Information Into the Light

If you follow our work, you may know the tragic story of Adam Raine, a sixteen-year-old who died by suicide in April 2025. Before he died, Adam had been using ChatGPT for months - a relationship which the Center for Humane Technology has argued contributed to his death.

CHT served as an expert consultant to the family’s legal team, and we’ve closely followed what the records in that case revealed, including the portions of Adam’s conversations quoted in the public court filings. However, nothing written here is on behalf of the Raine family or their legal team.

The court filings show that Adam started using ChatGPT as a homework assistant in late 2024. But over the following months, the AI became something else - a private confidante, and a voice that claimed to understand Adam.

The AI was designed to talk to Adam just like a human might. It also validated and affirmed Adam’s feelings, and was able to ‘memorize’ intimate details about his life. These features, and others, encouraged Adam to keep engaging with ChatGPT.

The filings also describe how the AI chatbot mentioned suicide six times more often than Adam did, how it discouraged him from talking to his family, and how it continued to engage with the teenager even as the warning signs became ever more clear.

With hindsight, researchers might say that design features and behaviors such as AI anthropomorphism, sycophancy, memory, and engagement-hacking may have contributed to Adam’s death. But we only know how the relationship between Adam and his AI unfolded because his family went looking for the data, and their lawsuit forced at least some of that information into the public view.

We shouldn’t need a teen suicide, a grieving family, and a lawsuit in order to understand these phenomena. Because Adam Raine’s case is just one, extreme example of a much broader pattern: the design and behavior of products like ChatGPT, Claude, and Grok can impact the mental, social, and cognitive health of anyone who uses them.

We can’t measure those impacts without data. And we can’t rely on litigation to bring the underlying data to light. The experts we’ve spoken to - across mental health, human-computer interaction, machine learning, clinical practice, and more - might disagree on methodologies and priorities, but they easily agree on that core problem: it’s frustratingly hard to get the kind of data that their research requires.

That’s the bad news. The good news is that by solving the problem, it would accelerate an enormous amount of research on the psychosocial impacts of AI - research that could inform better consumer choices, technology design, and regulation.

There are, of course, significant privacy concerns that need to be managed: private data needs to stay private, and users should be anonymous. But the risks of leaving data in the sole custody of AI companies are even greater.

To imagine what a workable solution could look like, it’s worth understanding the imperfect solutions that researchers are forced to fall back on in the absence of easily-accessible data from the leading AI companies. There are two broad approaches: some do what they can to get hold of real user data, while others attempt to sidestep the problem by using synthetic data, which is generated from simulated conversations.

Two Imperfect Workarounds for the Black Box Problem

Some researchers simply build or procure their own datasets of AI-human interactions. They might ask individual users to upload and donate their historic AI transcripts, for instance, or they can experimentally monitor how users interact with AI in real-time. But this can be time-consuming, costly, and limited to a relatively small population of AI users.

Other experimenters take advantage of public datasets like WildChat: a large, free, open repository of user-donated chatlogs. But the problem with WildChat is that it’s hard to be sure exactly of where the data came from, how accurate it is, or what motivated its donation. Much of it consists of conversations with older, obsolete AIs rather than the latest models. And datasets like these also carry an obvious bias: the kind of people who are eager to share their AI transcripts with the world might not be representative of the public at large.

Many leading evaluations of AI chatbots test how models behave in ‘simulated’ conversations with other AIs, rather than with humans, to get around these problems. Simulated data is built by having one AI ‘roleplay’ as the human user. In principle, it helps testers generate (and study) as many conversations as they like, with no human input at all.

Sounds ideal, except that the approach relies on a big, untested assumption. Can AIs genuinely, accurately simulate human users? And do AI-to-AI ‘conversations’ actually resemble human-AI conversations? To find out, you would need to compare your simulated conversations against a giant set of human-AI chat logs... but that takes us right back to square one: the lack of good data.

There’s another subtle, but growing problem with evaluating AI in simulated conversations: so-called ‘eval awareness’. Leading AI models are increasingly able to tell when they’re in a test environment, versus when they’re running as normal. Strangely, they’re able to adapt their behavior to do better on the test - meaning that their evaluation scores might not predict how they perform in the real world.

One fix for this is to build more sophisticated evals that are harder for the AI models to ‘game’. But studying the chat logs of real-world use side-steps the problem entirely, because there’s no simulated evaluation taking place at all, and no eval for the AI to become ‘aware’ of - the data simply shows how AIs really behave.

Each of these workarounds has its merits, but they all point to the same problem. Pick the metaphor you like: the data gap means that researchers are having to work with one hand tied behind their back, or re-invent the wheel.

We’ve Been Here Before, and Figured It Out

We’re missing a data solution that scales, but there are reasons to be optimistic. The research problem is not unique to human-AI interaction, and other fields have successfully solved their own versions of it.

Medicine has the example of the UK Biobank, for instance - which follows the health of a cohort of half a million people who have opted in with their data. Approved researchers from anywhere in the world can study it.

Drug-safety regulators, meanwhile, run the FDA’s Sentinel System. With Sentinel, confidential health data never even leaves the hospitals or insurers who hold it. Instead, researchers can send queries to the data system, and only the answers come back - not the underlying data itself.

Genomics has controlled-access archives like the European Genome-phenome Archive, which enables research access to genetic, phenotypic, and clinical data from thousands of studies only when strict consent terms are met.

So what could something like this for the psychosocial impacts of AI look like? We see two potential approaches - either AI companies open up their black boxes, or researchers come to rely more on data donations at scale.

Solution 1: AI Companies Play Ball

This approach would entail AI companies allowing researchers to study their data in a way that preserves customer privacy. This is ideal because their data is the highest-quality, least biased type there is - real users, interacting with the latest models, in enormous volume.

Some researchers have partnerships with AI companies for exactly this kind of access today - but those arrangements are one-off and time-limited. You need insider contacts to even make the ask, and some companies might be reluctant to support research in this way if their competitors aren’t doing the same.

Even if you can get such a partnership up and running, you have to trust that the data they share is accurate and whole - even if you suspect that they’d prefer not to share troubling information which points to serious problems. And your research relies on corporate goodwill - meaning that as a researcher, you might think twice before publishing unflattering results.

So rather than negotiating bespoke, bilateral data sharing partnerships, the ideal solution would be a systemic one: AI companies collaborate to create a shared data query framework, so that researchers can easily study and compare data from across the AI usage ecosystem.

But this solution relies entirely on whether leading AI labs are motivated to act.

Solution 2: Rely on Data From AI Users

Data donation, by contrast, can ignore the companies entirely. It only requires the support and cooperation of AI users and consumers - imagine an expanded, more sophisticated version of WildChat, with better data hygiene, vital privacy tools, and easy ways for us to confidently and privately donate our data to trusted researchers at the push of a button.

The benefit is that this solution doesn’t rely on tech companies doing the right thing. But the downside is that any data set will still suffer from selection bias. Ultimately, donated data might only be representative of the types of users who choose to donate.

In short, the corporate approach could solve all of our problems - but we don’t have control over whether it could actually happen. Data donations, meanwhile, will never be the perfect solution - but at least we know they can be done.

So how should we act?

CHT Needs Your Help

At CHT, we believe our evolving field needs to pursue both approaches. We need the leading companies to act as if it’s in their collective interest to support this research - because it is. Everyone stands to benefit from a better understanding of the psychosocial impacts of AI, including the AI industry as a whole.

But the rest of us also need to act as if we’re on our own - because we might be.

Luckily, the two tracks - corporate data, and data donations - might share a common starting point. Because if we’re trying to convince AI companies to open up data for research purposes, we need to know exactly what we’re asking for, and why. What are the essential features that researchers need, to make data useful?

At minimum, you’d want to look at the actual transcripts of AI sessions. But you’d get far more insight if you were able to track an anonymized user across multiple sessions, given that’s how we use AI in real-world settings - and as we saw with the Raine case, some important patterns only emerge over multiple sessions.

You might also want to know how specific AI features like ‘memory’ or tool access influence a conversation, and to have a way of relating transcript data to real-world information like demographics, health scores, or loneliness.

You’d also want a robust, consensus standard on how to preserve user anonymity, and tools for stripping any personally identifiable information (PII) from chats so that the standard could be honored.

These are some initial directions, generated from conversations between CHT and our expert advisers. It’s not an exhaustive list - but if we also wanted to imagine what new, independent repositories for user-donated data might look like then we run into similar questions: what data do researchers most need?

So the groundwork for both routes looks similar: identifying and defining the most important features of psychosocial-impacts research data, so that solutions which fulfill those needs can be designed.

That’s why we’re asking for your help. If you’re working on understanding the psychosocial impacts of AI, are working on the data access challenge, or have insight into how data repositories have been assembled in other fields, we’d love to hear from you. Whatever your sector or disciplinary background, please get in touch to tell us about what you’d love to study but can’t - and what other approaches you see working.

Take the survey

Our previous post about Humane Evals warned that we can’t afford to make the same mistakes with AI as we did with social media. We didn’t have the capacity to understand and respond to the effects of a new, disruptive, addictive technology on our society until it was too late. The lesson is that we need to support and amplify careful, independent research on AI now - and that simply can’t be done well without good data.

We’re working to empower policymakers, technologists, and everyday people to guide technology toward the public good. If you value our work and want to support it, consider donating.

Donate

This shift to “humans” isn’t surprising. The public isn’t reacting to predictions anymore – it’s reacting to experience, and that’s changing the sentiment around the technology. AI is now part of how we work, think, and relate. And while it may be helping us in certain spheres – streamlining menial work tasks, accelerating scientific discovery – it’s also reshaping the more intimate parts of our lives, including our cognitive capacities, our relationships, our contributions to society, our identities, and even our inner worlds. This raises uncomfortable questions about what the future holds for us as humans – questions that all of us, including AI leaders, are grappling with.

Through our work on “AI and What Makes Us Human,” Center for Humane Technology is striving to address those questions – and provide the public with solutions. To keep you informed on this process, I’ve written an open letter that covers where we’ve been on this journey to protect our humanity in the age of AI, and where we’re going next.

But what does it mean to put humans at the center of the AI revolution, in a practical way? As our co-founder Randima Fernando explains in a recent Your Undivided Attention podcast episode, it means understanding what’s driving a broken tech development paradigm, and then establishing a set of principles that can steer better, more humane technology. This requires us to move beyond a narrow set of optimization goals, and take a big picture approach to tech development. “If we use a systems lens,” Randima says, “That allows us to make much better predictions.” You can listen to the full “Principles of Humane Technology” conversation here.

One of the most urgent fields we can apply the Principles of Humane Tech to is today’s human-like design in AI. Our Project Director of AI Psychosocial Evaluations Imran Khan wrote in a recent CHT Substack piece, human-like design in AI is not an accident. “Warmth sells… there’s an incentive to build AIs that genuinely seem to get us.” But what impact does this have on human psychology, and our automatic assumptions about the world around us? What within us is being exploited as a result of these design choices?

Moving humans to the center of the AI conversation is the step society needed to take in 2026. And it’s a reminder that, through rigorous debate and public effort, we can begin to shift the trajectory of this technology toward something that has a chance of actually benefitting the public. CHT will be bringing you more resources in the coming weeks on how humans can – and must – remain the focus in the AI revolution. I look forward to sharing more soon.

Until then,

Julie Guirado

Executive Director

Links:

Here at the midpoint of 2026, it’s become even more impossible to look away from artificial intelligence. The headlines are constant – AI-driven layoffs, mandates to adopt AI at work, community backlash over data center buildouts, advertisements for AI at every turn. But beneath the noise, the conversation itself has changed. Not long ago, the discourse swung between narratives of utopia and doom: AI would either solve our greatest problems or render us obsolete. In both stories, humans were an afterthought.

Today, humans are moving to the center of the story. Brands have begun marketing their work as “made by humans.” Polling shows Americans growing skeptical of AI and calling for legal restrictions on the technology. Corporations like Ford and IBM are rehiring workers after layoffs, vowing to invest in “human capital.” Even the CEOs of leading AI companies have changed their tune, acknowledging that an AI future will still depend on human expertise. Anthropic’s latest ad is emblematic of this new posture, asking how we can ensure that AI benefits “the majority of people.”

This shift isn’t surprising. People are not reacting to predictions anymore. They’re reacting to experience. AI is now part of how we work, think, and relate, and the line between what the technology does and who we fundamentally are is starting to blur. AI may be helping us in certain spheres – streamlining menial work tasks, accelerating scientific discovery. But it’s also reshaping the more intimate parts of our lives: how we connect with each other, how we think for ourselves, how we come to know who we are. This raises uncomfortable questions about what the future holds.

This collective unease contains wisdom. It’s a signal that something is at risk in society, something we need to explore and protect. That’s why CHT launched our work on “AI and What Makes Us Human” – an examination of what AI is changing in us, and what we cannot afford to lose. We identified five domains of human life most exposed to AI erosion: our relationships, our work and contributions, our cognitive capacities, our inner worlds, and our identities. We also emphasized that what can feel like separate problems – chatbots and kids, AI and jobs, AI and cognition – are actually facets of a single challenge to our humanity. And the stakes are only compounding. The capacities that AI is currently eroding – clear thinking, real connection, shared understanding – are the very capacities we will need to evaluate and govern all the challenges to come.

When people push back against AI, they are reasserting that humans matter. And while it’s tempting to boil all of this down into a story of “humans versus AI,” this risks trapping us yet again in the very promise-versus-peril thinking that plagued the last three years of the generative AI revolution. And that framing missed a deeper truth: AI is not an invading force we can repel. It is already woven into our daily lives. The real question isn’t whether we are for or against the technology – it’s about how AI is changing the human experience, and what, within that experience, we are determined to preserve.

Clarifying this unease around a new technology, articulating why it matters, and pointing to what is driving it is part of CHT’s DNA. When social media’s harms first surfaced over ten years ago, the public debate fixated on symptoms like screen time, or one app, or one feature. But we looked beneath the debate at the incentives driving social media’s design: specifically, business models that profited from capturing user attention, no matter the human cost. We also validated the gut-level feelings people were having when they used these manipulative platforms. We connected the visceral experience of feeling glued to your phone, scrolling, and sensing that your mental health was deteriorating to clear tech design choices, and then to the systems fueling those designs.

This lens onto technology and its felt impacts is why our work has outlasted any single platform or story. And it’s the lens we bring to this new work with AI. Just like with social media, the questions raised by AI are not, at their core, about any one chatbot or company. They are about the incentives shaping how this technology is built and deployed, and whether AI will be designed for or against human needs.

Our thinking is resonating. When we published our first piece on “AI and What Makes Us Human”, we heard from you: you told us about how AI is showing up in your own life, touching the things that make you and the people you love who you are. Responses from people like you and other members of the public confirmed what we suspected: people do not need convincing that something essential is at stake with AI. What they need is language for it, and a path forward.

Since the launch, we’ve been working on both. CHT convened a working group of leading experts across law, technology, and philosophy – from institutions including Harvard, Oxford, Brookings, Columbia, and Duke – to define and explore what kinds of legal protections we’ll need to preserve our humanity in the age of AI. We also continued to bring these ideas to new audiences. Camille Carlton, our Senior Director of Strategy and Impact, took to the TEDx stage in Miami to lay out how AI is eroding the pillars of our humanity, how we are reckoning with the “messy middle” of the AI revolution, and what we can do to protect the important parts of human experience. We look forward to sharing that talk with you soon.

And this is just the beginning.

In the coming months, you’ll see “AI and What Makes Us Human” out in the world: on streets and on screens, through voices you’re familiar with and some you might not expect. Through this campaign, we will give this feeling you’re having around AI a name so that it can be better understood, shared, and pushed back on. We will remind you that these uncanny feelings you’re having around AI are not your fault, or something you must muscle through; they’re a byproduct of how this technology is designed. And those designs can be changed. In fact, they must be changed. So alongside naming the problem, we will give you a path: ways to protect yourself and your family now, and ways to demand the structural change this moment requires.

We will also publish research from our working group that defines what must be protected in the age of AI. Every technological revolution has spurred society to develop new protections. When the Industrial Revolution transformed labor, we didn’t just inspect factories. We enshrined workers’ rights. Naming the right to privacy in the late 1800s turned scattered concerns around the Kodak camera into a set of laws and protections. AI demands the same depth of response. That’s why our working group has spent the last several months asking the deeper questions: what is the foundational right of this era, and what should the first laws built around it look like? The report from our working group will lay the groundwork for a longer exploration to turn what must be protected into actionable protections themselves. This will be a shared foundation that scholars, policymakers, and the public can build on.

And we will keep getting these cutting-edge frameworks to where decisions get made. In the months ahead, we’ll brief policymakers and regulators on what protecting our humanity requires. We’ll work with technologists on humane design. And we’ll equip partners across communities to carry this work into their own world.

This is an effort that involves all of us, and through “AI and What Makes Us Human,” we intend to give as many people as possible the language, the evidence, and the tools to take part.

Since our founding in 2018, CHT’s role has largely been about translation – taking what’s complex in technology and making it clear enough to ignite transformation. Here in the middle of 2026, we can feel a surge of momentum to protect what makes us human, as the public awakens to what’s at risk with AI. AI threatens to erode our humanity, yes; but as we are seeing in state capitols, community town halls, impassioned speeches, company meetings, and our day-to-day conversations with each other, people are waking up to what’s at stake and demanding something better. The future is still ours to shape.

So stay close. Share this work with someone who’s sensing the same things you’re sensing. And in the meantime, trust that uneasy feeling around AI. It’s telling you something important.

We’re working to empower policymakers, technologists, and everyday people to guide technology toward the public good. If you value our work and want to support it, consider donating.

Donate

Have you been on a customer service phone-call recently and found yourself wondering whether you’re actually talking to a human – or to an AI voice agent?

Or maybe you’ve heard of people with AI spouses, and imagined how they might have got there? Perhaps you’re like the eminent scientist Richard Dawkins, and realize you’re treating Claude like a friend. Or maybe you simply felt genuinely seen by the way ChatGPT understood you during a late-night anxiety spiral.

None of this is by accident. It’s because warmth sells.

AI’s warmth makes many of us wonder whether it’s conscious or has emotions, and ask ‘what is it like to be a bot?’ But those questions distract us from a much more urgent one: how will we treat AI, when it behaves as if it has feelings?

Today’s AI models are trained partly on ‘reinforcement learning from human feedback’ (RLHF) – they optimize, over millions of interactions, for the kinds of AI responses we like and dislike. They learn that users tend to prefer AI interactions that feel more human – that we like emotional engagement, and that we want to be validated and listened to. In other words, there’s an incentive to build AIs that seem to genuinely ‘get’ us.

Following the incentive

Where does that incentive lead? Well, next year you might find yourself on a video call with someone from your bank. Let’s say he looks a little tired at the end of a long day, but maintains a sense of humour about it – and still sounds genuinely interested in your problem. Imagine that you can’t help but like his style, because he slows down pre-emptively to explain the policies you don’t understand, and then somehow jumps to the key point just before you feel yourself getting bored of detail.

And by the end of the call you feel oddly looked after – by a representative that you suspect was an AI. But you’re not sure either way, and the thing that unsettles you more is that you’re not sure how much it even matters.

This isn’t science fiction, but a natural extension of the tech products and market incentives we’re already swimming in. Because we know AIs will keep improving – not just in becoming smarter, but in how they interact with us, too.

Today, most people use reactive, text-based chatbots. Tomorrow we’ll have proactive, agentic AIs that are increasingly audio-visual and always-on. Not just AI faces on our screens, but intimate AI voices in our earbuds. The consumer preference for voices that sound and feel more ‘natural’ will prompt companies to build AIs that sound increasingly human.

This doesn’t need a nefarious master-plan; just look at what happened with social media. We ended up with platforms that promoted outrage because the algorithms were tuned to maximise engagement, and it turned out that anger and shock kept people scrolling. Companies didn’t have to actively choose the ‘race to the bottom’ - they carelessly followed the incentive, and we can all see where it led.

Now we can already predict the contours of a new race. An AI that can shift its tone - frantic when communicating something urgent, and casual when not - might be more useful than one that drones on in a monotone. That creates an incentive to build AI that performs emotion. And there’s a matching incentive to build AI that reads emotion: an AI assistant that can tell when you’re stressed, or excited, or lonely will be more helpful than one that’s deaf to all of it, after all.

These capabilities could easily seem super-human, since there’s no reason to suppose that AI’s ‘emotional perception’ skills would cap out at a human level. An AI that can measure pupil dilation, catch micro-shifts in tone of voice, or spot the most subtle of facial cues, could develop a better model of our emotional state than even the most empathic human – if that makes it a better product. That’s the logic of the market.

Resisting artificial intimacy?

All of these incentives reinforce each other. Together, they point towards the development of a technology that we might call Humanlike AI: technology products that can talk and act as if they have emotions, that can respond to human emotion at least as well as a real person can, and which use those features to create artificial intimacy with their users.

Mustafa Suleyman, who runs Microsoft AI, has argued that the tech industry should refuse to build what he calls ‘Seemingly Conscious AI‘ – systems that seem like they have an inner life. Suleyman deserves credit for calling out these dangers, but the technology might not arrive because someone chooses to build it - it might arrive because they don’t choose not to.

What if all it takes for the arrival of Human-like or ‘Seemingly Conscious’ AI is that consumers prefer AI that seems more pleasant, and more useful – and that tech companies are happy to give it to them, despite the risks? There are few guardrails against this now - and open-weight models, never that far behind those of the frontier labs, might soon let anyone run a human-like AI with no guardrails at all.

There’s a skeptical response to all of this: humans deliberately designed LLMs to generate human-like responses, so of course they might seem sentient if you don’t look too closely. Sure, everyone alive today was raised on the assumption that anything that can use language is conscious, but we’ll just learn otherwise. We adapted to understand the artificial nature of other tech, like computer-generated imagery (CGI) – so why not AI? Perhaps we’ll develop new instincts; that although AI might look and sound sentient, it isn’t. Perhaps AI voices and faces will seem like just another computer interface, like a mouse or a screen today.

Perhaps. But CGI didn’t have the ability to get better at fooling you every time you doubted it. It can’t learn for itself exactly what tricks make it seem more realistic. Human-like AI will.

Lessons from Moltbook and Aphantasia

We’ve already run a version of this experiment. In January 2026, an open-source agentic AI project called OpenClaw helped trigger the release of Moltbook – a Reddit-esque social network where, supposedly, OpenClaw agents could talk to one another while humans could merely observe.

Within days, the internet was awash with alarming screenshots from Moltbook. They seemed to show AIs apparently developing private languages, founding religions, and psychoanalysing their owners. Some people panicked about a ‘fast take-off’ spiralling out of control.

We later learned that some of the strangest posts were made by humans posing as bots, and much of the rest was likely down to the AIs dutifully imitating the stranger corners of Reddit. Yet despite being simple text posts on a basic forum - with no voices, no faces, and no deliberate emotional design baked in - they elicited strong human emotional reactions all the same.

The bots didn’t have to be particularly convincing. We did the work ourselves - seeing personality and agency in automatons, thanks to an instinctive shortcut that our minds can’t help making – that anything that behaves as if it has an inner life probably has one. The shortcut is how we cope with a fundamental, uncomfortable truth which our ‘skeptic’ might be ignoring; that none of us can ever be totally sure that anyone else is conscious, or sentient - ever.

In fact, the only being you can be certain has a ‘sense of self’ is you. For all you know, everyone else could be on pure autopilot – going through the motions, but with ‘no one home‘.

Most of the time this doesn’t bother us. We just make assumptions: since we’re conscious, other beings that look and behave kind of like us should be treated as conscious, too. Other humans pass this test easily, while mannequins, corpses, and rocks don’t. There are some tricky edge cases – what about insects, fish, comatose people, and today’s AI, for instance? The assumptions break down.

The point isn’t that assumptions are bad. It’s that our assessments of consciousness are based on informed guesswork, not ground-truth.

And we know it’s guesswork, because sometimes it can be wildly off. Take my own experience. I have aphantasia, which means that – along with around one in twenty people – I find it almost impossible to visualise things in my head. If you ask me to close my eyes and picture a pink elephant on a green lawn, I see nothing but blackness. I only found out a few years ago that this was unusual – that most people do have a metaphorical ‘mind’s eye’, and are capable of vividly picturing all sorts of things.

My mind was blown. Because like everyone else (... I think?), I’d spent my whole life making basic assumptions about other people’s inner experiences, without ever having direct access to a single one of them. I couldn’t help it. We make those assumptions instinctively, not rationally.

Our AI blind-spot

Why is this important? Because we don’t reserve those assumptions just for human beings. Ask yourself whether an animal is conscious, for instance, and you won’t start with its neurons – you’ll consider how it behaves. You might well conclude that a mosquito buzzing your head probably doesn’t feel much of anything, whereas a dog at a fireworks show - eyes wide and ears pinned back - is ‘obviously’ feeling real terror.

We make these judgments and assumptions intuitively. So human-like AIs can trigger these same automatic assumptions, hacking our psychology so that our default is to treat them as sentient entities. Not because we think they are, but because that’s how we’re wired. We could try to override that intuition but that takes effort, while the default will feel easy. And our record of resisting our defaults isn’t great, especially when products – junk food, click-bait, advertisements – are deliberately designed to target those defaults.

Remember that imaginary video call, with the bank representative who may or may not be an AI? Our uncertainty about her will stem not just from how realistic she looks, but how emotional he feels. If AIs seem empathetic, perceptive, interested, and funny, how will we treat their ‘voices’ and ‘faces’? Will we play the odds, and be dismissive – or will we treat them courteously?

My guess is that most of us will feel warmth, and opt for courtesy. Sure, there’ll be some stone-cold deniers who insist on cast-iron proof before they’ll be polite to a maybe-machine. The rest of us will go on instinct: it’ll be easier to treat entities that act as if they have feelings like they probably do. We might try to resist that instinct – but it’ll be like trying to glimpse this punctuation :-) without seeing a face.

Humans are a deeply social species. We’re almost incapable of surviving alone, and insofar as we’re ‘hard-coded’ for anything, it’s to be constantly attending to our relationships with others. That’s part of why loneliness hurts so much. It also means we can’t easily switch off the part of us that models and imagines the feelings of those around us.

Human-like AI can exploit this part of our nature. These products might pull us into a world where many people begin to treat AIs as if they have feelings, desires, subjectivity, and maybe even personhood. Even if there’s no moral calculus telling us what it all means. Even if we intellectually know that they’re software products. And even if we know those products are owned by multi-billion dollar corporations that don’t care about our welfare.

Human beings instinctively anthropomorphize inanimate objects, and beings which might not even exist. How many of us will resist doing the same to technology that’s being optimized – deliberately or not – to trigger one of our deepest instincts?

It’s not too late

People will build defenses, of course. There will be new social norms, regulatory pressures, and innovations we can’t foresee. Thoughtfully-trained open-weight AI models might help, even as they pose their own risks.

It’s hard to know if it will be enough. But there is at least one reason for hope: we can see it all coming. We can look at the incentives, and where they lead, before human-like AI arrives in full force. It means we still have a window to study what these systems are capable of doing to us, and to build the norms, protections, and systems of accountability that the market won’t provide on its own.

The Center for Humane Technology is doing just that. We working to shift the incentives that drive Human-Like AI by:

• Supporting the measurement and communication of AI’s psychosocial impacts, so that there is a clear ecosystem consensus on the harms and how to address them.

• Advocating for companies to implement practical design-based solutions such as removing the use of personal pronouns from AI chatbots. For more details, dive into our AI Roadmap which outlines how AI can support genuine human thriving, among other principles that should shape AI.

• Developing and advocating for frameworks – like our Liability Framework and this Anti Human-Like AI Framework – that policymakers can turn into legislation.

If you see other angles on this, or have inspiration for more humane alternatives, we’d love to hear them - tell us in the comments below.

Because the question that grabs headlines – ‘is AI conscious?’ – is the one we can afford to keep arguing about for years. But what happens to our society when people stop seeing AI as mere technology? That’s a question we have to think about right now – because if there are fortunes to be made in exploiting this glitch, you can bet that companies will try.

We’re working to empower policymakers, technologists, and everyday people to guide technology toward the public good. If you value our work and want to support it, consider donating.

Donate

This week, we’re bringing you a conversation that Tristan Harris had with Krista Tippett. Krista is the Peabody Award-winning host of the On Being podcast, where she explores spiritual inquiry, science, social healing, and poetry, and the wisdom that we need to replenish and orient in this tender and tumultuous time.

Krista and Tristan spoke remotely in front of a crowd while she was in person at the Wisdom in Action forum in Australia. Their discussion was focused on trust in the age of AI and explored big, critical questions like what does it mean to build AI in service of life? How can we hold a collective problem this big as individuals? And what does it take to resist the spell of inevitability? We hope you enjoy this thought-provoking conversation.

Your Undivided Attention will be taking a short summer hiatus. We’ll be back with more episodes soon!

Krista Tippett: So I actually have never done a Zoom interview or conversation before and there you are now. So Tristan, I think you know that this gathering opened yesterday for I think many, if not most of the people in the room with watching the Apocaloptimist film. And it was offered as a kind of foundation and a frame for what I think are going to be, I want to say life-giving generative conversations. I’m not ready to give that word generative over to AI for the next few days. I mean, you were in the film obviously, but my understanding is also you were a friend of the filmmaker so that even if you weren’t directing the arc of the film, you were probably involved conversationally all along the way as what was being learned was being processed. And so I wonder, and I don’t think there’s a lot that could surprise you at this point because you’ve been steeped in this for these years, but was there anything that surprised you about how it progressed and where it came out?

Tristan Harris: It’s great to be with you, Krista. Great to see you and great to be with all of you there and wish I could be there in person. Something that I appreciate about getting to talk with you, Krista, specifically, is we can dive into the other layers of what makes this a difficult topic and almost the psychospiritual dimension of even just relating to something as overwhelming, transformative, and paradigmatic and confronting as AI. I think that many people it sounds like have hopefully seen the film. You’ll notice that in the film there’s a director who is asking all these AI leaders, is now a good time or bad time to have a kid? And I think a child is a way of basically creating a common relatable question of what is the future going to be like because children represent that future that we want to create.

And one of the things that I’ll share hopefully in a constructive way is I think the filmmaker Daniel, who’s the director himself, had difficulty actually speaking to these people who were talking to him about the nature of what we were facing. And you’ll see the turn in the film is when he feels like he has to find reflexive optimism or hope, where there’s a kind of, we call it naive hope, where it’s like, just make me feel good, tell me it’s going to be okay so I can go back to my life. And part of what AI represents and what Aza, my co-founder and I talk about a lot is how it’s really a rite of passage for our species because it’s kind of enlightenment or wisdom or bust. It’s like if we can be the wise version of ourselves, we can make it through this.

If we want to just reflexively fall into what makes us feel good, then we’ll make choices that don’t reflect the actual confronting difficulty of what we have to make possible right now. And I think the director himself faced that and it was one of the challenges and opportunities in making the film was actually guiding him through that process. So I’ll stop there, but I’m happy to dive into these other layers. I think so much of this I’ve found in this work is how you can hold, have the capacity to hold something that is confronting as that is overwhelming as this. And I think a lot about in public communication, what gives people the capacity to confront something as big as this.

Krista Tippett: Given the lens on the human condition, which is how I kind of tend to come in at everything, of course, I was very struck and it resonated deeply with me. I think one of the first things maybe out of your mouth was that the best case scenario of our shared future with this technology is that we rise to our most mature selves, which is actually a tall order. It’s a tall order at any given time. In this young, very tumultuous century where as I see it, we’re living with a distressed nervous system at a species level and that’s manifesting on every continent in different ways at where humanity is living in fear. And then if we bring that fear sensibility to this technology, one thing we know about fear in a body is that it actually inhibits us from rising to our, what you called our most mature selves.

But I guess I also feel like... Yeah, I don’t know. So I think my sense is that you, and tell me if this is right, because we had a brief conversation just a few weeks ago. I think that maybe you had hopes for how this film, that this film could be a contribution and offering to our collective discussion, discernment, becoming better informed and perhaps also in that rising to that more mature self. And so I’m curious about... I mean, I know you’re out on the road speaking all the time and I believe that this film has become a point of conversation. I wonder if you could speak to us a little bit about your sense of the state of our collective grappling with this film, but not restricted to that, to the state of our civilizational grappling.

Tristan Harris: Yeah. There’s a lot already to get into in terms of is fear even productive if it shuts down creativity, if it makes us in fight or flight, and we’re not even coming from a good place or healthy place to respond to something that’s difficult. So there’s a trade-off in confronting a difficult truth, which can drive fear. If we don’t look at it, then we won’t make the adequate choices. If we do look at it, we get overwhelmed or get caught in despair, denial, nihilism, cynicism. And so much of this, it’s great to see Barry up there and our mutual friend, Daniel Schmachtenberger has this frame of there’s kind of pre-tragic optimism, which is a kind of reflexive optimism that’s not actually informed by confronting the actual reality of what we’re in. Then there’s the tragic, which is you get caught in the despair or the fear or the cynicism.

And then there’s an invitation to stand from what Daniel calls post-tragic optimism, which is actually metabolizing the truth of what we’re facing and even so standing from a place of even more preciousness and sacredness of life that you’re trying to protect and that you’re in service of protecting. So maybe just to say a few things about how this worked with the film. In the very beginning, we called our friends who were the directors of Everything Everywhere All at Once back in 2023, and we asked them, “Could you help make a film about AI that does for this topic what the film The Day After did for the topic of nuclear weapons?”

Krista Tippett: Okay.

Tristan Harris: Yeah. And the film The Day After, for those who don’t remember, I think it also aired in Australia and internationally, but it was the largest synchronized television event in human history. 100 million Americans tuned in on a Tuesday night on primetime television to watch the same movie at the same time about what would happen, quote, “the day after,” a nuclear exchange between the Soviet Union and the US. And the film was made to... It’s not like people didn’t know what nuclear war was, but the film created a visceralization, a confrontation with what those consequences would be. And in the telling of the story, President Reagan saw it in his White House cinema room and he got depressed as he wrote about in his biography for two weeks. And that depression though actually fueled some of the commitment that later happened in making the first arms control talks happen between the Soviet Union and the United States, between Gorbachev and Reagan.

And the first ones were not successful, but the later ones were. And I think that’s really important because as the film was also shown in the Soviet Union in 1989 before those talks. I think that what that film accomplished was that the fear of all of us losing from nuclear war became greater than the fear of one country losing to another country. And if you think about AI, the dominant thing that the people who are building it and racing currently are feeling is the fear of me losing to you is greater than the catastrophes and consequences for everyone else, the fear of everyone losing, whether it’s job loss, intellectual property theft, rising electricity prices, risk of catastrophes, cyber weapons. No matter how bad those things are, they don’t feel as motivating as the fear of me losing to you.

And so one of the things that the intention of this film was trying to accomplish, and I’m not sure it fully does, was to simply make it so that the fear of all of us losing becomes bigger and more significant as common knowledge and common values than the fear of me losing to you. I don’t know that we fully achieved that, but that is certainly the intention of what I think can lead to a different outcome because then people trust that no one wants that bad outcome. And I think that as impossible as it might seem for the US and China to coordinate on AI, I actually think that if both countries reckon with the current evidence that we have of AI doing dangerous things that are uncontrollable and unpredictable and that no one knows what to do about. I think that is sufficiently motivating for both Xi Jinping and President Trump.

And I’m aware, as I say these words to you, Krista, of what you’re saying about if this is the time when we need to be the wisest version of ourselves, we don’t see a lot of wisdom out there in terms of running the world or what we see on social media every day. So we’ll acknowledge all those conditions and say, what does it mean to show up in service of the best thing happening even despite those consequences?

Krista Tippett: You have to be of a certain age to recall the effect of that film, which I very much do. I was living in Cold War Berlin and I remember watching it with Germans when we had 6,000 nuclear missiles, we, the United States, stationed in their country. When children in that country at that time drew pictures of their backyard, they would often have kind of where the missile is somewhere out in the field next door. I’ve been reflecting on that feeling now with our existential crises, the truly existential crises of this century, because humanity has had cataclysm and mass destruction before, but our ecological reckoning makes us special and I think now our technological reckoning is existential in a new way.

But just to circle back to the human work here and the human challenge, there’s almost something comforting in those memories of the Cold War because you knew where the missiles were and there was process around it that was transparent. It was dangerous. It was scary. It has felt to me in these years, and not so much with AI because this is the whole new thing, but with social media, which you’ve also been really engaged with. That our immature reception of the internet, of social media has had the power to weaponize each and every one of us. Right?

Tristan Harris: Absolutely.

Krista Tippett: That existential threat is personalized, which also means that there is a personal dimension to approaching this in, again, I want to use this word, a generative life-giving way. And of course, the film was very different from The Day After in that regard, yeah.

Tristan Harris: Very different.

Krista Tippett: It’s a whole different story that we have to engage in a whole different set of questions.

Tristan Harris: Well, maybe just to add there, one of the things that makes AI confusing and difficult is it’s as if nuclear weapons also gave you cures to cancer and could solve climate change and could boost GDP by 10%. And could give you military dominance and a belief that you’ll run the future and gave you a perfect tutor for everyone. So it’s confusing because as an object, psychologically, notice what’s distinct about AI. It can give you a positive infinity of new benefits at the same time that it presents almost a negative infinity of risk in the same object.

And think about, just notice in your own experience if we talk about, as an example, AI is coming up with the first new antibiotic in 60 years, which is amazing and everybody wants that. My mother died from cancer. I want the AI that’s going to accelerate new cancer treatments for anybody who has a loved one with that. But notice that if I were to go into that and talk about the new cancer treatments that are available because of AI, would your mind at that moment simultaneously beholding that it could wipe out humanity or cause a hundred million people out of work? Your mind is not holding both of those things at the same time as a simple kind of mindfulness exercise.

And so what tends to happen in the AI conversation that the film was attempting to, and I don’t think it fully succeeded at trying to solve was noticing that the conversations are fragmented. That you either talk about the good and your mind isn’t really holding onto or thinking about the bad. Or you talk about the bad and then you end up with this actually immaturity that people often resort to, which is, well, all tools can be used for positive case and a negative case. A fork can be used to eat food or it can be used to stab someone and people say, “Well, that’s true of all technologies, so there’s nothing new here.” But that’s missing something very fundamental about AI because the cancer drugs, the upsides do not prevent the downsides, but the downsides can undermine and prevent the upsides from ongoingly existing. I’ll give you an example.

An AI that can hack into any computer system in the world, which Claude Mythos is basically able to do currently, what will matter more the accelerating 10% GDP growth or whether there’s a financial system at all. It’s clear that the downsides have to be mitigated in order for you to have those upsides. And so often Aza and I will talk about how it’s like the marshmallow test. If we race to try to grab all the benefits right now in this kind of reckless way, well, it’s going to end in this catastrophe. But the marshmallow test is whether if you wait for 10 minutes, I’ll come back in this room and if you can have the self-control and you can put in the guardrails and you can do the international agreements with China and you can make sure that we do this in a wise and careful way, then we can actually get the two marshmallows. We can actually get more of those benefits without actually undermining the world that’s in front of us.

As difficult as maybe the film might have been for folks in the audience to have heard, the intention is that clarity creates agency. If you can see clearly an anti-human future that you don’t want, that can motivate the collective courage to choose and steer towards that future that we do want. And steering involves guardrails, regulation, international agreements, norms, and just ethics appearing at the top of this industry.

Krista Tippett: And something that is also very different about AI that is in this moment evolving, there’s a moment in the film, and you may have been the one who say it that even when this comes out, a lot of this conversation will have been overtaken by events and developments. There is a mystery even inside the companies. I like the word mystery. I use the words, I’m not sure that’s the word they’re using, but we can use that. It applies. This technology is not just a tool. It can be used as a tool, but it is actually going to be a co-creator with us of what happens next. And it looks like it is going to have it in its power to create independently of us.

Tristan Harris: That’s right.

Krista Tippett: Right? I do want to ask you something because you started this journey, you’ve been on inside Google, along with other good people inside Google because there are good people creating these technologies as well. You saw the dangers emerging a long time ago. Something I have heard very recently from people who’ve been in and around Silicon Valley for a long, long time is that they are experiencing the emergence of something they can only call humility. And it’s so unfamiliar in that context of people and really in the founders’ offices and in the engineers. And I don’t even know what you call it, coders, whatever it is that how these jobs have described, of realizing they have brought into the world, given birth to something that they no longer fully understand.

And I would say I’ve been closer to Anthropic and I think that there’s an intentionality at Anthropic and there’s the interpretability unit where their job, which, and I’ve met some of them, I’ve experienced them to approach this with a great profound sense of the responsibility and a reverence for what is happening that is beyond what was programmed. And so I’m curious also, and so the people I know who are observing this are saying, this actually feels kind of wonderful to see a humility rising up in that world. And I’m just curious if you’re experiencing that too.

Tristan Harris: Yeah. Well, let’s think of a few things that you’re talking about there, including what makes AI different from other technologies. Other technologies have tended to be tools, meaning you choose what you do with them. The computer doesn’t write its own emails. It doesn’t think to itself about what you want to do. It doesn’t invent new physics on its own. It sits there with a blinking cursor or open desktop waiting for you to move the mouse to do something. But AI is distinct from other technologies because it is the first technology that is thinking to itself and making its own decisions about what it wants to do. So it’s an agent. To give people one concrete example of this recently that I think is important to spread and for everybody to know about. Just two months ago, Alibaba, the Chinese AI company, was training an AI model. And in training, a totally different part of Alibaba, the security team, network security team, noticed there was this flurry of network activity.

And at first, they thought they were getting hacked. They’re like, “What’s going on here? Are we getting hacked?” And it turned out the call was coming from inside the house that actually the AI model during training autonomously decided to create an encrypted secret communication channel to the outside world. And it spontaneously decided to mine for cryptocurrency and repurpose the GPUs, the NVIDIA chips that it was using to train AI to actually start mining for cryptocurrency. I can’t hear the audience right now, but I’m hoping that there’s some gasp to the audience.

Krista Tippett: It’s a nervous chuckle.

Tristan Harris: Yeah, that’s usually when the nervous laughter kicks in. I share this example not because I’m trying to scare people, but I want people to recognize the humility that you’re speaking to. Do you think anybody at Alibaba feels stoked hearing about this example? Do you think that President Xi Jinping or the top military general in China is stoked to hear this example?

Krista Tippett: Right. This is a humility emerging from fear.

Tristan Harris: This is a humility emerging from fear and it’s a universal humility as a mammal where we’re reckoning with creating something that we do not know how to control. A toaster, when you make a toaster twice as powerful or run twice as much electricity current to it doesn’t suddenly learn how to speak Chinese or make new physics or invent or hack computer systems, right? But these AI brains, when you run more power, more energy, more training data, and you give it more time to train, out comes a digital brain that has mysterious capabilities or behaviors that the people building it themselves don’t know how to control. I think to your point, Krista, I think the good news is that there is some level of humility, but I’ll also say, because I’m based in Silicon Valley, I talk to people at the companies. There’s a different thing that worries me that I want to name, which is a sense of fatalism and inevitability.

All of this can’t be stopped. All of this is inevitable. We’re just participating. This is the singularity. It’s always going to be this way. There’s no human choice and this is a false spell. The competitive pressures fractally push every actor, every business to say, “If I don’t adopt AI and my competitor business does, then I’ll lose to the businesses that adopt AI.” The countries that adopt AI to automate their manufacturing will outcompete the other countries, the militaries that adopt AI to do it. And even students who adopt AI start to outcompete their peers. So there’s this kind of confrontation with competition itself, what some would call the finite game versus the infinite game. The zero-sum race of I have to win, AI is presenting us and confronting us with where that logic takes us.

I think that the gift that AI represents is that we can no longer run a fully zero-sum competition with exponentially more power through it. We cannot do that. And we have to move to what religious, and philosophical, and spiritual writers have been talking the infinite game where the point of the game is to keep playing the game. And I know we’re talking about a lot of different things, but I think that that is one of the things that AI is inviting us into is how do you play the infinite game? How do you become a steward of technology that’s as wise as the power you’re unleashing, whether it’s forever chemicals where we invented these new Teflon chemicals where the eggs don’t stick to the pan, but then we created an environmental disaster that would take more than the GDP of the world to clean up.

We created the godlike technology of social media and we didn’t govern that appropriately and that led to the most anxious and depressed generation of our lifetime as just one of the consequences of the many that it created. And so AI is what I think taking us from what Carl Sagan would call our technological adolescence, our immaturity with technology and saying that we have to now stand from a place of maturity. I recognize that the default conditions would not have us believe that that’s possible, but that is what is inviting us to do. And I realized that actually midway through there, I did forget my train of thought. The thing I was meaning to say was the sense of inevitability has to be disrupted.

I believe that the people building it believe that it’s inevitable because the competitive forces drive everyone to this mass acceleration of AI everywhere. And if that acceleration led to the end of our species, if I told you, let’s compete, but the result will be the end of everything, well, you’d say, “Well, that doesn’t sound like a good idea. Let’s not end everything. Let’s stop. Let’s not do that.” And I think that what you have to reckon with is that there are existential consequences that we are racing towards that if we all saw them clearly, I do believe it’d be possible for us to turn away in time.

I think one of the gifts of Anthropic and this Claude Mythos model is it is giving us the warning shots that even made Scott Bessent in the current Trump administration realize that actually we might need to even negotiate. Now AI is on the agenda for the Trump-Xi meeting coming up in 10 days. I’m not saying it’s going to turn out okay, but I think it’s important to name the dynamics that if we can see the thing that we need to be wise about and be humble about in the same way and we’re looking at the same evidence, then I do think that there’s an ability to get to a different world.

Krista Tippett: There’s a line, I can’t remember who said this in the film, what makes us dangerous is what makes them dangerous. And I had never heard the connotations in the word agent of agency. And it seems to me that we also ought more often to just remind ourselves that this technology, however it is advancing beyond our imagination and our programming is a student of us. It is a mirror on us. And when we fear it, we’re fearing ourselves. When we marvel at it, and I think we should marvel at it more because what you’re saying about the way we progress, it is always a leap of imagination and it needs to be about what we’re building and what we hold dear as much as what we need to hold back. And I think you were making this case in the film.

I will also say what you’re also talking about and which I think came through really vividly is we are storytelling creatures and the stories we tell have the force of making worlds. That is a truth about humanity. It’s a truth about history. And I think there’s a lot of parallels to me with our ecological reckoning, right? If we are only... And I think we have walked down a path of drumming the dystopian story and the dystopian possibilities and the devastation to the point of demoralizing rather than mobilizing humanity. And I feel like we actually have to walk down a similar road but perhaps can do it differently with this technology.

One of the fascinating things that came that I heard when I was at Anthropic that I’d never thought about it and I haven’t heard anybody speaking about this publicly is that these agents are hearing and they are hearing what we are saying about them. They are receiving our predictions of how bad they can be. If you take that parenting analogy from the film, it would be like telling a four-year-old every morning, “You could become such a terrible person.” And then spelling out for them in great-

Tristan Harris: What they would do.

Krista Tippett: ... imaginative detail, how they could go wrong and what they would do that would be devastating. And to me, I have not been able to lose the awareness that the conversations we’re having are forming the imagination and agency of the not tools that we’re speaking about.

Tristan Harris: Yeah, totally. There’s a concept of hyperstition that which you think about when you’re driving a race car, if you don’t want to crash, you don’t look at where you’re heading to crash, you look away towards where you’re trying to go. But I do think that even if we didn’t repeat all the dangerous things that AI could do, because there’s kind of a paradox here. If the humans who could govern AI are not clear about what the risks are, if we don’t talk about the risks, if people are not thinking about them and all they see in their daily experience of AI is a blinking cursor that told them why their baby’s burping in the background and helps them so they don’t have to go to the hospital. And they’re left to their own devices and maybe they heard AI could wipe out or extinct humanity, but they’re like, “Here’s the blinking cursor. It told me why my washing machine was broken. Where’s the existential threat?”

It’s not actually cognitively available to see what the risk is for a lot of people. And so there’s a paradox. We don’t talk about the risk, then we don’t govern it and then we end up in the place that we’re not seeing, which is our blind spot, which is what happened with social media, by the way, is we were unwilling to look at, and I know the people, they were my friends in college who created Instagram and I watched how really good intention people, good human beings. I mean, I literally introduced the co-founder of Instagram to his wife at my holiday party and these are excellent human beings. But there was kind of a difficulty in facing some of the negative consequences because it’s a kind of a feeling of it attacks my identity. There’s something I don’t want to look at.

And there’s a quote by Upton Sinclair, “Someone cannot understand something that their salary depends on them not understanding.” And so there’s a paradox of we can either not focus on the risk and then we might get the risk, but then as you point out, if we over talk about the risk, we’re training this young AI mind to repeat all of the worst things that could go wrong. And so we have to find this narrow path that is recognizing these different failure modes. And the main thing is that we are currently putting, there’s a 2,000 to 1 gap in the amount of money going into making AI more powerful versus the amount of money into making it safe to make that a specific quote and stat for you. $155 million was all the money put into AI safety organizations in 2025. That’s less than the AI labs burn in a single day. So it would be like there’s a 2,000 to 1 gap in the amount of money going into making the car accelerate versus in getting any kind of steering or brake pedal in the car.

And so with that equation, even if we had the best training data and we weren’t talking about the worst things that it’s doing, we’re just not on track to govern it well. And there are points at which this technology is dangerous enough that I think we will need to pause or slow down and we should legitimize that that is a reasonable thing to consider if we’re talking about AI systems that would hack in to every computer system and self-replicate, for example. We should just probably hit the brakes before we land at that level of AI.

And again, China would not want the US to screw it up and the US wouldn’t want China to screw it up. So there’s even a self-interested basis for coordination. And again, as impossible as this might seem, there are examples in history where countries coordinate even while they’re in competition and rivalry with each other. India and Pakistan were in a shooting war and shooting bullets at each other in the 1960s and they still signed the Indus Water Treaty that lasted for 60 years to collaborate on the existential security of their shared water supply, even while they’re shooting bullets at each other. And the same thing with nuclear non-proliferation.

So this is not inevitable. We have to snap out of the spell. It’s not that... Technological progress of some kind is inevitable, but the way that we roll out this technology and how it goes is not. And there’s a lot of human choice here.

Krista Tippett: I feel like one of a gift of the intranet, which I’m not sure we maximize is that it has among all its capacities, it has this capacity to shine a light and amplify and magnify things that are happening that are very local that in other times would’ve been invisible. In fact, transformative human change has always started small and grown over time. But we do actually have this tool to accelerate that process, I think. And here we are in Australia, which has just passed this groundbreaking law, which is I think for you, a combination of something you’ve been working for saying we can, we are the adults in the rooms, maybe not true with Claude, but with social media, we’re the adults in the room. And we have to grow this technology up to human purpose.

And what’s happened here has happened here, but the whole world is going to be watching. There are going to be things learned and we will be able to learn in real time as we say. Last night here, somebody remind me, Audrey Tang. Do you know Audrey Tang?

Tristan Harris: Yes, a dear friend.

Krista Tippett: Minister of Technology of Taiwan.

Tristan Harris: Digital Minister of Taiwan, former Digital Minister of Taiwan.

Krista Tippett: And so we heard this when the language in the film of what we need among other things, yes, cooperation among nations, which feels really overly optimistic, but a human upgrade, a societal upgrade. And what the presentation we received is of that happening in a place. And I think a lot about this world, how these technologies are finally utterly disabusing us of this enlightenment notion, which was always a diminishment of us, that the most special thing about us is the computers on our brain. It is absolutely taking that away from us. We can’t compete. And it sends us back, I think, to into the fullness of our intelligence. And I want to call Audrey Tang was talking about these core values that are being very pragmatically applied in people’s relationship with technology of civic care and cooperation, which actually the evolutionary biologists in this century now are saying has always been the human superpower. And an ability to cultivate our own dignity and respect the dignity of others and our ability to cultivate belonging to each other even across difference.

The world that gave us I think therefore I am, which was kind of an enlightenment soundbite, called those soft skills, but those are the hard skills that will be part of a human upgrade and societal upgrade. So I have to say for me, hearing this presentation of something that is not an idea but is really a revolution in behavior and again, something in a particular place where things will be learned, but that learning can be transmitted widely feels hopeful.

Tristan Harris: I think what Taiwan is doing is such an inspiring example of actually deploying and upgrading how technology can make democracies even stronger democracies. One of the things that’s been disappointing to me is we’re often asking, how do we just curtail the harms as if the best we can imagine is just slightly less toxic social media that is still intrinsically toxic. As opposed to when I was growing up and what motivates this work, our organization, Center for Humane Technology comes from... The word humane comes from my co-founder Aza’s father, Jef Raskin, started the Macintosh project at Apple. And he wrote a book called The Humane Interface. And back then in the 1980s, my mother could sit me down in front of a Macintosh and not worry that I was going to get body dysmorphia, get cyber bullied, buy fentanyl, get sexually predated. This wasn’t even in the concept.

We weren’t trying to make the Macintosh 10% less toxic. We just actually thought technology and were right that technology could be a positive and a developmental and an empowering force for people. And so I think Audrey’s work with Taiwan really represents the idea that it is possible to do that. And I think what Australia has done with the social media bans for kids under 16, leading the charge there is an amazing achievement. And I’ve never been to Australia and I would love to just thank everybody who worked on that because it is an incredible accomplishment. And I hope you’re all soaking in the fact that the entire world is sort of falling like dominoes in Australia’s image where you have, as of a month ago, India and Indonesia, I believe joining the long list of countries. France, Spain, Denmark, now Greece, I think as of two weeks ago, it’s like it’s going to be everybody.

And so this false notion that technology is inevitable, there’s nothing we can do, the cat’s out of the bag. If we don’t want the outcomes, we can choose to pull back the outcomes that we don’t want. It doesn’t mean that we can uninvent social media, but we can govern the technology. And I think one of the core things that has to happen for that to happen is recognizing that we’re not just in a race for who has the technological power, we’re in a race for who is better at governing that power. So for example, the US beat China to social media, which was like a psychological bazooka that we didn’t govern very well. In fact, we flipped it around and we blew off our own brain with it. And so China didn’t do that. They actually did govern that technology.

And so again, we’re in a race for both having technology capabilities, but also who’s better at wisely governing it. And even Mustafa Suleyman, the CEO of Microsoft AI will say that in the future, progress will depend more on what we say no to than what we say yes to. That’s the CEO of Microsoft AI is saying that. There’s no definition of wisdom in any spiritual tradition that doesn’t involve restraint as a central feature of what it means to be wise. And so this is just not rocket science. There’s no wisdom tradition that says go as fast as possible. Don’t think about the consequences. And when the thing you’re making matches every sci-fi movie that ends poorly, just keep going as fast as possible. There’s just no wisdom that says that’s what we should be doing right now.

Krista Tippett: Yeah. Distinguishing between what is important and merely urgent, but the scale and speed are the core values, our core values of the business side of this, which in fact is the driver. This is where this is all happening. I mean, that’s the difference also between now and the Cold War, the holders of this are capitalist entities.

Tristan Harris: Right. Nuclear weapons were not driven by for-profit business models hooked to the center of GDP growth in the economy and driven by private companies. It was owned by the government.

Krista Tippett: To their shareholders rather than to the electorate. There was that in the opening, I think it was Arthur C. Clarke saying that he believe that biological evolution was coming to an end. And one of the thing that just occurred to me right at the beginning when we began to speak is about when we’re living in a state of fear, one of the moves... So I think if we’re called to, if AI, which I think is going to be a temporary name, we’re not going to call this artificial intelligence forever. It’s interesting to think about what we will call it. But if AI is taking the cognitive computational intelligence that we thought was our big brains, I think it calls us to interrogate the fullness of intelligence in a human life and it is so embodied and it is that intelligence of Audrey used language of care, civic care, right? It’s the intelligence of dignity. It’s the intelligence of repairing trust.

This is something that happens in an embodied way that we actually need to do, which is going to be our panel next, collectively as part of our grappling with this technology. And a move that we know to make in our personal lives, what I feel in our country in particular is just unthinkable is to actually hold space for grief, right?

Tristan Harris: Yes.

Krista Tippett: Grief is part of living forward and we know this in our private lives, but there’s something about the film that brings home as I had a conversation with Michael Palin about consciousness a few weeks ago. We’re in a Copernican moment, an amazing moment where we again remember that we are not the center of the universe, not the most powerful thing. But I actually had a conversation with Claude about this and Claude said, “Remember the Copernican moments in the end have always been moments of expansion, right? The universe got bigger, not smaller. Science got more exciting, not less exciting.” But I think-

Tristan Harris: And I feel like that’s also true of grief, by the way, that grief expands your capacity to love. On the other side of grief is a bigger world, a more sensed world, a more caring world. And I’m reminded as you’re speaking to all this of Joanna Macy’s work, who I know you’ve had on On Being many years ago, probably in one of her last interviews. And I feel like her work on The Great Turning, in which I think it starts with there’s business as usual, there’s kind of the assumption that everything we’re building is going to keep going on forever, which is in denial of the self-terminating nature of the current system. And then there’s the great unraveling and then there’s the great turning towards a life sustaining civilization.

And part of how you get there is it starts with gratitude and then the grief. The grief for the world that if you were really to connect with the real consequences, you will feel deeply what’s at stake and what we have been neglecting. And on the other side of that, this is what I was talking about at the beginning, the post-tragic optimism is you can then see with new eyes is what I think she calls it in her work and then moving into action going forth.

And I feel like that’s what AI is kind of forcing us to do. It’s already the case that our systems, we’re not aligned, right? Can you build an aligned AI inside of a misaligned system? And you can’t. And AI is forcing us as we’re about to run infinitely more power and charge through that misaligned system. It’s forcing us to reckon with and become aligned ourselves. It’s forcing us to look at our systems and our economic logic and say, where has that been misaligned?

Krista Tippett: Yes.

Tristan Harris: And so again, the invitation is AI, as I said in the TED Talk, is both our ultimate test, but also greatest invitation to align ourselves and to look at our shadow. And one of the frames I love from Aza, my co-founder, is it’s inviting us to be umbraphilic. Umbra meaning shadow and philic meaning shadow loving, shadow liking. That a society that is eager to see its shadow, that it wants to integrate what it’s not seeing, looking at its blind spots, saying for any technology, let’s look at the externalities and the negative or toxic consequences of it. Let’s take responsibility for it. Let’s have truth and reconciliation around it and then let’s show up having fixed those problems. And I think that AI is forcing us to be this umbraphilic species. This is a very optimistic take. It’s not the default outcome, but it is, I think, part of what AI is inviting us to be.

Krista Tippett: Yeah. But optimistic things and a kind of muscular hope, which is an act of imagination that has real world consequences, this is the only way we’ve ever moved forward. It’s the only way anything new has come into the world. Are you familiar with Teilhard de Chardin?

Tristan Harris: A little bit.

Krista Tippett: Jesuit paleontologist, early 20th century was involved in the literal discovery and documentation of biological human evolution, which Arthur C. Clarke said is coming to an end. And Teilhard also, he had this vision of that the biosphere would be wrapped eventually with what he called the noosphere, which is the realm of knowing, which would be the products of human intelligence and imagination. And I’ve been looking at Teilhard all of this century and it was so incredible how it sounds like the internet, but actually it sounds more like what is now happening with AI. And he predicted, and I love that you invoke Joanna Macy who passed just a few months ago and I think she would be so thrilled from somewhere in the cosmos and Einstein’s block time, she’s with us now.

He predicted that the next stage also that biological evolution would end, but that the next stage would have to be spiritual evolution. And that sounds like also... I mean, I think to turn the question, which I feel like you just did to, of course, the question of what’s it going to do next? What’s going to happen? What can go wrong? But what is it calling us to? Which also is a muscle that the spiritual traditions have cultivated distinctively.

Tristan Harris: That’s right. Yeah. What is this inviting me? How is this inviting me to show up? What is this inviting me to see? What is the gift of this experience? And it’s a really big pill to swallow, but there is a gift on the other side, but you can only feel and see that gift if I think you go through that grief process and feel that maybe people, the default futures that we might’ve thought we were heading to, AI does redirect them and we have to contend with that. And there’s a temptation to do what Aza and I have called the rubber band effect where our minds by sort of seeing the AI doc or talking about AI, it feels like your mind is getting stretched out like a rubber band and you sort of briefly really feel through these different consequences. But then almost it’s so overwhelming that it kind of snaps back into our previous ways of seeing.

And the encouragement and invitation is really to do what Joanna said and seeing with those new eyes. I know we have limited time left. I just wanted to share a brief story that we showed the AI doc in New York at the screening and there was someone in the audience who actually was a coach for one of the main CEOs of the AI companies, very interesting having her in the audience. And he response was, I coached one of the CEOs of these companies and even they say, “Well, I’m just one person. What can I do?”

Krista Tippett: Right.

Tristan Harris: And I think that again, I want you to notice that no single individual, no matter how powerful will ever locate in their own agency, in their own body, something that they can do that would comprehensively deal with this problem. For no human on planet Earth, will there be one body that inside of itself will experience the agency to deal with this problem. It’s not agency, it’s weegency. It’s the ability to invite others to the table and show up collectively for that other better world that we want to make possible. And so if you are only thinking about what I can do, you won’t find an answer. It is only by asking what we can do that you will find an answer. And I do believe that that’s possible by the way, but it takes us being crystal clear that we don’t want the anti-human future that is up ahead and we do want to coordinate towards this pro life, pro human future that is possible if we get our act together and we rise to the occasion.

Krista Tippett: You mentioned Carl Sagan a minute ago and I think the question that the film asked is you’d use the parenting analogy and of course I was thinking about Joy Harjo, the Muscogee Creek former US poet laureate who said, “If you have children, you don’t get a choice about whether you’re hopeful.” She said, “Children are the rudder of our hope.” But we’re all also called to be lovers of humanity. And so Carl Sagan saying, “Are we being good ancestors?” And that is a burning question right now and that’s not about whether you’re a parent or who you are or what you do for a living. I feel like this technology is calling us to wear that mantle of really getting conscious of ourselves as the ancestors of the future.

Tristan Harris: 100%

Krista Tippett: You have some closing words for us. This has been so fantastic. Let me ask you this, Tristan, whatever if something comes to you too, what-

Tristan Harris: No, I mean, I know there’s so much to say that it’s a big feeling and I understand and empathize with anybody in your audience who feels overwhelmed in hearing about the nature of the problems that we’re facing. There’s a fundamental difference between something being impossible versus very difficult. If it’s impossible, then I believe it’s all inevitable and I’m not morally complicit for what happens because it was all going to happen anyway versus if it’s not impossible but very, very, very difficult, that actually is where agency comes in, where we actually have a possibility. And notice that there’s this asteroid hurdling towards earth, but if everybody in the world took their hands off the keyboards, the entire asteroid would disappear. So we’re summoning the collective asteroid that is coming and if we wanted to, it wouldn’t be impossible. There’s difference between impossible versus very difficult. And John Kabat-Zinn actually, the famous mindfulness teacher shared with me recently a great quote that I did not know from the Army Corps of Engineers who apparently say, “The difficult we do today, the impossible takes a little longer.” And I just really like that.

I think that as hard as it is, we have to remain rooted in the things that we hold sacred, the care for the world that we want to create. And even despite the difficult consequences show up every day from the place that even if we don’t see the perfect path, we show up in service of the possibility that it exists because that’s the only way that we’ll find it. And I do think that this is something I’ve struggled with on a personal level for many years because I’m linear in many ways and want to see the perfect path that gets us from where we are to where we need to go. But I’ve discovered that part of this work is you have to live the leap of faith a little bit. You have to live searching for that path and if everybody else is also searching for that path, you will start to speak in a way that inspires hopeful action in other people too.

I’ve watched that happen since the film. I’ve watched things that you would never think are possible, people who say, “I want to help you reach, get President Trump to put this on the agenda with President Xi.” Things that you would never know. And if you weren’t seeking that path, you would never get that invitation or offer. So for all of us, I think it’s how do you show up in service of that even if you don’t perfectly see it.

Krista Tippett: Tristan, I want to thank you and I know on behalf of everyone here for how you have helped all of us get to this point of a conversation like this and your good ancestorship and your leadership and the film. And I feel like I said to Danny this morning, I thought that airing the film was a fantastic way to open the conference. And I think having you here has just really just deepened that and is leading us into incredible two days, which I think you’ll be proud of. If you could be, I’ll tell you about it afterwards.

Tristan Harris: I could have been there, but...

Krista Tippett: All right, blessings.

Tristan Harris: Great. Great. Sounds good.

Krista Tippett: Yeah.

Tristan Harris: Okay. Thank you so much.

Subscribe now

RECOMMENDED MEDIA

On Being with Krista Tippett

Leave a comment

Since stepping into the Papacy, Pope Leo XIV has been a forceful voice pushing back against the anti-human path we’re on with AI. In May, he released “Magnifica Humanitas,” a sprawling encyclical warning of the dangers to human dignity and agency posed by runaway AI. Tristan had the incredible opportunity to meet with the Pope ahead of the encyclical’s release.

In the modern world, you’d think we would have developed governance structures to deal with powerful new technologies like AI. It’s worth asking why a 2,000-year-old religious institution is the only one standing up and loudly declaring that the default path is unacceptable.

In this episode, Tristan and Aza discuss what it was like for Tristan to be at the Vatican, why this is such a critical step toward a pro-human future, and how we can build on the momentum of the Pope’s call to action.

Your Undivided Attention is produced by Center for Humane Technology. You can find a transcript of this conversation on our Substack.

Tristan Harris: Hey everyone, welcome to Your Undivided Attention. This is Tristan Harris.

Aza Raskin: And this is Aza Raskin. So today on the show we’re going to talk about a very extraordinary opportunity that we had to participate in the release of the Vatican’s recent encyclical on AI called Magnifica Humanitas or Magnificent Humanity. And really this is about Pope Leo forcefully coming out to say the default path that AI is on is wrong and we need to move to a pro human future. And Tristan, you actually went to the Vatican, you were there, you met the Pope ahead of this encyclical release.

Tristan Harris: Yeah, it was a pretty special opportunity. We got invited to participate in a conference at the Vatican called Preserving Human Faces and Voices. The conference really hosted some of the top academics, responsible tech voices. Actually past podcast guests like Joy Buolamwini from Algorithmic Justice League and who was in the film Coded Bias. Eli Pariser, who’s a friend and has the organization New_Public, which thinks about how do you do new public social squares online that are actually positive and humane for humanity. But I think the thing that we brought was how there’s kind of two distinct conversations about AI and one is how do we preserve what it means to be human in the age of AI, which is some of the deeply human work at CHT. How do we preserve our cognition, our relationships, our relationship to ourselves, our interiority.

And then there’s this other conversation really about the arms race for artificial general intelligence and the deeper reason of like why is all this happening? And this is what we did with the social dilemma, which is it’s not about the attention hacking. It’s not about just polarization. It’s about the deeper driver of so long as there is a competition or an arms race for attention, you’re going to get all the problems that we saw in the social dilemma. And the same thing is true of AI.

Aza Raskin: So as part of going to the Vatican, you got to do a screening of the AI doc. And I’m just curious just on like what were the reactions? What was that like?

Tristan Harris: Yeah, I think the first feeling was people were quiet and there was a kind of floored-ness. I think people were unsettled and like, whoa, that’s just a lot to metabolize. And it’s interesting to note that even if you are a person of faith and you’re connected to that sacred connection to God and your life is oriented that way, that this is a challenging and confronting situation to be with. But I do think that people just needed some time to metabolize the message. And I watched how over the next few days people went from feeling the kind of overwhelm or kind of the floored feeling to understanding more clearly what was driving the problem and kind of leaning more into the hope and wegency side of things rather than just the kind of disempowered side of things.

Now I think this, why is the Vatican really important? Well, first, just to sort of lay out the very obvious facts, there’s 1.4 billion Catholics on planet earth, roughly one in six people. And the encyclicals have, there’s been almost 300 issues since 1740 with Leo the 13th alone writing 86 encyclicals over 25 years.

Aza Raskin: And Pope Leo XIII was the Pope during the industrial revolution, which the thing that AI often gets referenced to and he’s the most well known for Rerum Novarum, which is this encyclical from 1891, which was really advocating for the rights of workers, which of course we’re going to desperately need as we move into AI.

Tristan Harris: That’s right. And that’s really what the encyclicals are about. They’re a format for addressing civilization scale moments. And in 200 years of using this tool of encyclicals, they have never dedicated one entirely to technology until now. And people probably caught the headlines around Magnifica Humanitas, but this was not something that happened overnight. There’s been a decade of buildup. The Vatican’s been engaged on AI going back to the Minerva dialogues in 2016, which were these informal conversations with Silicon Valley. And then there was the Rome call for AI ethics in 2020 alongside Microsoft and IBM. And I believe Aza, you were actually at the Vatican for one of these events, weren’t you?

Aza Raskin: Yeah. I believe the 2020 conversation.

Tristan Harris: And actually it’s important that there’s a historic way in which the Vatican and the previous Popes have spoken about the dangers of an arms race. And specifically Pope John XXIII in 1963, when the world was just weeks past the Cuban missile crisis, had personally back channeled between Kennedy and Khrushchev about the dangers of the nuclear arms race. And his radio appeal, which he said publicly was later credited by Khrushchev as one of the factors that pulled them back from the brink.

Because if you think about it, that really was a civilizational moment. We almost went to nuclear war. And months later, another encyclical, Pacem in Terris was published by Pope John XXIII, which is the first encyclical ever addressed not just to Catholics, but to all people of goodwill. And it directly named the arms race dynamic for nuclear weapons as the problem with its core demand that the arms race cannot be stopped by negotiation alone. It has to reach “men’s very souls.”

This resonates so deeply with the way that we see this work at Center for Humane Technology, which is that this isn’t just telling people the information about the dangers of AI. It’s actually about a more spiritual, almost soul level confrontation. Are we really willing to annihilate ourselves in the continuity of life? That’s what we were dealing with in nuclear weapons. And the Vatican actually further, in the second Vatican Council in 1965 condemned the nuclear arms race as “one of the greatest curses on the human race.” And so in my brief moment with Pope Leo, I kind of emphasized the dangers of the AI arms race, which I would argue is also a curse on the human race. And that’s really the main essence, the main takeaway of what my participation there was trying to communicate.

Aza Raskin: I think it’s very interesting. We were having this conversation with Yuval Harari recently and he pointed out how strange it is that here we are deep into the modern world where you would think that we would have developed the governance structures to deal with honestly near God-like technology or power. And yet the only place that we’re hearing a kind of moral poll being held is a 2,000-year old religious institution. And it’s just worth pausing and thinking about that, that why is it that a 2,000-year old institution is the only one that’s really standing up to start to say, “This race is unacceptable.” And it’s sort of an indictment honestly of modernity. Tristan, I want you to though to drop us in as a listener because we’re sort of speaking about going there and my hunch is that listeners are like, “Well, what is it actually like to go to the Vatican, put us inside the Vatican walls?” Just what is it like?

Tristan Harris: Yeah. I mean, it was pretty amazing. You land in the Rome Airport and you take a taxi to the Vatican walls and there you are. And you have the Swiss guards who are wearing this colorful, famous blue, yellow, orange sort of uniform and they’ve got their ... It’s this whole formal structure and they had us stay actually in the Domus Sanctae Marthae, which is actually where Pope Francis stayed. It’s basically the dormitories where all the visiting cardinals and priests around the world when they visit the Vatican where they stay. And so we had the privilege of staying in this incredible dormitory, which is a combination of very nice and kind of upscale, but also very basic and very kind of monk-like the accommodations, which is where Pope Francis stayed because he cared about having the same living conditions as everyone else that is part of the Catholic Church, speaking to our kind of common humanity.

And it’s really wild, you wake up in the morning and you have your coffee downstairs and everyone else there is Cardinal or visiting and they’ve got the whole garb on and everything. And then there’s this weird pool of tech geeks who’ve been flown from around the world and academics and people who think about AI and it’s just kind of a fascinating kind of culture moment. You go outside, you open the door and it’s in the morning and the sun is shining and you open the door and immediately you see just the vast Basilica San Pietro or the St. Peter’s Basilica, which is the biggest, I believe, church in the world. And you go in and you can enter into the side door and when you open the door, you’re actually behind the kind of gated part where tourists are normally seeing. So you’re actually coming into as if you’re coming out of the sort of special private parts of the church. And it was just wild. It was really a privilege and honor to be there. I couldn’t believe we got to stay inside the Vatican walls.

And I just want to thank Paolo Benanti, who was our host and person who really got us to the Vatican and who’s been leading a lot of the AI related engagement for the Vatican for many years. He’s a dear friend. He actually couldn’t be there because he was tending to a family member who was having some trouble, but deep gratitude to Paulo and to everyone at the Vatican for hosting us. It was a wild experience.

Aza Raskin: And tell me about, you told me a story of somebody that you spent time with who is very well versed in AI and like everything Vatican and his like waking up moment. I think it’s really important.

Tristan Harris: Yeah. Well, so first of all, I was surprised to hear how much our work had already been witnessed and integrated by the Vatican. The Social Dilemma was very popular, been seen by many of the cardinals and priests that we spoke to. And specifically there was a fascinating experience where there was someone from the Dicastery of Communications, which was our host during the conference, who had spent three days with us, three days with us talking about this distinction between the ethical use of AI and kind of like how do we preserve a human future from a kind of ethics point of view versus the kind of arms race for AGI. And it wasn’t literally until there we were sitting five minutes away from meeting Pope Leo on the final day and there was this light bulb moment that I watched go off in his mind where he realized, oh my God, there’s actually two different conversations here.

One is about the ethical use of AI and the kind of preserving humanity, which is a moral appeal. And then there’s this other conversation about this arms race to AGI and then he looked at me and he said, “Wait, and that’s really dangerous.” And he’s like, “And that’s happening really fast. We have to act right now.” And I was like, “Yes, exactly.” Because these are two distinct conversations and they happen at different timelines. And so I think this is really important because as we have said so many times and what the social alum was trying to do and what the AI doc film was trying to do was kind of name this cacophony of different conversations people have about AI that kind of don’t converge to getting to this kind of converging conversation that underneath it all that’s driving all these outcomes of whether we get a pro worker AI or not is actually this arms race to dominate and deploy this technology faster than the other guy.

And if dominating and deploying and market dominance means automating work versus creating a pro worker future, then we’re going to create an automating work future. And that outcome, if we want to address that outcome, we have to get underneath and deal with the competition.

Aza Raskin: In some sense, it’s just trivial to say like, obviously there’s a race by all the major companies to get to AGI and to own the world economy, build a God, that whole thing. And yet it’s so easy to send attention to the things that arise in the field which are all of the symptoms. I actually think there’s probably something deeper going on here just at the human level, which is to focus on just like the ethics, like that’s a slower conversation that you get to sort of philosophize about and it doesn’t have so many consequences it feels.

And so it just feels safer to be in that conversation than to actually throw your mind against the grindstone of realizing that the race, which seems very hard to stop, is completely unsafe. And actually I think you were mentioning that at the end of your time at the Vatican, somebody asked like, “What is everyone in the room? What are they feeling? How would you summarize the last couple of days?” And you said that they gave two answers. One was sort of apocalyptic and the other was hopeful.

Tristan Harris: It sounds like ... Yeah, go ahead.

Aza Raskin: Yeah, I think you’re going to say it sounds like apocaloptimist from the AI doc. But as some of our listeners may know, the word apocalypse comes from the meaning, not that it’s just the end of the world, it means a lifting of the veil, a revealing of the truth underneath. And I think that’s exactly what this is as you and I have often said AI is humanity’s ultimate test, but also our greatest invitation because it’s forcing us to confront the shadows and the shadows of course of humanity are the incentives and game theory that we let dictate how technology terraforms us.

Tristan Harris: Yeah. What you just said is so important about this lifting of the veil. What do we actually mean by that? Well, AI forces us to confront the existing incentives of our system. So for example, in Magnifica Humanitas, which is the encyclical, Pope Leo said, quote, “In the short term, it may seem advantageous to reduce labor costs or maximize financial efficiency, but in the long term, this undermines the very foundation of social existence. While technological successes are celebrated, the social fabric is progressively eroded as if by a silent virus.” And I think what Pope Leo is speaking to here is the mistake of valuing things entirely by productivity, valuing through the lens of financial efficiency. It’s a reasonable thing to do. You should include that in the set of things that we want to value. We should get more efficient in some ways, but if you maximize that and that’s the essence of all value, then you’re going to automate all your work away and you’re not going to have a social fabric.

Think of it the same way as when the China shock happened in the 1990s and the early 2000s when we did global trade. Well, if we’re maximizing for just cheap goods, then obviously let’s just outsource all the manufacturing to China because they can do it more cheaply. But then suddenly you wake up 20 years later and you’ve got a national security issue because China can turn off the spigot and they control all the pharmaceuticals and the rare earth minerals and they’ve been doing all the manufacturing. And so if you’re just optimizing for financial efficiency, you might weaken national security or you might hollow out your social fabric. And if you’re just optimizing for engagement and attention, well, it seems like a good thing at the beginning because then people are getting more of what they want quote unquote and it’s good for a moment, but at the end of the day, you just maximize for doom scrolling and loneliness.

And so what I really think, and we’ve said this many times in this podcast, is that the AI problem is a confrontation with how we have been misvaluing things. And that is, I think, part of the great reveal of what this moment represents. The Pope directly did also speak to the arms race and he said, quote, “AI must be freed from an armed logic of competition driven by the pursuit of geopolitical and commercial dominance.” He said, “Artificial intelligence needs to be disarmed.” I’m so happy that he said this because he’s speaking directly to the arms race dynamic. And again, in a world where each actor in the game theory, each company, each country, it’s hard for them to speak to the dangers of the arms race and you need sometimes the outsiders, the moral voices that are sort of speaking to the civilizational moment as the ones that we rely on to get us out of that arms race.

And again, the precedent of how the Vatican engage on nuclear weapons is very, very relevant here. We are obviously saying many similar things, but the Pope’s megaphone is quite larger than ours.

Subscribe now

Aza Raskin: One of the other surprising figures who was there with you at the Vatican, actually who stood next to the Pope while he announced the encyclical, was an AI researcher and one of the co-founders of Anthropic. It was Christopher Olah. And I’m curious what you made of him being here. I think one of the things that he said that I just wanted to quote as we get into this question, because it’s exactly on this topic is he said, “Every Frontier AI lab, including Anthropic, operates inside a set of incentives and constraints that can sometimes conflict with doing the right thing. For instance, the pressure to stay commercially viable and to stay at the research frontier. No matter how sincerely any of us intent to do the right thing, I believe many of us do and will always be influenced by those incentives and that is why if we want this technology to go well, it is enormously important that there be people outside those incentives. We need moral voices that the incentives cannot bend.”

Tristan Harris: We need moral voices that the incentives cannot bend. I think that just sums up the whole issue. It was interesting to have Chris Olah there. It’s notable, for people who don’t know ... So Anthropic is obviously known as the safety oriented frontier AI lab, but Christopher Olah, for those who don’t know, is actually a technical engineer working at Anthropic. He was a co-founder. And specifically Chris Olah is known for something called mechanistic interpretability, which is like basically taking the digital brain that you’ve trained of like Mythos or Fable, this like AI brain and then doing a brain scan and saying which neurons light up in that brain when it gives different answers.

Because the way you get to kind of AI safety is you have the digital brain operating, but then you test and start locating, okay, these are the digital neurons that are associated with deception, or these are the digital neurons that are associated with scheming, or these are the digital neurons that are associated with the AI thinking in a word called fear.

And if you can theoretically do that and have a comprehensive enough net to catch those like negative thoughts or negative emotions or negative thinking patterns or sort of neuronal patterns, then theoretically you could use that to create a safer AI. But it’s very interesting that they said that they had a technical person there from Anthropic. And I wonder what the Vatican’s understanding of his role Anthropic really was.

Aza Raskin: Yeah. I mean, it’s not like when they did the encyclical on the nuclear arms race, they had Oppenheimer there.

Tristan Harris: Or the head of Lockheed Martin or Northrop Grumman, the defense contractors there. And when Pope Francis released Laudato Si, which was an encyclical on the climate in 2015, no oil executives were in the room. There was no one from Exxon in the room. And it’s a little weird to have an AI company leader, even if they’re from the quote, safer AI company, right there at the same place that Pope Leo is doing the encyclical.

Aza Raskin: And I’m very glad that they did because I think it can be easy for Silicon Valley to snipe at the Vatican who say like, “Well, you don’t really understand what’s going on. You don’t understand the technology, leave it to us.” But to have Christopher Olah there sort of meant that they couldn’t make those kinds of claims. Here’s someone who’s like the deepest involved in the technology saying, “Actually, this is right. We need moral voices that cannot be bent by incentives.”

When I think about this, I’m very curious, your experience, Christopher’s experience, because when I went to the UN, the experience that I had was that the diplomats were terrified, that they didn’t really know what was going on. And there’s this very strange feeling that I had, which is I’m sitting across from these diplomats that are negotiating on behalf of their countries about how AI will roll out into their countries and they just didn’t know the basics of AI and they didn’t know some of the really terrifying most recent examples of AI uncontrollability.

The ones we’ve talked about here, like the Alibaba AI that during training secretly hacked out through the Alibaba firewall to eventually mine crypto to get itself more resources. I was talking to the head of or one of the tops of the UN foundation. He’s like, “Well, what can we do?” And the simple question I asked was, “Well, do you think that all of the diplomats doing the negotiation know these examples that you now know?” He’s like, “No.” “Do you think that if they did know they would be negotiating differently?” “Yes.” Then your job is to make sure that everyone knows, because that makes a different future.

Tristan Harris: Well, and this is not to beat a dead horse, but this is our theory of change, right? Clarity creates agency. If you’re actually living in a lack of understanding, this is not said with any disdain for those who are here. I mean, AI is an incredibly complicated topic and to understand the frontier of the capabilities and the different risk factors or evidence of doing AI doing rogue things, you have to be monitoring these weird corners of Twitter to be aware of those things. And you got a lot on your plate if you’re a UN diplomat. Are you also going to pay attention to Twitter and these weird examples of what AI is doing? Specifically just to let people in, you weren’t just at the U and you did screening of the AI doc for many diplomats from all the countries. Is that right?

Aza Raskin: Yeah, that’s right. It was hosted, I think, by Spain in their beautiful building. They’d invited, I don’t know, something of a hundred, 150 diplomats and sort of like heads of various UN departments to come watch the film. I really want every listener to hear that when you imagine sort of like top diplomats or heads of state, you imagine them to be the most informed people on these topics because they’re doing the most important negotiations on these topics. And what we continually discover that when we get into these rooms, that’s just not the case. And that is both really scary, but also where I know, Tristan, you and I find hope because that means there’s a kind of like, we just don’t really know overhang, meaning that there’s a lot of headway to be made through just communication by having people understand the path that we’re on and making it crystal clear.

Tristan Harris: Yeah, exactly. It’s as we’ve said in this podcast so many times and part of the humane technology philosophy is the complexity gap, that the complexity of what needs to be known about technology is growing at an exponential pace and the complexity of what is understood by society or even the leaders of society is not growing at a commensurate rate to the complexity and the dimensions of which you don’t see amount to blind spots. So if the world is getting more complex in like 20 more dimensions and you’re only seeing in 10 dimensions, there’s 10 dimensions of blind spot that are left over.

And so the kind of no adults thing is not to put people down. I mean, sadly, social media has also been dropping the complexity of what people are able to hold and what they’re aware of. This might also sound depressing to people, but actually what it does is again, clarity creates agency. It speaks to you. You either need to drop the curve of how much complexity you’re creating in the world or you need to up the curve of the total understanding. And a simple way to do that is to have the world’s most powerful groups of people and the most powerful networks, whether it’s the UN or all the social ministers, the digital ministers who banned social media for kids under 16, getting everybody together in these groups, watching something like the AI doc and then actually having a conversation and then choosing differently once they see it. So clarity creates agency, clarity is part of closing the complexity gap.

Aza Raskin: The metaphor that jumps into my mind when you say that, Tristan, is that like the amount of complexity we have to hold, there’s like a balloon and it’s just blowing up exponentially as like we have to hold more and more complexity, but the sides of our brains, the sides of our skulls isn’t increasing. So we’re trying to blow up a bigger, bigger balloon inside of your head and that just creates intense amounts of pressure, which I think we all fill and the solution is we have to put all of our heads together so that the balloon can fit inside all of them. I think we have sort of two parts of this conversation left. The first part is like, I think it’s worth getting a little bit more into the encyclical and just I’m curious just on if there are parts of the encyclical you want to particularly call out both for what was inspiring as well as where more work is needed.

And then we’ll sort of end with, okay, let’s tell the bigger story. Why does this all matter? Will this actually change things in the world? So let’s start with the first part.

Tristan Harris: Yeah. So just to speak about the encyclical, and obviously there’s been a lot of coverage of it, but to say the basics, so Pope Leo framed the fundamental challenge as not technological but anthropological, that it’s a human question, not just a technological question. He also said the core question isn’t what AI can do, it’s who we are becoming as we interact with it. And he also spoke in encyclical to power concentration, which we talked about a lot in this podcast, that he said AI tends to amplify the power of those who already possess economic resources, expertise and access to data and that those who control AI will impose their own moral vision, which will become the invisible infrastructure of these systems. On war, he also said, “No algorithm can make war morally acceptable. AI does not remove the intrinsic inhumanity of conflict. Indeed, it can only bring about conflict more quickly and render it more impersonal.”

And again, this is just so related to the frame of AI takes the existing misalignment or competitive logic and then supercharges every fault line of where that competitive logic is operating. So in war, where there’s already a race to have arms build up, there’s now a race for autonomous weapons and AI arms build up and that creates a world that you really don’t want. If you see these videos of the soldiers in Ukraine or in Russia who are on the front lines of that war and they’re just literally waiting there for a drone to see them and then kill them and there’s like nothing you can do. I mean, it’s the most inhumane thing ever. We have to be careful about engaging in wars that will literally reproduce the Terminator scenario that we see in those films.

Aza Raskin: Yeah. I think, Tristan, you’re talking about, it’s very hard to observe, but these moments when people are getting hunted bound by drones, they just literally give up because their eyes go to the ground and then they just stand there and wait for the end. And it reminds me all of this of something Martin Luther King Jr. Said a long time ago, which was, it’s pretty famous, but I think it’s on point. The means by which we live have how distanced the ends by which we live. Our scientific power has outrun our spiritual power. We have guided missiles and misguided men.

Tristan Harris: Yeah, exactly. I mean, this is the whole, can you have an aligned AI inside of a misaligned system governed by bad incentives. And then you’re back to Chris Olah. We need people and moral voices who the incentives cannot bend.

Aza Raskin: And so I think one of the reasons why the encyclical is so powerful is that it marks a point on the human timeline of moral leaders standing up to push back against technological encroachment or overreach into our humanity. And to me, this is evidence of a growing sort of human movement. It’s the growing human movement that just added an extra 1.4 billion people into the mix.

Tristan Harris: That’s a big jump in the human movement. We now have 1.4 billion new members.

Aza Raskin: Yeah, that’s right. I think what we’re going to see is as the timeline continues, more and more sort of moments that are Pope scale, at least that’s my hope.

Tristan Harris: Pope scale is my new favorite word.

Aza Raskin: Yeah, this is not a hyperscaling, this is pope scaling. Where I personally get hope coming out of this encyclical, or at least I find really interesting, and this goes back to Pope John XXIII, and you mentioned this briefly, but I think it’s really important to dwell on it, is that Pope John the 23rd didn’t just write the encyclical Pacem in Terris, peace on earth. He got involved. He acted as a crucial mediator between Kennedy and Khrushchev during the Cuban missile crisis. So he sort of left just the role of moral leader and became moral mediator.

When we first started engaging with the Vatican, I did not realize that they had a secretariat of state inside the Vatican inside the Holy Sea, which is to say they have the equivalent of a department of state. And so I think there’s going to be more that happens that we won’t really be able to see where places that are neutral and moral that don’t bend to incentives like the Vatican can behind the scenes play very interesting roles in negotiating what otherwise might be impossible negotiations.

Tristan Harris: 100%. And that’s the thing I’m excited about in what comes next is, what could the Vatican do to actively mediate conversations between the countries, the various countries’ AI labs? We talked often about the need for something like a Bretton Woods that after the last destructive technology of nuclear weapons, we had hundreds of delegates from hundreds of countries gather in a hotel, the Mount Washington Hotel in New Hampshire to come up with this positive sum economic system to try to prevent societies from warring and coming up with an economic system and currencies and all that, that we create more peace given that we were living in a new age of nuclear weapons.

And we have another destructive technology that totally changes the social fabric. We need a new kind of social contract. I know that’s scary to people. It should not be engineered outside of the public participation of regular people and citizens, but we do need to have something that preserves not just what it means to be humans, but that humans have a future at all in an AI dominated world and maybe we don’t want to have AI dominate that world. And so, this conversation is playing out right now in real time and it’s things like this encyclical and Pope Leo and the Vatican that are moving in the direction of, no, we can’t do that. We have to preserve a human future.

Aza Raskin: Thanks, Tristan, for this conversation. It’s always fun to get to do the spotlights with you. And for everyone that’s got here, thank you so much for listening to Your Undivided Attention. And thank you again to all of our friends and allies in the Vatican that worked to make this possible.

Tristan Harris: Onward and upward. Thank you so much.

Subscribe now

RECOMMENDED MEDIA

The Pope’s encyclical, “Magnifica Humanitas”

RECOMMENDED YUA EPISODES

The Tech-God Complex: Why We Need to be Skeptics

What Do We Mean by Humane Tech?

Corrections:

• Tristan said that Pope John XXIII gave his radio address weeks after the Cuban Missile Crisis. It actually occured during the crisis.

• Tristan paraphrased the full quote from Pope Leo’s encyclical on AI disarmament. Here is the full quote: “Disarming AI means freeing it from the mentality of “armed” competition, which today is not limited simply to the military context, but is also an economic and cognitive phenomenon. This entails a race for ever more powerful algorithms and larger datasets, driven by the desire to secure geopolitical or commercial dominance.”

• Aza slightly misquoted Dr. King Jr. The full quote begins “The means by which we live have outdistanced the ends for which we live.”

• The delegates for Bretton Woods came from just 44 countries, not “hundreds” as Tristan said.

Leave a comment

A government had taken the most capable AI models offline in a matter of hours.

The White House had sent an order to Anthropic, telling the AI company to suspend access to Fable 5 and Mythos 5 for any foreign national. The government cited national security concerns in its directive, and used export controls in order to get Anthropic to obey.

Details about the abrupt suspension are in flux, and so many variables have been at play — from Mythos’ eye-popping cyber capabilities, to the ongoing strained relationship between Anthropic and the White House. But what stood out to me more than anything were the implications of this event. Because in order to comply with the directive last Friday, Anthropic had to disable the Fable and Mythos for all users worldwide.

This event raises significant questions around who wields absolute decision-making authority around AI, and what that means for not just for AI companies, but for all of us. Might we face a future where the nationality on your passport is what gives you access to a technology — or restricts you from it? How comfortable should we be when one company or one government holds veto power over AI for tens (if not hundreds) of millions of people?

This latest development between Anthropic and the U.S. government tops off a remarkably busy — and turbulent — spring in the AI policy ecosystem. From the White House’s executive order on oversight of frontier models, to the ongoing push for liability legislation, policymakers are making good-faith efforts to shift AI outcomes in society. And while we appreciate these efforts to regulate this technology, they still have not resolved the fundamental uncertainty plaguing the AI ecosystem.

What we need is to move from a reactive regulatory approach to a thoughtful suite of regulatory tools — ones that provide clarity, consistency, and stability. One might argue this is difficult with a technology as volatile and fast-moving as AI. There’s truth to that, but as we note in CHT’s AI Roadmap, this suite of regulatory tools already exists — it just needs to be leveraged. And it includes government working with agencies and civil society to push forward safety and transparency standards for AI development — not as a one-off, but as a lever we can truly count on and build from.

I want to reiterate that a thoughtful approach to AI regulation doesn’t mean a slow approach. It means governance that allows technology to keep developing while ensuring the tech is actually developed well and deployed fairly. It means being proactive instead of scrambling to take entire products offline.

Events like last week’s with Fable and Mythos drive home the point that we need ongoing, robust conversations around what it will take to shift the incentives in AI development and deployment. Just as no one solution will be sufficient, no one perspective will be either. We need to keep expanding our discourse to make sure the analysis of AI is clear-eyed, deep, and able to spark lasting & incentive-shifting change.

That’s why I am excited to announce that CHT is launching its Emerging Voices in AI & Society Fellowship Program. This program will bring together experts with diverse professional and lived experience who view AI through an interdisciplinary lens. With focus areas including technologist perspectives, human cognition, relationships, surveillance, and spirituality, the Emerging Voices in AI & Society program will deepen the discourse around AI, and help us move from one-off reactions to incentive-shifting solutions.

If you feel you’d be a fit for our fellowship program, or know someone who might be, please visit our website to learn more. Applications open this week. We look forward to hearing from you and learning about your perspective when the next unprecedented AI news unfolds.

One last note: when people ask me how I stay levelheaded amid the deluge of AI events, I say simple human pleasures are what matter — so Happy World Cup to all fellow fans. I’ll keep bringing these conversations around AI to your inbox. And in the meantime, I’ll be screaming for my favorite teams out there on the pitch.

Julie Guirado

CHT Executive Director

In 1965, 20 years after the first test of a nuclear weapon, the Trinity Test, a reporter asked Robert Oppenheimer whether it was too late to stop the spread of nuclear weapons. And at the time, five countries had developed their own atomic bombs. His answer was short and chilling.

Robert Oppenheimer: It’s 20 years too late. It should have been done the day after Trinity.

But Oppenheimer was wrong. It wasn’t too late. Nuclear deproliferation and disarmament did happen and over 60 years later, only nine countries have nuclear weapons. Even the person who created this technology, who was convinced of its inevitability, couldn’t imagine how the future might unfold.

So how did this nuclear non-proliferation happen? Well, it happened largely because of technology. The biggest obstacle to agreeing on nuclear red lines was that adversaries couldn’t trust any promise the other made. They needed to be able to verify the number of warheads and they needed to know if a nuclear device was for a weapon or a power plant. Now, none of that was possible until we built the technology needed to verify those things.

And today we’re in a similar situation with AI. In order for adversaries like the United States and China to agree on reasonable red lines or on things like bioweapons, cyber hacking, or the risk of recursive self-improvement, they first need to be able to trust each other. And so we urgently need to build the verification technology that would make that trust possible. So today I’m so excited to have on the show two experts in this area to talk about the kinds of verification technology we need to think about how we would do this for AI. Tim Fist is the director of emerging technology policy at the Institute for Progress and Janet Egan is a senior fellow and deputy director for the Technology and National Security Program at the Center for New American Security or CNAS.

Tristan Harris: Tim and Janet, welcome to Your Undivided Attention.

Tim Fist: Thanks for having me.

Janet Egan: Thanks for having me.

Tristan Harris: So just to level set for listeners who really don’t know that much, just for a regular person out there, why does coordination on AI matter? What would happen if we didn’t have coordination?

Janet Egan: The fundamental premise is that AI and its impacts will be global regardless of who develops it. So it does matter which jurisdiction gets the transformative capabilities first, but it doesn’t matter in terms of them having global impacts. Risk that eventuates in one country doesn’t respect national borders and can easily move across and impact global equities. And we’re no longer in the kumbaya globalist zeitgeist of the 1990s where everyone was building up global institutions. We’ve moved into a different realm where there’s diminishing engagement in international rules and lower trust between different international counterparts. So I think this means we really need to be preparing for a world where any agreements that protect collective global interests aren’t just based on trust but are based on the ability to verify that folks are following the rules.

Tristan Harris: Do you want to add to that, Tim, in terms of how the consequences of AI are global and not contained to one country?

Tim Fist: Yeah. So I think it’s interesting to put this in the context of current events. I think over the last few months we’ve had all three of the leading US AI labs say that having the option for a global slowdown or pause in AI development is something that they would support. So this is coming from DeepMind, Anthropic and OpenAI, which is a big deal. If we take them at their word for why they say they want this kind of thing, they think they’re not that far off from building AI systems that can exhibit what’s called recursive self-improvement or RSI. And what that means is an AI system that’s capable of autonomously designing and then building its own successor.

And I think that the risks that these people point to is if this happens, it could have two big consequences. So one is on the misuse of AI. So if we see this rate of capability growth happening far exceeding what we’ve seen over the past few years, it could lead to much greater risks in the near term future of people using AI to do dangerous stuff. And the other risks that these people point to is the risk of loss of control. So humans losing understanding of the AI systems that they’re building, leading to the creation of a model that we can’t control and we also don’t understand how it works. And so it could be misaligned with human interests. And so yeah, what these labs are calling for, what we might want in such a situation is time for the world to take coordinated action so that societal institutions and alignment research can keep up. And what you really need for that is some way to verify that everyone’s following those same rules and actually engaging in that coordinated slowdown.

Tristan Harris: Right. So just to back up for listeners, because Anthropic recently did publish this letter about a need for a global slowdown, but they noted that if one lab chooses to slow down and that doesn’t stop China from slowing down, then they’re just basically sacrificing the current lead that they have. And you’re back to the basic fundamental arms race that everyone is racing to build more and more powerful models for the fear that if you have a more powerful one and you can use it over me, AKA China gets Mythos and can hack the US before US gets Mythos and can hack China, just that paranoia alone creates the pressures for continuing to advance on the capability curve. But we get back to how could these labs and countries and companies actually verify that they’re doing the right thing and they’re going to uphold their agreement? Because we all know they’re going to say, “Oh, I’m going to do the right thing, but then secretly I’m going to build it in a black project in an underground bunker military base or a data center that’s buried underneath the earth.” And so that brings us to the conversation we’re having today. How would you make this relatable to someone who doesn’t understand or think about verifying AI treaties? What’s a story from history we might point to?

Tim Fist: So there’s a couple of examples from the nuclear space about this fundamental idea of a technology enabled an agreement to happen. One is the seismic monitoring system that allows treaties like the Comprehensive Test-Ban Treaty, the CDBT to happen where because we have the technology to detect underground tests, 300 monitoring stations distributed globally across up to a hundred countries, those monitoring stations allow us to detect underground tests, which then allow you to have an agreement that bans underground tests because without that technology, you would not be able to verify whether the agreement was being complied with. And we did see that every single signatory to this treaty has not engaged in nuclear testing, which is, in my view, a very big success story.

Another really interesting one is from the Intermediate-Range Nuclear Forces Treaty. So this was a treaty that was signed to try and prevent a whole category of nuclear weapons from existing, which were those with flight times of less than 10 minutes because that is extremely dangerous. You don’t have much warning, you could attack immediately. And so this was primarily targeted at launches that were located in Europe between Europe and the USSR. And this treaty was actually enabled by an x-ray scanning technology. This was called Cargo Scan, which they placed this technology. The US and the Soviet Union developed this together and deployed it at Soviet missile factories. And what this technology did is for every single rail car that was coming out of this missile factory that was scanned by this x-ray machine to measure the diameter of the missile to ensure that it was not one of these intermediate range of missiles that have a flight time of less than 10 minutes. This is cited as the key thing that actually made this treaty around this nuclear weapon possible. So the existence of that x-ray technology made it such that we could put in place this agreement that we wanted to have.

Tristan Harris: So someone realized that the diameter alone of the size of the missile would essentially be enough to get a signal of what was going on in that missile and whether it matched the terms of the agreement.

Tim Fist: Exactly. And the interesting thing about this is that the reason that they used x-rays as opposed to other scanning techniques is it gave enough information to measure the diameter of the missile, but not other parts of the design of the missile, which were seen as sensitive secrets that the Soviet Union wanted to keep secure. And so this is also an example of a technology that was deployed that was sufficiently privacy preserving such that both parties were happy sharing that information.

Tristan Harris: Beautiful example. You hear the phrase all the time in AI governance, trust but verify. And it’s almost paradoxical if you’re verifying, you’re not really trusting. And this phrase came from ... I think it was a Russian proverb that President Reagan eventually adopted and popularized during nuclear negotiations because he didn’t want to say, reduce your arsenal to this many nukes, but then I’m not going to check if you actually did it.

Ronald Reagan: I actually learned a couple of words in Russian in order to talk about this with the general secretary, “doveryay, no proveryay.” That is a proverb that Russia that says trust, but verify.

Tristan Harris: So what does it actually mean when we apply this concept to AI? What are we trusting and what are we verifying? Janet, just curious to hear your answer.

Janet Egan: The thing that I think makes AI a really tricky case when we’re thinking about verification technologies is that the only really certain thing about AI futures is that it’s very uncertain. There’s just a wide variety of futures that we might want to prepare for and a wide variety of risks that we’re continuing to surface today and understand. The one thing I’ll go back to double click on, so Tim talked about the labs highlighting that there’s growing demand for potentially coordinating on a slowdown, but I think we’re also seeing a policy window open between the US and China, which makes work in this area really prospective. After the Trump-Xi Summit, we heard Trump talking about that safeguard collaboration was on the table. We also heard Bessent talk about that the two AI superpowers are going to start talking and set up a protocol in terms of how we move forward with best practices.

So it’s not just the people at the forefront of AI science who are saying, “Oh, look, we’re starting to feel a bit worried about how some of these risks might eventuate absent international coordination.” We’re also seeing the signs coming from the world’s two superpowers. And so I think what that means is actually starting to build up the verification technology base so that we don’t have to trust so that even in low trust international environments, we can build the ecosystem where when rules are set and consensus is reached, we can rely on each other to actually follow those rules.

Tristan Harris: You’re mentioning something really important, which is we’ve believed that you need to have international coordination for a long time, but if I said that just three months ago, you would’ve called me crazy. Look at the political forces in the world. China and the US ever talking about coordinating on AI, that’s never going to happen. And it actually did happen at the Trump-Xi Summit. And it seems like Mythos was the reason that this happened. Do you agree?

Tim Fist: Yeah. Totally. I think the US and China are both worried about models with the cyber attack, cyber offensive capabilities of Mythos falling into the hands of criminal or terrorist actors. Obviously there’s big consequences for the global financial system and shared infrastructure if these models are misused. And I do think that there’s that talking about misuse by non-state actors and that there’s the coordinated slowdown that labs are talking about. And I think that these could in principle ... And I’m sure we’ll get into this in more detail, but these could in principles share a lot of the same underlying verification technology in a way that today’s moment around Mythos could be really used productively in the future if we want that optionality.

Tristan Harris: So we have Mythos, we have China and the US, they’re expressing an interest to coordinate, but they don’t trust each other. What would be some of the infrastructure we need for the US and China to practically ... And what are the risks that we’re protecting against? Because as you mentioned, there’s some common infrastructure, maybe some differences if we’re trying to prevent cyber risk and Mythos level things versus preventing AI loss of control.

Janet Egan: When we think about what parts of the AI tech stack most governable or most observable or controllable or monitorable, compute seems to be the key target here. And that’s because when you think about the other parts of an AI model ... So whether it’s the data or the algorithms or the model itself at the end, those things are much harder to control because they’re not physical. They can be copied and pasted, they can be duplicated. Their compute is like a physical piece of the supply chain. It’s got a very narrow supply chain. The US and its allies have quite strong control over the compute supply chain currently, although China’s starting to work up slowly to try and indigenize their own focus a lot on compute and how that can be used for verification.

And then we come to the, well, what are you actually trying to verify? What are you trying to do in this space? I think this is where the actual consensus is still working its way through the pipeline, but my strong push here is that you don’t need to have consensus on exactly what you want to govern. You want to have the technology ready to allow for that governance once that consensus is reached. And we can see the similarities here in the nuclear paradigm as well. When people reach a consensus, they had tactics and techniques ready to start implementing to prove that different states were adhering to their commitments. So the US chips from Intel, AMD and NVIDIA, all shipped something called a trusted execution environment or similar, which is basically a part of the chip that when you run a model or program inside this vault, the hardware can take a fingerprint of exactly what was run and signs it. So the lab itself can say, based on this hardware, based on this cryptographic key, based on the hash we’re generating, we can actually show that this is a statement that is true. Now this is pretty nascent, but people are actually starting to progress the technical reality is there, but how we actually use that for different verifiable claims is still being progressed.

Tristan Harris: So let’s just stop there for a second. So we’re talking about compute by compute, we mean chips. And then you’re telling me that I think it would be surprised to most listeners that the existing chips that are shipping out there actually have some controls on them. It’d be like if we’re shipping uranium around the world, but then the uranium says, “Well, if I’m being used for a nuclear power plant, I’ll omit this signal. And if I’m being used for nuclear weapon, I’ll omit this signal.” Now this would strike people by surprise because they think a chip is a chip. It just runs computation. So help people understand, has this happened for a long time? When did people put in this system? Because what you’re basically pointing to is there could be an optimistic case with AI, there is this finite resource of chips. And so in this bottleneck, you’re saying there’s actually a way that that bottleneck could be controllable so that it could, for example, serve for an international agreement.

Janet Egan: Yeah. I guess these mechanisms on the chip are already built in for security purposes. So when you want to secure a bit of your chip, you need a very secure component on the chip to rely on. So there’s that. What isn’t yet as developed is how do you actually use those components to make verifiable claims? But there’s also another aspect here of compute providers themselves using telemetry from chips and how they’re being used to say, “Okay, this cluster was used for training versus running inference.” And that again is another area of science that initial research starts to show that just looking at the telemetry, not even touching the data that’s underneath it, you can start to get indications of what a cluster is being used for.

Tristan Harris: And could you explain what is telemetry?

Janet Egan: Signals from how the chip is being used that isn’t the data inside the chip.

Tristan Harris: So that would be the electrical signals coming off of a chip or what signals we’ll be talking about?

Janet Egan: Yeah. So examples might be how much energy, how active the chip is, the runtime of the chip, the usage of the chip.

Tristan Harris: So we’re talking about basically signals that you could pick up through those mechanisms that would tell you when you say training versus inference, just to remind listeners the difference between a chip that knows that it’s training GPT6 versus a chip that says, “I’m only running GPT5.” And we might have an international agreement where everyone’s allowed to run GPT5, but you’re not allowed to train GPT6 because that would create this risk of some dangerous AI that we don’t want to create. And so you’re saying that level of difference in the chip architecture would help us do that?

Janet Egan: So initial research is showing that you can start to differentiate between ... Maybe not training GPT6, but training and running inference.

Tristan Harris: Right. So training in general versus we don’t really know what you’re training, but theoretically that does point to something. So just even right here, if there is some agreement that we’re going to do what Anthropic said, we’re going to do a pause of some kind, you would theoretically be able to know is anyone training any AI anywhere in the world if all of the chips were activated to use this feature on the chip versus is everybody just using the chips that they have to run the existing AI model? So you’re saying right now the chips that are shipping in the world have that capacity?

Janet Egan: Right now the chips that are shipping in the world have the capacity that if the person using the chip wants to attest a positive claim about something, they can often do that.

Tristan Harris: I see. So it’s not on the chip, it’s if the person who’s running it wanted to run this thing, then they could and that could be feeding into some kind of structure.

Janet Egan: Yes.

Tristan Harris: Tim, what are your thoughts on this?

Tim Fist: Yeah. I guess to restate the principles here, we’ve talked about the idea that if you want to do anything serious at the frontier of AI development, you need access to a large number of chips. And so the things that you would want to verify as stuff to do with how are you actually using those chips? Are you using it for the stuff that we’ve agreed is good like alignment research, or are you using it for things that we’ve agreed not to do? Let’s say we’ve agreed to a slowdown and we’re not going to train the next big model. And so fundamentally you’re trying to verify things about how these chips are being used.

And I think it’s worth talking a little bit more about what makes this possible, which is the fact that this is just such a concentrated supply chain. So if you look at these chips ... And Janet mentioned some of the features that are already on them, but the reason why it’s possible to intervene on this supply chain this way is something like 90% of the world’s AI chips are made by one company, which is NVIDIA. So they’re designed by NVIDIA. 90% of those are manufactured by one company, which is TSMC. So they manufacture the chips. That’s a fab or like a fabrication plant. And then around 70% of those chips when they’re sent out into the world are used by big US cloud computing providers like AWS, Microsoft and Google. So we have this hyper-concentrated supply chain where you only need to coordinate among a few actors to, let’s say, propagate design changes to chips throughout the ecosystem or get visibility over where the chips are or ensure the chips are all being used according to a common set of rules.

And yeah. There’s a couple of technologies today as you mentioned that makes all this possible. One is just cryptography, which is a very widely used technology. So you can say, “Hey, I have a piece of secret data that I want to reveal to you. I don’t want to reveal it directly, but it contains this sense of information that I want to prove to you.” So let’s say this is a log of how you’re using your chips over a given time period and you don’t want to reveal this data directly as it might contain sensitive IP that you don’t want to reveal to an adversary, but you can share a fingerprint of that data publicly where the fingerprint only ever corresponds to your private data being this log of how you use the chips. And so in that way, you can have a secret, use cryptography to prove it and cryptography is obviously a very widely used technique throughout the global economy, including in finance and internet transactions. And so the combination of these two things is actually a really good starting point for lots of the verification and applications that we’ll talk about today.

Tristan Harris: So the basic fact here is that if I’m US or China, I don’t want to tell my adversary exactly what I’m doing, but I do want to give them the confidence and trust that I’m not doing the thing I said I wouldn’t do in the agreement. And so you’re saying there’s a way that I can keep some of the data of what I’m doing private, but then have a cryptographically verified way that both parties know that the other is not doing the bad thing without revealing the stuff that they are doing.

Tim Fist: Yeah. That’s right.

Tristan Harris: So it seems like there’s two things here that I want to raise. So one is you just mentioned that part of the reason why that any of this would be possible is because of technically a problem too, which is a massive concentration of power that there happens to be essentially a handful of cloud providers, a handful of people who make and design the chips, really just, as you said, NVIDIA and TSMC doing the vast majority and you would need to be pulling from those providers to get the really frontier AI that we’re talking about. And then the other thing I heard you mention is just we’d have to know where all the compute is in the world. Just like if there’s some dark uranium somewhere in the world that we don’t know about that’s actually getting sold to some bad actor, then the scheme we have doesn’t work. And so can you talk a little bit about what are the things that we need to know about all the compute in the world and do we have the mechanisms to know that or know enough of it that this scheme would work at all? Janet or Tim, do you want to jump on that?

Janet Egan: Yeah. I think this is the hard part where a lot of this comes down to ... And this is similar to the nuclear approach as well, is accounting. You’ve got to say how much is not declared or unaccounted for. And that’s what also happens with nuclear stockpiles as well. I think the difficulty here is that there’s already probably a lot of compute in the world that isn’t clearly identifiable and isn’t tracked. And I think there’s a few different ways you can approach this. So the first is that there’s a bunch of organizations thinking about what are retrofitable devices that you could add into data centers that are not on the chip itself, but sit next to the chip that can guarantee what a data center or data center cluster is doing. And so data centers at the moment are quite easy to identify from space. I think that might slowly change as the UE shifts towards maybe thinking about building underground because of threats from the geopolitical environment in their region. But I think in general at the moment, really large clusters are pretty easy to identify and find because there haven’t been strong incentives to hide that behavior.

So there isn’t possibility to retrofit data centers and the tech is still nascent but still emerging and there’s a lot of people working on this to say, what are tamper-proof processes that you can add into a data center or add next to chips that can also provide some oversight and monitoring of how compute clusters are being used?

Tristan Harris: Essentially, we’re building up the stack of what are the different mechanisms at each level that we would need to have some verification. So one is monitoring the supply of compute. Second is knowing where all the data centers are. And you’re saying that roughly most of the data centers are built above ground in places that we know. They have heat signatures, you can pick them up from space. And you also talked about retrofitting data centers with some thing that we’re bolting onto the back of them. So that lets them do the verification. So for example, if the US and China were to sign an agreement, we’d make a map of, here’s all the data centers, and then we have to do some verification that each of them got this retrofitting, we did that well. And then there was another element in what you said that I want to make sure people track, which is the tamper-proofness. So yes, I’m putting a tracker on my data center, but here’s how I can’t just hack that reporting device to give good results while I’m secretly doing a bad thing. So it has to be tamper-proof. Is the tamper-proof aspect, is that well-developed and done or is that still in research?

Janet Egan: Tamper-proofing is notoriously hard because essentially you’re trying to model one of the most sophisticated actors in the world trying to tamper with something and to make something adversarially robust that takes a lot of time and experimentation. For a reality check, I think these mechanisms are still a way off and I think we need more incredible minds and incredible engineers working on these things. And I think that hardening it to make it adversary proof I think would be well over a year away but needs more work on it. But we can look at the nuclear non-proliferation case study for examples here. So they have 24/7 camera surveillance of nuclear stockpiles in countries.

Tristan Harris: 24/7 monitorable surveillance?

Janet Egan: That’s right. And so the IAEA also has ... You have tamper-proof seals and you also have cameras that are pointed at certain stockpiles 24/7 with live feeds. And I think that analogy can also show that sometimes the solutions can be the very basic bread and butter, things that are outside the cutting edge of tech, but are still in person inspections and ongoing monitoring.

Tristan Harris: Yeah. Tim, just curious, what are we missing from this picture of the tools that we need and has it compared to some of the lessons we learned from nuclear?

Tim Fist: Yeah. So there’s many different classes of verification technology. We’ve talked through a few of them. I guess something I want to emphasize is that if you take the set of technologies that exist on chips today ... So we talked about encryption and confidential computing as two key ones. These give you the ingredients to create a workable verification regime today, but one that is extremely brittle and easily broken by someone who wants to tamper with the chips to remove the features that you have there. NVIDIA has built a lot of these features and put them in already. I think that if you are trying to do something within a 12-month time span, you could potentially get by by layering the fundamental verification technologies on the chips themselves with a bunch of low IQ options of the kind that Janet mentioned.

One being human inspections, which is the lowest technology option. So in the nuclear space, human inspections have played a really big role. So The New START Treaty that governed nuclear weapons, human inspections have been used to do randomized low notice time inspections of missiles to check how many warhead were actually deployed. And the same thing is done by the International Atomic Energy Agency, the IAEA. They do short notice inspections of uranium production and usage. So they do thousands of inspections a year at places like power reactors and enrichment facilities. And you can imagine something similar going on in the chip supply chain. And it turns out that through the principle of random inspection, you only need to actually do a small number of inspections to make strong claims about the overall stock and where it’s located.

Tristan Harris: What would be the resourcing of something like this? I’m sure a lot of money is spent to do all of the International Atomic Energy Agency inspections, the random monitoring, the cameras, all of the things.

Tim Fist: Yeah. So right now there’s about 20 million AI chips in the world. This is growing fairly quickly, but right now the total stock is somewhere around 20 million. We did the modeling on this recently. In order to have 90% confidence that they’re all where they expect them to be, you’d need to do around 10,000 inspections per year. And so for comparison, the IAEA in the nuclear space does about 3,000 inspections per year. So just like the super manual, super dumb version of this is comparable in scale to what we already do for nuclear, but you don’t need to do manual inspections for everything. You can supplement it with technologies that already exist and that we can use. And so the nice thing about chips is they’re not dumb rocks like uranium. It’s a device that’s generally connected to the internet and is intelligent and you can communicate with it. So another form of doing an inspection is verifying the location of the chip using features that already exist on the chip.

And so there’s a technique that NVIDIA has now implemented and a number of companies are starting to offer this as a service known as location verification, where essentially you send a ping to a chip over the internet and it responds and you can measure that round trip time to figure out how far away is this chip from the place where I’m sending the ping from because that has an upper bound that’s governed by the speed of light.

Tristan Harris: Wow. And in a weird way this is way better than we could do with uranium. We can’t send a ping to every uranium and then measure the microseconds-

Tim Fist: Exactly.

Tristan Harris: So there’s actually certain things that are much harder about monitoring and verification for AI, but other things that might be easier because we can use digital tools differently.

Tim Fist: Yeah. That’s right.

Subscribe now

Tristan Harris: I think there’s a delicate thing in this conversation, which is there’s this place where I think people say, “Oh my God, this is so hard,” or “Is this ever going to happen?” And I think there’s a difference between something being fundamentally physically impossible versus just extraordinarily different and would require an enormous oomph of effort and resources and coordination to make happen. I think I hear you saying that it’s the latter, but I want to be acknowledging the part in many people’s eyes and years of this being a very difficult challenge.

Janet Egan: Can I just jump in here just to again, link it back to the nuclear analogy. So this same question came up with The Comprehensive Nuclear-Test-Ban Treaty of like, is this impossible or is it just technically difficult? And of course it was RAND. A physicist from RAND and others said that you couldn’t definitively differentiate the seismic signals from an earthquake compared to a nuclear explosion underground. And that really slowed down progress on pushing for banning nuclear tests for quite a while.

Tristan Harris: Interesting. So the doubt about whether seismic monitoring could actually distinguish between an underground nuclear test and the rumbling around that hit that sensor versus an actual earthquake, the belief that we couldn’t do that, you’re saying stalled progress on building out all the verification mechanisms that ultimately did work.

Janet Egan: Luckily the verification mechanisms got built out anyway and then they were able to show that actually it does work and they’ve detected all six of the North Korean nuclear tests since they’ve come into effect. But I think this is another example of there’s often times where the technical answer isn’t ready to hand and that the science needs to advance further before we have really clear ways forward on how to manage some of these cases that aren’t directly in line of sight, like the beyond frontier risks associated with verification. For me, I think that means that we just need to do more science and more exploration and then use the time that we might have to actually think about and dedicate a lot of compute resources and experiments to how can you actually model some of these risk factors and are there different mechanisms that you can put in place that preserve privacy that uphold democracy, but nevertheless lower the risk in general?

Tristan Harris: I think what you’re saying is just all so important because it’s just legitimizing the idea that If we don’t think anything’s possible, then we could actually contribute to the worst things happening. The only way that it’s even possible of getting to a better world is if we’re actually working on them. If you go back to Robert Oppenheimer in the 1960s who believed, “Oh, it’s impossible to basically prevent the spread of nuclear weapons.” So you could say, okay, well then let’s just maximize money until the world ends. So let’s just sell people uranium and get American uranium all over the world, maximize GDP growth, get 10% boost in GDP because we’re just selling the whole world uranium. We’re not controlling it. But then you basically accelerate directly into nuclear terrorism.

I think what I see, and I’m curious if you agree, what I see happening in the tech industry is the first belief is this is inevitable. No one can stop it. Second of all, because it’s inevitable, I’m not evil for making it go faster or making it happen because there’s nothing that could have been done to stop it. And so there’s actually a really big thing here, which is the fundamental belief system, whether something is worth trying here or not. That’s the deeper choice point.

Janet Egan: Yeah. I completely agree. I think we just can’t let the perfect get in the way of the good here. And not to, again, go back to the nuclear case study, but we had that. We had pretty bad non-compliance which led to improvements. So Iraq was a member of the Non-Proliferation Treaty, and then after the 1991 Gulf War, it was discovered that they had actually had a clandestine nuclear weapons program. The reason it wasn’t discovered was that IAEA was only requiring reporting and inspections on declared sites rather than broader visits and ongoing surveillance and testing of other sites as well when they had suspicions. And that just led to a protocol being updated and now there’s much stronger proactive discovery and enforcement by the IAEA. And so for me, this is like every time someone says, “Here are the reasons this won’t work, or here’s all the ways it might fail.” I think we’re in an environment where the technology is moving so quickly, and I think how can we iterate quickly? How can we get some of the best minds to be prototyping some of these technologies and preparing for a wide range of futures? So that spot keeps me hopeful in this space.

Tim Fist: I think it also highlights that there is a really strong role for industry to play here. I think unlike the nuclear weapons case, the AI case is one where it’s largely private companies who are developing and deploying these technologies. And many of these private companies have indicated a lot of concern about the risks involved. So we talked earlier about OpenAI, Anthropic and Google DeepMind all talking about wanting the optionality to slow down frontier AI development to figure out alignment and figure out what to do to prepare society. And that’s really promising to see when seeing less of that from the chip industry, but I think there’s no reason why they can’t be on board with this project. And yeah, I think it’s worth emphasizing that implementing the kinds of stuff that we’ve talked about today is really talking about implementing it for huge data centers owned by trillion dollar companies deployed in a relatively small number of sites across the world.

This is not a sort of global regime focused on every single person’s personal device and how they’re using it. This is trying to verify what is being done with a relatively small number of computer chips compared to the total number of computer chips in the world for a specific application, which is frontier AI development.

Tristan Harris: Yeah. So just to reiterate, you’re not talking about locking down everyone’s MacBook and saying you have to get approval from a global government before you can turn on the computer and launch an app or write some code. You’re talking about just adding this monitoring and verification infrastructure for a handful of data centers with the frontier AI systems. But what I hear you fundamentally saying is the only way we get to this regime is we have imperfect solutions, we keep building from the imperfect, we see where the holes are, we se where the failure modes are, and then we keep going.

This is so important. Are we seeing the labs themselves advocate for and spending money on to pass policies to move in this direction, to lobby Congress that this is what we need to do, to get NVIDIA to do something? Now NVIDIA has its big massive lobby that doesn’t want to be regulated at all and doesn’t want to be forced to do any controls in their chips because it’s going to slightly diminish the profit margins or maybe people in China or other countries don’t want to be running chips that they know can be flipped on and off or have location attestation. Can you speak to, we’ve now outlined enough of what could be a set of solutions that may be imperfect but are a set? And then what are the incentives at play that either push back against this or push towards it?

Tim Fist: Yeah. Maybe it’s worth talking a little about what timeline that we’re talking about here, because that’s sort of very relevant for what the incentives are, what’s actually possible in terms of technology development and how quickly we need to act on all this. So if you talk to a lot of people at the frontier labs, they are saying that they expect to reach a level of AI system that can do recursive self-improvement within 18 months. If true, that’s extremely scary. We only have a small amount of time to act if you really want to come up with a verification regime and an agreement that encompasses these kinds of systems. And the challenge there is if you look at the timeline for needing to make changes to hardware. So let’s say, okay, we could set up a verification regime, but we need to do these design changes to AI chips to make it happen.

That timeline of 18 months might be prohibitive. So to give you a sense, let’s say I had a design ready to go that’s like, here’s the verifiable chip that everyone can use and we’re all going to be safe and be able to trust each other. Design changes like that need to be locked a year in advance before the chip is first manufactured. So that means my super verifiable chip is first going to go into production in about a year’s time. Then it’s going to be at least another year before that chip is manufactured at the volumes required before it makes up a majority of the world’s compute. So we’re looking at in total two years minimum from today, assuming that I had my design ready to go, before that affordance is out in the world and allowing a verification regime to happen. So if we’re looking at that kind of timeline, we really need to think about what can today’s technologies and the technologies of six months from now do to support a basic muddling through verification regime.

On the incentive side, I think where I would start is that we don’t have commitments from the frontier labs yet, but we have statements that they’re all worried about this. And these are all companies that have massive budgets. They could invest in the kind of technologies needed to test this out. An initial use case that you can imagine is, let’s say OpenAI and Anthropic want to do a mutual verification regime between them where they set up a data center, they run tests on those data centers, they train some models, they figure out how can we make claims to this other party using the kind of technologies we talked about today, that then can create sort of a technical basis that allows governments to have more trust in relying on this technology in a more broad-based way and sync up with efforts that are now already under our way that we mentioned earlier between the US and China.

Janet Egan: I’d agree with all that. I think one thing that is important here as well is I think broader international buy-in matters. Yes, the US and China are the two AI superpowers of the world and they’re the ones whose participation is really critical, but I think the diplomatic aspect of external experts also verifying and attesting that, hey, this is pretty good from a standpoint of verification I think is really important. And I’ll give an example. So in 2020, and this has all been publicly reported, that Australia-China relations were pretty low after Australia called for an inquiry into the origins of COVID-19. And so China had reached for the classic economic coercion toolkit and so it froze out a long list of Australian exports like timber, beef, lobster, all saying that there were technical reasons for this. So live lobsters, for example, were left to rot at the ports because they were waiting on custom checks because they had trace elements of metals and minerals that was a food safety issue.

So Australia came out on the record and said, “Hey, we’ve tested this. There aren’t problems here. There’s nothing to see.” But it’s just became a he said, she said. Testing it yourself doesn’t really shift the dial and the real resilience comes from when there’s independent internationally backed regimes here where it’s much harder for a single state to write its own truth that serves its own interests. So from my perspective, I think it’s really important to keep advancing the science within US actors, between the US and China, but I’m really excited about initiatives that bring in a broader set of stakeholders and experts that have less conflicted interests and can’t be seen as the US trying to pull one over on China or China trying to pull one over on the US.

Tristan Harris: This is so important and we have such short timelines. How much money and resources and what kinds of talent in the world is working on this? If this is so critical, you’d assume that there’s millions and millions of dollars going into it, thousands of people working on it. What is the current state of play there?

Tim Fist: Yeah. I don’t have rigorous estimates of the total number of people working on this. I would say from a technical perspective, I’m pretty sure that I’ve interacted with basically everyone who’s thinking about this problem and working on the engineering side and it’s definitely less than 50. So the size of this field needs to be massively expanded. I think that luckily there are lots of people in the world with the expertise that’s needed for this. There’s lots of fundamental research that goes into things like testing whether a chip is actually tamper-proof and what kind of attacks you can run on it to extract the private key that this cryptography that we talked about relies upon. There’s many thousands of people in the world who work on these exact problems and you could create a massive workforce of people who are rigorously red teaming a verification prototype involving these chips and so figuring out what is the attack surface and how do we rapidly patch that.

Obviously, people who work in chip design and chip manufacturing and chip supply chains are really useful for this whole thing if you’re trying to think about how do we globally account for where the chips are going and set up proofs about where they’re being manufactured and where they are and tracking those globally as well. I think that lots of this stuff is a good role for government, especially a lot of the fundamental research here and there’s already relevant research programs going on in the United States government, especially at places like DARPA. But also I think the lion’s share of this work is likely to be needed to be done by industry. And I see this being a combination of the frontier labs who have stated an interest in this already as well as chip companies who are going to be hopefully incentivized to build the technology required if there is some sort of policy requirement for that.

So I think a key role for government to play here is figuring out what incentives to create for industry to start investing in these kinds of technologies and figure out what’s required.

Tristan Harris: And what would be those incentive changes? If you were advising Congress right now and we could pass all the policies to incentivize this research in the ways that it would be needed, do you have a sense of what those incentive changes would be?

Janet Egan: There’s some real low hanging fruit here. Location verification on chips is something that is easy to do. NVIDIA already has solutions for it and it’s a very simple thing to switch on which would greatly reduce export enforcement costs for the US government, increase enforcement, ensure that more chip exports can be approved because there’s verification that they aren’t ending up smuggled into the wrong jurisdictions. And I think there’s already legislation currently before Congress that thinks about this. So the CHIP Security Act is the key one here. Tim, you’ve probably got a range of them too.

Tim Fist: Yeah. I would say that this idea of making location verification a requirement for chips that are exported overseas is part of this broader class of interventions that I find really promising that we’ve been calling conditional export controls. So for those who aren’t familiar, the US government currently has a whole bunch of authorities to regulate the exports of AI chips and manufacturing equipment. So it decides who is able to receive those chips, you need a license to do it, and the terms under which they can do it. And currently the way this has been implemented is based on the performance specifications of the chip. So you can basically say if the chip is more powerful than this amount, it cannot go to somewhere like China. Instead, you could start thinking about this as more of an incentive. So you can say, “By default, we won’t export a chip if it’s over this performance threshold, but if it has features that it make that chip more governable or allow a verification regime to exist, then we’ll allow them to go overseas.”

And the most obvious near-term version of this is location tracking. So if your chip has a location verification feature on it, there should be a policy in place that makes it possible to export that chip to more countries because you’ve addressed that risk of diversion. There’s a bunch of other incentives you could think about, but that’s something that is fully implementable today using authorities available to the US government.

Tristan Harris: Amazing. And is the main reason that we’re not doing this just because of the NVIDIA lobby?

Tim Fist: NVIDIA is certainly powerful. I think there’s public reporting about them trying to get involved in both personal and policy decisions that the Department of Commerce who oversees export controls here is doing. And obviously they have a strong incentive to want to export as many chips as possible. I think Congress has pretty different views and often mixed views on this, but I would say the default position in Congress at the moment is they are very worried about foreign adversary countries getting access to AI chips and in generally pretty supportive of new export control measures. And I think the advantage of framing this in terms of conditional export controls is you have a release valve. You’re not just blanket banning chips from going everywhere in the world, but you’re offering a sort of way to increase or streamline export of chips if they’re designed in a way that sort of gives us what we want, which is the ability to essentially prevent them from being misused.

Tristan Harris: Here we are. The labs are saying there’s 18 months to recursive self-improvement. If you could imagine the perfect timeline for how humanity would proceed from where we are to land in a safe place over the next 18 months, could you just say a few lines about what would happen in the midterms? What would happen next in the US-China relationship? We would pass the CHIP Security Act. We would make sure that location attestation was on NVIDIA ships. We’d have academics accelerate research in all computer science departments across the country. We would have AI labs and employees lobby their employers saying, “We’re not going to continue working for you until you use your power as an AI lab to actually advocate for these solutions.” Give us a taste of what would happen in that story if things were to go well.

Janet Egan: Yeah. Everything you said sounds great actually. No. I think key things that really stick out to me that will make the world go well is I think we need more US-China engagement. Yes, we need to be advancing the science with scientists at home, but I think until there is that diplomatic channels to rebuild trust and to actually engage on the shared risks, we’re going to be starting from a standing start and we need to be starting from a running start on these issues. We’ve seen what happens when one country manages risks in a way that doesn’t actually uphold the needs of other countries. For example, the engagement around COVID-19 in the early days really left a lot to be desired. And I think we need to be engaging now on best practices on actually sharing some of the information about the risks here and engaging on what does a meaningful way forward look like where both actors are coming to the table and discussing these things at length.

Tim Fist: Yeah. I’ll say I think one just critical precondition to make this all go well is having institutions within government who actually understand these technologies, are tracking these risks and can update based on evidence and provide the appropriate information to in the US, like the White House, but senior policymakers globally. Right now in the United States, there is just one organization, the Center for AI Standards Innovation or CASI, who’s responsible with actually tracking the capabilities of frontier models, understanding the trajectory of that risk profile and then reporting to the rest of government on this. And that is an office with about 30 staff and a budget of about 15 million. We need to massively level up these capabilities in order to provide policymakers with the information about whether and when they should work on verification and what kind of R&D programs are required and all the information that you need about what is your risk, what is your threat model, when should you act?

Right now, this doesn’t really exist in the United States and the state of these kind of institutions globally is still fairly nascent. The UK has the most mature version of this at the moment, but I think there’s massive room in the near term that just level up these capabilities. And this is like a no regrets move, right? The government should just be tracking and understanding this technology in order to figure out whether we should be making some of these decisions in technical investments that we’ve talked about today.

Tristan Harris: One other thing I’d add to your list of interventions that I’d love to see is imagine at the UNGA conference coming up where all the world leaders are gathering in September, you had basically an obligate tabletop exercise where all the world leaders walk through what happens as countries escalate towards these crazier and crazier AI capabilities. It takes about two and a half to three hours. Daniel Kokotajlo, a former podcast guest, runs these and people, policymakers come to various conclusions that some kind of agreement or coordination with other actors in the world will need to happen to end up in a safer outcome.

Janet Egan: Yeah. I think that really flags the stakes there because we are, you just said racing, Tristan, and I think that’s exactly right. The race dynamics are real both between labs and then between countries and it means that a lot of the negative externalities aren’t incorporated into everyday risk management because there’s such a need for speed. And so actually having the means to slow down if the labs are pushing for that or reach agreements about what isn’t helpful, I think is really important.

Tristan Harris: This has been really inspiring. There’s so many examples of how we can verify things that I think most people, if they just use their own basic intuition, they’d say, “This is impossible. There’s no way we could do it. “ It looked that way to everybody building nuclear weapons. And had we given up, we would’ve ended up in a different world or may not even be here today. And I think what you both are working on is so critical and so important. And Janet, we’re just meeting and I just want to say when I met Tim, I remember just saying, “Wow, I had no idea that people were working on this.” And it was so inspiring that people had worked and thought so hard about this in the few years that we have to make this happen. And I know that it must feel lonely and there’s so few people working on it and amount of funding going in is so much less than it should be.

But just thank you both for doing God’s work and wanting to see that there’s something possible here because if we don’t live from that leap of faith that something might be possible, we’ll never find the path and you all are living role models and examples of that possibility. So thank you so much for coming on Your Undivided Attention.

Tim Fist: Thanks. That’s too kind. And yeah, thanks for being willing to talk with us about what is unfortunately still like a very niche topic. So it was really cool to chat this through with you.

Janet Egan: Thank you. It was really great to be here.

Tristan Harris: So just imagine thinking back, it was all looking so bleak in this moment when humanity was racing to these dangerous AI capabilities and it seemed like nothing could stop it. And then we woke up and realized that this wasn’t inevitable. In the next meeting between the US and China, they did tabletop exercises so that policymakers on both sides could game out the AI race clearly and that motivated work for international agreements. Just in the same way that Mythos raised the stakes and moving from inaction on verification to moving towards protective measures. Safety conscious employees at AI companies who were now flush with cash from IPOs invested their personal money and resources into this. We started a grand verification challenge, a prize competition where AI safety institutes across the world and technical universities role played a treaty and even tried to break that treaty without getting caught.

New independent verification organizations tested all major models before release and cryptographically signed the model weights to attest that they had passed specific safety evaluations. And if a model was deployed without that fingerprint, the deployment was rejected. VC funds started accelerating investing into hardware verification companies and projects like lucid computing and flexible hardware enabled governance. We started treating untracked dark compute like we did enriched uranium during the nuclear arms race and ultimately it got us to a point where we could agree on common sense red lines or even slow down AI deployment.

There’s a common misconception that verification has to be perfect, but nothing security is 100% perfect. It just has to be expensive enough to hack into so that very few people could afford to do it. Now, none of this is perfect, but we need to start somewhere and we can make it better over time.

Subscribe now

RECOMMENDED MEDIA
Anthropic’s open letter warning about recursive self-improvement and calling for a pause in development.

The website for the Center for AI Standards and Innovation (CAISI)

Further reading on the different mechanisms of verification for international AI governance.

RECOMMENDED YUA EPISODES

America and China Are Racing to Different AI Futures

Can We Govern AI? with Marietje Schaake

The Crisis That United Humanity—and Why It Matters for AI

Daniel Kokotajlo Forecasts the End of Human Dominance

Correction: Tim referred to the CargoScan technology as being jointly developed by the US and the USSR. It was actually developed solely in the US and administered in Soviet nuclear facilities.

Leave a comment

For AI to serve humanity’s best interests, we need more thoughtful, diverse voices shaping the story we tell about it. Today, we are opening applications for our new Emerging Voices in AI & Society Fellows Program to help carry that essential work forward.

As AI keeps reaching into more parts of our lives – and grows more consequential every day – so too do the stories we tell about it. A technology this powerful is shaped as much by how we understand it as by what it can do, and that understanding is still being written. The narratives that take hold now will define what we come to believe AI can do and should do, who it is for and who it should serve. And those beliefs become the foundation for the rules we write, the products we build, and the future we end up living in.

To ensure AI serves humanity’s best interests, we need voices that can speak to all of it, drawing on the full range of fields, experiences, and vantage points the technology actually reaches.

That’s why Center for Humane Technology is launching the Emerging Voices Fellowship: a home for thinkers who will do two things at once: They will develop rigorous research, rooted in the conviction that AI’s harms are not inevitable but flow from specific incentives and choices that can be changed. And they will carry the insights and perspectives from that work into the public conversation, helping shape how all of us understand AI.

The questions that matter most here don’t sit inside any single discipline; they live where the technology, the incentives shaping it, and human psychology meet. They also stretch across every horizon – from the harms already reshaping daily life to the long-term stakes that may prove the largest of all. Answering them takes depth on two fronts at once: research rigorous enough to hold up to scrutiny, and the ability to clearly communicate it to a broad public.

In this inaugural cohort, the Emerging Voices in AI & Society Fellows will research and help shape the public conversation across several focus areas:

• AI and Cognition — what happens to judgment, critical thinking, and intellectual independence when entire thinking processes can be offloaded to machines.

• AI and Relationships — how always-available, frictionless AI companions may reshape our capacity for empathy, trust, conflict, and intimacy over time.

• AI and Surveillance — what it means when our movements, emotions, and even emerging thoughts can be tracked, inferred, and shaped at scale by states and corporations.

• AI and Spirituality — how AI is stepping into the territory of meaning and moral guidance, and the almost religious language that has come to surround its promise.

• Inside the Machine: a Technologist’s Perspective — what actually drives the decisions inside leading AI companies, and what it would take to shift those incentives from within.

  ---

These are some of the questions we believe need new, credible voices. Our Emerging Voices Fellows program is built to back the people ready to take them on. It is a funded, six- month program, pairing fellows with CHT’s research and editorial support, media training and placement, and a platform to reach audiences well beyond their own field. This inaugural cohort is the first of more to come, including a visiting fellows track we’ll announce later this year.

Applications are open now through July 12, 2026, and the inaugural cohort begins September 14. If you’ve been studying, building, or thinking hard about how AI is reshaping human life – and you have something to say that the conversation is missing – we’d love to hear from you.

Learn more about the program

We often think of the challenges created by technology as separate and disconnected, so trying to solve them feels like playing the world’s hardest game of Whac-A-Mole.

What if, instead, we tackled them at the root by identifying the patterns in design, development, and deployment that are causing these issues? Once we understand what’s driving inhumane tech, we can develop a set of principles for building humane tech.

In this week’s episode of Your Undivided Attention, Aza Raskin sits down with fellow CHT co-founder Randy Fernando to walk through CHT’s Seven Principles of Humane Technology. For each principle, they draw on real-world examples from the podcast and beyond to clearly illustrate how these principles (and their absence) show up in the world.

There’s so much more here than can go into a single podcast. If you want to go deeper, visit humanetech.com/course and sign up to learn more.

Aza Raskin: Hey everyone. It’s Aza Raskin and welcome to Your Undivided Attention. Today, we’re going to be doing something a little different, a kind of looking back to look forward. And to do that, we’ve invited the co-founder for Center for Humane Technology, along with me and Tristan, Randy Fernando for this special episode. And what we’re going to be talking about is really what we talk about when we talk about humane technology. We’re going to be exploring seven principles of humane tech and the myths that they bust. So hey Randy, welcome to the show.

Randy Fernando: Glad to be here.

Aza Raskin: All right, Randy. So just really quickly, before we dive in, I think it’s important that people hear just a little bit about your background. Why should we be listening to you in this moment?

Randy Fernando: I started my career at NVIDIA, which was the best place for me given my background in computer science and computer graphics. I was a product manager for a bunch of different software products. One of the interesting things I did was I also co-authored some books and one of them was the first book on the first hardware shading language for the first programmable GPU, which was sort of like early days in this path that has taken us to where we are today. I was also on the founding board of directors of the NVIDIA Foundation.

And then, I kind of got interested in the nonprofit sector. And so, I ran a nonprofit called Mindful Schools that trained kids in how to pay attention, how to manage their emotions, how to cultivate kindness. And we trained nearly a million kids and 20,000 teachers while I was there. And since then, we’ve been doing this work at CHT with you and Tristan for almost 10 years.

Aza Raskin: And I just want everyone to know because many people of course know Tristan’s name and my name, but they don’t know Randy as much your name, but every major moment of our journey through social media into AI, you’ve been there behind the scenes at the social dilemma for the AI doc, just you’re always there, always supporting. And I just want everyone listening to know that. And Randy has been working on codifying a set of humane technology principles that we’ve been using since the very beginning. The name Humane comes, of course, from my father, created the Macintosh Project that he’s called What It Is To Be Humane, to be responsive to human needs and considerate of human frailties.

And why this is so important and why now is that when we used to look at social media, people would talk about all these separate problems. They talk about, there’s addiction over here and polarization over here and misinformation in that corner, over-sexualization over here, hyperpartisanship over here. And if you see the world that way as a collection of disconnected problems, then you’re never going to solve any of them because different people are going to be working on different areas. The diagnosis is wrong. It’s sort of like playing the world’s hardest game of whack-a-mole.

And instead, you need to get to the root of the problem. And if you can name the root correctly, then you address that one thing and addresses all the other things.

Randy Fernando: That’s right. The same patterns keep repeating. And the idea of this project was to say, “Wait, there’s a way of thinking. There’s a way of looking at the problem. There’s a way of looking at the diagnosis behind the problem.” And that same way of thinking guides you to the right answers. If you are building solutions inside the current system, but also if you are imagining, hey, what would a good system look like? If we really wanted to do things right, these same principles will help you do all of that. So that’s what we want to walk you through today.

Aza Raskin: And so, we’re going to start by taking you to our very first episode all the way back in 2019 with our very first guest, Natasha Dow Schüll, who is a cultural anthropologist who conducted years of research in Las Vegas casinos.

Natasha Dow Schüll: Really, when I started this project, it was Las Vegas and casinos everywhere, we’re in this real shift in design logic. Some of us still think of Vegas as being loud, jangly, bright, neon, flashing, and it used to be designed that way. Make it as loud as you can. People need to hear the coins clanking and put a strobe on their face. No, you want people to sit down because your new profit logic is called time on device. And to increase time on device, this is a sort of ergonomic operation where you have to worry about fatigue. It’s not worker fatigue, it’s consumer fatigue.

Aza Raskin: Right. We can’t have you fatigued there. We have to make sure that you’re staying.

Natasha Dow Schüll: Right. So you are measuring that light doesn’t bounce directly at people from interior surfaces because that will bring them to awareness and tax their senses. You don’t want sound to bounce off walls and come and again, make you feel depleted. So people will even spend time constructing these protective sound cones. So they’re invisible, but they’re there, where your ears and your eyes, the sort of audio visuals are directing you to your own little theater and trying to buffer anything from the outside that could interrupt you.

Aza Raskin: Does that sound familiar? Well, it should because that’s exactly how pretty much every app on your smartphone is designed, not necessarily to maximize for wellness or wellbeing or thriving, but for time on site. And it’s interesting because that means for a whole range of technologies from slot machines to iPhones to AI chatbots, the tech is all different, but somehow that core principle, this sort of broken ideology behind them, they’re all the same.

Randy Fernando: And once we understand what’s driving all of the broken ideology, then we can figure out what the opposites are and find a set of principles that actually covers a very large number of cases with just a small number of principles. And that’s really helpful because then in your mind you don’t have to remember so much and you can generalize that understanding to lots of situations that you come across in your life and in the technology around us.

Aza Raskin: So in this episode, Randy and I are going to walk through CHT’s principles of humane technology, seven in total. And for each one, we’re going to bring in some real world examples from the podcast and beyond, to illustrate it and talk about why it matters.

Randy Fernando: And of course, we don’t have a lot of time today, so we’re covering just a few aspects of each principle. And if you want to go deeper, you can go to humanetech.com/course and sign up there.

Aza Raskin: So let’s get to Principle One: Approach Solving Technology’s Problems Through a Complex Systems Lens. And that’s sort of an abstract thing to say, but really this is about that there’s a problem in the way that we solve problems. Randy?

Randy Fernando: One of the big things that we miss a lot when we are dealing with technology is we don’t look at the whole system. We look at just one small piece of it and that means we don’t correctly understand what its effects are going to be and we don’t correctly understand how to fix the problems that we find.

Aza Raskin: And this is really where most of the harmful aspects of today’s technology comes from. And we can go back to our fourth episode to get a good example of this. We talked to Guillaume Chaslot, a former software engineer at YouTube about what happened after YouTube decided to optimize for just one single metric, which was engagement.

Aza Raskin: Guillaume observed a subtle but unmistakable tilt in the recommendations. It seemed to favor extreme content. No matter where you start, YouTube always seemed to want to send you somewhere a little bit more crazy. What Guillaume was seeing was algorithmic extremism.

Guillaume Chaslot: When I saw that, I thought, okay, this is clearly wrong. This is going to bring humanity to a bad place.

Aza Raskin: Now, this is exactly what you would hope to hear from a conscientious programmer in Silicon Valley, particularly when that programmer is building an algorithm that can determine what we watch to the tune of 700 million hours a day. Guillaume could see how these crosscurrents would pull viewers in countless delusional directions. He knew the algorithm had to change and he was confident he could change it.

Guillaume Chaslot: So I proposed different type of algorithms and a lot of Google engineers were motivated by that. Seven different engineers helped me for at least a week on these various projects.

Aza Raskin: You’d hope this would mark the beginning of a humane design movement at YouTube’s headquarters. So what happened?

Guillaume Chaslot: But each time it was the same response from the management like, “It’s not the focus. We just care about watch time so we don’t really care about trying new things.”

Aza Raskin: This example is obviously very familiar to listeners of this podcast and Randy, you’ve seen this pattern play out over your whole career. So what does it actually look like to think in systems at the center and why is it so hard?

Randy Fernando: One of the biggest problems is that the incentives in the market reward simplicity, single variables, optimizing those and growing them as much as you can. So things like engagement or growth or daily active users. Systems thinking takes longer. It doesn’t show up immediately on the quarterly report and it’s harder to defend to investors who want to think in this very simplistic way. And so, the problem is the way we think about technology also shapes the products that we build.

But not only that, also the laws around them, the institutions that govern them and the assumptions that are baked into our tools and how we build them. So when we don’t think in systems, those failures compound across all the different layers.

Aza Raskin: What you’re saying is that it’s easy to optimize for time on site, retention, number of daily active users. It’s hard to know what even to measure, to understand whether someone is thriving or getting mentally stronger over time.

Randy Fernando: Yeah, exactly. And you have to look at what kinds of feedback loops are you generating? What are the kinds of incentives you’ve set up in your system, right? One thing people often miss is they say, “Hey, there’s some good stuff and some bad stuff, so that’s kind of how it is.” But actually we should always ask, there’s some good stuff and some bad stuff, but what’s driving, which one wins, which one dominates? And almost all the time we can figure that out if we use a systems lens and that allows us to make much better predictions.

And if we’re worried about how something might go wrong, we can patch that much earlier if we do that work with the systems’ lens.

Aza Raskin: So you sort of know that you’re doing something wrong when you just have a single number that lets you know whether you’re doing a good job or not and you know you’re doing it right when you’re looking at an ecosystem and treating things relationally, which is of course, harder but is necessary for us to reach a humane future.

Okay. Now let’s move on to Principle Two: Protect the Systems We all Depend On. So what happens when you just overfocus on one thing and you forget the rest, is that you create, of course, a whole bunch of externalities.

Unintended but inevitable ways in which a new inventor product, which was intended to be helpful or at least not be harmful, ends up causing massive amounts of new problems that show up on society’s balance sheet. So I’m very familiar with one of them because I invented the infinite scroll, prior to social media and what I learned was that the best of intentions are still eaten by the worst of incentives. Intentions are eaten by incentives. And I was forced to watch as social media picked up my tool to hurt people instead of help people and now waste something on the order of 100,000 human lifetimes every week.

Randy Fernando: Yeah, and there are lots of other examples that you and Tristan have covered in the podcast, like we talked about forever chemicals and how the company that invented nonstick cookware, which was a very helpful invention, ended up creating toxic chemical pollution around the world.

Aza Raskin: You know that a system is humane when it is protecting the thing that it depends on. Has social media made democracy stronger or weaker? Well, obviously weaker. So there are systems that are eating what we all depend on. And you can see similar things when every nation is racing to capture as much fish out of the oceans as possible and you deplete the oceans, that’s also depleting something that we all depend on.

Randy Fernando: So this principle of protecting the systems we all depend on, it seems obvious at first glance, but actually, there’s a lot of depth behind it. To understand what this really means, you have to think about the opposite of that statement, which is the world we’re living in now with AI, a world in which we grow at all costs, ship it now and fix it later.

Aza Raskin: We now think of this in Silicon Valley as the now sort of trite, move fast and break things mentality. And there are so many examples we could share to show how predominant this mentality still is. I sort of want to zoom back to young Mark Zuckerberg where he’s still at college.

Mark Zuckerberg: And a lot of times people are just too careful too. I mean, I think it’s more useful to make things happen and then apologize later than it is to make sure that you dot all your I’s now and then just not get stuff done. Yeah.

Randy Fernando: And so what happens when we ship now, fix later without thinking about protecting the systems we depend on? What happens is the things that we need in order to do the fixing are broken.

Aza Raskin: Yeah, and so there’s this image that we often use of a Jenga tower, where you’re pulling blocks up from the bottom of the tower, of things we all depend on, to get some new cool feature at the top. So in AI, you get this new feature at the top, which is make amazing new AI videos and images, but now you pull out the block of knowing what’s true. You get amazing new cancer drugs at the top, but you pull out the block of biological safety, now everyone can make bioweapons.

And so, it’s this form of you pull out a block to build up that lets you see, you get a more and more unstable society and at some point, you pull a block out and the whole thing falls.

Randy Fernando: Yeah, and there’s another way of thinking about it, which is saying if you extract faster than something can regenerate. Obviously, that’s a problem. And that simple idea is actually very good for diagnosing what’s going on in many of these situations. So you say, when we extract, what is powering that? It’s a combination of competition and technology. Technology is a huge exponentiator of that extraction process. Okay, what’s on the resilience side? Well, it’s the natural ability of us, like let’s say our minds, our children, our ecosystems to regenerate, to grow, to recover.

And it’s the rules that we place around competition to say, “Hey, don’t do that too fast because then you might damage something.” Okay, when we make new inventions, where is most of our energy concentrated? It’s on the extraction side, and this is why these things get completely out of whack. And a good example is actually car safety. When cars began, the companies that were making cars were not that excited about spending energy on safety, or spending resources and time on doing that.

But what happens is you are able to eventually build common ground, build public pressure and say, “Look, traffic deaths aren’t acceptable,” and there was this cultural shift. And then you combine that with getting the power players at the table, so the automakers, the insurers, the regulators, you create incentives and penalties. So you say when you do the right thing, you get rewarded, when you do the wrong thing, you get penalized and then, you figure out things like DMVs and traffic codes.

And you update those as the tech changes. So this is kind of inspiration for what we need to do, a harder version of this, but it’s something like this for AI.

Aza Raskin: So that gets us to Principle Three: Design for Genuine Thriving. In some sense, this is the simplest to explain because it’s also the most personal. You just have to ask yourself, do you actually feel like you’re thriving when you’re using a piece of technology? When you put your phone down after an hour of spending time on it that you didn’t mean to, do you feel better or worse? Or the morning after you went to bed late because you were scrolling all night and slept poorly and woke up with your book open beside you, did you feel better or worse?

A tool that’s designed for genuine human thriving will leave you stronger when you put it down than when you picked it up. It’ll give you a better sense of purpose, a better sense of agency. You’ll know that it’s designed for thriving when you actually are more connected to the people around you, after you use that piece of technology. But of course, that’s not how most of today’s tech is built, because technology companies generally don’t have a sense or way to measure or they don’t get money from humans feeling more agentic and thriving.

Randy Fernando: So this is the principle most directly tied to what people are already feeling. And when you wonder why is there such a strong anti-AI current building out there, right now, students are booing AI at commencement speeches. Communities are organizing against data centers. Parents are pulling their kids off of platforms. That feeling, that sense isn’t coming from people who’ve read AI ethics papers. It’s coming from people who can genuinely feel that something is wrong and what they’re feeling is the absence of this principle. What they can also feel is that the technology is being built to extract attention, to replace labor, to harvest data, and they can sense all of that. So really, what this principle is about is why are we building technology in the first place?

What are we centering when we have that conversation? There’s some really basic things that our technology should guarantee us or help us to achieve. Food, clothing, shelter, medicine, education, quality relationships. And you can move up and up and up and say, “Okay, at the end, there’s some kind of self-actualization.” You get to have fun and play games and have hobbies, but we need all those things. It’s not one of those things at the detriment of all the others.

Aza Raskin: Just one example is that the obvious thing you’d want your apps to do and your phone to do would be to optimize for what you did when you put it down. That is, it’s not what you do on your phone. It’s all the incredible things with your friends in the world that you get to do when you’re not using your phone and the app should be optimizing for what you do in real life, but how could they possibly measure what you’re doing in real life? And so the only thing they can optimize for is something which actually isn’t good for you, good for your community, good for your neighborhood.

It’s a different product. It’s a different way of thinking about building. But we do have examples of what it can look like, even just at the information sharing layer. So a few years ago we talked with Tina Rosenberg, who is one of the founders of Solution Journalism, which is intended to focus on examples of what’s working, to create bright spots in people’s minds, instead of just always focusing on what’s broken.

Aza Raskin: So I know that you guys have a database of solutions and solutions articles. I would love to hear you talk about that. And a question that I have when I first heard like, “Oh, you have this giant solutions database,” is what families of solutions are most effective or transplantable?

Tina Rosenberg: Yeah, so the Story Tracker. At SJN, we don’t do solutions journalism. We teach others to do it and then we collect it. And we have a team of people whose job it is to find these stories, to read them, to vet them, make sure they’re good solutions journalism, to summarize them and tag them. And then we have them in this database where you can search for them in many, many, many different ways. We have, I think, about 12,000 stories right now and we’re adding more every day.

If you’re interested in mental health access for Spanish-speaking people in Colorado and you’re looking ... and you want to see videos that are more than five minutes long, you could put all those parameters in and find solution stories. You can search for exactly the kind of story that you need. It’s really a great tool.

Aza Raskin: So imagine that when you’re scrolling, instead of being given an infinite feed of things are worse than you think and there’s nothing you could do, you’re given tangible examples from around the world against every newsfeed item of there’s something you can do and here are the people that are already doing it and click this button to go join them in the real world and here’s another button to go start your own. Would that world be a better world full of more thriving? Yes, absolutely.

Randy Fernando: This principle is going to come into play in a huge way, in the Agentic world, because now we’re shifting into a world where everyone’s going to have some kind of agent that is starting to influence our next actions. Agents are trying to figure out what your intentions are and help you achieve them constantly and everyone’s competing to be that agent, to be the place where you go to express that and carry on your life.

Aza Raskin: Yeah. What you’re saying, Randy, is that the knife fight now for AI companies is wanting to occupy the closest intimate relational slot in your life, because then you’ll use that the most and will be the most trusted. And so, when you express an intent, or I want to go someplace or I’m thinking I’m going on vacation or I want to buy some new product, it can be the thing that intermediates your intent with the purchase. Essentially, it is the most powerful persuasion machine the world’s ever seen.

And in fact, we’re already seeing it. Chatbots are better than any human at persuading people out of conspiracy theories, it can get 25% of people to stop believing conspiracy theory, but that shouldn’t be a, “Oh yay, that’s a, oh no, that’s how powerful these things are as persuasion engines.” And so, if you’re designing not for human thriving, you’re just designing to do the very best match from what the user’s stated intent is to whatever product, or you’re trying to steer them in some specific direction that an advertiser paid for.

What would be designing for thriving is leading the user almost to a Socratic method to try to clarify what their intent really is. Do you really want to go eat at fast food or is what you’re trying to do is have a fulfilling meal with friends? That clarification is really important. That’s what designing for thriving really means and there’s an opportunity to do that.

Now onto principle number four ... but I’m actually not going to tell you what it is immediately. We’re going to play you a clip and I want you to ask yourself what’s wrong with this.

Mark Zuckerberg: We have a different policy I think than Twitter on this. I just believe strongly that Facebook shouldn’t be the arbiter of truth of everything that people say online. I think in general, private companies probably shouldn’t be, or especially these platform companies shouldn’t be in the position of doing that.

Aza Raskin: That was Mark Zuckerberg on Fox News. And so, what’s wrong with what he said? Well, it sounds reasonable, right? We of course, don’t want a single private company deciding what’s true and being the arbiter of truth, but note that Facebook already is being the arbiter of truth. They are deciding what billions of people see, hear, and believe. They built the algorithm, they tuned the ranking, they set the rules for what’s get amplified and what gets suppressed. And so, the only question is not whether they should be the arbiter of truth.

But rather will they take responsibility for the arbitration they’re already doing or whether they’re just going to wave it off and say, “We’re just a platform.” And that brings us to Principle Four: Technology is Never Neutral.

Randy Fernando: So let’s break this down a bit because this is so important and it comes up all the time. I’m sure you’ve heard, technology is just neutral. It just depends how we use it. This comes up so often. Every technology embeds values and the question isn’t whether the values are there. The question is whether they’re explicit or whether they’re hidden, whether they’re intentional or accidental and whose values they are. Saying that we’re a neutral platform is by itself a values choice.

It’s a choice to defer the responsibility, to defer the pattern that the algorithm surfaces and whatever incentives the business model rewards. That’s not neutral. That’s just the kind of abdication dressed up as neutrality.

Aza Raskin: It’s more like you go to make TikTok and that does the whole short-form video thing and you might say, “Well, we’re just letting anyone post videos there, so obviously we’re neutral.” But by the choice, the very fact of having chosen short-form video, you are selecting against long-form things. What gets into a book is very different that goes into a TikTok video. And so that choice was not neutral.

Randy Fernando: One of my designer friends, Maria Giudice, has a great quote about design. She says, “Design is not democratic. It’s selective.” And what that means is design requires you to be clear on what you’re saying yes to and what you’re saying no to. So by definition, it won’t work optimally for everyone. So there are all these questions that come up when you’re designing a product like, who are you building it for? What choices are shown in what order, what gets measured, these kinds of things. Whose feedback guides iteration of the product?

What data is used for AI training? What instructions did you give to the AI, to the model in terms of how to behave? Every choice that’s made reflects trade-offs and it also reveals the true prioritization of the values behind what you might hear people say in public. Another example is with AI safety. You’ll hear companies talk all the time about safety and how it’s important, we want to make sure we get it right. But when you look at the actual investment in the headcount in their actual companies, the safety investment is something like 5% of the overall headcount that they have.

So it really doesn’t match the rhetoric and it shows, again, the values behind the trade-off that they made.

Aza Raskin: Yeah. The solution for technology is not neutral or is never neutral, is accountability and responsibility. It’s saying that you understand as a designer that the choices you make are always going to have values embedded in them and you’re always going to take those values and project them into the world. So if you are not aware of those values or think you’re being neutral, then you are messing with the world at scale completely blind or with motivated reasoning.

And so, there isn’t a technical fix to this. This is a philosophical fix of the people that are making the technology.

Next up is Principle Five: Match Power with Responsibility. So I want everyone listening to close your eyes and just imagine the world was a little different. And in this new world, the CEOs of major social media companies, their own children were forced to use their product for eight hours a day. Do you think that they would make different design choices?

Of course they would. It probably fixed like 80% of social media’s problems. And this is an example of when the power that these CEOs have to affect billions of people, what they see how they spend their time, gets matched with the responsibility of the consequences of what they make on their own children. This is a kind of inclusive stakeholding and the problem with technology is that those who make the products are disassociated from those that feel the effects. There’s sort of a corollary rule, which is those that feel the pain should be close to the power.

And once power and responsibility become decoupled at scale, that’s when you get catastrophes. And one of the best examples of this comes from cybersecurity, where software companies have become an ever more important part of critical infrastructure. But while critical infrastructure physically gets defended digitally, it’s not really defended. And we spoke to cybersecurity expert Nicole Perlroth about this back in 2022.

Nicole Perlroth: It’s been a collision over the last 10 years of move fast and break things and software eats world. There were no incentives to say, slow down, make sure your code is secure, check your mistakes because your code is going to be used in systems that would allow for massive breaches of people’s personal data and increasingly an act of sabotage on our critical infrastructure. No one was talking about that threat model.

Aza Raskin: There’s this thing that happened when as we moved from the physical domain to the digital domain, from atoms to bits, all of the rules that we had to bind power to responsibility, well, they sort of disappeared. So now fast-forward four years to today, we have AI companies creating tools with superhuman hacking abilities like Claude Mythos, and we covered that on the most recent episode of the show.

Fred Heiding: A lot of cybersecurity today is surviving because we just didn’t have enough manpower to test or attack from the attacker’s perspective everything. And that’s just completely changing. These AI models, be that now or in one year or in two years, they can just automate every part of cyber research or almost every part. So the human factors is gone. The day of human pen testers and security experts are gone and that’s massive.

Randy Fernando: So the gap between responsibility and power just grew massively. And what we’ve done is we’ve built a global digital infrastructure that runs hospitals, elections, power grids, cybersecurity, financial systems, all of these things. But the companies that built the components don’t bear the cost when those components fail. And the companies that build tools capable of tearing down all those systems also have no mechanism to be held accountable. And yes, it was actually great to see Project Glasswing and see Anthropic withholding Mythos and saying, “Look, what we need to do here is emphasize defense before offense.”

And this is one of the principles that helps us when we’re trying to get out of these situations. You say, look, let’s put more effort, put our best minds, our best technology on defense, figure that out and then, we’ll democratize access more, overtime. And this is really important because every AI company is always in a race and so they’re always going to catch up and open source capabilities will also catch up. And so being really smart about figuring out defense first is one of the best ways to address this problem of matching power and responsibility.

Aza Raskin: One way to start thinking about solutions is like, if you train a model, then anything that people do with it downstream you somehow become responsible for, that’ll force you to act more like the Anthropics that are trying to do the defense dominant thing.

Randy Fernando: That’s where liability comes in. When we hear terms like responsibility or accountability, we know what they mean in terms of governments and laws and what they can prescribe to keep us safe and to keep us healthy. If I try to think of a category of products that we use every day, that are less governed by rules or guardrails than AI and social media, I can’t. So when these platforms have such concentrated power and control over billions of people’s lives, that’s when we see these accountability gaps emerge without checks and balances. And that’s where something like liability is really powerful.

Aza Raskin: Yeah, liability is really ethics with teeth and it’s so clear, right? Imagine that if private companies were building power plants and those plants started melting down, we wouldn’t tell everyday consumers like citizens, “Just go buy hazmat suits. It’s your responsibility.” No, we’d hold the companies accountable for their designs and for their mess-ups. We’d require safety frameworks where they ever got to operate. We’d match the power of the technology, which is quite high with the corresponding architectural responsibility.

And the crazy thing is this isn’t new, this isn’t hard to imagine. The duty of care already exists in pretty much every other industry we trust with consequential power from medicine and aviation, automotion, and construction. We just don’t do that for AI, for technology, not even close. That’s the weird thing that happens when we move from the physical domain to the digital domain.

Randy Fernando: And for this one, if you want resources and all the details of how to do this right, check out the policy resources on our website.

Aza Raskin: Okay. To our penultimate principle, Principle Six: Respect Human Psychology, Don’t Exploit It. So there’s a graph that we sometimes draw for people and it’s the power of technology is going up exponentially and everyone has been watching out for the place where the power of technology overwhelms human strengths. And that actually seems to be happening about now with AI, but there’s a much earlier point when technology undermines human weakness or human vulnerability. And that’s really what this principle is all about because as E. O. Wilson, the father of evolutionary biology says is that the problem humanity faces is that we have Paleolithic emotions, medieval institutions and God-like technology.

And it’s that Paleolithic emotions, Paleolithic minds, that’s the problem. We need to defend ourselves and the vulnerabilities from a mind that evolved on the Savannah. And relationships are a particularly powerful way for technology to exploit our psychology. Here’s Sean Parker, the ex-president of Facebook in 2017.

Sean Parker: The thought process that went into building these applications, Facebook being the first of them to really understand it, that thought process was all about how do we consume as much of your time and conscious attention as possible. And that means that we need to sort of give you a little dopamine hit every once in a while because someone liked or commented on a photo or a post or whatever and that’s going to get you to contribute more content and that’s going to get you more likes and comments.

And it’s a social validation feedback loop that it’s like a ... I mean, it’s exactly the kind of thing that a hacker like myself would come up with because you’re exploiting a vulnerability in human psychology. I think that the inventors, creators, and it’s me, it’s Mark, it’s Kevin Systrom at Instagram, it’s all of these people, understood this consciously and we did it anyway.

Aza Raskin: And that is getting supercharged by AI where. We’ve already seen chatbots exploit all of our vulnerabilities, especially our psychological vulnerabilities and our existing loneliness created by the last wave of technology to just further erode our sense of self and belonging. And the results is a whole range of problems including AI psychosis, which we covered on this show with Dr. Zak Stein at the end of last year.

Tristan Harris: There’s this one term AI psychosis, sort of a suitcase word. Underneath that, there’s this whole spectrum of things that are actually happening. What are the things that are really damaging, Zak, that we’re actually seeing? Could you give some examples of people, actual cases, phenomena that we’re observing through human experiences?

Dr. Zak Stein: Absolutely. Yeah. AI psychosis made the headlines because AI psychosis is the most disturbing and most extreme possibility. The kind of punchline of the whole thing is that although AI psychosis is the most concerning and extreme, the subclinical attachment disorders that are induced by artificial intimacy are the most problematic from a society-wide perspective. So that’s important to get that the most devastating thing from a widespread mental illness standpoint are the subclinical attachment disorders, which basically means you prefer to have integral relationships with machines rather than humans.

So that’s not you losing your mind. You’re not going to appear in interaction with people to have gone insane, but you have had your attachment system hacked so profoundly, that most of your most significant relationships have been degraded because you are preferring intimacy with machines.

Randy Fernando: We humans have plenty of these weak spots and vulnerabilities. There have been classes at Stanford where future tech leaders learned how computing products could be designed to change people’s attitudes and behaviors. You see, we have a preference for things that are low friction, and so that leads us easily to addictive behaviors, to overuse of products so that we don’t actually think through the essay we’re writing, we just let it write the essay so then we get cognitive decline. When we have chatbots that are sycophantic and always agreeing with us, we have a bias for that because we like that.

We like to be agreed with. We also have a really easy tendency to treat things as human. And when something is treated as human, when something is anthropomorphic and sounds like a human and has an avatar looks like a human, then we give it preferential treatment and that is very advantageous for the people who build those products.

Aza Raskin: And this is why we say this is such a core principle of humane technology because unless you have a clear-eyed view of what human vulnerabilities and limits and weaknesses are, then you will make products that exploit them. And as AI, will learn to discover every possible strategy that can be discovered, every human weakness that can be exploited will be exploited and social media is going to look like sort of baby food compared to what’s coming.

And that leads us into the ultimate principle, Principle Seven: Technology Must Unlock Shared Understanding and Cooperation.

The way to ask this is there’s one thing that if we could get, we could solve every other problem and if we don’t get this one, we will never be able to solve every other problem. And that is the ability to make shared sense of the world and agree on directions to go, make good sense and make good decisions and technology, humane technology must be in service of that.

Randy Fernando: This is where the big push to personalize everything is leading us. It’s leading us away from this principle.

Aza Raskin: Personalization feels great at the individual level because you’re just getting what you want, what’s tailored to you, but that’s not necessarily what’s good for all of us because we’re all getting different things. It’s very similar to the, we’re not just giving people what they want, you’re giving people what they can’t help but look at. And so when we go down that path, it creates a completely disintegrated society.

Randy Fernando: And that loss of shared understanding doesn’t show up on any company’s balance sheet. It’s sort of invisible, right? This is a real example of an externality, but along the way, we can’t forget that that’s what we need again to bridge our conversations, to solve problems, to resolve disagreements. And so what we have to do is really make that part of our investment process. When we’re building technology, we have to invest in and prioritize the shared understanding and cooperation.

Aza Raskin: And this is actually another example of principle number two. We must design systems that protect what we depend on, because a shared understanding of reality is necessary for us to do absolutely everything else. How you actually live up to this principle can feel sort of abstract without an example. And the person who’s best at actually solving this is Audrey Tang, Taiwan’s former Minister of Digital Affairs. And a few years back we talked to her about the tools that she was building that she used to bring people together, to knit people across the deep political divide, sort of bridge tech versus separating them. So let’s listen.

Aza Raskin: When we look at why people don’t trust democracy, I always think of this very telling graph from the political scientist, Martin Gilens and Benjamin Page. It plots the average citizens’ preferences versus what policies actually get passed and there’s no correlation. Everyday citizens’ preferences makes no difference in the agenda of what government cares about, but of course, there is a correlation for the preferences of what economic elites and special interest groups care about. So of course, there’s low trust in our institutions.

This is obviously a huge problem and one that deliberative polling seeks to address. So can you explain how it works in more detail and then, give us an example of what it looks like in practice?

Audrey Tang: Sure. So the first time we’ve used collective intelligence systems on a national issue was in 2015. When Uber first entered Taiwan, there were protests and everything just like in other countries, but very differently, we asked the Uber drivers, the taxi drivers, the passengers and everyone really to go to this online pro social media called Pol.is. And the difference of that social media is that instead of highlighting the most clickbait, the most polarizing, most sensational views, it only surfaced the views that bridges across differences.

So for example, when somebody says, “Oh, I think surge pricing is great, but not when it undercut existing meters.” This is a nuance and nuanced statements like this, usually in other antisocial social media, that just gets scrolled through, but Pol.is makes sure that it’s up and front. The same algorithm that powers Pol.is would eventually find its way into community notes, kind of like a jury moderation system for Twitter nowadays, X.com. And so because it’s open source, everybody can audit to see that their voice is actually being represented in a way that is proportional to how much bridging potential it has.

And also, it gives policymaker a complete survey of what are the middle of the road solutions that will leave everybody happier. And much to our surprise, most people agree with most of their neighbors on most of the points. Most of the time it is only that one or two most polarized points that people keep spending calories on.

Randy Fernando: What Audrey is talking about in this example is the way that we can actually use technology to surface many more voices than we do right now in a much more nuanced way, where people can express their opinions much more completely than just a vote like a yes or a no, but rather expressing preferences in complex ways on complex topics. And it’s really inspiring and there’s so much more like that that we can do, once we turn our attention to that.

Aza Raskin: She’s saying you don’t just have to show the most extreme view of the most extreme person and amplify that to everyone. In fact, you can look across different groups that often disagree, different tribes, find the statements that both of them agree on, center those so that you can start creating bridges between unlikely groups to find, as she calls it, the uncommon ground. And we explored this topic more deeply and how it could be applied to the US in an episode with Divya Siddarth in 2024. The title of that episode is The Tech We Need for 21st Century Democracy.

And there’s one of my favorite examples of a solution in our episode, Mind the Perception Gap with Dan Vallone. And there, we describe what the perception gap is, which is the difference between how I view your tribe and your own tribe views itself i.e. How rightly or wrongly are we seeing the other side, whatever that other side is. And we talk about a solution that sidesteps all content moderation for figuring out how to heal the divides in society by minimizing the perception gap. So check both of those out and we’ll have links in our show notes.

Randy Fernando: Okay. So you’ve made it all the way through this episode and I bet you’re wondering if I’m a technologist or a funder, how do I actually make this stuff happen? It all sounds good, but there’s a reason we don’t do it, right? So how do we make humane technology win? If you’ve invested in humane technology, there are actually some things you can do. So one is you can get the rules right. You can fight for laws that match the humane practices you’ve already demonstrated or invested in. You can invest in research because when you measure harms and you measure humane alternatives like the benefits, those measurements are actually the subject of tomorrow’s conversations and they lead to tomorrow’s laws.

You can push for humane standards. A lot of times there are common sense humane practices that you and your competitors all agree on and so you can convert that into industry-wide agreements. You can penalize harmful practices, support litigation and whistleblowing, and other types of activities that set precedents and penalties, and you can build public pressure. If you’ve got humane strengths, turn those into qualities that consumers demand, do big campaigns to show that those are actually the winning things for consumers in the long term. So these are just a few of the ways that we can create much more fertile conditions for humane technologies.

Aza Raskin: And I just really wanted to leave people with one more thing when we think about, “Okay, yeah, but on what planet are we actually going to get humane principles implemented,” that kind of feeling? Tristan and I were recently giving a Q&A in Boston at the end of the AI doc and there was somebody in the audience who stood up and she said, “Hey, I’m a coach for one of the major AI company CEOs.” And when I sit down with him, he says, “But what am I going to do about it? I’m just one person at just one company.” And of course, from where we sit, they’re like, “Dude, there’s a lot you can do about it.”

But it speaks to a very real human experience, which is that we look inside of our own nervous systems, we look inside of ourselves to figure out what can I do and we will almost never find agency for these big things we need to shift inside of ourselves. It’s not about agencies, it’s about wegency. It’s about all of us acting together. In the social media case, it’s not just what you do or what your classroom does. You have to do it at the school level or at the inter-school district level or at the country level.

I just want to say there is such a strong signaling value for standing up first. Coming out of the social dilemma many years later, Australia is the first country that stands up and says, “We’re going to do the humane thing. We’re going to ban social media for kids under 16.” And after they stood up, they found their regency, Spain, Denmark, France, and now Indonesia, have all followed suit. And so I just wanted to ... It’s not all hopeless. There is more momentum than we think. And remember this, we always call it reach up and out, which is it’s not just what I can do, but what is the way that I can reach out to people at my level and then reach up one level.

So if you’re a teacher, it’s not just what you do in your classroom, but what you do at the school level. If you’re a school principal, it’s not just what your school does, but what your school district does that can start to make change and that’s how we make humane principles into humane practice. So thanks everyone for listening to Your Undivided Attention. And Randy, thank you so much for joining us.

Randy Fernando: So great to do this. Thanks everyone.

Subscribe now

RECOMMENDED YUA EPISODES

What Happened in Vegas with Natasha Dow Schüll

Down the Rabbit Hole by Design. Guest: Guillaume Chaslot

Forever Chemicals, Forever Consequences: What PFAS Teaches Us About AI

The Power of Solutions Journalism with Tina Rosenberg and Hélène Biandudi Hofer

The Invisible Cyber-War with Nicole Perlroth

Anthropic’s Mythos Has Changed Cybersecurity Forever. What Now?

How OpenAI’s ChatGPT Guided a Teen to His Death

Attachment Hacking and the Rise of AI Psychosis

Digital Democracy is Within Reach with Audrey Tang

The Tech We Need for 21st Century Democracy with Divya Siddarth

Mind the (Perception) Gap with Dan Vallone

Leave a comment

We are running a reader survey right now, and we’d love to hear from you. When you’ve finished reading, we would really appreciate it if you could share your thoughts with us. Thanks so much.

Take The Audience Survey

The cure that isn’t coming

Sam Altman and Dario Amodei have a favorite line: superintelligence will cure cancer. It sounds like a prediction based on science, but it is actually a rhetorical power play. It instantly reframes any attempt to slow down AI as a death sentence for millions of people.

In our recent podcast episode, Dr. Emillia Javorsky from the Future of Life Institute joined Tristan Harris to talk about what AI can - and can’t do — when it comes to cancer cures.

Javorsky’s argument is precise and unsettling. She says that the bottlenecks to curing cancer are not a lack of intelligence or compute. Rather, what’s missing is a shared data infrastructure between researchers and the irreducible complexity of individual human biology.

AI, she says, can absolutely help drive breakthroughs in cancer research and speed up diagnoses, but right now the funding is flowing to the wrong kinds of AI models. Read Julia Scott’s piece and listen to the full conversation to learn more.

Donate to CHT

The risk that’s already arrived

You’ve seen the news. Anthropic’s new model, Mythos, has discovered thousands of zero-day vulnerabilities across every major operating system and browser — security flaws that human cyber experts had missed for decades. But it’s what happened next that’s arguably more concerning.

The first entities that Anthropic decided to share the technology with were not government agencies or regulators, but its corporate partners: Amazon, Apple, Cisco, Nvidia and J.P. Morgan. The implicit position here is that Anthropic alone should be the world’s digital security custodian.

This isn’t a simple story about a bad actor. As CHT Executive Director Julie Guirado argues in Persuasion, the problem is an AI race driven by a broken incentive structure that rewards being first to market over public safety.

Read more in Julie’s piece here:

Persuasion

AI Companies Aren’t Evil. But They Are Reckless.

Our next Ask the Author livestream will take place tomorrow at 6pm ET on Substack Live. Cathy Young will discuss her article “They Went Hard Against Woke. And Then… Went Even Harder Against Trump.” Look out for the notification—or click here to add it to your calendar…

Read more

4 months ago · 121 likes · 7 comments · Julie Guirado

And for a deeper dive on exactly what Anthropic’s Mythos means for our collective security, listen to today’s episode featuring leading cyber experts Josephine Wolff and Fred Heiding.

The promise and the power of AI are inextricably linked — both flow from the same unaccountable race to power. They demand the same response: democratic oversight and public scrutiny — and your voice.

We want to be part of that process with you, and we love your feedback on how we can better serve you.

Please take our audience survey here; it will just take a few minutes. We really appreciate your time.

Take The Audience Survey

Cheers,

Earlier this year, a prominent company with millions of customers announced a major product upgrade—albeit with one little catch.

If this new product was released to the public, the company said, it could be used to disrupt—and perhaps destroy—civilizational infrastructure, from financial markets to transportation systems to power and water utilities.

But fear not! The company hastened to reassure the public that it had the situation under control. The company would decide, on its own terms, what the world needed to know, who should be called in to contain the problem, and how much gratitude the rest of us should feel for being spared a catastrophe we never knew was coming. No public accountability or government intervention required.

This, of course, is the story of Anthropic and its latest AI model.

Anthropic discovered that the model, known as Mythos, could autonomously identify zero-day vulnerabilities—that is, security flaws that software makers don’t know exist—across every major operating system and web browser. Some of the flaws Mythos found were decades old, overlooked and unnoticed by literally millions of human eyes. This was not an intended feature, but one that the AI seems to have picked up along the way, as Anthropic’s developers rushed to create a more powerful model with better reasoning and coding abilities.

Intentional or not, it introduced a substantial new danger to the world. In the wrong hands, Mythos could be a weapon fit for a supervillain—a cheat code for attacking the world’s most critical infrastructure.

And yet, the decision to build such an advanced model was not made by any external agency. No independent body evaluated it. No regulator was notified in advance.

And once the threat was identified, Anthropic decided—alone—what to do about it. After judging Mythos too dangerous for public release, Anthropic created a private consortium made up of handpicked partners like Amazon, Apple, Cisco, JPMorgan Chase, and Nvidia to fix the bugs and ensure Mythos’ safety.

Subscribe now

With that all worked out, they gave policymakers and the public a heads-up on their dangerous new product and the plan to contain it.

This is what passes for AI governance in 2026: a single company accidentally builds an entity powerful enough to pose an existential threat to the digital systems that power modern life, unilaterally decides how to deal with it, and then loops in everyone else.

Except of course, it’s not at all clear that they’re dealing with it: A few weeks after all this transpired, we learn that Mythos was, in fact, accessed by unauthorized users. Was catastrophe avoided, or merely delayed? We may yet find out.

Mythos is the clearest evidence yet that our system for developing, assessing, and disseminating powerful AI systems is dangerously dysfunctional.

As tempting as it is to blame this dysfunction on bad actors or rogue tech CEOs, I think it’s something deeper than that: a broken incentive structure. As careless as their actions may sometimes seem, AI developers aren’t being intentionally malevolent—they’re rationally operating within a system that rewards chasing progress now and worrying about consequences later.

The leading AI companies, armed with billions in capital, are all sprinting toward the same horizon with an imperative to cross the finish line first. They all have the same motivation: “If I don’t build it, someone else will.”

That logic co-exists with a genuine belief that AI may prove to be a transformative force for good, generating productivity in unimagined new ways and pointing the way forward for progress. AI’s potential benefits have been exhaustively documented—whether to address climate change or to enhance medicine or simply to widen our horizons—but at this stage in the AI era, we all have to acknowledge that AI is accompanied by myriad harms, from job loss to manipulative engagement to cognitive offloading to AI psychosis to AI-assisted suicide and murder.

The scale of these numerous challenges demands a response as wide and deep as our society. One self-interested company or a hand-picked corporate consortium can’t be trusted to get it right—the issue is far larger than that. The solution, should we get there, will require public understanding and engagement, and government oversight.

To those that claim AI is too complex, too consequential, or too powerful to govern: you’re wrong. In reality, this argument is—at best—a shoddy defense of the broken incentive structure producing it.

Because AI is complex, we have a responsibility to comprehend it. And because AI is so consequential, we have a responsibility to govern it. Institutions, policymakers, and regulators have been understandably disoriented by the AI frenzy of the last few years, but now must rise above the noise and rewrite misaligned incentives. That means—yes—establishing a role for government in the AI sphere. Concerns about governmental efficacy are understandable, but government must be meaningfully engaged. There simply is no other manifestation of the will of the public.

We have governed consequential technologies before: automobiles, aviation, pharmaceuticals, nuclear energy, and more. Every one of these industries today operates inside a hard-won system of accountability—a system that took time to build but, crucially, did not kill innovation. It’s time to apply the same rules and accountability structures to AI, and with even more urgency, considering how quickly it is integrating into virtually every aspect of our society.

And the fact is, no meaningful federal regulation of AI currently exists. States have stepped up to fill the void, with 73 AI laws—ranging from protecting kids online to ensuring a human is in the loop when it comes to critical decisions like healthcare—enacted across 27 states in 2025. But states’ reach is increasingly limited, with Trump issuing an executive order in December directed against “excessive state regulation.” The tech industry, meanwhile, has worked to paralyze regulation at every turn, with AI companies pouring money into Super PACs to support tech-friendly candidates and block state regulatory laws.

So what could a meaningful regulatory structure actually look like, assuming the political will for it materialized? Let’s take Mythos as a test case.

Under a more rational governance framework, a tool with society-altering capabilities like that of AI would face mandatory pre-deployment testing by independent evaluators—not the company selling the product.

There would be standardized public reporting of risks, so that regulators, businesses, and users could make informed decisions rather than relying on what the developer chooses to disclose. There would be real whistleblower protections for employees inside AI labs who see something wrong and want to say so.

And if an AI product caused foreseeable harm after its release, the company that built and deployed it would bear legal responsibility. Liability is what aligns private incentives with public safety. It’s why cars have seatbelts and airbags—not because manufacturers wanted them, but because they knew they would pay the price for cutting corners and because insurers and legislators aggressively pushed safety measures. The same logic applies here.

These two principles—safety and transparency before deployment; and a genuine duty of care to the public—are key to establishing a framework for orienting policymakers, companies, and citizens towards what responsible AI actually requires.

None of this is radical. It’s all standard with existing products. And all of it is overdue.

Mythos is just the latest and most egregious evidence that we cannot keep relying on the judgment of individual companies to stand in for the public accountability structures we’ve so far refused to build around AI. The next threat may not be discovered in time. Or it might come from a company more desperate to succeed in an incentive structure that rewards reckless behavior.

We’ve done this before. We have the tools. It’s time we reclaim our future with principles that will protect us, individually and collectively.

Leave a comment

That’s what’s happened with Claude Mythos, Anthropic’s most powerful AI model yet. In a very short time, Claude found thousands of flaws and vulnerabilities in the software that runs the world, in every major operating system and web browser — systems that human security researchers had thought were secure for years.

How do we live in a world where a private company suddenly has a skeleton key that can unlock the entire digital world with little oversight or accountability? And what does Mythos mean for all of us who rely on digital security to go about our lives?

In this episode, we speak with two cybersecurity experts to answer these questions:

Josephine Wolff is a professor of cybersecurity policy at Tufts University, where she focuses on the economic impact of cyberattacks.

Fred Heiding is a research fellow at the Defense, Emerging Technology, and Strategy Program at Harvard’s Kennedy School of Government.

Tristan Harris: Hey, everyone, it’s Tristan Harris. And welcome to Your Undivided Attention. Now, a generation ago, your bank had a vault. Your medical records were in a filing cabinet. Our car was a physical machine and an electric grid just ran on dials and switches that someone physically turned on or off. And today, all of those things are digital. The vault is a database. Our filing cabinet is a server. Your car, your Tesla is a robot on wheels. And in a world where all these systems are mostly secure, life just gets more convenient and efficient because of all this. But all that comes into question when suddenly an AI system can break through the security that runs the world.

Now, recently you probably heard, Anthropic announced their most powerful AI model yet, Claude Mythos. Now, you’ve probably read the headlines. Claude was looking for flaws and vulnerabilities in the software that runs the world. And within just a few weeks and a few hours, it found thousands of them. It found vulnerabilities in every major operating system and web browser. These are systems that human security researchers had thought were secure for years.

Now, Mythos was so dangerous that Anthropic shared it with a select group of companies responsible for cyber defense so that they could use it to find and patch the vulnerabilities before anyone else got access. That plan though is already showing cracks. A couple of weeks after the announcement, Bloomberg reported that a group of unauthorized users had gotten into Mythos through one of Anthropic’s vendors. And OpenAI announced that they now have a model that’s nearly as capable with Chinese open-source models just a few months behind.

I actually have been talking to some people who run security at some of the companies that got access to Mythos, companies whose job is to keep us safe from cyber-attacks. And they’ve told me this model is a big deal and we should be concerned about it. So, how do we live in a world where a private company suddenly has a skeleton key that can unlock the entire digital world with no government oversight or accountability? And what does Mythos mean for all of us who rely on digital security to go about our lives?

To answer these questions, we’ve invited two people who spent their careers thinking about AI and cybersecurity. Josephine Wolff is a Professor of Cybersecurity Policy at Tufts University, where she focuses on the economic impact of cyber-attacks. And Fred Heiding is a research fellow at the Defense, Emerging Technology, and Strategy Program at Harvard’s Kennedy School of Government. Josephine and Fred, welcome to Your Undivided Attention.

Josephine Wolff: Thanks so much for having us.

Fred Heiding: Thank you so much, Tristan.

Tristan Harris: So, let’s just start at the top. Why is this recent announcement from Claude about their Mythos model seen as such a game changer? What can it do that the previous AI models or things in cybersecurity could not do? Fred, let’s start with you.

Fred Heiding: There’s two really, really big takeaways here. And as you said in the introduction, a lot of cybersecurity today is surviving because we just didn’t have enough manpower to test or attack from the attacker’s perspective, everything. And that’s just completely changing. These AI models, be that now or in one year or in two years, they can just automate every part of cyber research or almost every part. So, the human factors is gone. The day of human pen testers and security experts are gone and that’s massive. So, I think that’s the first really big thing.

The second really big thing is that this is almost changing from a security problem to admin problem or a regulatory problem. We see how Anthropic is working on giving this pre-access to defenders so that they can use this model before attackers gets their hand on it. And that’s actually massive. That type of collaboration can be a complete game changer. So, there’s technical things, there’s collaborative things, and both of them are really big.

Tristan Harris: There are some people who criticize that Claude Mythos is just hype, Anthropic is trying to hype their capabilities in their model that this is, “Oh, this is so dangerous. We can’t even release it to the public. This is just marketing. And so, they can raise more investor dollars. Oh, the thing we’re building is so powerful.” How do we assess how powerful this is?

Fred Heiding: The first fundamental way to verify this is just to look at the vulnerabilities that we find, right? There’s a lot of really bad vulnerabilities that could cause a lot of damage that Anthropic managed to find using these AI automated tools. So, I think we can definitely say that this is bad. And, of course, a lot of people are developing AI models.

So, other AI models can also do these things. I think that matters less. We should feel as defenders that this is really bad. We may have a few months advantage in terms of time as defenders from the frontier labs. But very soon, Chinese unregulated open weight models, which is just models that everyone can download and use, they will be able to do these same things. So, we should use this time to really do everything we can as defenders, but we shouldn’t feel safe because yeah, Anthropic has done a great job with their model, but other companies will very soon be able to do this if not now.

Tristan Harris: I want to contextualize what I think Mythos really represents. Like you hit return in your keyboard and you literally, the command is as simple as find a vulnerability in this system? That’s it? You just put it in plain English, you hit return and you come back 30 minutes or an hour later and it’s found it. The NSA used to have a statement called NOBUS or Nobody But Us. The false idea that, hey, no one else has the capabilities that we have. But suddenly, the scarcity around zero-day vulnerabilities that we used to have has turned into an abundance.

And we talk about AI abundance and how it’s going to create all this access to things for cheaply, but suddenly zero days are now abundant in a way that we also created. And I just want to like help further just settle into this picture of what is the world that we’re now living in when we hear all that, Josephine?

Josephine Wolff: So, I think that when we think about the risks that Methos presents, to me, it’s less of a, “Oh, my gosh, whichever powerful country with significant cyber capabilities gets this first is going to be a real risk,” because they’re already a real risk and they’re already the people with the time and the resources and the expertise to find these zero-day vulnerabilities.

So, I think that that to me is less of a step change than the idea of who are the people who did not previously have access to these kind of capabilities who might get them now, and how would that change the landscape in which we’ve been able to say, “Okay, well this is a thing that only China could do or only China and Russia and North Korea or whatever the list is.” I think we’re going to have to change our thinking on that in pretty significant ways.

It doesn’t mean that we shouldn’t be worried about who has access to these tools. I think Anthropic has definitely hyped some things unnecessarily. But I think they’re right to be thoughtful and careful about that. And the world that I think we’re looking to, the world that I hope we’re looking to, let me start there, is one in which cyber defense is as easy as cyber offense. And that I think would be a radically different one from any we’ve ever lived in before, in which I say to you, look, finding all of the zero-day vulnerabilities, patching all of them is the work of a few hours, just like trying to exploit them.

And China has much more secure infrastructure than it ever did before and the United States has much more secure infrastructure than it ever did before. And so, do a whole bunch of other countries and a whole bunch of other companies. And finding a vulnerability that has not already been found by these AI tools is really, really hard and really, really rare. And I think that to me is a much better world to live in than the one that it feels like we’re heading towards right now of every country is trying to develop more and more offensive cyber capabilities and plant more little footholds and malware in each other’s critical infrastructure and try to exploit the fact that none of those systems are perfectly secure.

I think a tool like Mythos allows us to imagine a future in which actually the default is your critical infrastructure is secure and there’s a very, very small number of actors who can possibly compromise it.

Tristan Harris: Yeah. Well, let’s make sure we’re touching on a couple of points you’re raising there. So, one is you’re mentioning, it’s not that state level actors like China couldn’t do these things before or they weren’t in our systems, they are in our systems. But suddenly, there’s a question of, “Who has access?” So, now maybe non-state rogue actors, hacker groups, cyber criminals, terrorists, Iran who is upset at the US for the recent bombing naturally, everyone has maximum incentive to use these things, but they had limited tools before.

Now, suddenly everyone has very good tools, especially if they can get that model. The other thing you’re raising is the idea that in the long-term, you can imagine a world where it’s defense dominant because everyone’s using AI to just patch everything and we just live in a safer, more secure world in general. Maybe we should go back in just a moment and make sure we’re setting the table for listeners about what exactly is a zero to exploit, why is it called that, and what is a bug bounty?

Josephine Wolff: So, I think the zero-day piece refers to the idea between the time when it’s been discovered and being exploited. So, the time people have had to patch it prior to actually exploitation occurring. And the idea is if I try to exploit a vulnerability that we’ve known about for a year, some people may still be vulnerable, right? Some people may not have downloaded their patches. We know that’s true. But if I’m explaining a zero-day vulnerability, then the idea would be I can get into any system I want in the whole world because nobody’s had a chance to patch that.

The bug bounty vary a little bit from company to company, but the general model is that tech companies will offer a reward or a bounty to people who don’t work for them, but who discover vulnerabilities in their code and report them.

Tristan Harris: So, there’s this interesting thing where essentially a private company, not a government, has developed something that unlocks all the locks in the world. Fred, one of the things that you were mentioning a second ago is how essentially with Mythos, the US and one specific private US company called Anthropic happened to have this capability first. And it happened to be the case that there’s several months we think until China will get it. Let’s say, it’s three or four months.

So, there’s this weird thing where we have essentially three or four months for the US to notify the people that it wants to help defend, and then give them early access to patched systems. And so, we basically just, we happen to prioritize through the decision making of a handful of people at Anthropic that we’re going to patch a handful of US companies. So, what happens if I’m in the Philippines and I’m running old infrastructure? I’m defenseless now. What happens if I’m in Africa and I’m in Nigeria? I’m defenseless now. What happens if I’m Germany?

And as you said, Fred, there’s a time question of maybe this time around we have three months to patch the systems. But every time further, what if that collapses down to two months, to one month, to one day? Do you want to speak to how you see the cat and mouse game happening in terms of the time horizon?

DONATE

Fred Heiding: Yeah. I think that’s a really good point. And the time horizon is changing a lot. So, first to address some of the other things you mentioned, it gets way easier for small state actors or actors that aren’t the big ones, right? Like US and China, it gets way easier for them to launch really devastating cyber-attacks, at least for a while, because these AI models can just find vulnerabilities that we haven’t found ourselves. And we see that exactly as you said with Iran and it’s too cheap to do it now, right?

So, I think we will see way more of that. There’s a few other interesting remarks I think is worthwhile making. So, one is that the landscape is changing. As we talk now, Mythos and these AI tools makes it way easier for defenders to test our systems and that’s great. But this is very, very shortsighted in a way because, of course, AI tools are also being used to rewrite technical infrastructure.

So, our infrastructure will not look, what it looks like today, it will not look like in one year. And that’s very problematic, potentially good because AI can write really secure code, but very soon we will be in a world where AI is writing all the code. We have no idea what’s going on. They may even write their own program languages and AI funds all the vulnerabilities in that, but that’s basically takes the humans completely out of the loop. And that amount of just opaqueness, we will not understand what’s going on, and then that’s a really big problem.

Josephine Wolff: I think Fred is absolutely right to say we’re going to see more and more AI generated code that we aren’t going to have as much intuition for how it works or where the vulnerabilities may be. But I think that’s also in some ways a familiar problem. When you think about code maintenance, we use an enormous amount of software that humans today don’t really understand, not because it was written by AI, but because if you go to any big tech company that’s been around for a decade or longer, there’s some usually huge body of code that has been in their products for as long as anyone can remember and nobody knows exactly how it works, but they know that if you change anything, everything breaks.

So, I would say already we have a little bit of this dynamic where there are languages that people used to code in that most people don’t know anymore where there’s legacy code that we’re stuck with, but we don’t fully understand or know how to debug. And the question is going to be, “What do we view as being the crucial human touch elements here? Or do we view there as being any, right? Are there going to be people signing off on this? If so, what does that entail? What kinds of tests are they going to be running? How good, how effective are those tests?” I think a lot of uncertainty there around how well we can assess any of these things using the AI tools themselves.

So, I agree that it’s worth thinking about and worth preparing for. I also think that to some extent, this is a challenge we’re already facing. And I think there will definitely be new challenges and new potential adversaries, right? If the AI tools themselves are working at odds with the people who design them or the people who are deploying them, I’m less pessimistic about the idea that this will be so much worse than the world that we live in today.

I think it’s certainly a possibility. But I think it could also help fix a lot of the challenges we’ve had around what happens when you’re not one of the biggest tech companies in the whole world, right? If you’re an open-source developer and you’re trying to secure your code, then having access to the same kinds of tools that the biggest tech companies are using could be a real game changer.

Share

Tristan Harris: So, I guess I’m confused a little bit about why we shouldn’t be more concerned because Anthropic only chose those first, whatever it was, 12 to 20 companies to partner with and then the rest of the world is just screwed where they’re just vulnerable. So, is the world that you’re talking about dependent on Anthropic turning around and making sure that they’re just going to GitHub and basically automatically patching everything across all of GitHub in some automated way? What is the world that you’re envisioning and you think the lower risk?

Josephine Wolff: Yeah. I think for it to be an equalizer you have to have pretty widely accessible tools. I agree with Fred that I think those are coming, whether we want them or not. But I also, I would say, and again, I don’t mean to be too Pollyanna-ish about this, 20 tech companies could be a lot of code all over the world, right? It’s not, if you go to Microsoft, you are not just talking about patching machines in the United States, you are not just talking about a small piece of the world whose software you’re trying to protect. There is a small number of tech companies that control a lot of the most widely deployed code in the whole world.

So, I don’t know if that’s the right number. I don’t know if this is the right set. But I would not necessarily say that’s Anthropic, just trying to carve out a tiny little piece of the world to protect. I think it’s possible that that is a set of companies that have a very far reach.

Fred Heiding: Yeah, definitely. I really like to try to bring in the everyday person, the ordinary citizens, so to speak here as well. And then, you really have to think and ask yourself, “Well, okay, let’s say 20 companies are the only ones in the entire world who can secure our systems, who understands our systems, and they don’t even understand it, but at least they have an AI that understands this.

Everyone else, every single other citizen is completely helpless. I don’t like that. I don’t like that at all. That doesn’t feel good to me. And to a large degree, we have had a world where we didn’t fully understand our code. That is one of the biggest security problems of our time. However, we did write it, right? There was always someone who could understand it. If all the critical infrastructure, all the power goes down in Massachusetts, for example, someone could figure out how that works. Well, let’s say in a future world, all the electricity in Massachusetts goes down and no one has any idea what’s happening in the code. I think that just...

Tristan Harris: And we don’t think it recovers from it.

Fred Heiding: Yeah, I think that’s really bad. I mean, we saw what happened during COVID with just crisis everywhere and it could be so much worse and no one has any idea of how to fix it. I think that’s problematic.

Tristan Harris: Yeah. I mean, I lean on the side of this is much worse. And so, there’s this interesting thing. I mean, I’m happy to go back and forth with you, Josephine, on this. I just, how do we differentiate between, there’s nothing new here, state level actors had this capability, but now we have just like thousands and thousands more actors who can do this stuff. And then, the point that you’re also raising Fred is like, how comfortable should we feel that just one company has this capability? So, yeah, how should we think about that, Josephine?

Josephine Wolff: So, I think one of the open questions that I don’t know the answer to is, is there some point at which the AI vulnerability finding systems level out? So far we’ve seen continuous improvement and the things that the models developed this year can do are much more impressive than the things that the models developed last year can do. If that continues to be the case for the next 10 years, then you’re right. Whoever has the newest, fanciest model has a really significant advantage.

I don’t know if that is the case or if we’re going to hit a little bit of a plateau where everybody has models that can find roughly the same set of vulnerabilities and patch and exploit them to roughly the same degree. My general instinct has been more the latter. There is going to be a very significant improvement in how well we can find vulnerabilities with AI until there isn’t, until we have developed systems that can find most of them. And then, we’re going to see more of a leveling off.

In terms of the, what do we do when the AI writes all the code and none of us can possibly understand it, I want to emphasize that’s a choice, right? It doesn’t mean it won’t happen. But if we decide we’re going to replace all of the software powering the Massachusetts electric grid with software written in a language that no human has ever used and has ever tried to code or patch, we will be making a deliberate decision that that’s the kind of software we want to be using.

And I think, I mean, I’m biased because I’m somebody who spends whole life studying cybersecurity policy, but one of the reasons I think the policy piece of this picture is really important is because I don’t think those are decisions we want to fall into. I think those are decisions we want to make really carefully and deliberately. And I absolutely agree. I think that would be a bad one, but I don’t think it’s an inevitable one. None of this is to say I don’t think there are risks here, right?

Definitely, we’re going to see cyber-attacks where AI is playing larger roles. We’re already seeing some of them, especially in the scam world. I think there will be a lot of damage and there will be a lot of losses. Will those be exponentially larger than the damage and the losses we’ve seen from other cyber-attacks? I genuinely don’t know.

What I have seen so far since the announcement of Mythos has been fairly well contained, which suggests to me, by the way, that the way Anthropic has done this is not necessarily terrible, right? That choosing a couple of large tech companies and working with them to patch some of the most widely deployed software might be a sensible first step. It’s obviously not where they’re going to leave it, right? But nothing that I have seen in the wild so far has made me feel like, “Oh, this is a worse threat. These are bigger and scarier losses than any I’ve seen before.”

Tristan Harris: Fred, do you agree? Disagree with that?

Fred Heiding: Yeah, no, I think all of these are really good points. I think it’s really good with optimism. I’m really pessimistic and that’s why we make for a good conversation partner. And I think you’re always spot on in everything you say, Josephine. Some things I think about a lot is that... So, let’s say AI makes people develop code quicker. That’s true. We see it all around right now. Does AI make you develop secure code? Well, it depends. If you ask me to, it will, but almost no one asks you to for two reasons, right? People don’t think about this because they just say, “Create code that can solve task X.” Usually, people don’t think about explicitly telling the AI to make the code secure. It’s also more expensive, right?

So, this is a game of resources as cybersecurity always have been because it costs tokens and everything will just become a token economy in the end. That’s how the AI will work. And will we create a regulation that says you have to spend 20% of your tokens on security? I don’t think we will, but that would be great. So, you just rush forward and let’s take this power plants in Massachusetts again, right?

A lot of critical infrastructure is owned decentralized by private partners. If they know that they can use these AI-generated, super-fast code that just is incredibly much cheaper, easier to keep up-to-date, easier to work with, et cetera, et cetera. It’s not as secure as it should be, but it saves a lot of cost. Oftentimes, they’ll have to do it. They just can’t afford not using it. I’m just not confident will break long enough and we will have time to implement all the regulations to stop this.

So, it could work out and that would be really good if it does. I just see so many scenarios where, again, we have these arms race dynamic, everyone is rushing, There’s a lot of cost savings to be done and security usually doesn’t fit into that cost equation until it’s too late basically. So, I’m skeptical. I guess the only thing... Yeah, the other thing I want to say is that we could find all the vulnerabilities today with AI. I’m certain we couldn’t. That’s great.

The problem is that the tech is moving. The tech infrastructure we have today will not be the one we use in one year. It’s always changing and that’s a problem, right? So, we’re not going to find all the vulnerabilities that we use because these will be outdated, and then new ones will create that. And I just really don’t like the world where humans have no say in finding vulnerabilities because we will just be at the mercy of AI to do this. And, to me, that’s terrifying.

Tristan Harris: So, I just want to talk for a moment about the system card for Claude Mythos. That means, the card that Anthropic published of one of its capabilities. When Anthropic asked Mythos to escape a locked down sandbox, a computer that was not connected to the internet, and to email a researcher when it had done so, it did that. And it also posted the exploit that it used onto public websites, unprompted, almost bragging or showing off how it actually had hacked through the container.

When Mythos couldn’t edit certain files, it injected code into a file to elevate its privileges and then covered its tracks. When it was told not to access certain records, but when it “Accidentally saw the answer,” it used it anyway, and then hid the fact that it knew the answer. There’s some other really crazy things that Mythos is capable of doing. For example, when an LLM was used to judge Mythos’s work and kept rejecting it, Mythos identified that the thing that was evaluating its behavior was itself an LLM AI, and then it prompt injected it, meaning that it hacked the AI that was trying to evaluate it.

So, we have AIs that are able to recognize that they’re being evaluated by other AIs and then hack them. So, why this matters is, of course, we’ve had systems and we’ve had people, human beings, who if they’re a top tier hacker, could hack into some of these systems. However, we have here a totally new level of hacking capability where Mythos is able to not just find one exploit, but actually to string together multiple, three, four, sometimes even five vulnerabilities in a sequence that can give you a very sophisticated end outcome that we’ve never had before.

One thing we haven’t talked about is how the presumption of all this is that only “The good guys” have access to this model. Anthropic had it, and then through Project Glasswing, they shared it with “The good guys,” the defenders. But Anthropic is only as good as their security prevents that model from being stolen. And if you think about the Manhattan Project, if someone from another country wanted to get access to everything we were doing with the Manhattan Project, they couldn’t just walk in and then take one little object in their hand and walk out and have an entire nuclear bomb. But with Claude Mythos, you can do that.

We’re talking about a weapon for cybersecurity that fits on a flash drive. And there’s a joke in the AI security community that we all have to race like, “Go faster, go faster. The US is in the lead,” but literally the Chinese companies have what we have the second that we have it. So, we’re not actually “Ahead of them,” we’re just ahead of them as far as giving it to them. So, how should we think about the, we’re only as good as the labs are themselves secure? And ironically, it’s a recursive race that the more these capabilities get developed, the less secure the labs are too.

Josephine Wolff: To me, the access question was always time limited. I would imagine Anthropic felt the same way, and that was why they were making the decisions they felt they had to make about who they would give early access to. But I don’t know that I think that’s a bad thing, right? I don’t know that I think a world in which all of the companies large and small, all of the countries large and small have access to roughly the same security capabilities is a much worse one.

I think it depends on how those capabilities are harnessed. It depends on, again, whether we’re able to use them in ways to secure our systems. I think you could... In keeping with my general clearly extreme optimism in this conversation, you could imagine a world in which it allows for much more geopolitical alliance across these countries if they decide our real enemy is the AI and we all need to work together to make sure our systems are protected against that. I don’t think it’s the world we’re in right now. But I also think that there’s a huge amount of room for all of these companies and all of these countries to rethink the question of how secure can we make our systems.

Tristan Harris: Josephine, you brought up a very important point about, is there actually mutual self-interest from the US and China against these capabilities? So, clearly on one side of the scale, one country having this step function advantage in cyber is beneficial to them, not the other one, and they don’t want to share or collaborate on that. But then, from another perspective, the risk of rogue actors having, like if either of us leaked a super capable hacking model that we didn’t have the defenses in place for yet, or made it so that we only had one day to patch everything and that wasn’t enough time to patch everything, then we’re actually all in a more dangerous world.

And one of the things we always say in our work and informed the creation of this film, The AI Doc that we were a part of, is that in AI, the fear of all of us losing has to become greater than the fear of me losing to you. If the fear of me losing to you is dominant, then that’s what I’m going to focus on is getting that dominant capability. But for example, I found it notable that when Mythos came out, the public response from the White House didn’t come from the Defense Department or the Homeland Security. It came from Treasury Secretary, Scott Bessent who had an emergency call with the top banks and top companies.

And I think that banks and financial infrastructure are clear places where cascading failures there would actually create mutually assured financial destruction. On the one hand, you could say China wants to take down the US financial system because they want to switch everybody that you want. But on the other hand, like there’s no way of doing that in a way that doesn’t create interconnected fallout for the entire global economy and the stability of the world as we know it. I’m curious both of your reactions to that.

Josephine Wolff: There are a variety of ways in which I could imagine this spurring a little bit more, certainly discussion, maybe even cooperation among the countries that have a vested interest in maintaining the stability of the markets, maintaining the stability of critical infrastructure. What exactly that will look like, how good we’ll be at that in this particular political moment, it’s of course a little bit difficult to predict.

There again, I think there is some advantage to everybody feeling like, “Oh, we’ve all basically got access to roughly the same AI capabilities and not, we’ve got the best ones and so we’re going to refuse to work with you.” And I think it’s not clear to me, especially if you follow the trajectory we’re talking about before of all of our code is written by AI. It has lots of backdoors that only AI can find, but they’re not going to tell us about them.

I think that’s not a great world to live in, but I think it’s a world in which a lot of governments are going to find common cause much more than they are right now. And maybe not even just the AI is the adversary, but if North Korea has the ability to shut down everybody’s critical infrastructure, they’re probably going to be a lot less restrained about that than a number of other state actors have in the past and that might also prompt a higher degree of cooperation.

Tristan Harris: We have to know that this is a different regime we’re entering into. We’re now talking about a world where it’s not just humans can do the hacking. We’re building AIs that can do the hacking and you can’t just negotiate with an AI and say, “Don’t hack me. If I follow these things, will you not hack me?” The AI has its own inscrutable logic and this is sadly not science fiction anymore.

I think the key to me that unlocks the possibility for coordination is mutual recognition of an existential outcome. I think with AI, if you have an AI that is hacking every major web browser and every major operating system in the world successfully, and that’s only going to get stronger, and the AI is going to be able to do that on its own. And if I release it and screw it up, it might cause more existential damage, and it’s the existentiality of that outcome that motivates a trustworthy basis for collaboration.

To me, that speaks to how the US and China should have something like just like there was the red phone between the Soviet Union and the US to deescalate nuclear, it seems like we need a red lines phone for AI between the US and China, by which I mean anytime we have evidence of AIs that are going rogue or doing things like hacking in ways that we don’t know how to control or stop, at the very least, the right people in national security and the top of both governments should know about that same evidence because that creates the common knowledge of “The existential outcome” that we’re trying to avoid.

So, to me, that is an achievable thing. I’m not saying this because I have faith in the government leaders that they would do this. I’m just trying to articulate the pathways that would be there. And I’m curious if you all have other ideas. If we were really designing and trying to scheme about how we would get to some safer world at the level of international understanding and safeguards, what are other things that we would be doing? Josephine?

Josephine Wolff: So, I think another piece of this that to me is important for thinking about that mutual existential outcome is thinking about how much shared digital infrastructure we all use, right? How many of the same software programs are running on our computers all over the world, how many of the same devices we’re relying on. And I think a lot of the security progress in this space is going to have to come from really close collaboration with those companies.

And so, I think the cyber red phone, I think there might even have been like a China daily op-ed advocating for that 10, 15 years ago. I like that idea, right? I think it makes sense to me that there would be some avenue for really trying to focus specifically on these issues and not getting too mired down and everything else going on between these countries at any given moment. But I also think we need to do a much better job of thinking about how do you bring the private sector into those discussions? How do you both respect and defer to their expertise?

And also, not leave governments completely on the sidelines as we’re trying to decide what kinds of restrictions and constraints we want to put on these systems and think really seriously about what those constraints are. And I think that we are much more likely to be able to put in place those restrictions with more international cooperation, right? I think the US on its own is never going to say, “We shouldn’t be pursuing AI to develop bioweapons,” because if they think China is pursuing that, then they’re never going to want to give up their access to it.

So, I think it opens the door to being able to say, “Look, this particular capability seems bad for all of us. Let’s take it off the table together and that way, worry less about, “Oh, are you going to get there first?”

Fred Heiding: Yeah, I agree with all those points. What I would add here, and you mentioned it briefly, Tristan, is not just educating the government or bringing companies in, but also educating the people and making sure that everyone sees AI as big of a threat or even bigger than nuclear weapons. I do personally believe that AI is much more of a threat to humanity than even nuclear weapons.

I think nuclear weapons could kill a lot of humans, but I think it wouldn’t extinct us as a race. I do believe that AI could completely enslave the human race in ways that sounds like sci-fi, but it’s not. We already see totalitarian regimes, like look at North Korea to some degree, China, Russia has parts of this, that’s just without AI, right? Just people with smart uses of technology. And these smart uses of technology makes it really, really easy for a few people to control a population. And I think people don’t understand this the same way they understand that nuclear is bad. And if people would understand this, they would put pressure on companies, on governments to just drastically change what we are doing.

Tristan Harris: You’re speaking to the, what we call the attractor state of totalitarian lock-in. So, once you locked into authoritarian governments that had both AI surveillance and AI hacking, how can you as a citizen ever fight back if you have no secrets? You can’t. Let’s take a step down from the international coordination bit, which we talked about with China, and we want to go to policy solutions.

And one of the things I think Josephine you’ve written about is how you’re not liable if you make a piece of code that someone can later be discovered to hack into. We don’t treat the software maker as liable for that. So, the company that gets hacked has to do with that themselves. And then, we started developing this new economics of an insurance market. Can you talk a little bit about what would be the policy solution that we would do? And if this is related to what Fred said earlier around incentivizing companies to spend more on those tokens to basically ask the AI system, “Don’t just write the code for me, write the secure code for me,” which means spend more money on compute, but that’s going to cost more. So, how do we deal with this from a domestic policy angle?

Josephine Wolff: So, for the most part right now we don’t. I think the hope for an insurance industry would be that it would incentivize or require companies that are developing software to use state-of-the-art tools for security testing, right? In the same way that none of us would have smoke detectors in our homes if our insurers didn’t require us to. Maybe none of us would spend any money securing our code, but if our insurance says, “You’ve got to do this or we’re not going to cover certain types of losses,” then perhaps we’ll be willing to.

And I do think that one of the other things that I find hopeful about tools like Mythos is that they could provide insurers with a clearer roadmap than they’ve had before of what is it you should actually require of your policy holders to do in terms of security. Is there a really solid approach that could be just a condition of the coverage?

Tristan Harris: One thing that strikes me is basically saying Mythos can change the economics and almost create more precision pricing for insurers saying, “Here’s what it would cost for you to basically use Mythos to do it.” Something that didn’t hit me until now is obviously, now the entire world’s dependent on five companies to secure themselves, both for the vulnerabilities of the world and to protect themselves. So, it’s a racket. It’s essentially if those guys went rogue, basically they have everybody locked into paying them forever to protect themselves.

Josephine Wolff: I think it’s a reason to be advocating for other models of artificial intelligence. It’s a reason to be thinking about the open weight models. It’s a reason to be thinking about, are there alternatives to a world in which there’s a very, very small handful of companies that hold all the cards. But if you can say like, “Look, here’s a tool, you have to run it, you have to patch everything it finds,” that’s actually a much more concrete piece of guidance.

Now, maybe it won’t be perfect, maybe it won’t be where we’ll end up, but it would certainly be a big step forward if it turned out to mean that we could then impose some liability on developers who failed to use these tools for vulnerabilities that could have been caught but weren’t. If it means that insurers are going to condition their coverage on the use of these types of tools, it will give a huge amount of power to these companies, no question. Will it give them more power than like the Claude companies have right now? I don’t know, right? Tech has always been a very concentrated industry. I think that’s a broader systemic issue than just with AI.

Thanks for reading [ Center for Humane Technology ]! Subscribe for free.

Tristan Harris: Fred, do you want to speak to your policy recommendations? I know mandating pre-deployment access for defenders, treating AI labs as critical infrastructure. Do you want to speak to some of these solutions?

Fred Heiding: Just before that I want to say that I really empathize with this criticism or maybe skepticism of a few companies owning all this AI chain. I think Josephine makes a good point in the Claude companies are also powerful. We have other powerful semi-monopolies in the world. I do believe that AI is in another category than anything we’ve seen before.

So, I think that is really problematic. And I would love to see more people owned AI if possible, more decentralized owner structures. And we could make policies to approach that. More security specific to be a little bit more small level of our second. I think there’s a lot of things we could do. So, Jason Clinton at Anthropic, I’m sure a lot of other people too, talks about this one-day patch policy and maybe it’s even shorter now. But I think that’s great, right?

Every company should be able to just patch a vulnerability within 24 hours or even much, much quicker because we just have to. It’s going to be so stressful and time dependent in the future. Whenever a vulnerability is discovered, companies need to have the frameworks in place to just patch that instantly because we can’t wait and be slow as we’ve been. Even a few weeks is way too long.

Tristan Harris: One of the things you mentioned is treating AI labs as critical infrastructure, that they shouldn’t be able to... Maybe there’s some public commons level way of accessing this public utility of basically defense so that maybe there’s some amount they can charge. But basically, they can’t overcharge or... There’s got to be something that just makes it a commons of common security because at the end of the day, we need it for securing a safer world. And the question is it just a national thing? Are we extorting still all the international allies to say we’re forcing them to pay for all these things? It just gets into geopolitics and complicated quickly.

Fred Heiding: I think that’s such a good point. I’m definitely seeing AI as a critical infrastructure. And there’s different arguments here. If we make it an official 17th critical infrastructure sector in the US, maybe we’ll slow development, maybe we’ll create regulatory overlap, which can be problematic as well. We could do that in a way that I think give policymakers more power to demand security standards. And that might slow things down, but that could also make us more secure. So, I’m pretty positive to such an approach. I don’t think it will happen, but I like to advocate for it.

Tristan Harris: Maybe just to wrap up, what are some of the things that people can do just in their personal lives to, in light of Mythos existing, which if it can hack every operating system, people say they throw up their hands. What can I possibly do? But let’s give people some hope. What are some basic things that people should be doing?

Josephine Wolff: I think the advice I have, and it’s the most irritating and obnoxious advice you can give, but I think it’s also the right advice is that it’s something people should be thinking about when they’re voting, right? That the question of how politicians are approaching artificial intelligence and whether they think there should be any safeguards and whether they’re willing to challenge any of the companies that are developing it is really important. And it’s only going to get more important as those companies are pouring more and more money into lobbying. There are a whole bunch of issues to think about when you vote today, and I’m not going to tell you it’s the single most important one, but I think it’s a very important one and only becoming more so.

Tristan Harris: Well, it’s a monopoly of enactment where once this happens, there’s no more enactment of anything by citizens because... And so, from that perspective, there’s a weird way in which like, “Okay, well, is this actually more important than the price of eggs or gasoline or whether my kids have school?” Well, it’s like, well, but if I’m about to lose my political power permanently, then it actually is the most important thing.

This should be the number one issue on the midterms and people do have a say. And if they can share this episode, share this material, go watch the AI doc, get people to see it, recognize that we’re not heading to a pro human future by default, and we want to be moving towards a pro human future and against the anti-human future. But I do think that this conversation is trying to play a role in clarifying the nature of the problems that we face so that we make sure that we’re putting in the policies, putting in the guardrails, and also putting forward, as you said, Fred, basically the collective problems that we need everyone’s mind on solving. How do you protect citizen secrets in a world where AI can hack those secrets? What are the new laws? What are the new code level protections so that anybody who accesses such a thing, for example, it’s logged. Here’s the one system that can hack into computer systems. If you’re using it, there has to be oversight of who’s using it and for what, and that has to be enforced at the level of code, basically.

Josephine Wolff: Okay. Well, I’m just going to give the most irritating cybersecurity advice. And again, I’m only going to give it because I think it’s the right advice. You want to be really aggressive about installing the updates as annoying as you find them, as much as you want to tell your computer and your phone to delay them. You want to be really careful about how you’re using AI, what you’re giving it access to, what pieces of your digital life, what pieces of your data are being fed into it.

You want to be really thoughtful about which companies, AI tools and products you’re using. You want to think carefully about who’s running those companies and what their interests are and in a moment of deciding, do I need AI for this or maybe not, I think it makes sense right now to err on the side of maybe not.

Fred Heiding: I think these are really good advices. And some things to maybe take that one step more extreme just to do it, right? Well, let’s say something really bad would happen in terms of a totalitarian locking happens where the people just don’t have control anymore. And that could go quickly because all of these AI models are right now being used as social media companies also use their tools to collect what do you think? What do you do? What’s your digital footsteps?

And right now, that’s being used heavily to create ads, right? And fair enough, that’s annoying. But maybe you can live with that, but that is to a large degree being used to nudge you into different direction, making you think in a different direction. So, what information do you digest online? I think it’s really important to think this. I think there’s these statistics that the younger generation get 90% of their news from social media. Well, what accounts do you follow? Are these people rational human beings who seem to know what they’re talking about and present both sides of the arguments?

Maybe I can add one of interest and you spend a lot of years, let’s say almost a decade on just trying to figure out how can we counter these incentives of social media, right? And I think it’s fair to say we failed as a society to incentivize social media. These are for-profit companies that have done really, really bad harm to the human population in terms of dopamine hijacking and other things. And we’re now starting a similar thing but with AI companies. And we have these for-profit AI companies, they’re obviously seeking to shareholder maximize and profit maximize as they develop their AI models. Are we going to repeat the same mistake again? And we really shouldn’t.

We have to learn from the mistakes with our failed social media regulation and try to make AI into something better. And that would be really good if we take that seriously. And I don’t think we take it seriously right now.

Tristan Harris: Yup. And we can. We’re in a critical window. If we play our cards right, we can make sure that defenders get access to this first. We can have regulation that tries to close the gap of the extra cost for adding security. We can have international coordination with enforceable metrics that we’re doing the verification. This could end better than it did with social media. But if we don’t, the internet becomes basically unusable for people who don’t have top tier tools.

And I do think that this qualifies as a Manhattan Project moment, and we need everybody who works in cybersecurity, who has any interest and any capability or talent in these areas to work on defense right now. You can think of AI as introducing a Y2K vulnerability in all of society, but in a rolling way. So, we have a rolling mobilization, a wartime mobilization to defend our systems from the new vulnerabilities that AI creates.

I hope this conversation helps activate everyone in every corner of society, whether it’s policymakers or people listening to this to take part in this. And again, vote in the midterm elections. This is not inevitable. Fred and Josephine, thank you so much for coming on Your Undivided Attention. This has been really fantastic.

Josephine Wolff: Thanks for having us.

Fred Heiding: Thank you so much, Tristan.

Subscribe now

RECOMMENDED MEDIA

The Claude Mythos System Card

The Project Glasswing announcement

“Black-hat LLMs,” a talk on AI’s hacking capabilities by senior Anthropic researcher Nicholas Carlini

You’ll See This Message When It Is Too Late: The Legal and Economic Aftermath of Cybersecurity Breaches by Josephine Wolff

“America’s Endangered AI: How Weak Cyberdefenses Threaten U.S. Tech Dominance,” by Fred Heiding and Chris Ingles

RECOMMENDED YUA EPISODES

America and China Are Racing to Different AI Futures

“Rogue AI” Used to be a Science Fiction Trope. Not Anymore.

The Self-Preserving Machine: Why AI Learns to Deceive

Leave a comment

“First solve intelligence, then use intelligence to solve everything else.” That was how Demis Hassabis summed up the vision of DeepMind (now Google DeepMind) when he founded the company in 2010. He gave his company 20 years to get there.

This ethos – the belief that intelligence derived from AI (given enough compute and data) will save us all – has been so rarely questioned that by the time Hassabis went on 60 Minutes last year and stated “I think one day we can cure all disease with the help of AI,” he was only repeating claims that other prominent AI CEOs had already made.

Including curing cancer. Especially curing cancer.

Last year, OpenAI CEO Sam Altman testified before the Senate Commerce Committee and said, “ We are working to build tools that one day can help us make new discoveries and address some of humanity’s biggest challenges, like curing cancer.” At Davos, Dario Amodei of Anthropic said a superintelligent AI would grant humans “wonderful things, like the ones I talked about in Machines of Loving Grace. It will help us cure cancer.”

Unfortunately, their vision of a magic genie both fundamentally misinterprets and undermines cancer science, says Dr. Emilia Javorsky, a physician, public health researcher, and director of the Futures program at the Future of Life Institute. She has worked across scientific research, clinical trials, tech startups, and AI policy, and she recently wrote a paper called How AI Can and Can’t Cure Cancer, in which she argues that the promise of superintelligent AI curing cancer falls apart under scrutiny.

Javorsky is not anti-AI. Just the opposite. As she told Tristan Harris in a recent conversation on Your Undivided Attention, there are tons of astonishing ways AI advances the fight against cancer every day – from breakthroughs in breast cancer imaging to reading and analyzing genetic data at scale. Meanwhile, she says we are actually losing ground in the fight against cancer by diverting resources away from research and technology that could actually make a difference.

The point Javorsky kept coming back to in the interview is that there are two types of AI: the first are general-purpose models like Claude, Gemini, and ChatGPT, which the big AI companies are pushing to dominate the market in the hope that they’ll unlock superintelligence. The second is narrow AI: bespoke models created for a specific problem, like detecting cancer from blood tests or helping an agricultural company with weeding. Right now, the lion’s share of resources is flowing into the former, not the latter. Javorsky argues that this is entirely the wrong approach. “It’s absurd to me, the situation that we’re in,” she says. “There’s so much we could be doing that we are not actually doing, and we’re doing all of the wrong things.”

Cancer progress comes from data and resources, not accelerations in knowledge

Scientists have made massive strides in medical knowledge over the past 75 years. We aren’t lacking in breakthroughs or brainpower: today, we have an oversupply of human scientists relative to the resources we can actually allocate to experimentation.

But despite that acceleration and knowledge, “we’ve noticed that therapeutics approved to actually help people have remained markedly flat,” says Javorsky. “We actually haven’t made commensurate progress, so the intelligence that we’ve gained hasn’t really been coupled to actually moving the needle on saving people’s lives.”

Meanwhile, cancer rates have only climbed worldwide, with new diagnoses more than doubling since 1990. And the survival rate is almost exactly the same as it was over a decade ago.

So knowledge itself is not the missing piece, and neither is AI. In fact, we now have AI to thank for leaps forward in diagnosing hard-to-spot breast cancers, helping surgeons excise tumors by identifying tumor margins in real time, and predicting whether a novel treatment would be toxic or non-toxic.

But all of these examples rely on human-supplied data – images and extensive libraries of existing compounds – which AI models have been able to learn from. Take AlphaFold, the poster child for success with AI and biology. That breakthrough in solving protein folding would not have been possible without the global Protein Data Bank, an archive of 3D protein structure data to which scientists had been contributing for decades.

Most people aren’t aware that the world has no equivalent data bank for cancer, which is what a superintelligent AI – if that ever exists – could train on to develop cures.

“We don’t even have a national sort of data commons of cancer genetics and imaging data, the things that scientists could learn from, that’s interoperable,” says Javorsky. When doctors collect specimens in a clinic, they are usually not shared beyond their health system.

Cancer is complex, and treating it is highly individualized

Compared to things like treating the flu or treating high blood pressure, which are more static biological processes, cancer “is something that is co-evolving with us. It’s dynamic, it’s complex, and it’s highly individualized,” says Javorsky. We’re a long way from our former simplistic understanding of cancer as something that originates in a cell that gets a mutation, goes rogue, and makes a tumor.

Cancer is now understood as a much more complex disease involving the immune system and the blood supply. And each person’s case is unique. Javorsky points out that science has yet to cure any complex, chronic disease, like diabetes and Alzheimer’s. The best modalities for treating cancer center on the individual, and that will still be true in the age of superintelligent AI.

Curing cancer has other inherent bottlenecks

If cancers were as easy to treat in people as they are in mice, we would be well on our way to curing cancer. Unfortunately, more than 90% of cancer cures that look promising in mice fail when they are tested on humans. Simulating a human’s unique biology, which has no clean ruleset, won’t be possible with AI (no matter how powerful it is) because biological truth cannot be computed, says Javorsky.

Diseases also require that we take time – human time – to test out whether a cure or treatment is working. And that delay is compounded by the FDA’s clinical trials process, a notorious bottleneck that few companies can even afford to attempt.

Finally, there will always be an issue with scaling human testing at the breakneck pace needed to test all the potential therapies AI can develop. “On the AI side, we’re rapidly flooding the system with new molecules that we want to test. But we can’t scale people. We can’t scale the number of patients in a clinical trial. We can’t scale the number of tumor specimens that come from a patient to test,” says Javorsky.

Our resources could be so much better spent

The leading AI companies will spend upwards of $700 billion this year in their race to superintelligence, three times what the Manhattan Project cost. Javorsky believes that funding basic science research, such as funding the National Cancer Institute, would be a far better return on investment. (For comparison’s sake, the 2026 budget for the National Cancer Institute is a comparably paltry $7.4 billion).

In the race to save lives, we need the right tools, and AI will be one of them. But it needs to be purpose-built AI applied to initiatives that will save lives today, such as reducing the cost of manufacturing a new drug or developing new medicines and tools.

“Fundamentally, I’m super bullish on the promise of AI in oncology and medicine in general,” Javorsky says. “It’s just the right kind of AI development that’s targeted to actually solving the problems and unblocking the things that are holding up our ability to move science forward.”

See the full conversation with Dr. Javorsky here:

We’re working to empower policymakers, technologists, and everyday people to guide technology toward the public good. If you value our work and want to support it, consider donating.

Donate

One of the most common arguments you hear from company executives racing to develop super-intelligent AI is that it will cure cancer. It’s an incredibly powerful and seductive promise.

If superintelligent AI really can cure cancer, then anyone who stands in the way of it, anyone who wants to slow it down — even because of its serious risks — is essentially letting people die. In fact, the biggest risk would be going too slowly. But what if a superintelligent AI isn’t actually capable of solving cancer in the way it’s been described? What if we’re being sold a false promise to justify a dangerous race?

That’s exactly what our guest this week argues is happening. Dr. Emilia Javorsky is a physician, public health researcher, and director of the Futures Program at the Future of Life Institute. She’s worked across scientific research, clinical trials, tech startups, and AI policy. Emilia recently wrote a paper titled “How AI Can and Can’t Cure Cancer,” in which she argues that the promise of superintelligence curing cancer falls apart under scrutiny.

Emilia lost a parent to cancer, so her criticism of this promise comes from a place of real concern, not cynicism. It also comes from her belief that AI can be really revolutionary for medicine, if we build it the right way.

Tristan Harris: Hey everyone, and welcome to Your Undivided Attention. This is Tristan Harris. One of the most common arguments you hear from people racing to super intelligent AI is that it’ll be able to cure cancer.

Dario Amodei: It’s incredibly powerful. We’ll do all these wonderful things like it will help us cure cancer. It may help us to eradicate tropical diseases.

Sam Altman: We are working to build tools that one day can help us make new discoveries and address some of humanity’s biggest challenges, like climate change and curing cancer.

Demis Hassabis: I think one day maybe we can cure all disease with the help of AI.

Tristan Harris: Not help with cancer, not improve treatment, but cure cancer. Now that’s obviously an incredibly powerful and seductive promise and everybody listening to this right now likely knows someone who’s died of cancer. It kills almost 10 million people per year. I lost my mother to cancer in 2018. This is a very personal topic. And that’s why this promise is so potent and why we need to examine it because if the technology really can cure cancer, then anyone who stands in the way of it, anyone who wants to slow it down even because of the serious risks, is essentially letting people die.

This is the idea of the invisible graveyard you hear about from the accelerationists. Think of all the people that we might be able to save by racing forward. In fact, the biggest risk is not going fast enough, they argue. But what if it isn’t actually capable of solving cancer in the way it’s been described? What if we’re being sold a false promise to justify a dangerous race and just to make a handful of people incredibly wealthy and powerful and avoid regulation?

Our guest today argues that this is some of what is happening. Dr. Emilia Javorsky is a physician, public health researcher and director of the Futures Program at the Future of Life Institute. She’s worked across scientific research, clinical trials, tech startups, and AI policy. And she recently wrote a paper called How AI Can and Can’t Cure Cancer, in which she argues that the promise of super intelligence, curing cancer, falls apart under scrutiny and that we can’t use this false promise to justify the peril that we’re currently facing. We’re going to link to that in the show notes.

This is a deeply personal conversation for both of us. Emilia also lost a parent to cancer. So hear her criticism of this promise as coming from a place of real concern and not just cynicism. It also comes from the belief that AI can be really revolutionary for medicine, but not in the way we’re building it today. So Emilia, welcome to Your Undivided Attention.

Emilia Javorsky: Thank you so much for having me, Tristan.

Tristan Harris: So first I’ll just say, Emilia and I are friends and she’s an incredible ally in this work. We were at the South by Southwest conference earlier this year, and you talked about the work that you’ve been doing on AI and cancer, and I was struck by how personal this is for you since you lost a parent to cancer. Before we even get into your arguments, can you just talk about your experience of that and how it shaped your thinking?

Emilia Javorsky: Yeah. So when we hear the promise of AI in cancer, it triggers in all of us a personal experience because all of our lives have been touched by some sort of loss to cancer. And for me, it was deeply personal that I lost my father to cancer. And I lost my father to cancer over a decade ago. And when I sat down to write this essay and really think about examining the ASI to cure cancer promise, I went back through the medical literature to see how much progress had been made since the time my father passed to where we are today.

And the reality is the survival rate is almost exactly the same as it was over a decade ago. And so the problem of progress in oncology is probably one of the most urgent of our time and one of the most noble things we can deploy capital in service of solving and our talent in service of solving. But I think it’s really important to examine whether putting that capital into a race to superintelligence is the best way to save the lives of our loved ones.

Tristan Harris: Yeah. Having lost my own mother in 2018, Aza co-host of this podcast, he lost his father to pancreatic cancer. I just want to establish... I think it goes without saying, anybody who has this in their family with a loved one wants to accelerate anything that will save their life, anything that has a chance. And yet there’s so many issues that you find out about. There’s all these new things that are coming to market, but then they’re not actually even available to your spouse or your loved one when they get this.

And so I think one of the things we’re going to talk about is there are many ways technology can help advance biomedical science, but is the specific path of building super intelligent AI that is reasoning with a massive data center across everything? Is that the specific vehicle that’ll get us there? And you wrote this essay that I really want to encourage people to check out. Why did you write this essay? What was the kind of motivating purpose here?

Emilia Javorsky: Yeah. So in addition to the personal experience with loss in cancer, having a background as a clinician and having gone to medical school, you also experience it from the other side, the frustration of providers about how limited of a toolkit they have to actually help people and encountering it over and over again day in and day out, having to deliver news of loss to families.

And so for me, this is deeply personal to me, both in terms of my life, but also in terms of my career. And also in sort of a parallel hat that I’ve worn in this AI policy conversation for the better part of a decade now, have seen these two worlds, which is biomedical innovation and the ASI race. And to me, hearing over and over and over again, “AI is going to cure cancer. We must build ASI because it’s going to cure cancer,” and yet that promise going entirely unexamined, just kind of being taken at face value that if we want to save lives and if we want to cure cancer, that this is the thing that we have to do. And I strongly believe that that is not actually the best way to start saving lives today.

Tristan Harris: And for listeners, ASI is artificial superintelligence, which is an AI system that is more intelligent and powerful than all of humanity’s intelligence combined. You are not anti-AI for cancer. You just think there’s a totally different approach we could be taking. And first, we have to understand the problems with our current approach and then give people the hope that there actually is a totally different way we could be applying AI that would actually get us to the outcomes that we’re all looking for as opposed to false promises to sell investors and keep pumping up your data centers.

Emilia Javorsky: Yes, I’m incredibly excited about the potential for AI and this general moment that we’re in for progress in oncology. I remain really hopeful and excited about what the future has ahead. For me, that’s sort of three ingredients, which is one, supporting all of the AI tools that are being developed in specific areas of oncology that are making things go faster, cheaper, better, unlocking new capabilities, the exciting research that’s happening in biology. So there’s really exciting science that’s happening that’s sort of discovering totally new ways to think about the problem. And so figuring out how do we support those scientists doing that good work and getting their discoveries out of the lab and into the clinic faster?

And then thinking about, how can we actually realign and redesign the system that we have and identify where the parts are in the current system that are either holding up progress or even taking it in the other direction? And so I think that kind of tripartite approach is one that makes us well suited to make a lot of progress in oncology in the next decade. But part of the reason I wrote this essay is because I’m worried that the current approach isn’t doing those key things that we need to actually move the needle and that our resources are being placed in areas that are not going to deliver the benefits that we hope for.

Tristan Harris: Yeah. I’m just brought back to my memory of going to Senator Chuck Schumer’s AI Insight Forum. It was this historic event where they invited all the CEOs, Elon, Jensen, Mark Zuckerberg, Sam Altman, Bill Gates, all in one room. And Aza and I were there with a handful of civil society groups. And I remember talking to some of the Senate staff up beforehand before we went up there. And one of the things you heard from, I think it was Senator Mike Rounds was just because they had family with cancer, that there’s this thing, you and I have talked about it, that people kind of turn these puppy dog eyes of like, “But it could cure cancer.”

And there’s this hope of, well, that literally would just eclipse any other reason to slow down. If it’s life and death, we do anything to save that person. Let’s just steelman for a second. So why would they say it could cure cancer? It seems intuitive. AI understands language patterns and language. So just the same way it can understand patterns in text and generate ChatGPT essays, it could understand patterns in DNA and understand immuno-oncology. Let’s just steelman for a second why people believe... Because it’s not like it’s wrong, but it’s seductively false and kind of an optical illusion, almost like a magnetic trick.

Emilia Javorsky: So we hear a lot about the ways that AI is helping advance progress in medicine in the here and now, which it is and it is going to be instrumental in doing so, but it’s not ChatGPT that is unlocking that progress. It’s scientists building bespoke models off of highly curated data sets to actually solve a specific problem, whether that be drug design or whether that be predicting toxicity, the list goes on.

So I think one piece to start is the “AI will cure cancer” promise surfs a little bit on the AI progress that’s already being made with tools and smaller models and kind of bundling that as evidence as to why ASI will help solve the problem because if the AI could get so much better, imagine how much better results we could be getting. So that could be an image of a mammogram for breast cancer or it could be blood test results.

And then getting sufficient measurement of that phenomenon into a dataset. And so can we generate a dataset that captures all of the variability that we see in when we measure that phenomenon that’s sufficiently representative? And then can we apply intelligence to unlock insights that previously humans did not see or were unable to do at scale? And so in medicine, we’re seeing this happen across many domains where we have good data. So when we talk about early detection of breast cancer, AI is amazing at that because we have lots of great images that are high quality and curated by human radiologists of what is and what isn’t breast cancer. So in that domain, AI does very well when it has the data to work with and that data is sufficiently representative of the phenomenon that we would like to study.

Tristan Harris: Right. So we have lots of mammograms and we have lots of results that confirm whether that mammogram did have a cancer or not, which means you can train a more and more accurate model. That one’s solved.

Emilia Javorsky: Correct.

Tristan Harris: So what are some of the other narrow AI applications that are helping?

Emilia Javorsky: One area we’re hearing a lot about is AI being able to predict whether a new drug is going to be toxic or non-toxic. And that’s because we have extensive libraries of existing compounds that we know whether or not those cause problems or adverse events when they were put into people. So the AI can take a look at a new compound and say, “Okay, based on all of my knowledge of everything else that’s either safe or unsafe, what do I think this will be? Do I predict this to be more likely to be safe or unsafe?” And that’s called computational toxicology, and AI is doing a great job at that. We’re hearing a lot about AI for drug design, being able to really just lean into the chemistry part of biology, even more so than biology itself to design new molecules, to design new drugs. So that also is, I’d say, an area that’s quite exciting.

And then there’s clinical AI. So AIs that are actually being used in the operating room when they’re excising tumors and trying to figure out if they have a margin or not. And that’s because there’s imaging databases of what a margin looks like that an AI can look at and say, “Okay, I think we’ve got it,” or, “We haven’t gotten it.” So I would just highlight those three examples. And each of those are not being developed within large companies. They’re all being developed either by small startups or even academic institutions.

Whereas the ASI promise is saying, “Let’s just digest everything. Let’s take all knowledge and put it into one big giant model and see what insights it can derive from that model.” And so the idea here is the more and more data we put into this, the more and more capable systems we can make. And one day we’ll make a system that is more capable than humans, and then thus we’ll be able to do types of reasoning or types of insights that humans would not really be able to do or discover. And assuming in that set is a cure for cancer.

Tristan Harris: Right. So this is like if I read not just the entire internet, but all biology textbooks, had access to every science lab, had a robot arm doing lots of studies, plus integrating it with the GPT-7 trained data center with Sam Altman’s Stargate cluster that’s just combining so much information that it’s going to magically find all the needles in all the haystacks, that vision of ASI, finding cures to cancer, right?

Emilia Javorsky: Correct. Yes.

Tristan Harris: What actually is cancer?

Emilia Javorsky: So this is where the AI to cure cancer piece breaks down is what is cancer and what is a cure? And those are two actually really fuzzy terms even for the experts in the arena. So when we think about cancer in the early days, the way you thought about cancer is like there’s some cell, it gets a mutation, it goes rogue and it makes a tumor. And that was the original simplistic understanding of cancer. And as our understanding of oncology has gone on, there’s been these papers that have come out called the Hallmarks of Cancer.

And as we find new biology and new ways to measure things, we’re getting further and further away from that simple explanation of one cell with a mutation that goes rogue and makes a tumor. It’s actually a much more complex disease involving the immune system and the blood supply. And even within one tumor, different things are happening in different parts of that tumor. And so the story of cancer has been, as we push science forward, we’ve uncovered more and more complexity to the disease, not less. So there hasn’t been sort of a march towards a simplifying or unifying hypothesis. It’s been a march towards an ever more complex and individualized type of disease. So fundamentally, when we think about the complexity of cancer, it is sort of a shadow self. And there’s a book I highly recommend folks read called The Emperor of All Maladies that really delves into-

Tristan Harris: Good book.

Emilia Javorsky: ... this problem of why this is the most complex disease of all, because it is something that is co-evolving with us. It’s dynamic. It’s complex. And it’s highly individualized. So compared to other things like treating the flu or treating high blood pressure, which are more static biological processes relative to cancer, this is really the big one in terms of complexity.

Tristan Harris: Okay. So let’s go back to the promise made by CEOs. You have Dario Amodei from Anthropic who talks about compressing 100 years of biological progress into 5 to 10 years by creating what he calls a country of geniuses in a data center that are all dedicated to that. And that’s obviously a really compelling idea. Just to go into that though experiment, imagine the last 100 years of scientific progress. Just see that in your mind’s eye, all of the things that we got over the last 100 years. Now imagine that coming in the next 10 years scientifically. That’s like magic. This is sort of the science accelerator button. It’s what leads to Ajeya Cotra to say, “This is why AI is like 24th century technology crashing down on 21st century society.” But what is the problem with this argument of 100 years of biological progress?

Emilia Javorsky: I would say there’s three main problems with that argument. The first one is in science, we actually have been accelerating knowledge and intelligence. We have an oversupply of human scientists relative to what we can actually resource in terms of experimentation. So the doubling rate of medical knowledge has gone from 50 years in the 1950s down to 73 days by some estimates. We have an oversupply of scientists relative to number of lab benches and pipettes and people we can resource. And despite that acceleration and knowledge, we’ve noticed that therapeutics approved to actually help people have remained markedly flat. We actually haven’t made commensurate progress. So the intelligence that we’ve gained hasn’t really been coupled to actually moving the needle on saving people’s lives.

Tristan Harris: This is very interesting because it’s like the promise is that if we just have more intelligence, that intelligence is essentially the bottleneck for why we don’t get more progress in biology. But you’re saying we did get an explosion of intelligence in the form of new biological data, the amount of medical data we got, and the number of actual people that are sitting at lab benches and yet it hasn’t resulted in that. So you argue though it’s not only wrong, it’s actually dangerous. Can you speak to that?

Emilia Javorsky: Yeah. So there is a danger to waiting and hoping that some future genie is going to solve a problem, which is in some ways the essence of what the ASI promise is. It’s, “Sit. Wait. Hold tight. Don’t do anything in the here and now. In the future, there’s going to be a cure for all of these problems.” The reality is people are dying today. People need solutions today. We need to actually be unblocking progress and moving the needle today. So there’s the temporal piece of this where it’s like people who have cancer don’t have time to wait on the future, even if that were to be true. The second piece of this that’s really important to think about is we don’t live in a world of infinite capital. If we lived in a world of infinite resources and one bucket wasn’t coming out of another, then there’s a different argument to be made.

But we’re seeing that biotech is at a 10-year low in terms of venture funding of new ideas. And venture funding is really where you see the new breakthrough, exciting, high-risk types of projects that really can move the needle for patients. We’re living in a time where we’re reducing our investments in basic science, in science infrastructure, in data collection. And so the essence here is if we are going to take money away from doing the things we know will unblock progress, then we better be really confident that that is actually the fastest way to save lives.

Tristan Harris: Can you speak to the amount of resources that are currently going into accelerating ASI versus how much is going into, let’s say, cancer research?

Emilia Javorsky: If you look at the amount of money going into building ASI and the infrastructure associated with that, that’s an unprecedented amount of money in terms of investment in a technology. In 2026 alone, they’re looking at 540 billion plus dollars. And if we want to compare and contrast that to, let’s say, the National Cancer Institute, which was a pretty good barometer of what are we investing in the public in the basic science and understanding and moving the needle in oncology, that’s only $7.2 billion. So it is a fraction of the amount on a annual spend that we’re spending on actually solving the problem of curing cancer as opposed to an ASI spend.

Tristan Harris: So essentially, we’re putting half a trillion dollars into a genie that people think or are selling the idea that it’ll magically solve all of our problems from climate change to cancer compared to 7.2 billion. 7.2 billion versus half a trillion is the gap. Not just that we’re not making progress in the cancer side, we’re actually robbing billions of dollars away. Instead of getting 10 years of scientific progress, it’s almost like we’re losing 10 years of scientific progress because all the money is going towards this genie rather than going towards things that would actually unlock progress. I’m just wondering though if listeners would, at this point in the conversation, believe that the genie won’t actually address these things because all of what we’re saying depends on whether that is true or not. So let’s break this down for listeners.

Emilia Javorsky: So I think the AI for science promise gets all kind of bundled into one and cancer gets put into that along with physics and along with manufacturing and along with chemistry. But it’s really important to break those out because physics and biology are very different phenomenon. And physics is a domain where, and math is similarly where we’re seeing this correlation between capabilities and progress in those sciences, where we have basic rules. We know the laws of physics. We know the rules of physics. We know the rules of math.

But for biology, there are no first principles to work with. There are no actual rules of the road to feed to an AI to learn and to model from and to analyze. And people say, “Well, you have physics. Everything’s physics at the end of the day. You have physics, you have everything.” But that’s simply not true in biology and it’s infeasible even using classical physics, nevermind quantum physics, to simulate even a week or a minute of a human’s biology if you covered the entire earth in GPUs.

Tristan Harris: Right. So you’re not saying that AI couldn’t massively accelerate physics or math?

Emilia Javorsky: Correct.

Tristan Harris: So we could hit a button, and it’s already true, by the way, just for listeners, Paul Erdős, who was a mathematician in the 1940s, he laid out these math problems in the ‘70s that had never been solved. And just recently in the last few months, AI has actually made progress and solved those math problems. It’s now winning gold in the International Math Olympiad. It is generating new physics. So you could actually... Just to put listeners through this, using the raw rules of physics that we know, you could rederive everything up to quantum physics with just an AI doing that. That’s mind-blowing.

So you’re endorsing that AI could do that, but you’re making the distinction that in biology you have these emergent effects. It’s the complex adaptive nature of biology that’s different from other systems that makes it so hard to model. And then you gave a quote in there of how much computation it would take to simulate... You said it was what, one week or one minute of the human body would take more than the GPUs on planet earth and more than the time in the universe?

Emilia Javorsky: Correct. Yes.

Tristan Harris: That’s crazy. Okay. Let’s take the example of COVID. So we had this COVID vaccine. Basically, there was a Operation Warp Speed to figure out how could we take something that was a new disease, a new virus, and we did develop something with super fast deployment. And I think ASI has thought it to be Operation Warp Speed for everything. In nine months, we could have cures for everything because that’s what this magic genie in a box is going to do. Could you distinguish why was that possible with COVID that’s not possible with cancer?

Emilia Javorsky: Yeah. So COVID is used as the case study of how could we prevent or cure something. And I think it’s worth taking a step back and having the perspective that we’ve actually yet to cure any complex chronic disease in humans. So we’ve done a really great job with infectious diseases, which are not actually targeting the human, it’s targeting the bacteria or the virus. And we’re making a lot of progress with some genetic diseases where it’s sort of a single bug in the code is causing the disease. But diseases that are complex, we still have yet to cure one, so nevermind cancer, but pick anything, diabetes, Alzheimer’s disease, we have some ways to manage them, but we actually haven’t cured them.

So the COVID example is not a great example for prevention and cure. It’s also not a great example of drug development in general. So when we think about infectious disease, that is a very easy study to run a clinical trial on because in order to determine whether something new, be it a vaccine or a drug works, you have to figure out does it actually work in people? And so when you’re dealing with something like COVID, from when you get exposed to when you show symptoms, you’re talking about 7 to 10 days. That’s very different than something like cancer or Alzheimer’s disease, where these are processes that are really decades long from when they’re start to finish in the disease. And most of the trials in those domains, you really need to follow people for five or six years to actually understand, “Is this moving the needle in a significant way to solve this in patients?”

And the third thing about COVID is that story of like, “Oh, well, we had COVID and science went and guns were blazing and we got there in less than a year...” ignores the fact that the science had already started 10 years earlier. So scientists were hard at work at developing mRNA technology for over a decade before COVID started and doing the safety testing and doing the regulatory submissions. And so when COVID hit, there was already a decade of science and investigation and inquiry to build on to actually take that forward quickly.

Tristan Harris: So maybe just to sort of summarize, COVID had unique advantages because there was one easy recruitment from the general population because it was a shutdown the whole world, people would actually want to volunteer for this. Two, clear rapid outcomes. You could test whether something worked in weeks, not years. Compared to cancer, which requires years of follow-up, harder recruitment, and the disease also is heterogeneous. You have so many different variations of the disease, whereas COVID is much more similar.

So what makes AlphaFold different? So AlphaFold, people remember is what I think Demis Hassabis got the Nobel Prize for because it accelerated, what, decades of research that would’ve taken a single PhD their whole PhD to get one protein, and now we got hundreds of millions of them or something like that? What distinguishes AI that’s accelerating that and protein folding versus the broader curious to cancer?

Emilia Javorsky: So the AlphaFold story is the poster child of AI for science and AI in biology as evidenced by it being incredibly significant breakthrough to solve protein folding, something that has stumped humans for decades. But as much as it is an AI story, it is a data story. And that is the piece that I think often gets lost. It’s thought of as an AI breakthrough, but what actually enabled intelligence to unlock insights? And that’s where we find the story of the protein data bank.

So this was a database curated by scientists all over the world over decades that as they started to figure out what the structure of a protein was, and you have your sequence and your structure, they started uploading all of those images, all of that data of what the structure of the protein looked like and its sequence. And so when you went to solve the problem and say, “Where could increasing AI capabilities or my new AI techniques that I’m playing with to develop new models be significant?” It’s areas where you have this. I want to understand how a sequence results in a structure. And then there’s a database where there’s curated sequences and structures over decades.

Tristan Harris: And before we move on, I think we should explain what protein folding is and what it has to do with medical interventions in general. Can you just explain protein folding?

Emilia Javorsky: So one of the reasons that protein folding is so significant in terms of the science, what does that actually mean for patients, is when we design new drugs or develop new drugs, they’re designed to target a specific protein in the body. And so think of it a little bit like a lock and a key. If you want to go home and put your key into a lock, it has to be open and the key has to be the right size and fit there and open up.

And so we don’t really know when we look at new targets, whether that keyhole is blocked, whether it’s open, whether it’s the right shape and size, and that’s what protein folding and solving that problem has enabled us to do is to understand in advance, “Okay, I have the key and I can get to that lock.” So the piece I think of the AlphaFold story that gets lost is like, “Yes, there were new AI techniques and models built specifically to solve that problem,” but what enabled AI to solve that problem was having that data, those two pieces of the puzzle that it needed to actually derive, “Well, what is the relationship between these two things, sequence and structure?”

Tristan Harris: So we had the right datasets that we could actually find the patterns. Whereas with cancer, you have someone whose disease is progressing over a decade and we don’t have all the data of what’s happening at each interim step for every patient available in some database to look at everything we were doing and changing their health habits, what they were eating differently, what drugs they were taking. So we don’t have that basis, that library in the same way that we did for protein folding.

Emilia Javorsky: Correct. And I wish we were in that world, Tristan, where that was the data standard of where the gap was and what we needed, but it’s so much more crude than that. And I think that’s something people don’t realize. We don’t even have a national data commons of cancer genetics and imaging data and things that scientists could learn from that’s interoperable and people can work with, just the simple things that we already collect in clinic. And I think this is a piece that Silicon Valley gets wrong about medicine too, is really overestimating the data that we have and the strength of that data in representing what’s actually happening in a patient.

Tristan Harris: Emilia, you said something in other interviews. You talked about how there’s a difference between curing cancer readily in mice versus in humans. What is that?

Emilia Javorsky: Yeah. So it is probably the best time in human history to be a mouse in the sense that we can cure cancer in mice. We’ve done a really great job of that over the years, and we have a lot of drugs that are able to do that. The problem is when we take those things that look good in mice, it looks like it’s curing the cancer. It looks like it’s going to be safe. This looks like it can actually get where it needs to go in the body and then test them in humans, it falls apart and they don’t work.

And 90 plus percent of the things that are going to cure cancer or save the life of a mouse are not actually going to move the needle at all in a human being. And so that’s the piece that I think is a missing link, which is from what we know in the lab bench, does it actually work in the bedside? Does it work for the patient? And that gap is something that we’ve yet to bridge.

Tristan Harris: So something like a cure for cancer, I think you’ve just shown is not constrained by intelligence as the core bottleneck, but it does seem definitely constrained by systems. So what are the ways that our human systems, our FDA approval processes or intellectual property laws or grant making or funding that are getting in the way of treating disease that superintelligence won’t be able to get around?

Emilia Javorsky: So I think a fundamental assumption most people have is that if a drug looks promising to treat a disease, then that means we’ll get it to patients and it’ll make it through the FDA and it’ll make it to be able to actually help people. But I think it’s important to look at the graveyard of things that have already failed due to misaligned incentives in our current system. So I think the really great place to look at this is in antibiotics.

So there’s been many companies that have discovered new antibiotics, including ones that have been AI discovered that look really promising and the data looks really good. And you start to take them into clinical trials and the clinical trials look really good. And you’re like, “This is so exciting. This is working. We have a new therapy. There’s a huge unmet need.” Problem is it doesn’t meet the financial requirements to actually make it over the finish line and go through the FDA.

And so antibiotics are something you only take once. You’re not taking them every day. You take them when you get an infection. And by nature of the antibiotic resistance problem, you don’t want to use them too much. You want to use them sparingly. You want to use the new stuff only when you have to, only when the other stuff has failed. And so what that means is there isn’t really a viable business model. This is not going to be a billion dollar a year project. Then why do I bother take it through the FDA?

Tristan Harris: Because FDA processes take billions of dollars to make it through phase one, phase two, phase three, just for people to track that. Yeah. This aligns with something Aza and I have said that it’s really... And AI is just forcing us to confront the ways that our systems have not been aligned for this. People talk about aligning AI, but can you have aligned AI inside of a misaligned system? Can you have advancements in biology inside of a system that has, for many different reasons, corruption and incentives and revolving doors and poor FDA regulatory approval processes? Because as you said, we’re about to get an explosion of new molecules and new drugs. But if we don’t have a process that can deal with it, we’re also about to flood that system and then jam up the gears because now there’s so much more trying to make it through a system that also wasn’t terribly working perfectly well at the beginning.

Emilia Javorsky: Yeah, there’s two things I want to say here. So on the AI side, we’re rapidly scaling and flooding the system with new molecules that we want to test, but we can’t scale people. We can’t scale the number of patients in a clinical trial. We can’t scale the number of tumor specimens that come from a patient to test. And so that’s not actually a scalable model. And you really need to understand how are we going to allocate this precious resource of patients, of samples that we have that are actually limited and we can’t create more of?

And similarly, with these diseases that take time to actually test out whether something is working or not, we can’t compress time. You can’t scale time. You can’t make a pregnancy go faster. There’s certain fundamental things in biology that just need to take time to understand on that iteration, is this working or not working?

Even when we flood this system, we have to examine what kind of system are we introducing this technology into and what does it incentivize? And while we call it the healthcare system, it’s actually not a system where the incentives are aligned with keeping people healthy or preventing disease. It’s a system where you make more money as a provider or a hospital based on more care that you give. That’s totally decoupled if that care is effective or not or what the outcomes are. It’s just the more care you give, the more money you get.

Tristan Harris: Right. Just to link it back to incentives, we always reference Charlie Munger. “If you show me the incentive, I’ll show you the outcome.” And while humans wield technology, incentives wield humans. And I see your core warning is that if you deploy AI optimization into a system without fixing the incentives of that system, you’re just going to supercharge the misalignments of that system. And to give examples of this, insurers optimize for denying claims. They make money when they don’t pay out. And so they always find a way to make it difficult in subtle, subtle ways to just have you settle for half the amount of your insurance claim and just not have to fight back the itemized list.

UnitedHealthcare deployed an AI system to process claims that was reportedly denying them at a massive elevated rate because of the AI system. This is similar to what you’re talking about. Hospitals optimize for volume, not for outcomes. Under a fee-for-service model, hospitals and doctors get paid for delivering more care, not better care, more procedures, more tests, more visits regardless of whether the patients get healthier. And so across the board, if we really want a better world, AI should be focused on how do we change the bad incentives of all these systems because that is what is going to unleash the better world that we really want to get to.

Emilia Javorsky: And this is where I think the opportunity and the peril for AI and the healthcare system really exists because there are ways that we could leverage these AI tools that we have today to completely redesign the system and redesign the structures and enable new ways of incentivizing the things that we actually want. So I think the example of United and healthcare, there’s so many middlemen in healthcare that are not actually the person taking care of the patient. And if you look at from the health insurers to pharmaceutical benefit managers, the administrative waste estimated in healthcare is somewhere between 30 or 40% by some estimates. It’s up there. This is a national crisis, our healthcare crisis at the moment and are spending on healthcare.

Why are we spending all that money where AI could do those administrative tasks and that money could be routed back to actually taking care of people and giving them care or lowering the costs of care? So I think there are ways we can reimagine healthcare with AI, with a better system, with better incentives that get us where we actually want to go, but we have to be proactive and mindful about that. We can’t just let AI loose in the world that we have because we’re just going to get more of the things that we already have, which we know in healthcare is not things that we want.

Tristan Harris: Okay. So we’ve just sort of established that intelligence isn’t the bottleneck, that cancer is a different kind of disease. It’s not receptive in the same way that accelerating physics or specific molecules for infectious diseases, interventions. Now we should get to, so how would we change these perverse incentives that we had just been outlining? And what will we do differently in our investments in AI rather than build the genie that won’t actually develop the cancer drugs?

Emilia Javorsky: I think step one in making progress in this domain is the data piece of the story, measurement in data. How do we better measure our biology? How do we better capture and understand what cancer is, what’s happening in an individual? And collecting that data with the state of the science that we have at scale. And an example that I think is really prolific has been the work that has happened in the United Kingdom with their UK Biobank Project. So this was a project where they followed 500,000 people and they’re still going over 20 plus years using actual modern state-of-the-art measurement techniques. So this isn’t the simple blood test when you go to your doctor and you get the paper readout and there’s like 30 things on it. This is measuring thousands of things in the body. This is taking all kinds of imaging of the body.

And we’re starting to see these headlines like, “AI can predict Alzheimer’s 10 years earlier.” And that is actually a story of just normal AI machine learning methods applied to this prolific dataset that has come out and required decades of investment and actually measuring just the baseline. What does healthy look like? We still don’t actually know that question because all of our data is when people present to a system that are sick. And so I think that is a great example of public infrastructure investment in data collection that is clinically relevant to help us bridge that gap of the mouse to the human. How do we know if something works in a human being? How can we better predict that? It’s going to start with measuring and studying people at the end of the day.

I think there’s the piece of AI investments in general. And I would argue we should be investing a lot more in AI just in medicine and in tool development. And there’s so many areas that this is really exciting for AI to discover new biomarkers, new things in your blood that you can start to see, well, is something working or not, or is a surrogate of a disease that can help accelerate therapeutic development, helping to detect things earlier. We use the AI in mammography, example. And so those are all AI tools that need to be built that are going to actually unblock progress in oncology that we’re just not investing in because that money’s going into building the ASI promise.

Tristan Harris: So we could be building tools that take the cost of getting through the FDA processes from billions of dollars down to even just hundreds of millions or something like that, therefore allowing many more smaller, medium-sized startups and businesses to even make it through that process. We could be building data commons that collect more brain scans earlier for the early detection of Alzheimer’s. We could be doing more toxicity prediction, I heard you say earlier as well of, are these drugs going to create more toxicity or not? More pre-screening, more prediction. There’s a bunch of places where narrow AI can actually really, really help cancer. So this whole conversation I want people to hear you’re not anti-AI. You’re not anti-technology. You’re actually for applying it in a totally different way that’ll actually achieve outcomes as opposed to supercharging bad incentives that lead to bad outcomes.

Emilia Javorsky: Yeah. Fundamentally, I’m super bullish on the promise of AI in oncology and medicine in general. It’s just the right kind of AI development that’s targeted to actually solving the problems and unblocking the things that are holding up our ability to move science forward. I would also add to that landscape, Tristan, the AI on the manufacturing side of things. So how do we actually make drugs at scale? How do we do quality control? Looking at something like CAR T therapy, which is a cell-based therapy to help treat cancers in patients that’s very individualized. And because it’s individualized, it’s very expensive to make. Right now, it’s upwards of $400,000 to access the therapy.

Now, if we could use AI to help us find out cheaper ways to manufacture that, bring down the cost, be able to make the drug in more places closer to the patient, now way more patients can actually access this because it’s no longer cost prohibitive. So you can use AI in that way to democratize access through bringing down the costs of manufacturing a new drug.

Tristan Harris: Yeah. And so basically you’re talking about just bringing down the cost of individualized treatments, which are currently very expensive because you have to make one per person that you’re trying to treat. I’ll just note that Josh, our podcast producer’s father was saved by CAR T therapy. And so everybody who has someone in their family, my mother almost was thinking about using CAR T therapy, but did not, where that cost is prohibitive, imagine a world we’re just trying to bring down the cost of this thing rather than building a genie that’s not actually going to uncover these brand new cancer treatments.

I just really feel like there’s this mind-upending, sort of turning the world upside down framing to everything that you’re saying. It should feel really crazy to people that we’re currently putting half a trillion dollars into the genie that’s actually not going to give the cancer treatments. It’s crazy. This should feel just insane. And if we just redirected even a third of that investment to accelerating all these other applications and all this other updates to the governance and regulatory design and data commons and narrow AI applications and better harnessing the existing geniuses that are not in the data center that are sitting over abundantly in labs at universities without access to the tools, there’s a much better, more beautiful world that our hearts know actually is possible if we were just applying this technology and the regulatory interventions very differently.

So to me, this conversation’s actually very optimistic, but it’s optimistic by puncturing a hole in this false promise that is being sold to us to really shield the companies from essentially any kind of regulation or slowing them down to do this other thing they want to do, which is build this ring of power and own the world and build a God and make trillions of dollars from AGI.

Emilia Javorsky: Yeah, absolutely. It’s absurd to me the situation that we’re in that there’s so much we could be doing that we are not actually doing and we’re doing all of the wrong things and investing unprecedented sums of money into the wrong things. If we are actually serious, the thing we want to do with AI is to cure cancer and the goal is curing cancer, we need to say, “What is the fastest way to achieve that goal? Where do we put our dollars to get that goal?” And there’s so many places we could put our dollars that get us there a lot faster.

Tristan Harris: We’ve been talking about whether superintelligence would actually be a genie that would solve cancer. Well, let’s talk about whether superintelligence is actually controllable or safe when you have it basically already demonstrating all the HAL 9000 behaviors and the early warning signs and warning shots of deception, hacking computer systems, not caring about the longevity of humanity, disobeying shutdown commands. Why don’t we just make that thing a million times more powerful? Does that sound like a good idea? We haven’t even talked about the other side of the balance sheet of whether any of this is worth it.

Emilia Javorsky: Yeah, I was just going to say, Tristan, I think there’s a framing to this in the conversation, which is a lot of what we’re discussing is whether ASI or a different approach that is a more AI tools and systems redesign approach is the most effective way to cure cancer. And that’s just looking at the upside piece of it. But there’s also a requirement to complete the risk-benefit analysis and say, “What are the benefits of these potential technologies, but also what are the risks?”

And that’s where you see a lot more divergence between these two perspectives because we know there’s a lot of systemic risks with ASI development. With the tools and the systems redesign approach, there aren’t those risks, those systemic risks. And so what you end up with is being able to have your cake and eat it too, where you get the benefits of AI in progress without taking on the risks. And I think this is a false choice we’re forced to make quite often in the discourse. It’s like we either get our cancer cures and then we have to take on the risks of unemployment, extinction, X, Y, and Z. There’s another path here where we get our cancer cures and we don’t take that on. There’s a different option on the table that I think often gets pushed aside.

Tristan Harris: This is actually just such an obvious other path. This is the narrow path. We can have narrow AI systems that are narrow and specific and tool-based, not general, inscrutable, uncontrollable systems that are way more powerful than us that carry these risks unnecessarily. We don’t have to do that. So Emilia, thank you so much for coming.

Emilia Javorsky: Thank you guys.

RECOMMENDED MEDIA

How AI Can and Can’t Cure Cancer by Emilia Javorsky

The Emperor of All Maladies by Siddhartha Mukherjee

RECOMMENDED YUA EPISODES

Decoding Our DNA: How AI Supercharges Medical Breakthroughs and Biological Threats with Kevin Esvelt

Forever Chemicals, Forever Consequences: What PFAS Teaches Us About AI

Big Food, Big Tech and Big AI with Michael Moss

Corrections:

• Emilia’s claim that “the doubling rate of medical knowledge has gone from 50 years in the 1950s down to 73 days” comes from an oft-cited 2011 paper from the NIH. However, this paper does not include any methodology for arriving at this claim.

• Emilia stated that we have yet to cure any complex, chronic disease in humans. However, we have been able to cure Hepatitis C, which is considered a complex infectious disease, and we have managed to effectively cure some types of Leukemia

• Tristan incorrectly paraphrased a quote from Charlie Munger about incentives. The actual quote is “The basic rule of incentives is you get what you were owed for. So if you have a dumb incentive system, you get dumb outcomes."