The moral critic of the AI industry—a Q&A with Holly Elmore

Since AI was first conceived of as a serious technology, some people wondered whether it might bring about the end of humanity. For some, this concern was simply logical. Human individuals have caused catastrophes throughout history, and powerful AI, which would not be bounded in the same way, might therefore pose even worse dangers.

In recent times, as the capabilities of AI have grown larger, one might have thought that its existential risks would also have become more obvious in nature. And in some ways, they have. It is increasingly easy to see how AI could pose severe risks now that it is being endowed with agency, for example, or being put in control of military weaponry.

On the other hand, the existential risks of AI have become more murky. Corporations increasingly sell powerful AI as just another consumer technology. They talk blandly about giving it the capability to improve itself, without setting any boundaries. They perform safety research, even while racing to increase performance. And, while they might acknowledge existential risks of AI, in some cases, they tend to disregard serious problems with other, closely related technologies. 

The rising ambiguity of the AI issue has led to introspection and self-questioning in the AI safety community, chiefly concerned about existential risks for humanity. Consider what happened in November, when a prominent researcher named Joe Carlsmith, who had worked at the grantmaking organization called Open Philanthropy (recently renamed as Coefficient Giving), announced that he would be joining the leading generative AI company, Anthropic. 

There was one community member on Twitter/X, named Holly Elmore, who provided a typically critical commentary: "Sellout," she wrote, succinctly.

Prior to seeing Elmore's post, I had felt that Carlsmith probably deserved, if not sympathy—making a decision that would presumably be highly lucrative for himself—at least a measure of understanding. He had provided, in a long post, what had seemed to me an anguished reasoning for his decision. "I think the technology being built by companies like Anthropic has a significant .. probability of destroying the entire future of the human species," he wrote. But for Elmore, this didn't matter. "The post is grade A cope," she concluded.

Elmore's response made me ask myself whether I had been overly forgiving. And in the last several years, everyone concerned about the existential risks of AI has had to ask themselves similar questions. Therefore, rather than stirring up controversy, Elmore's perspective has tended to feel clarifying, at least for me personally. Whether you agree or disagree with her opinions, they allow you to evaluate your own opinion with greater certainty.

I wanted to interview Elmore for Foom for several reasons, more broadly. First, because a core purpose of this website is to provide news and analysis on research in AI safety. And, in deciding what research to write about, it is essential to understand the conflicts faced by researchers at leading AI companies, who, confusingly, also produce some of the most important technical studies. 

Second, Elmore has become an important figure in fighting for an alternative, non-technical solution to the problem of AI safety: To pause or temporarily halt AI development, completely. Towards that end, she founded a non-profit organization in 2023/2024 called Pause AI US. Anyone interested in the science of AI must also understand where non-technical solutions might need to come into play.

To understand Elmore's positions better, and how she came to them, I spoke with her in November and December. But before I get into our interview, I want to explain a little more of her backstory.

A changing community

Compared to some, Elmore is a relatively new entrant to discussions of AI safety. She first started thinking seriously about the topic at some point between 2013 and 2020, when she was a PhD student in evolutionary biology at Harvard University.

At Harvard, she regularly attended and eventually co-organized meetups for discussions about how to approach various problems facing humanity. These meetups were part of a growing movement and philosophy called effective altruism. During this time, Elmore also began regularly posting on an EA-affiliated forum called (unsurprisingly) the EA forum. Her early posts tended to revolve around themes like personal experience, psychology, and animal welfare. 

To my knowledge, Elmore first posted publicly on the topic of AI safety in July 2022. This was in a very different internet forum, called Less Wrong, founded in 2009 by the AI researcher and writer Eliezer Yudkowsky. It was called Less Wrong, in part, because Yudkowsky sought to do focused writing on the nature and problems with human rationality, which he saw as a key reason why the existential risks of AI were being underappreciated by many.

In 2010, when I had been a graduate student in physics at the University of Wisconsin, I had randomly come across the Less Wrong forum and began participating. In those early days, a perhaps peculiar characteristic of the forum was that the existential risks of AI were not discussed with a great deal of anxiety. Powerful AI didn't exist yet, and moreover, no one knew how to build it, or how long it would take to be built, one day. Therefore, you could largely just have fun with it as an interesting theoretical problem.

The situation started to change in May 2020, when the company OpenAI released the program called GPT-3, which looked surprisingly similar to a real human language intelligence in a machine. Further, there was now a plausible pathway to increasing the program's capabilities. This plausibility was soon proven by the release of much more capable programs, like the same company's November 2022 release of ChatGPT. 

Participants in public discussion forums like Less Wrong, including Elmore, started feeling more worried. And, these worries were increasingly being channeled into formal activities, like grantmaking and community organizing. 

In March 2023, an organization called the Future of Life Institute (FLI) published an open letter calling for a literal pause in the accelerating development of AI technology. Signed by many distinguished scientists, the letter also received widespread recognition in online discussion forums like Less Wrong and the EA forum.

Indeed, while the letter was likely only intended to send a signal, or to serve as a token of consensus, Elmore took the idea of pausing quite seriously. A few months later, she wrote her own post on the EA Forum laying out her own case for pausing. Her view was heavily influenced by public polls of US citizens, released shortly after FLI's statement, which showed a majority of the public would actually be in support of something like pausing.

However, there were numerous reasons to doubt or criticize a pause strategy. Was it really realistic to try to resist what felt like an inexorable goldrush to a new technology? It would have been easy to see the whole notion of pausing die away.

Instead, a small handful of people, including Elmore, kept going. In 2023 and 2024, Elmore and another individual, Joep Meindertsma, founded (or co-founded) closely related organizations called Pause AI US and Pause AI Global, respectively. According to Elmore, Pause AI US is now funded through a mixture of grants from organizations such as FLI, major donations, and smaller contributions. 

It would be fair to question how much progress these organizations have made towards pausing. Their staple activities are more incremental in nature, including encouraging members of the public to reach out to representatives, organizing protests, and doing public educational work as well as messaging.

In our interview, Elmore said she sees the biggest accomplishment of her organization to be its shifting of the boundaries of what is considered socially acceptable to call for in discussions of AI safety. These boundaries of social acceptability are sometimes referred to as an Overton Window, a concept that Elmore often referenced in our discussions.

Overton windows are hard to measure, as is their direct impact on society. But, it is at least true that calls for pausing—or what Carlsmith referred to in his blog post as "capability restraints"—have not gone away. Arguably, they have strengthened, and they remain part of the public discussions both inside and outside of AI safety. Last week, Bernie Sanders, the Vermont senator, separately called for a moratorium on building datacenters, acknowledging AI and robotics as being "the most transformative technologies in human history," something that AI safety proponents have long been saying.  

In what follows, I discussed with Elmore how she became concerned about AI, how she wound up founding Pause AI US, and the influences that have shaped her. The interview has been condensed and edited for clarity. 

Author's note: Initial funding for this website was provided in October 2025 by a grant from the Effective Ventures Foundation, an organization associated with the EA movement. In November 2022, the author separately received an independent journalism grant from the Open Philanthropy organization. Foom is an editorially independent publication. 

How did you first start thinking about AI safety?

My introduction was through effective altruism (EA). I was very involved in EA, starting in grad school. And for six years I played a leading role in Harvard University EA. My primary issue going in had always been animal stuff, but I was also very concerned about poverty. AI safety was another issue people thought about, and I was always very confused by that. But what I was doing for my PhD was studying evolutionary biology. And it leads to a lot of 'black pills' about just how easy it is for species to be edged out, and how much this is the default story. In history, of all of the species that have ever existed, most of them have gone extinct, often because of direct competition with other species. And so eventually I was like, well, 'What do I think is going to happen if we build smarter than human intelligence?' And the risks became very believable to me.

Do others in the evolutionary biology community tend to think similarly?

Absolutely not. They don't know enough about AI. I think a lot of them probably don't realize how much the method of training AI programs (called gradient descent) can be seen as an evolutionary process, for example. A collaborator of mine, Oliver Sourbut, wrote a proof showing the conditions under which natural section and gradient descent have the same outcome. They're very similar. And there's not an appreciation of this, generally. There's a belief that we have created AI programs that are highly predictable and controlled, when we haven't. And the only thing that would prime you for understanding that is having an empirical knowledge of evolution [and the way it creates highly unexpected outcomes]. I do think that pausing AI would be a conclusion that a lot of evolutionary biology people would reach, if nothing else was intervening. But industry is very against people coming to that conclusion.

It can be a depressing problem to think about, the idea of AI ending the world; as you've written about in the past. Something that keeps you from getting out of bed, even.

Well, when you go through that, you'd think if you accepted that thought, it would just end everything. But it doesn't, and the day comes again, and it gets to that point where you've got to go to the store, or whatever. And just because you accept this horrible possibility, that itself doesn't end the world. 

What caused you to develop such a strong interest in moral and ethical issues such as animal welfare and then AI safety?

I can't ultimately say where it came from, but I had that interest early. The earliest issue for me was vegetarianism. My mom says that I wanted to become a vegetarian because my kindergarten teacher was vegetarian. And I remember her getting tofu dogs for me for a class cookout and me thinking, 'Well, is this okay?', because definitely all of my favorite foods were meat, up until then. But I got a sense of relief at learning there was something I could do. And thinking about how if I was in the same situation as an animal, then all I could hope for was that somebody else would care about me, in the same way. I didn't know how to actually eliminate all the animal stuff from my diet, for a long time. But my parents did respect my moral feelings. They thought it was important that I be able to follow my conscience, even if it wasn't their own. If I had not had that support, I think things would have turned out differently. 

How did you go from seeing the March 2023 letter from FLI calling for a pause to actually forming your own advocacy organization?

I started out not at all intending to leave my job at the time. I was just so excited by the public backing of pausing, shown in polls, that I wanted to help volunteer somehow. I assumed there would be a public campaign, or public protests, and I wanted to be part of them. I just assumed they would be led by someone more knowledgeable about AI safety. So, I started trying to engage people, and then, I did a protest, and I was like, 'Hey, come do this protest!' And I just kept looking around, and having interviews with people, and then being very disappointed with their answers. The one person I connected with that ended up being the most important was ​​Joep Meindertsma. We met on Slack [a messaging platform], where we both realized we were calling our own separate ideas the same thing, Pause AI, and we had both got the idea from the FLI letter. And we both had a very similar vision, initially.

You say you experienced a lot of disappointment when you started to explore the idea of pausing. 

I just had no idea that I would come to represent the public face of the entire thing. When people started talking about pausing, I started asking, 'Why not do this? This seems great.' For example, it was clear what you were talking about, compared to something like technical AI alignment research, which is such a complicated solution to the existential risk problem. Like, alignment to whom? And I was so happy and relieved that there was this other possibility. But other people were not happy about it. I was living in Berkeley at the time, and I was talking to a lot of experts and trying to understand why they weren't excited. And to be clear, it's not that I expected everybody to agree, at first. Some people might legitimately just have a different perspective, or think the answer is different. But there was a missing mood. For all my time in EA and rationality, everybody had been talking about preventing the deaths of everyone from AI. And it made me think, 'We didn't all want to save everyone's lives, did we?' 

Where was this pushback against pausing coming from? 

It's quite hard to say with the groups I participated in, like the EA and rationality groups, because they are quite decentralized. And some of the people in these groups did join me very early on, like a lot of the animal EAs. Some of them moved very quickly. But I would say the more central you got in the EA network graph, the more I was seeing a missing mood. I had also assumed there would be an Open Philanthropy sponsored organization working on pausing—but there wasn't. In retrospect, I think for some of these people, it's not that hard to explain. I think a lot of early stuff built around Eliezer's AI safety ideas, what I'll call the Less Wrong account, was actually upholding other ideas as well, like libertarianism. And a lot of people influenced by this account didn't want to bring the issue to the public and just have them say, 'Let's make [an AI race] against the law.' Because they did want to build powerful, smarter than human intelligence; they just wanted it to be done right, or done in a way that somehow threaded the needle. 

While you have seen resistance, you've also seen a lot of support, especially on social media. How have you managed to use social media so effectively? 

Anytime I got on a new social media platform I would grow pretty organically—and have fun doing that. I guess I never did it on purpose really. I was kind of late to Twitter (in 2021). Before that, I had used Facebook extensively. Facebook helped me with my depression because it offered a level of interaction that I could control. I could feel safe engaging, even when I had limited energy. On the other hand, I did happen to get introduced to AI safety through in-person discussions at Harvard EA meetings, and I think that helped me. People who encounter the idea of AI safety online often interpret it as if there's some kind of orthodoxy. It's like a little mystery you're initiated into, and if you don't like the way it sounds, then you're made to feel like it's a problem with you, not AI safety.

You've also become well known on social media as a critic of public figures, as with your recent criticism of Carlsmith. Was it stressful to make that post about him? 

I had been trying this strategy of directly calling people out for a while,and it didn't seem like it made much of a splash, but it did with Joe Carlsmith. A lot of people really didn't want to hear that. A lot of people did not like that. And they scolded me about it—and that hurt. It made me question why I'm in this adversarial position, and go through all my thinking again. But I know that's how it feels, also, to people like Joe, to be scolded. And whereas I get challenged constantly, with people being super upset about pausing, people like Joe tend to get zero pushback. And, I thought, maybe it's not too late. If being called a sellout hurts, and he has this occasion for reflection, then maybe he'd quit. And that would be pretty big. He's a load-bearing personality in the AI safety community. And, (all I did was) I called him a sellout. I don't think it's that big of a deal.

In his post, he provided a lot of rationale for his decision. He explicitly considered—and disagreed with—an idea similar to pausing, which he referred to as 'capability restraint.' And he directly acknowledged the existential risks posed by the company he was joining.

He expects to get credit for saying that—but then there's no action that's expected. All that shows is that he knows better than anyone what he's doing. 

Nonetheless, how did you cut through all these layers to come to such a firm judgement that he was wrong; that he deserved, in your eyes, to basically be publicly shamed for his decision?

I personally have always liked Joe. I love his writing on most stuff. But just from knowing Joe, a little bit, I had a sense that emotionally, he was coming from a place of feeling that something  was wrong. I just got the vibe that he knew better. Like even if that was his best attempt to figure things out, why would the answer, after writing that post, be to work at Anthropic? The preponderance of evidence in that post goes against that. It's also not clear that Joe as a philosopher really has a plan for solving safety or alignment—but he's just casting his lot in, anyways. It felt like he was gish galloping himself—a debate fallacy where you just lay out a bunch of stuff, and even if it's bullshit, it takes so long to debunk that the other person just can't deal with it. And it felt clear to me from his post that he probably felt the right, but difficult, thing to do would be to miss out on Anthropic. And he didn't do that. 

Is there an alternative where employees at leading AI companies could take action internally, or take action by unionizing?

I disagree with the insurgency theory of change. If your goal is to join an organization to change it, how are you equipped to resist it changing you? Surely, everything is stacked in its favor. You're entering it. You're being paid by it. You're surrounded by its people, its culture. 

Unionizing is more of a strategy. Tech workers unionizing would be huge. I'm interested in the tech unions that already exist. I would love to get those people seeding unions at AI companies. A challenge is that industry is extremely incentivized against unions. And in academia, I don't think unions would be the top priority. Within academia, you could just start having petitions, proposals, or ethics boards at your school, even if it did not directly influence industry.

You're almost alluding to the idea of academics taking up a kind of Hippocratic oath for AI.

Yes, and they wouldn't need the permission of industry, which is the problem that Pause runs into constantly. To get a pause to happen, you would need the permission of the people who are the most intent on building AI. But to take a Hippocratic oath, you don't need anybody's permission. That's something that computer science professors could get together and do on their own.

Is there something wrong with the culture of computer science and machine learning, that there tends to be so little questioning on safety and ethics?

I think for computer science they're just very used to not having ethical issues. There's a different paradigm, for example, in psychology, where you have to think about this stuff, or you have to get institutional review board (IRB) permission just to give people a survey. There's an attitude in computer science that these things are just seen as getting in the way and inefficient. It's very much a part of recent history. People feeling, 'I don't have to follow any rules,' or 'I'm here to disrupt things, I don't have to work with society.' And when you come in with an ethical concern, you're seen as a scold, and then they complain you're slowing people down or you're not letting them build. And that's a huge dynamic even within AI safety, because it is one of these fields where people really want to be at the frontier of things. 

Looking back at the example of Carlsmith, do you think we would all actually benefit from being a little more confrontational about these sorts of problems? 

Yes. You don't think of yourself as confrontational if what you're saying is considered righteous by the people around you. And at that point, it goes from seeming like trouble making to seeming worthwhile. It's only seen as confrontational when you're in the minority. But when you're in the minority, it's also really useful. I think we really should be more confrontational, now, because now is the time when it really matters. These confrontations push the idea of pausing further into the center of the Overton window. And when something is in the center of the Overton window, it's just what we do as a society.