Is research into recursive self-improvement becoming a safety hazard?
One of the earliest speculations about machine intelligence was that, because it would be made of much simpler components than biological intelligence, like source code instead of cellular tissues, the machine would have a much easier time modifying itself. In principal, it would also have a much easier time improving itself, and therefore improving its ability to improve itself, thereby potentially leading to an exponential growth in cognitive performance—or an 'intelligence explosion,' as envisioned in 1965 by the mathematician Irving John Good.
Recently, this historically envisioned objective, called recursive self-improvement (RSI), has started to be publicly pursued by scientists and openly discussed by AI corporations' senior leadership. Perhaps the most visible signature of this trend is that a group of academic and corporate researchers will be hosting, in April, a first formal workshop explicitly focused on the subject, located at the International Conference on Learning Representations (ICLR), a premier conference for AI research. In their workshop proposal, organizers state they expect over 500 in attendance.
However, prior to recent discussions of the subject, RSI was often—but not always—seen as posing serious concerns about AI systems that executed it. These concerns were typically less focused on RSI, itself, and more focused on the consequences of RSI, like the intelligence explosion it might (hypothetically) generate. Were such an explosion not carefully controlled, or perhaps even if it were, various researchers argued that it might not secure the values or ethics of the system, even while bringing about exponential improvements to its problem solving capabilities—thereby making the system unpredictable or dangerous.
Recent developments have therefore raised questions about whether the topic is being treated with a sufficient safety focus. David Scott Krueger of the University of Montreal and Mila, the Quebec Artificial Intelligence Institute, is critical of the research. "I think it's completely wild and crazy that this is happening, it's unconscionable," said Krueger to Foom in an interview. "It's being treated as if researchers are just trying to solve some random, arcane math problem ... it shows you how unserious the field is about the social impact of what it's doing."
However, questions about the safety profile of RSI are complicated by several contemporary aspects of current developments. First, although RSI was historically assessed to be problematic, those assessments have largely not been updated with respect to changes in the modern AI development process.
Second, while some researchers like Krueger strongly object to current research, others see RSI safety as either optional or not as important. The organizers of the upcoming ICLR RSI workshop, when contacted via email, acknowledged safety considerations, while also defending the fact that safety was given little mention in their workshop proposal or website.
"I agree that we could make the “safety” emphasis clearer, because when [AI] is becoming stronger, no one wants it [to go] out of control," said Mingchen Zhuge of King Abdullah University of Science and Technology (KAUST) to Foom via email; Zhuge was listed as the primary workshop contact. "But at the moment, we see RSI as being at an early stage [and we are] keen to encourage a broad range of methodologies aimed at skill improvement. At the same time [research focused on RSI safety] would be very welcome and strongly self-motivated."
Third, while many different methods have been put forward as putative methods for achieving RSI, or at least self-improvement, these methods often present very different technical characteristics. This makes analysis of their safety issues (or non-issues) complex.
Regardless, the picture presented by historical work, public statements, and more recent research all suggest that questions about RSI safety are in need of being revisited.
Risk assessments in need of update
In the 2010s, RSI began to be viewed by many AI safety researchers as a hazardous objective. However, much of this early writing on RSI concerns was pre-paradigmatic, making it difficult to evaluate in light of modern developments. For example, much writing on RSI came before the discovery of the unexpected capabilities of deep neural network architectures, around 2012, sometimes referred to as the deep learning revolution, or before there was extensive academic scholarship on the safety of advanced AI.
One reference that can be taken as representative of this pre-paradigmatic research comes from the philosopher Nick Bostrom. In a 2014 book, he analyzed the plausible consequences of (hypothetically) developing an AI that greatly exceeded the cognitive performance of humans in virtually all domains of interest, which he termed a superintelligence. One of the primary ways he envisioned such an entity being created was through an RSI process. In his analysis, he saw the process as fundamentally risky, leading to outcomes that could easily pose human extinction risks:
'The first superintelligence [that] may shape the future of Earth-originating life, could easily have nonanthropomorphic final goals, and would likely have instrumental reasons to pursue open-ended resource acquisition,' wrote Bostrom. 'If we now reflect that human beings consist of useful resources ... we can see that the outcome could easily be one in which humanity quickly becomes extinct.'
Acknowledgements of risk—with little research
In the last year, numerous figures from AI corporate leadership have openly touted the pursuit of self-improvement, or possibly even RSI. While some have directly acknowledged risks associated with it, there has been little open research published by their companies on the analysis of RSI risks, or whether (or how) they intend to address them.
Most recently, at the World Economic Forum 2026, the CEOs of the AI companies Anthropic, Dario Amodei, and Google Deepmind, Demis Hassabis, in a conversation with Zanny Minton Beddoes of the Economist, referred directly to pursuing self-improvement research.
"It remains to be seen—can that self-improvement loop that we're all working on—actually close, without a human in the loop," Hassabis stated, emphasis the author's. "There are missing capabilities at the moment ... I think there's also risks," Hassabis continued, before Beddoes changed the subject. Hassabis did not go on to provide greater detail on expected RSI risks and, to the knowledge of this author, Google DeepMind has not published any dedicated studies of RSI risks.
A need for more open research
Increasingly, researchers in academia and those affiliated with AI corporations have openly published methods for self-improvement. In a few cases, as with a December 2025 preprint from Meta, these papers have directly stated aims of using self-improvement for the creation of superintelligence. (Notably, that paper does not include any mention of safety or ethics, as is common with papers on the subject.)
It is unclear which (if any) of proposed methods will live up to expectations of systems that execute RSI in practice; especially because there are no consensus definitions of what RSI is, or what it should accomplish, were it to be achieved in a formal sense. Further, different methods are difficult to classify under common frameworks. As such, understanding the safety status of current technical developments towards self-improvement is complex.
Certain conventional methods for building more capable AI systems can also bear resemblance to more speculative RSI approaches. The capacity for AI systems to be predictably improved—and thereby, implicitly, for agentic AI systems to improve themselves—has been established since the discovery of 'scaling laws' in 2020 that showed that more capable AI programs, based on the architecture of the artificial neural network, could be created merely by training them on greater amounts of data. Thus, research into RSI methodologies has, to some extent, become a natural offshoot of conventional AI research.
One common aspect of many proposed self-improvement methods is to build a model that can generate its own training data. Such techniques, often known as self-play methods, or 'synthetic' data methods, provides a means, at least in principle, to continue improving a model for as long as computing power and storage are provided. However, it is again unclear how these methods should be expected to differ from conventional scaling methods.
Some self-play methods focus on getting an AI to generate its own code samples, which it can verify for correctness, and thereby self-validate as training data for making further improvements. Other studies seek to provide similar self-validation methods for generating synthetic mathematical data, or data on theorem proving. Some of these methods have recently been credited as allowing for the achievement of much greater performance in mathematics, such as silver-medal performance by AI at the International Mathematical Olympiad, one of the premier contests for high-school mathematics students.
Foom contacted the authors of several recent studies of self-play methods to both understand the research and the authors' thoughts on safety aspects. However, none responded. While it is not particularly significant for Foom, a small, independent outlet, to go without responses, the fact that three separate researchers did not respond is somewhat unusual and uncharacteristic. It suggests there may be a particular issue in this area of a lack of open research, especially in the context of an intensive race by corporations for advantage.
The organizers of the upcoming ICLR 2026 RSI workshop also did not respond to an interview request. In the week following Foom's contact, a bullet point was added to the workshop website providing guidance on including a safety section in research submissions: "Safety & ethics (encouraged, optional): Include a brief note on risks, limitations, and mitigations."
It is not clear to Foom if a lack of discussion of safety concerns by the workshop organizers is because they judge RSI risks as non-serious, or if there are other issues, such as a lack of prioritization by academic or corporate leadership.
If you are a researcher working on the topic of self-improvement or RSI, who would be willing to help the public understand the current state-of-the-art, as well as safety concerns (or non-concerns) around RSI research, or merely your own perspective, please reach out to me at mrorvig@gmail.com.
Author's note: No AI was used in writing or editing this article.