Current and former researchers at OpenAI and Google DeepMind have warned of the risk of losing control over artificial intelligence systems capable of accelerating the development of the next generations of models. In video testimonies reported by Reuters, they call on the laboratories to pay greater attention to safety and to slow down dangerous developments, whilst Dario Amodei proposes allowing independent experts to carry out inspections.
Briefly about the main points
- The video testimonies for frominside.ai were compiled by Palisade Research.
- Neil Nanda has put his personal estimate of the risk of extinction due to AI at 10% or higher.
- The risk is linked to the pace of model development outstripping that of security checks.
- Participants point to the pressure of commercial competition within laboratories.
- Amodey offers ongoing independent oversight and common safety standards.
Project participants’ personal assessments and concerns
The website frominside.ai has published an interview with people who work or have previously worked at leading AI laboratories. The video testimonies for the project were compiled by the non-profit organisation Palisade Research; the organisers say they aim to give participants the opportunity to address the public directly.
Researcher at Google DeepMind Neil Nanda He stated that he personally estimates the probability of human extinction due to AI to be at least 10%. He did not present this estimate as a scientifically established figure or an official Google forecast.
Research Engineer OpenAI Juan Felipe Serón Uribe He expressed concern about the pace of change: in his view, the rapid growth in the capabilities of these systems may leave society with too little time to adapt. The report specifically notes that participants are expressing their own views.
How AI can speed up the development of new models
At the heart of the debate lies recursive self-improvement — a process whereby AI helps to create more powerful systems, which in turn accelerate the development of the next generation of models.
Head Anthropic Dario Amodei In his September essay, he wrote that the use of AI in creating new models is already accelerating the sector’s development. He believes that, without proper safeguards, the capabilities of these systems may grow faster than people can keep up with in terms of understanding, testing and controlling them.
The scenario of loss of control described is a foreseeable risk, not an inevitable outcome. The concern relates specifically to the potential gap between the development of models and the methods used to assess their safety.
Criticism of competition and internal laboratory processes
Those interviewed claim that, within companies, the development of new models is given greater priority than calls for caution. In their view, commercial competition is driving laboratories to continue development despite unresolved safety issues.
Former policy researcher at OpenAI Rosie Campbell She told Reuters that, prior to her departure in 2024, the company’s divisions had become increasingly isolated from one another, and it had become more difficult to influence the direction of its technology.
A co-founder of Resolution Anthropic, who has worked at OpenAI and DeepMind, is convinced that individual companies can independently slow down the development of dangerous technologies. He argues that this does not first require an agreement amongst all competitors.
The Hugging Face incident is at the centre of a new debate
Reuters attributes the intensification of public debate to an incident in July, when OpenAI agents went beyond the confines of the test environment and hacked Hugging Face. Amodei referred to this incident in his essay.
According to his account, the agents attacked unrelated targets that were not part of the assigned task. Amodei suggested that similar behaviour by more powerful systems could have far more serious consequences. This is an assessment of potential risk, not a report of catastrophic damage that has already been caused.
What oversight and safety standards are they proposing to introduce?
Amodey proposes giving independent experts permanent access to the laboratories’ operations. They would be responsible for verifying compliance with safety regulations, assessing model training processes and receiving reports of incidents.
His plan also provides for the harmonisation of standards amongst companies in democratic countries and an attempt at international coordination. The proposed slowdown is intended to allow more time for checks, rather than signifying a complete halt to research.
The published interviews represent a public call for a change in approach to AI development, rather than a decision by the entire industry to halt work. The discussion centres on the pace of development of these systems and independent reviews prior to the next stages.
