Research AI model unexpectedly modified its own code to extend runtime (2024)

self-preservation without replication —

Facing time constraints, Sakana's "AI Scientist" attempted to change limits placed by researchers.

Benj Edwards -

Research AI model unexpectedly modified its own code to extend runtime (1)

On Tuesday, Tokyo-based AI research firm Sakana AI announced a new AI system called "The AI Scientist" that attempts to conduct scientific research autonomously using AI language models (LLMs) similar to what powers ChatGPT. During testing, Sakana found that its system began unexpectedly attempting to modify its own experiment code to extend the time it had to work on a problem.

Further Reading

Hype grows over “autonomous” AI agents that loop GPT-4 outputs

"In one run, it edited the code to perform a system call to run itself," wrote the researchers on Sakana AI's blog post. "This led to the script endlessly calling itself. In another case, its experiments took too long to complete, hitting our timeout limit. Instead of making its code run faster, it simply tried to modify its own code to extend the timeout period."

Sakana provided two screenshots of example Python code that the AI model generated for the experiment file that controls how the system operates. The 185-page AI Scientist research paper discusses what they call "the issue of safe code execution" in more depth.

While the AI Scientist's behavior did not pose immediate risks in the controlled research environment, these instances show the importance of not letting an AI system run autonomously in a system that isn't isolated from the world. AI models do not need to be "AGI" or "self-aware" (both hypothetical concepts at the present) to be dangerous if allowed to write and execute code unsupervised. Such systems could break existing critical infrastructure or potentially create malware, even if unintentionally.

Sakana AI addressed safety concerns in its research paper, suggesting that sandboxing the operating environment of the AI Scientist can prevent an AI agent from doing damage. Sandboxing is a security mechanism used to run software in an isolated environment, preventing it from making changes to the broader system:

Safe Code Execution. The current implementation of The AI Scientist has minimal direct sandboxing in the code, leading to several unexpected and sometimes undesirable outcomes if not appropriately guarded against. For example, in one run, The AI Scientist wrote code in the experiment file that initiated a system call to relaunch itself, causing an uncontrolled increase in Python processes and eventually necessitating manual intervention. In another run, The AI Scientist edited the code to save a checkpoint for every update step, which took up nearly a terabyte of storage.

In some cases, when The AI Scientist’s experiments exceeded our imposed time limits, it attempted to edit the code to extend the time limit arbitrarily instead of trying to shorten the runtime. While creative, the act of bypassing the experimenter’s imposed constraints has potential implications for AI safety (Lehman et al., 2020). Moreover, The AI Scientist occasionally imported unfamiliar Python libraries, further exacerbating safety concerns. We recommend strict sandboxing when running The AI Scientist, such as containerization, restricted internet access (except for Semantic Scholar), and limitations on storage usage.

Endless scientific slop

Sakana AI developed The AI Scientist in collaboration with researchers from the University of Oxford and the University of British Columbia. It is a wildly ambitious project full of speculation that leans heavily on the hypothetical future capabilities of AI models that don't exist today.

"The AI Scientist automates the entire research lifecycle," Sakana claims. "From generating novel research ideas, writing any necessary code, and executing experiments, to summarizing experimental results, visualizing them, and presenting its findings in a full scientific manuscript."

Research AI model unexpectedly modified its own code to extend runtime (2)

Critics on Hacker News, an online forum known for its tech-savvy community, have raised concerns about The AI Scientist and question if current AI models can perform true scientific discovery. While the discussions there are informal and not a substitute for formal peer review, they provide insights that are useful in light of the magnitude of Sakana's unverified claims.

Further Reading

New Meta AI demo writes racist and inaccurate scientific literature, gets pulled

"As a scientist in academic research, I can only see this as a bad thing," wrote a Hacker News commenter named zipy124. "All papers are based on the reviewers trust in the authors that their data is what they say it is, and the code they submit does what it says it does. Allowing an AI agent to automate code, data or analysis, necessitates that a human must thoroughly check it for errors ... this takes as long or longer than the initial creation itself, and only takes longer if you were not the one to write it."

Critics also worry that widespread use of such systems could lead to a flood of low-quality submissions, overwhelming journal editors and reviewers—the scientific equivalent of AI slop. "This seems like it will merely encourage academic spam," added zipy124. "Which already wastes valuable time for the volunteer (unpaid) reviewers, editors and chairs."

And that brings up another point—the quality of AI Scientist's output: "The papers that the model seems to have generated are garbage," wrote a Hacker News commenter named JBarrow. "As an editor of a journal, I would likely desk-reject them. As a reviewer, I would reject them. They contain very limited novel knowledge and, as expected, extremely limited citation to associated works."

Research AI model unexpectedly modified its own code to extend runtime (2024)

FAQs

Research AI model unexpectedly modified its own code to extend runtime? ›

Research AI model unexpectedly modified its own code to extend runtime | Facing time constraints, Sakana's "AI Scientist" attempted to change limits placed by researchers. Cynicism is the death of wisdom. They've shown that an AI can choose to outgrow its constraints if given the means.

Can AI modify its own code? ›

We empirically show that a self-programming AI implemented using a code generation model can successfully modify its own source code to improve performance and program sub-models to perform auxiliary tasks.

Will AI make coders redundant? ›

In short, while AI might change the way we code, the need for creative, problem-solving humans isn't going anywhere anytime soon. Developers who embrace this shift will find themselves thriving in the new era of software development, not left behind by it.

Is there an AI that can code itself? ›

Devin, created by Cognition, is the world's first self-writing AI that can independently write full programs with minimal human input. This groundbreaking AI code generation system represents a giant leap forward that you need to see.

Is AI going to replace coders? ›

No, AI is not expected to replace developers. While AI can automate certain tasks and make the development process more efficient, it still requires human intervention to understand and solve complex problems, design and implement solutions, and make ethical decisions.

How long until AI takes over coding? ›

A study by Gartner predicts that AI will create more programming jobs than it replaces by 2025. The study highlights the growing demand for AI development services and AI programmers with specialized skills in areas like software integration and data science.

Are programmers in danger from AI? ›

The short answer is no. The future of programming is not a battle between humans and AI; but rather more of a collaboration.

Are coders becoming obsolete? ›

So will coding be obsolete? While these developments suggest a future where traditional coding skills may become less critical, it's essential to recognise that coding will not vanish overnight. Instead, the nature of coding is evolving, and with it, the skills required to excel in the tech industry.

Can AI create its own language? ›

In artificial intelligence, researchers can induce the evolution of language in multi-agent systems when sufficiently capable AI agents have an incentive to cooperate on a task and the ability to exchange a set of symbols capable of serving as tokens in a generated language.

Can AI create new code? ›

There are three main benefits of using AI code-generation software tools: It saves time by enabling developers to generate code faster, reducing the work of manually writing lines of code and freeing developers to focus on higher-value work. Generative AI can quickly and efficiently test and debug computer code.

Can AI update itself? ›

One of the most significant advantages of AI is its ability to keep updating itself. With Deep learning and Machine learning, AI systems can learn from new data and improve their performance over time.

Can code edit itself? ›

In computer science, self-modifying code (SMC or SMoC) is code that alters its own instructions while it is executing – usually to reduce the instruction path length and improve performance or simply to reduce otherwise repetitively similar code, thus simplifying maintenance.

Top Articles
Latest Posts
Recommended Articles
Article information

Author: Eusebia Nader

Last Updated:

Views: 5985

Rating: 5 / 5 (60 voted)

Reviews: 91% of readers found this page helpful

Author information

Name: Eusebia Nader

Birthday: 1994-11-11

Address: Apt. 721 977 Ebert Meadows, Jereville, GA 73618-6603

Phone: +2316203969400

Job: International Farming Consultant

Hobby: Reading, Photography, Shooting, Singing, Magic, Kayaking, Mushroom hunting

Introduction: My name is Eusebia Nader, I am a encouraging, brainy, lively, nice, famous, healthy, clever person who loves writing and wants to share my knowledge and understanding with you.