Artificial intelligence (AI) agents that can autonomously perform scientific research have emerged. This agent generates research ideas on its own, designs and executes experiments, analyzes the results, and writes papers. The researchers called this “the beginning of a new era of scientific discovery” and said that it is necessary to wait and see what the actual results will be.
Venture Beat reported on the 13th (local time) that the startup Sakana AI is collaborating with researchers at the University of Oxford and the University of British Columbia to develop an AI agent called ‘AI Scientist’ that can autonomously perform end-to-end scientific research. It was reported that the paper was published in the archive.
Sakana AI is a company founded in Tokyo, Japan last year by Lillian Jones, one of the authors of Google’s ‘Transformer’ paper. It is famous for its ‘merge evolution model’, which improves model performance through model crossbreeding over hundreds of years. In addition, a large language model (LLM) specializing in Japanese was launched and is receiving attention in the Japanese market.
The AI Scientist they released this time is a fully automated pipeline that generates scientific papers using LLM.
Given a broad scientific research direction with a simple initial code base, AI then proceeds with subsequent processes without human intervention. In other words, it goes through the entire research life cycle, from generating a research idea, writing the necessary code, running the experiment, summarizing and visualizing the experimental results, and presenting the research results in a complete scientific paper.
Additionally, by introducing an automated peer review process to evaluate generated papers, write feedback, and further improve the results, the generated papers can be evaluated with almost human-level accuracy.
In addition, it boasts high-cost efficiency. According to the researchers, the computing cost for an AI scientist to write a research paper is only about $15 (about 20,000 won).
AI scientists go through four main processes: ▲idea generation, ▲repeated experimentation, ▲paper writing, and ▲paper review.
In the idea generation stage, when a starting code ‘template’ that describes the desired research topic is given, the AI scientist brainstorms a new research direction. During this process, search the AI academic search engine ‘Semantic Scholar’ to check whether your idea is novel.
Given a template and an idea, a plot is created to visualize the results after executing the proposed experiment in the subsequent iterative experiment phase. Write a memo explaining the contents of each plot and provide all information necessary for writing the paper, including saved diagrams and experiment notes.
During the paper writing stage, AI scientists write concise, information-rich progress reports in the style of standard machine learning conference papers. It also autonomously searches semantic scalars to cite related papers.
In the paper review stage, peer review is generated through an automated LLM-based reviewer that can evaluate the generated paper with accuracy close to the human level. The reviews generated can be used as feedback for improving the project or generating future ideas. This allows AI scientists to iteratively improve their research results.
According to the researchers, the AI scientist wrote a paper that, as evaluated by automated reviewers, exceeded the acceptance criteria of a top machine learning society.
The researchers used a variety of closed LLMs, such as GPT-4o and Claude 3 Sonnet, and open source LLMs, such as DeepSeek and Rama-3, and found that Claude 3 Sonnet produced the highest quality papers.
However, the researchers said that AI science has several drawbacks, and it remains to be seen whether it will actually help scientific development.
First of all, I was told that I couldn’t solve the paper’s visual problems or read the plot because I didn’t have any visual capabilities. Therefore, it was announced that there are plans to add multimodal in the future.
They also said that AI can implement one’s ideas incorrectly, compare unfairly with other content, and lead to misleading results. In addition, he pointed out that, like a typical LLM, mistakes can sometimes be made in the process of evaluating numbers.
However, he emphasized that “AI Scientist represents an important step toward realizing the full potential of AI in scientific research,” and “opens the door to endless possibilities for innovation and problem-solving in the most challenging areas of science and technology.”
“It is still unclear whether we will ultimately be able to propose ideas that can change the paradigm,” he said. “We believe that AI scientists will be excellent companions to human scientists, but only time will tell what the results will be.”