Unleash AI's Potential: Automated Agentic Design

Generated by DALL-E

One of the most exciting new developments in the rapidly evolving field of artificial intelligence (AI) is the Automated Design of Agentic Systems (ADAS), described in a new research paper on arXiv. This approach promises to create more powerful, versatile, and adaptable AI agents through automated processes.

From Handcrafted to Automated Design

Designing AI systems has historically been labour-intensive and heavily reliant on manual tuning and expert knowledge. Researchers and engineers painstakingly craft every component, from the architecture of neural networks to the specific prompts used by models like GPT. However, as the field matures, there's a growing recognition that many of these manually designed solutions may eventually be surpassed by those learned and optimized by the systems themselves.

This is where ADAS comes into play. The idea behind ADAS is to automate the creation of AI agents by allowing them to evolve and improve through a meta-agent—a system that designs other agents. By leveraging programming languages and foundation models like GPT, ADAS aims to explore the vast space of potential agent designs, combining and optimizing various components such as prompts, tool use, and control flows.

Introducing Meta Agent Search

A cornerstone of the ADAS approach is the Meta Agent Search algorithm. This tasks a meta-agent with iteratively creating new agents, testing their performance, and refining them based on an ever-growing archive of previous discoveries. The meta-agent acts as a researcher, continuously experimenting with new designs and learning from past successes and failures.

The power of Meta Agent Search lies in its ability to explore a virtually unlimited design space. Because it operates in a code-defined environment, the algorithm can theoretically discover any possible agentic system. This includes novel combinations of building blocks that human designers might never consider. The result is a set of agents that outperform state-of-the-art hand-designed models and exhibit remarkable robustness and generality across different tasks and domains.

Real-World Applications and Implications

The potential applications of ADAS are vast. From coding and science to complex problem-solving, agents developed through this automated process have demonstrated significant performance improvements. For example, agents designed by Meta Agent Search have shown superior results in math and reading comprehension tasks, outperforming traditional methods by substantial margins.

Moreover, the transferability of these agents across different domains is particularly noteworthy. For instance, an agent optimized for mathematical reasoning has been successfully adapted to tasks in reading comprehension and science, showcasing the versatility and adaptability of the designs generated by ADAS.

Examples of Discovered Agents

The Path Forward

While ADAS offers immense promise, it also raises important questions about the future of AI development. As we move towards increasingly automated design processes, ensuring these systems' safety and ethical deployment becomes paramount. The research community must explore ways to safeguard against potential risks, such as unintended behaviours or harmful actions by autonomous agents.

Despite these challenges, the emergence of ADAS marks a significant step forward in the evolution of AI. By automating the design of agentic systems, we are not only accelerating the pace of innovation but also opening new avenues for creating AI that can learn, adapt, and improve in previously unimaginable ways.

As we continue to explore this exciting frontier, the possibilities are promising. Whether in enhancing scientific research, solving complex problems, or developing new technologies, the Automated Design of Agentic Systems could play a crucial role in shaping the future of AI.

Mustafa Suleyman, CEO of Microsoft AI, agrees with me!

Mustafa Suleyman, CEO of Microsoft AI and co-founder of DeepMind, said in a recent interview on Defining Intelligence with Seth Rosenberg on YouTube that Microsoft Copilot, and by extension, all AI assistants, must retain a memory of all their conversations. This echoes what I have been saying for over a year. An AI assistant needs to have an episodic persistent memory to remember important details from conversations potentially for years and even decades. As AI assistants gain the power of agency, as they indeed will, they must also retain memories of their interactions with other agents and the outcomes of their actions. 

We recognise that memory is a crucial component of human intelligence, and we have various medical definitions for different types of memory loss. ChatGPT currently has a relatively severe example of anterograde amnesia. OpenAI and Microsoft need to look at case-based reasoning, the branch of AI that has been handling episodic memory since the 1980s. Roger Schank's initial work on scripts laid the foundation for episodic memory management, which was then blended with ML techniques in the 1990s.

Clip from Defining Intelligence with Mustafa Suleyman

A workshop on Case-Based Reasoning and Large Language Model Synergies is being held next week in Mérida, Mexico, with the 32nd International Conference on Case-Based Reasoning (ICCBR 2024).

Google Illuminate - creates a radio interview from a research paper

Google Labs has a long history of inviting users to experiment with cutting-edge tech. Gmail was once a private beta project. Illuminate is a project that turns academic papers into AI-generated audio discussions in the style of an NPR podcast. The idea is simple: Google's LLM Gemini generates a paper summary and a Q&A. Two AI-generated voices, a male interviewer and a female expert, will guide you through a short interview describing the paper. You can listen to some of the samples on the Google Illuminate website. This is useful, letting me listen to engaging summaries of the ever-growing stack of research papers I must read as I exercise or drive. It can also be easily adapted to other narration forms for different use cases. Illuminate is in private beta, and you can join the waitlist here.

Recreating the DEC PDP-10 at the MIT AI Lab

 

I came across this today: a modern replica of the Digital Equipment Corporation PDP-10 mainframe computer. What makes this so wonderful is that it's not just a simulation of the PDP-10's OS and software running on a Raspberry Pi but also includes a facsimile of the original front panel.

The PiDP-10 front panel is not just a mock-up but allows you to control and interact with the PiDP-10 exactly as an operator would have done back then. I used a PDP-10 when I did my MSc in AI at Essex University in 1985. The PDP-10 was popular with "university computing facilities and research labs during the 1970s, the most notable being Harvard University's Aiken Computation Laboratory, MIT's AI Lab and Project MAC, Stanford's SAIL, Computer Center Corporation (CCC), ETH (ZIR), and Carnegie Mellon University. Its main operating systems, TOPS-10 and TENEX, were used to build out the early ARPANET. For these reasons, the PDP-10 looms large in early hacker folklore". 

Thus, the PiDP-10 comes with MIT’s Artificial Intelligence Lab, "the PDP-10 formed the heart of a large array of connected hardware, and its ITS operating system became a playground for computer scientists and hackers alike. MACLISP, emacs, the earliest AI demos, they were born on the 10, running ITS." I'm particularly interested to see SHRDLU - the first AI to understand a 3D blocks-world. I remember doing assignments in LISP on that and how, in the mid-80s, it was the considered the cutting edge of AI.

There's a waiting list to buy the PiDP-10 from Obsolescence Guaranteed, which I have eagerly joined.

GraphRAG - Using Knowledge Graphs to Empower LLMs

LLM-generated knowledge graph built from a private dataset using GPT-4 Turbo (Microsoft, 2024)

Back in the 1980s, I did my PhD in AI using Sowa's Conceptual Graphs, what we would now refer to as knowledge graphs. We've known for a while that providing LLMs with specific knowledge in the form of RAGs improves their accuracy. However, we've experimented with providing knowledge to LLMs in more explicit formats, for example, as cases in case-based reasoning augmented RAGs. Now, Microsoft has announced  GraphRAG, its Knowledge Graph-augmented LLM tool. The interesting thing about GraphRAG is that the knowledge graph is created by an LLM before being used to guide the LLM's retrieval. The LLM is, therefore, bootstrapping itself and "By using the LLM-generated knowledge graph, GraphRAG vastly improves the “retrieval” portion of RAG, populating the context window with higher relevance content, resulting in better answers and capturing evidence provenance."

Read Microsoft's announcement GraphRAG: Unlocking LLM discovery on narrative private data. For more information about the GraphRAG project, watch this video.

ChatGPT now has a memory - but it's naive

Screenshot - OpenAI

Last year, during the hype surrounding Large Language Models (LLMs), I published a position paper and wrote in this blog that LLMs, like ChatGPT, would need a persistent memory of their conversations to be most helpful. It's tough to converse intelligently with somebody with no memory. We value old friends so much because we know they recall events relevant to us, both good and bad, going back many years or even decades. 

However, managing that memory responsibly is a nontrivial task. Moreover, if virtual assistants based on LLMs become part of our daily lives, as it seems they will, their memory may have to be maintained over many years, perhaps even decades. I don't believe ChatGPT's memory management will be sufficient for this task.

My research has primarily focussed on case-based reasoning (CBR), a memory-based method. Interestingly, as a research community, we didn't consider how our case-bases (the memory) should be maintained over time. This was because we'd been focused on building systems in the early years of the discipline. Only when the systems matured did we realise our memories needed to be maintained. This happened in the late 1990s and centred on the work of Wilson and Leake, for example, "Categorizing case-base maintenance: Dimensions and directions". This work sparked a new line of research within CBR, leading to "Remembering to Forget" becoming a memorable paper title.

Consider this scenario: you've asked ChatGPT to remember your partner's name and that they like dark chocolate. You subsequently break up and acquire a new partner who prefers milk chocolate. You later ask ChatGPT to advise on buying a present. ChatGPT recommends dark chocolate in a gift box. Its memory is out of date, and the recommendation is inappropriate. The event of breaking up with your previous partner should have triggered a memory management process. These triggers are detailed in Wilson and Leake's paper in a comprehensive framework for maintaining memories. 

OpenAI's memory for ChatGPT is described in a FAQ webpage that is naive in its simplicity. The memory is described as a "notepad" with individual memories jotted down sequentially on it. Users can review and delete individual memories. But this is far too simplistic an approach to manage an AI assistant's memory that may have to span many years. An AI Assistant's memory must be structured, and policies and procedures will be required to manage it. OpenAI and others who build AI assistants with long-term memories should draw upon the expertise of case-based reasoners who have been managing memory for decades. Otherwise, they are in danger of reinventing the wheel.

Coincidentally, I've just been reading Why We Remember by Charan Ranganath. This book provides a fascinating insight into how the brain processes memories and highlights how little we currently know about this crucial aspect of ourselves.

Call for Papers: Workshop on CBR and LLMs

Generated by Gemini

Last year was the most remarkable year in AI that I can recall. Large Language Models (LLMs) like Chat-GPT changed the public perception of AI, and what had previously seemed like science fiction was now a reality. I was only tangentially familiar with LLM research, having been working on emotion recognition in speech with a PhD student. However, last year, I started diving into LLM research in-depth, which, as one commentator said, was like trying to drink water from a fire hydrant, such was the volume of publications through places like arXiv.

I view all problems through a lens coloured by case-based reasoning (CBR), my long-term AI research speciality. I quickly saw synergies between CBR and LLMs where both could benefit from each other's approaches, and I wrote up my initial thoughts and published them on arXiv.

CBR has an annual international conference, and I proposed the idea of a workshop at the conference on CBR-LLM synergies to some colleagues, who all thought this was a great idea and agreed to co-organise the workshop with me. The Case-Based Reasoning and Large Language Models Synergies Workshop will take place at  ICCBR 2024 in Mérida, Yucatán, México on July 1st 2024. The Call for papers can be accessed here, and submissions are via EasyChair.