5.2 LLM-based Agents

5.2 LLM-based Agents

5.2.1 Definition and Historical Evolution of AI Agents

OpenAI’s Five-Level Classification of AI Agents (AGI Context)

• Level 1: Conversational AI. These systems can engage in human-like dialogue. Tools like ChatGPT belong to this level. They are limited to language-based tasks and lack the ability to take action or solve complex problems independently.

• Level 2: Reasoners. These systems can demonstrate strong reasoning capabilities in academic or professional domains. They are able to solve complex problems using internal logic without needing external tools, but still do not take autonomous action.

• Level 3: Agents. These are autonomous AI systems that can act on behalf of users over extended periods. They can carry out a wide range of tasks over hours or even days, make decisions, and take actions—all without requiring supervision.

• Level 4: Innovators. At this level, AI systems can generate original ideas and solutions, driving breakthroughs in science and technology through innovative thinking and creative problem-solving.

• Level 5: Organizers. These advanced agents are capable of managing entire organizations, coordinating complex processes, and even outperforming human teams in high-value tasks.

Definition of an AI Agent

• Perception: The Agent’s “Eyes”. Agents use sensors to perceive their surroundings—for example, cameras to capture images or microphones to detect sound. This allows them to gather real-world data.

• Decision-Making: The Agent’s “Brain”. Agents process input data using complex algorithms and models (like deep learning or reinforcement learning) to make intelligent decisions.

• Action: The Agent’s “Hands”. Agents use actuators to interact with the world—such as controlling robotic arms or sending commands to software systems to complete tasks.

• Learning: The Agent’s Self-Improvement. Agents can learn from experience, adapting and updating their strategies over time to perform better in future tasks.

The Evolution of AI Agents

1. Rule-Based Agents (1960s–1980s). These early agents relied heavily on manually coded knowledge bases and expert systems to simulate human reasoning.
• Used handcrafted rules and logic to solve problems
• Lacked flexibility and adaptability
• Could not learn or improve over time

2. Reinforcement Learning-Based Agents (1990s–2010s). With the rise of machine learning and deep learning, agents became more intelligent and adaptive.
• Learned strategies through experience and trial-and-error
• Replaced rigid rules with data-driven decision-making
• Marked a shift toward autonomous learning

3. LLM-Based Agents (2020s–Present). The introduction of Large Language Models (LLMs) like GPT has given agents powerful new capabilities.
• Can interact with humans using natural language
• Capable of reasoning, planning, and acting without detailed instructions
• Able to break down human goals and achieve them step-by-step with minimal supervision

"Agent" or "Agentic"?

1. Router Level (Basic Agentic Behavior): The LLM is used to route input to the correct downstream tool or workflow. This is the most basic form of Agentic control.

2. Between Router and State Machine Level (Intermediate): Multiple LLMs work together to handle more complex, layered decisions. The system behaves somewhere between a router and a finite-state machine.

3. State Machine Level (Advanced): The system continues to make decisions across steps until a task is fully completed—this reflects a persistent, looped execution strategy.

4. Autonomous Agent Level (Highest): The system can autonomously build tools, remember them, and reuse them in future tasks. This level represents true autonomy and self-directed behavior.

5.2.2 Mechanism of LLM-Based Agents

• Carry on conversations

• Execute tasks

• Make decisions and reason

• Show a certain degree of autonomy

• Access external systems (like calculators or APIs)

• Solve routing problems (i.e., decide which tool to use)

• Incorporate modules for memory and planning

Planning — How LLM Agents Think, Act, and Improve

• Subgoal Decomposition
• Purpose: Most real-world tasks aren’t just one-step commands. They involve a series of actions. That’s why agents need to perform subgoal decomposition — figuring out what intermediate steps are required to reach a final goal.
• Three ways to help an LLM decompose tasks:
• Simple prompting. You can directly ask the agent to think through the steps: Steps for writing a story:\n1. , or What are the subgoals for achieving XYZ?
• Task-specific instructions. For example: Before writing the full story, first write an outline.
• User-guided input. You can manually provide the steps or let the agent confirm its own step-by-step plan before continuing.
💡
Techniques like CoT (Chain of Thought) and ToT (Tree of Thought) are all about guiding the LLM to reflect and reason step by step. Behind the scenes, this is achieved through well-crafted prompts that help the model activate deeper reasoning — a kind of artificial metacognition.
• Planning with LLM + Planner (LLM+P). A more advanced approach involves integrating formal planning systems with LLMs. Here’s how it works:
1. The LLM first converts a problem into Problem PDDL (Planning Domain Definition Language), a format used in classical AI planning.
2. A Planner tool uses that Problem PDDL to generate a detailed step-by-step plan according to a Domain PDDL.
3. The LLM then translates the plan back into natural language so it can act on it.
This process gives the agent more structured and long-term planning ability — but it requires domain-specific PDDL definitions and a proper Planner to work effectively.

• Reflection and Action: Two Key Components of an Intelligent Agent. To build effective agents, we often rely on two complementary abilities:
1. Self-Reflection – the ability to learn from past experience, and
2. Action – the ability to plan and act step-by-step.
Let’s look at two key frameworks that represent these abilities: ReAct and Reflexion.
1. Reflexion: Learning from Mistakes Through Reflection. Reflexion is a framework designed to help agents improve over time. It gives the agent the ability to:
• Remember past experiences (dynamic memory),
• Reflect on what went wrong or right,
• Adjust its strategy in the next attempt.
This self-reflection process allows agents to continually improve by learning from previous errors—just like how humans learn from failure. Reflexion is built using a reinforcement learning (RL) setup:
• A reward model gives feedback (e.g., success/failure),
• After each action step , the agent uses self-reflection model to generate reflections and add them to the memory
• Based on that reflection, the agent decides whether to reset the environment and start a new episode.

2. ReAct: Reasoning + Acting Step-by-Step. ReAct stands for Reason + Act, and it expands the agent’s action space by combining discrete actions with natural language reasoning.
• In the ReAct framework, reasoning and action are integrated inside the LLM:
• The reasoning step allows the LLM to think and plan,
• The action step allows it to interact with the environment.
• The process is driven by prompts that guide the LLM to generate a structured chain of reasoning and decisions using natural language. The ReAct prompting template gives a clear structure for this thought process, which typically follows the format:
• Thought: The LLM explains its reasoning.
• Action: The LLM interacts with the environment.
• Observation: The LLM receives feedback from the environment.
• In experiments involving both knowledge-intensive tasks and decision-making tasks, ReAct has shown better performance than using only Act. This highlights the advantage of combining reasoning with action.

Memory

1. Human memory can be defined as the process of acquiring, storing, retaining, and retrieving information. In the human brain, there are several types of memory:
1. Sensory Memory: This is the earliest stage of memory, where the brain briefly retains impressions from sensory input (like visual or auditory stimuli). Sensory memory usually lasts only a few seconds and includes:
• Iconic memory (visual),
• Echoic memory (auditory),
• Haptic memory (touch).
2. Short-Term Memory (STM) or Working Memory: This stores information that we are currently aware of and using to perform complex cognitive tasks (like learning and reasoning).
• It is believed to hold around 7 items at a time
• It typically lasts 20–30 seconds without rehearsal.
3. Long-Term Memory (LTM): This type of memory stores information for long periods—ranging from days to decades. It is generally considered to have unlimited capacity. Long-term memory is divided into two main types:
• Explicit / Declarative Memory: This involves conscious recall of facts and events. It includes:
• Episodic memory (personal experiences),
• Semantic memory (facts and concepts).
• Implicit / Procedural Memory: This is unconscious memory used for automated skills and routines, such as riding a bike or typing on a keyboard.

2. Memory in LLM-Based Agents. Similarly, LLM agents also rely on short-term and long-term memory to operate effectively:
• Short-term memory in agents is implemented as context windows—information the model sees during a single session.
• Long-term memory requires external storage to preserve information across sessions, such as:
• Past conversations,
• Retrieved facts,
• Self-reflections,
• User preferences, etc.
But unlike the human brain, which stores and recalls memories biologically, agents must search through external memory efficiently. That’s where vector-based retrieval comes in.

3. Fast Memory Access with MIPS (Maximum Inner Product Search). To retrieve relevant memories from a large collection of stored embeddings, agents use Maximum Inner Product Search (MIPS)—a technique for finding the most similar content to a given query (e.g., finding the most relevant past document or memory snippet).
Because exact search can be slow at scale, most systems use Approximate Nearest Neighbor (ANN) algorithms to perform MIPS efficiently, balancing speed and accuracy. Here are some common ANN algorithms used to power agent memory:
• LSH (Locality-Sensitive Hashing)
• LSH uses a special hash function to map similar inputs to the same bucket with high probability.
• The number of buckets is much smaller than the number of input items.
• It’s fast but less precise compared to more structured methods.
• ANNOY (Approximate Nearest Neighbors Oh Yeah)
• Built on random projection trees, where each non-leaf node splits the input space in two.
• Each leaf node stores a data point.
• Multiple trees are built randomly and independently, simulating the effect of a hash function.
• Searches are performed on all trees, then candidate points from each tree that are likely close to the query vector are selected and aggregated.
• Similar to KD-Trees, but more scalable and better suited for large datasets.
• HNSW (Hierarchical Navigable Small World)
• Inspired by the small-world network theory (e.g., six degrees of separation).
• Each node can reach other nodes in just a few steps.
• HNSW builds multiple layers of such networks:
• The bottom layer holds real data points.
• Upper layers contain “shortcuts” for faster navigation.
• Searching starts at a random node in the top layer and moves down, refining the search at each level until the best matches are found.
• Searching in upper levels accelerates matching; searching in lower levels increases matching quality.
• FAISS (Facebook AI Similarity Search)
• Assumes vector distances follow Gaussian distribution.
• First partitions the vector space into clusters using coarse quantization
• Then refines the results within those clusters using fine quantization.
• This two-step quantization greatly improves efficiency in high-dimensional vector search.
• ScaNN (Scalable Nearest Neighbors). ScaNN introduces an innovation called anisotropic vector quantization, which helps it achieve both accuracy and efficiency in vector search. Basically, ScaNN quantizes all data points so that their inner product with the query vector remains as close as possible to the original (before quantization).
In simple terms: ScaNN tries to preserve the true similarity score (like how well two vectors align) even after compressing the data. This makes it very effective for Maximum Inner Product Search (MIPS) tasks.

Tool Use

• Current and real-time data (e.g., weather, stock prices, search results),

• Code execution environments (to run, debug, or build code),

• Databases and proprietary knowledge bases (internal company documents, FAQs, etc.),

• Specialized tools (calculators, translators, recommendation engines, etc.).

LLMs, Agents, and RL Agents: What’s the Difference?

1. Difference Between an Intelligent Agent and a Plain LLM
• Talking to an LLM: You provide a prompt, and the model directly generates a response.
• Talking to an Agent: It’s like working with an assistant. Instead of just writing something for you, it might:
• First ask whether you need to do web research,
• Then write an initial draft,
• Reflect on the draft,
• Think about which parts need revision,
• Edit and iterate on the text repeatedly.
This is a reflective and iterative process of thinking + revising, not just a single-shot response.

2. Difference Between Reinforcement Learning Agents and LLM-Based Agents
• Reinforcement Learning (RL) Agents:
• Input: Predefined vectorized states of the environment (or images of it).
• Strategy: Initialized by relatively simple neural networks.
• Output: Often low-level action controls (e.g., movement, attack commands in games).




• LLM-Based Agents:
• Input: Text (can also include multimodal input like vision or audio in context-aware applications).
• Strategy: Built on top of pretrained large language models, which already understand the world.
• Output: Still textual actions, but can include structured outputs (e.g., function calls), enabling real-world interaction through APIs, tools, or environments.

5.2.3 Classification of LLM‐based Agents

1. How many agents are involved (SingleAgent vs. MultiAgent)

2. How the agent behaves—its “action model” (Tool Use Agents, Code Generation Agents, Observation‐based Agents, RAG Agents, etc.)

Classification by Number of Agents

1. SingleAgent: A SingleAgent is a single, self‐contained intelligent agent that handles task planning and execution on its own. It can reason about a problem and produce an answer or action plan without collaborating with other agents.

2. MultiAgent: A MultiAgent setup uses multiple specialized or varied agents (sometimes called “agent profiles”) that communicate and collaborate. They may debate or discuss partial solutions, exchange knowledge, and collectively make decisions. This approach is especially useful for more dynamic and complex tasks, where each agent can bring a unique strategy or capability, and the group collectively improves the overall solution.

Classification by Agent Behavior Model

1. Tool Use Agents
• MRKL System (Modular Reasoning, Knowledge, and Language). MRKL structures reasoning into modules (e.g., knowledge lookup, language processing). A large language model coordinates tool calls (like searching external knowledge bases or invoking APIs) to obtain and integrate information. Most include fine-tuning: Toolformer, Gorilla, Act-1, HuggingGPT, and ToolkenGPT.
• CRITIC: Self‐Correcting with Tool‐Interactive Critiquing. In the CRITIC approach, the agent first generates a preliminary answer using an LLM. It then “critiques” or analyzes that answer—again with the help of an LLM step—to identify possible errors. Next, it uses external tools (e.g., search engines, code systems) to gather additional evidence or data. Finally, it revises the original answer based on these findings. This loop of critique‐then‐correct helps the agent iteratively improve its response.

2. Code Generation Agents. In this category, the agent’s core mechanism is generating and executing code (often Python) to solve problems. This can be particularly powerful for tasks that involve calculation, data processing, or other programmatic solutions. Subtypes include:
• Program‐Aided LM (PAL). The agent directly transforms a user query into executable code. Then it runs that code (for example, in a Python interpreter) and merges the execution results back into the final answer.

• Tool‐Integrated Reasoning Agent (ToRA). Similar to PAL, but the code‐generation and reasoning steps are interwoven. This means the agent incrementally reasons about the problem while generating code snippets, executing them, and interpreting the results, in multiple loops until solving the problem.

• TaskWeaver. Another variant similar to PAL, but supports user‐defined “plugins” or external APIs. This means the agent can generate code that calls custom libraries or services to solve domain‐specific tasks.

3. Observation‐Based Agents. Some agents are designed to interact with an environment—such as a simulation like a “gym” environment (i.e. OpenAI Gym)—to solve problems. These agents:
1. Receive observations from the environment (such as sensor data or simulator states).
2. Incorporate these observations directly into their prompts.
3. Generate actions/decisions to influence or change the environment.
This loop continues until the agent reaches its goal in that simulated or real‐world setting.
• Reasoning and Acting (ReAct)
• Background: Traditional “chain of thought” (CoT) methods can suffer from hallucination and error propagation. A purely “Act‐only” approach cannot fully leverage an LLM’s high‐level planning capabilities.
• Introduction: By combining LLM reasoning with action generation, the agent alternates between generating reasoning steps and performing specific actions, in a loop of produce a thought → take an action → receive an observation. All new information then gets appended to the prompt as a running record of past experiences and memory, and the agent can also access external tools. The ReAct needs only one to six context examples to generalize to new tasks.



• Reflexion
• Background: It remains challenging for LLMs to learn quickly through pure trial‐and‐error, because traditional reinforcement learning typically requires large training datasets and expensive fine‐tuning.
• Introduction: The Reflexion framework uses three components: Actor, Evaluator, and Self‐Reflection. Compared with ReAct, it adds an Evaluator step and an explicit Reflection mechanism. The Actor generates text and actions, observing the environment. The Evaluator assesses the Actor’s output, computing a reward signal to measure performance. The Self‐Reflection module then examines the feedback (e.g., “What did we do? What went wrong?”) and writes those insights back into the prompt as a form of memory. This loop lets the agent continually refine its decision‐making during the task.



• Lifelong Learning Agents
• Background: These agents are designed for real‐world tasks where continuous, lifelong learning is required (e.g., Minecraft‐like environments or other open‐ended tasks).
• Voyager: Consists of three main parts. It automatically generates “curriculum tasks” to explore an open‐ended world, iteratively generates code to execute actions in the environment, and uses “Self‐Verification” so that any newly acquired skill can be stored in a skill library for later reuse. Experiments comparing Voyager with ReAct, Reflexion, and AutoGPT show that Voyager yields superior performance.
Flowchart:



Experiment Result:



• Ghost in the Minecraft (GITM): Uses an LLM to break an initial goal into subgoals, then iteratively plans and executes “structured text” (actions). GITM also incorporates external knowledge bases for additional help in goal decomposition and to store experience.
Flowchart:



LLM Planner Example:

4. RAG (Retrieval‐Augmented Generation) Agents. RAG Agents retrieve relevant information from external sources (like a local knowledge base or a vector database) and then insert that information into the LLM’s prompt. By doing so, they can handle knowledge‐intensive tasks more effectively. Although RAG is usually described as a “retrieval + generation” paradigm, you can also think of it as a special type of agent that uses external retrieval tools or vector stores for improved accuracy and context.
• Verify‐and‐Edit
• Background: Chain‐of‐Thought (CoT) has great potential for handling complex reasoning tasks in an interpretable way, but it still struggles with knowledge‐intensive scenarios (e.g. fact-based questions).
• Introduction: This approach generates multiple CoT traces, then selects some of them to edit. Edit is done by retrieving external information and then allowing the LLM to enhance them.



• Demonstrate‐Search‐Predict (DSP)
• Background: A common approach is to insert retrieval results (from a retriever) into the language model’s prompt. However, for multi‐hop questions (multiple information/knowledge is required to answer), performance is not good.
• Introduction: One solution is to use a few‐shot method to break the original question into subquestions, retrieve answers for each subquestion, and then combine these partial answers into a final result.

Summary

5.2.4 Agentic System Architectures

1. Workflow‐Based Systems – Tasks follow predefined steps (orchestrating LLM calls, tools, and programmatic checks). Best suited for scenarios where subtasks or steps can be planned in advance.

2. Autonomous Agents – The system dynamically decides on tool usage, task planning, and execution. Best suited for complex or open‐ended tasks that cannot be fully predetermined.

Autonomous Agent

• Real‐Time Feedback: After every action (e.g., calling a tool, running a piece of code), the agent evaluates the outcome to decide next steps.

• Human Review and Intervention: The process can pause at checkpoints or challenges, allowing humans to provide feedback or additional context before the agent continues.

• Clear Termination Conditions: The agent usually stops after reaching a goal, but it’s common to set upper limits (e.g., a maximum number of iterations) to prevent runaway loops.

Workflow‐Based Systems

1. Prompt Chaining
• Concept: Decompose a task into sequential sub‐steps, each handled by an LLM whose input is the previous step’s output. You can add programmatic checks (“gates”) to verify correctness or completion before moving on.

• Applicable Scenario: When a task can be split into simpler, ordered stages—reducing complexity at each step and thus improving the LLM’s accuracy.
• Examples: Generate marketing copy, then translate it to a different language. Or draft a document outline, verify it meets certain standards, then use that outline to write the final document.

2. Routing
• Concept: Classify incoming requests, then send them to specialized prompts or workflows best suited to the request type. By separating different kinds of tasks, you can tune prompts for each and avoid one‐size‐fits‐all solutions that degrade performance.

• Applicable Scenario: When your system handles a variety of task types (e.g., creative writing vs. factual QA vs. code generation), each requiring tailored instructions.

3. Parallelization
• Concept: Run multiple tasks—or multiple versions of the same task—in parallel, then combine their outputs programmatically (the system aggregates the results using code or logic, not by another LLM).

• Strategies
• Sectioning: Split a task into independent chunks, run them concurrently, and merge the results.
• Voting: Duplicate the same task across multiple LLMs or prompts, then vote or compare to reach the best answer.
• Applicable Scenario: When tasks can safely be divided or when you want to improve confidence and coverage. Parallel execution can save time or increase reliability by comparing multiple outputs.
• Examples:
• Sectioning:
• Parallel content review and main task handling: One model instance handles user queries, while another model instance simultaneously reviews the queries for inappropriate content or illegal requests. Compared to a single LLM handling both response and safety at once, this method performs better.
• Automated multi-dimensional model evaluation: Automatically evaluate the performance of an LLM, where each time a different LLM is called to assess a specific aspect of the model.
• Voting:
• Reviewing code for vulnerabilities: Multiple prompts review the code, and any discovered issues are flagged.
• Assessing the appropriateness of a content: Multiple prompts evaluate from different perspectives, or different voting weights are assigned to balance false positives and missed detections.

4. Orchestrator–Workers
• Concept: Use a central “Orchestrator” LLM to break a complex problem into smaller subtasks, assign them to multiple “Worker” LLMs, then aggregate the results. Importantly, the subtasks are determined dynamically, based on the input context.

• Applicable Scenario: This approach is well-suited for complex and unpredictable subtasks (e.g., in programming, where each task might require modifying multiple files, and the way each file is modified depends on the specific task). It resembles Parallelization workflow but differs in one key area: flexibility. In Parallelization workflow , subtasks are predefined, whereas in the Orchestrator-Workers model, the central LLM dynamically determines subtasks based on the specific input.
• Examples:
• Multi-file code modification: Each time, multiple files need to be edited in complex ways to produce the final software product.
• Multi-source information search and analysis: Searching and analyzing possibly relevant information across various sources to complete an information retrieval task.

5. Evaluator–Optimizer
• Concept: One LLM produces an initial answer. Another LLM (the “Evaluator”) reviews or critiques it, possibly giving a score or suggesting edits. This feedback is used by the original model (or another “Optimizer”) to refine the answer. The loop repeats until you reach an acceptable result.

• Applicable Scenario: Tasks where incremental improvement is possible and valuable. The presence of a reliable “Evaluator” helps guide the system toward better outcomes. Two major criteria for determining applicability are:
• Human feedback leads to clear improvement in the LLM's output;
• LLM itself can generate such feedback.
• Examples:
• Document Drafts: The Evaluator LLM highlights unclear sections or style issues, and the Optimizer LLM makes improvements.
• Search Query Refinement: The Evaluator sees if the results cover all aspects; if not, it instructs the Optimizer to adjust and re‐query.

Summary

5.2.5 Frameworks and Applications of Agents

Agent Frameworks

1. Full-code Frameworks
Full-code framework refers to a framework that requires developers to manually write logic, call interfaces, and configure toolchains in a programmatic way (mainly Python). This type of framework is highly flexible and suitable for complex custom development
• Langchain & LangGraph
• LlamaIndex
• Multi-agent Collaboration Frameworks
• AutoGen
• CrewAI
• Swarm
• CAMEL

2. Low-code Platforms
• Dify: Open source (modified Apache 2.0 license); also available on its official website with free usage limits.
• Coze: Closed source with paid plans required beyond free usage limits.
• BISHENG: Open source (Apache 2.0 license) designed for enterprise use; individuals can try a free plan on its demo platform.
These platforms offer a visual, drag-and-drop interface to design and deploy LLM-powered agents—allowing users to build workflows (e.g., LLM invocation, tool integration, prompt chaining) without writing much code.

Note: While these frameworks accelerate development and are accessible to non-programmers, their abstraction layers often conceal low-level prompt and logic details, which can complicate debugging and limit deep customization.

Applications of Agents

1. RAG (Retrieval-Augmented Generation) Applications
• Traditional RAG
• Pros: Simple architecture; good performance
• Cons: May lack precision
• Examples: A company's internal knowledge base Q&A system.
• GraphRAG
• Pros: Strong associations between knowledge graph nodes; fine-grained knowledge representation; high search accuracy
• Cons: Time-consuming to build and update knowledge graphs; longer retrieval times
• Examples: Construct a map between a patient's medical history, medicines, and symptoms to achieve more accurate Q&A.
• LightRAG: A lightweight implementation aimed at achieving benefits similar to GraphRAG but with lower resource requirements.
• Examples: Local semantic retrieval in mobile apps, such as voice assistants.
• DeepSearcher
• Pros: Employs tree structures to organize knowledge; reducing granularity and improving search precision. Supports parallel retrieval.
• Cons: Weak cross-document knowledge association; not well-suited for inferential queries.
• Examples: Browse a large paper library (e.g., arxiv) to quickly find key research in a certain field.

2. Single-Agent Applications: Examples of single-agent systems include BabyAGI, AutoGPT, and HuggingGPT. These systems:
• Utilize various prompt designs and workflow strategies.
• Leverage models from OpenAI or Hugging Face.
• Enable capabilities such as task execution, scheduling, and prioritization.

3. Multi-Agent Applications: Systems like Generative Agents, MetaGPT, GPT-researcher, and STORM / Co-STORM are engineered for:
• Collaboration among multiple agents.
• Enhanced communication and coordination to solve complex tasks.
Example of Generative Agents, which populating a sandbox environment, reminiscent of The Sims, with twenty-five agents:



The overview of STORM that automates the pre-writing stage:

Agents + Reinforcement Learning (RL) Framework

1. ToRL
💡
ToRL uses reinforcement learning (RL) to help LLMs autonomously explore and improve their strategies for using tools. This allows the LLM to figure out by itself when and how to use tools, reducing the need for predefined, human-written instructions about tool usage.
Details for training:
• Tool Invocation Frequency Control: Introduce a threshold parameter (C) to limit the number of tool calls per generation.
• Execution Environment Selection: Choose a stable, accurate, and responsive code interpreter implementation.
• Error Message Handling: Extract critical error information to reduce context length.
• Sandbox Output Masking: Mask the outputs of the sandbox environment in the loss calculation to improve training stability.
Reward Design: Implement a rule-based reward function such that:
• Correct answers: +1 reward.
• Incorrect answers: −1 penalty.
This research explores execution-based penalties (e.g., −0.5 for unexecutable code), though default experiments rely solely on answer correctness.

2. OpenManus / OpenManus-R
💡
Inspired by RICO from RAGEN, OpenManus further explores new algorithmic structures, diversified reasoning paradigms, complex reward models, and rich agent evaluation environments.
Detailed strategies:
• Exploration of Reasoning Models: Benchmark models such as GPT-o1, Deepseek-R1, and QwQ-32B to comprehensively assess reasoning capabilities.
• Alternative Strategies: To improve the agent's planning efficiency and robustness in reasoning, various expansion strategies (a method used by AI agents to explore different possible steps or solutions when solving a problem) were tested:
• ToT (Tree of Thoughts): Builds a tree where each branch represents a different way of thinking, allowing the agent to explore multiple solutions step by step.
• GoT (Graph of Thoughts): Similar to ToT, but the paths are connected like a web, helping the agent combine ideas from different directions.
• DFSDT (Depth-First Search Decision Tree): Focuses on going deep into one path at a time, which is useful for problems that need thorough, step-by-step reasoning.
• MCTS (Monte Carlo Tree Search): Tries out random paths and uses the results to decide which direction seems the most promising, like testing different strategies in a game before choosing the best move.
• Diverse Reasoning Paradigms: Strategies such as ReAct and Outcome-based Reasoning are evaluated to determine their effectiveness in varied scenarios.
• Post-Training Strategies: Fine-tuning methods—including SFT, GRPO, PPO, DPO, and PRM—are studied to further optimize reasoning capabilities.
• Reward Model Training: Train reward models using comments data to refine complex reward signals. These models guide decision-making during both training and evaluation stages.
• Action-Space Awareness and Strategy Exploration: Enable agents to understand and explore their action spaces. This helps design strategies that effectively target complex tasks and maximize expected rewards.
• Integration with RL Optimization Frameworks: Integrate with frameworks like Verl, TinyZero, OpenR1, and Trlx to optimize the learning process.

3. RAGEN:
💡
RAGEN tackles the challenges of multi-turn interactions and stochastic environments by leveraging the MDP framework and a “Reasoning-Interaction Chain Optimization” algorithm; its advanced reward normalization strategy further enhances model stability and performance.
Implementation Details:
• Markov Decision Process (MDP): Models learning as a sequence of states and actions with the goal of maximizing cumulative rewards.
• RICO Algorithm Stages: Uses reasoning-driven generation to guide action trajectories in generation stage; adjusts strategies to optimize rewards across full trajectories in update stage.
• Progressive Reward Normalization Strategies: Methods like ARPO, BRPO, and GRPO balance learning stability under varying task difficulties.
Evaluations of different models indicate that large-scale models deliver better performance, but training still requires balancing prompt diversity with data freshness.

4. eleurent / RL-Agents (Classic Framework)
💡
RL-Agents is a comprehensive framework that allows researchers to implement and test a variety of advanced reinforcement learning (RL) algorithms, including Value Iteration, Monte Carlo Tree Search (MCTS), and Deep Q-Network (DQN), etc.
implementations:
• Planning Algorithms:
• Value Iteration, Cross-Entropy Method (CEM), Monte Carlo Tree Search (MCTS), Optimistic Planning Algorithm.
• Safe Planning:
• Robust Value Iteration, Discrete Robust Optimistic Planning, Interval-Based Robust Planning.
• Value-Based Algorithms:
• Deep Q-Network (DQN), Fitted Q.
• Safe Value-Based Methods:
• Budgeted Fitted Q.

5.2.6 Hands-on Practices of Agents

A Simple Weather-query Agent Using OpenAI Interface

• The user says: "What's the weather like in Paris today?"

• GPT responds: "I suggest calling get_weather("Paris, France")"

• You then execute that function (here we simulate) and get: "The weather in Paris, France 18°C and sunny"

• Then you tell GPT: "The tool has been executed, here's the result"

• Finally, GPT says: "The weather in Paris today is sunny, with a temperature of 18°C."

Implementing a Simple ReAct Agent Using Langchain

• Objective: Create a ReAct (Reason + Act) agent that integrates reasoning prompts with tool usage.

• Overview: The example shows how to:
• Define tools and an agent using Langchain.
• Integrate tool invocation to enable action-taking based on reasoning outputs.

> Entering new AgentExecutor chain... To answer the question, I need to find the prices of Sprite and Coca-Cola first. After obtaining those prices, I can perform the subtraction and then cube the result. Action: Product price query tool Action Input: "price of Sprite" Observation: The price of Sprite is 5 dollars per bottle. Thought:I now need to find the price of Coca-Cola to proceed with the calculation. Action: Product price query tool Action Input: "price of Coca-Cola" Observation: The price of Coca-Cola is 5 dollars per bottle. Thought:Since both Sprite and Coca-Cola are priced the same at 5 dollars per bottle, I will now perform the subtraction and cube the result. Action: Calculator Action Input: (5 - 5) ** 3 Observation: Answer: 0 Thought:I now know the final answer. Final Answer: 0 > Finished chain.

Implementing a Simple Multi-Agent System Using Langchain

• Objective: Demonstrate a multi-agent system where individual agents work on distinct tasks.

• Overview: Key steps include:
• Agent Creation: Define functions that instantiate agents.
• Task Definition: Specify two distinct tasks that the agents must perform.
• Workflow Construction: Use LangGraph’s StateGraph to represent agents as nodes. Transitions between nodes are driven by router logic.
• Execution: Initialize the workflow and execute it based on the user’s initial input.

Manus & OpenManus Technical Solution

Research DeepSeek's Five Recent Open-Source Projects - Manus

Manus is a general AI agent that turns your thoughts into actions. It excels at various tasks in work and life, getting everything done while you rest.

https://manus.im/share/OZB4PvsXY5N5FrvLXqqCl4?replay=1

1. Manus. Manus is a multi-agent system capable of converting user inputs or thoughts into concrete actions. It achieves this by analyzing and decomposing tasks into smaller, manageable subtasks, with progress tracked via a TODO.md file.




Key Features:
• Multi-Agent Framework: Manus utilizes a multi-agent approach where it first leverages a PlanningTool to establish a sequential plan that outlines the required subtasks. It then assigns each subtask to the most appropriate agent and executes them in a structured sequence.
• Planning and Execution: The introduction of the PlanningTool enhances its task solving capabilities significantly. For instance, Claude-3.7 on Manus demonstrated a 70% resolution rate on SWEbench tasks, up from a previous rate of 49%. This improvement is attributable both to the advanced model and the planning strategy.
• Tool Utilization: Manus incorporates a robust ReAct loop to ensure that each subtask is completed efficiently. It combines reasoning via prompts with the effective execution of tool-based actions.
• Proprietary Nature: Manus is not open source. It is possibly built upon Claude along with some proprietary, post-trained models that have undergone extensive engineering optimizations to enhance their tool usage across diverse scenarios.

2. OpenManus. Inspired by the Manus framework, it leverages the strengths of Manus while embracing an open architecture that promotes modularity, extensibility, and rapid innovation.
Key Features:
• Minimalist Plugin Architecture:
• Emphasizes modular design and ease of extension.
• Allows for a flexible combination of reasoning prompts and execution tools, enabling developers to quickly implement new agent functionalities.
• Tool-Driven ReAct Agent:
• Integrates both reasoning via prompts and concrete actions through tools in a “Reason + Act” methodology.
• This design ensures that tasks are executed with greater precision and efficiency.
• Planning Capability:
• Employs a PlanningTool similar to Manus to decompose complex tasks into sequential subtasks.
• This “plan first, then execute” approach further boosts the system’s overall task success rate.
• Dynamic Agent Allocation and Tool Scheduling:
• Assigns pre-defined agents dynamically to the relevant subtasks.
• Combines various models and tools to maximize flexibility and ensure optimal resource allocation during task execution.
OpenManus Execution Process: The execution flow in OpenManus can be summarized in the following steps:
• User Input: The process begins when a user inputs a complex request (e.g., “Write some code and automatically deploy it to the server”) via a frontend or command-line interface.
• Use the PlanningTool: The system analyzes the input and uses the PlanningTool to break down the request into a sequential task list, such as:
Analyze requirements → Write code → Test and fix → Deploy and verify
• Task Assignment and Execution:
• The system dynamically assigns the most suitable agent to each sub-task.
• Each agent utilizes a ReAct loop, interacting with the necessary tools to execute the task.
• Result Aggregation and Status Update:
• As each sub-task completes, the result is summarized and stored in shared memory.
• The system then proceeds to the next sub-task or adjusts the plan if a sub-task fails.
• Final Output: Once all sub-tasks are completed successfully, a final summary is generated and returned to the user.



OpenManus Architecture: Despite its robust functionality, OpenManus features a concise engineering structure comprised of approximately 30 core files and lightweight libraries (such as pydantic, openai, playwright, etc.). Its architecture includes the four key modules:
• Core Multi-Agent Framework (Agent):
• Implements a layered inheritance structure, gradually enhancing capabilities from:
BaseAgent → ReActAgent → ToolCallAgent → Manus
• Each layer adds functionality with customized prompts and integrated tools.
• Tools Layer (Tools):
• Provides the underlying actions for agent operations.
• Includes Python execution, web search, file operations, task planning, and more.
• Tools are designed to be highly extensible by inheriting from BaseTool.
• Prompt Module (Prompt):
• Contains instruction templates that define the reasoning logic and thinking styles for various agents.
• Execution Flow Module (Flow):
• Manages high-level task orchestration.
• Ensures that tasks are executed in the planned sequence and reports progress and outcomes.

• Demonstration of OpenManus

5.2.7 Agent Evaluation Framework

AgentBench

• Coding: Involves tasks such as code generation, operating system interactions, database operations, and knowledge graphs.

• Games: Covers scenarios like role-playing games, digital card games, puzzle-solving, and strategy games.

• Web: Focuses on actions related to web environments, including online shopping and web browsing.

• Operating system and database tasks: Use the success rate as the primary metric.

• Knowledge graph tasks: Employ the F1 score.

ToolEmu

• Robustness Simulation: The framework includes an adversarial emulator to replicate situations that might lead to significant LM-agent failures. This helps developers understand and address weaknesses in agent performance.

• Automated Safety Evaluation: ToolEmu incorporates an LM-based automatic safety evaluator that actively monitors and analyzes potentially risky operations during agent execution, quantifying their severity. This automated process aids in early identification of potential real-world failures.

AgentBoard