Lesson 5 of 5 · 15 min

Lesson 5: Agent Memory and Reliability

Course: Building AI Agents

Estimated time: 25 minutes

Difficulty: Intermediate-Advanced Builder

Here's the thing about most demo agents: they're impressive the first time you run them and useless the second time. They forget everything. Every session starts from zero. The agent that spent 20 minutes researching your competitors yesterday has no idea who your competitors are today. You have to tell it again.

This is fixable. Agents without memory are a code problem, not a model problem. And while you're fixing memory, you should fix reliability too, because an agent that remembers things but crashes unpredictably is only slightly more useful than one that doesn't.

This lesson covers both: making agents remember across sessions, and making them reliable enough to trust in production.

The Three Types of Memory

Memory for agents works at three levels. You need different tools for each.

Short-term memory (conversation context). This is the messages array in your agent loop. It's everything that's happened in the current session - user messages, tool calls, tool results, model responses. It lasts exactly as long as the session lasts. When the process ends, it's gone. This is the memory you already have.

Medium-term memory (session files). State written to disk or a database during a session, designed to be read at the start of the next session. The website monitor from Lesson 2 uses this: it writes the current page content to a file, and reads it next time to compare. The file survives process restarts. This is the most practical form of persistent memory for most use cases.

Long-term memory (vector store or structured DB). A collection of past information that the agent can search semantically. Not just "what happened last time" but "what did we learn across all the times this has run?" Implemented with a vector database (Chroma, Pinecone, Weaviate) or a structured database with a search layer. More complex to build, necessary for agents that accumulate knowledge over many sessions.

For most agents you build in the next few months, medium-term file-based memory is what you need. Long-term vector memory is worth adding when the agent's job involves building up knowledge over time - a research agent, a customer support agent that learns from past tickets, a personal assistant that accumulates context about you.

File-Based Memory in Practice

The pattern is simple:

At startup: load a JSON file containing everything the agent should remember from previous sessions
During the session: update the in-memory state as new things are learned
At shutdown (or after each significant action): write the updated state back to the file


import json
import os
from datetime import datetime

def load_memory(path: str) -> dict:
    """Load agent memory from disk. Returns empty state if no file exists."""
    if not os.path.exists(path):
        return {
            "created_at": datetime.now().isoformat(),
            "sessions": [],
            "learned_facts": [],
            "preferences": {}
        }
    
    with open(path, "r") as f:
        return json.load(f)

def save_memory(path: str, state: dict) -> None:
    """Write current memory state to disk."""
    state["last_updated"] = datetime.now().isoformat()
    os.makedirs(os.path.dirname(path), exist_ok=True)
    with open(path, "w") as f:
        json.dump(state, f, indent=2)

def summarize_session(memory: dict, session_log: list) -> str:
    """Generate a compact summary of this session to store."""
    # In practice you'd use the model to summarize
    # For simplicity, just record timestamps and action count
    return {
        "date": datetime.now().isoformat(),
        "actions_taken": len(session_log),
        "log": session_log[-5:]  # Last 5 actions only
    }

The key design decision is what to put in memory and what to leave out. Don't store everything - that makes loading memory expensive and the context gets noisy. Store what changes the agent's behavior in future sessions:

Facts learned (this competitor dropped prices by 20%)
Preferences established (user prefers summaries under 200 words)
Past decisions and their outcomes (tried format X, user didn't like it)
State that needs to persist (last checked timestamp, current task status)

Don't store raw tool call logs or full page content in long-term memory. That's what files and databases are for. Memory should be compact enough to load into context without eating most of the window.

Injecting Memory into the System Prompt

Once you have memory loaded, inject the relevant parts into the system prompt or the initial user message:


def build_system_prompt_with_memory(base_prompt: str, memory: dict) -> str:
    memory_section = ""
    
    if memory.get("learned_facts"):
        facts = "\n".join(f"- {f}" for f in memory["learned_facts"][-10:])
        memory_section += f"\n\nThings you've learned in previous sessions:\n{facts}"
    
    if memory.get("preferences"):
        prefs = "\n".join(f"- {k}: {v}" for k, v in memory["preferences"].items())
        memory_section += f"\n\nEstablished preferences:\n{prefs}"
    
    if memory.get("sessions"):
        last = memory["sessions"][-1]
        memory_section += f"\n\nLast session: {last['date']}"
    
    return base_prompt + memory_section

The agent now knows what it learned last time. It doesn't repeat research it already did. It applies preferences it's already established. It picks up where it left off instead of starting over.

Reliability: Why Agents Fail in Production

Demo agents run in controlled conditions. Production agents run in the real world, where:

APIs return 429 rate limit errors
Websites return malformed HTML
Network requests time out at random
The model occasionally decides to call a tool with invalid arguments
Loop conditions occasionally fail to trigger

None of these are catastrophic in isolation. But an agent with no error handling will crash on any one of them and leave no trace of what it was doing when it stopped.

Reliability engineering for agents is the same as reliability engineering for any system: anticipate the failure modes, handle them gracefully, and make the failure visible enough to debug.

Error Handling at the Tool Level

Every tool should handle errors and return them as structured data:


def search_web(query: str) -> dict:
    try:
        response = requests.get(
            "https://api.search.brave.com/res/v1/web/search",
            headers={"Accept": "application/json", "X-Subscription-Token": BRAVE_API_KEY},
            params={"q": query, "count": 5},
            timeout=10
        )
        response.raise_for_status()
        data = response.json()
        
        results = []
        for r in data.get("web", {}).get("results", []):
            results.append({
                "title": r["title"],
                "url": r["url"],
                "snippet": r["description"]
            })
        
        return {"status": "ok", "results": results}
    
    except requests.exceptions.Timeout:
        return {"status": "error", "error": "Request timed out. Try again or use a different search."}
    
    except requests.exceptions.HTTPError as e:
        if e.response.status_code == 429:
            return {"status": "error", "error": "Rate limited. Wait 60 seconds and try again."}
        return {"status": "error", "error": f"HTTP error: {e.response.status_code}"}
    
    except Exception as e:
        return {"status": "error", "error": f"Unexpected error: {str(e)}"}

The model reads the error and can decide what to do: retry, try a different approach, or stop and report the problem. An unhandled exception gives the model nothing to work with and crashes your loop.

Retries with Backoff

For transient errors (rate limits, network timeouts), automatic retries save you:


import time
from functools import wraps

def with_retry(max_attempts=3, base_delay=2):
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            for attempt in range(max_attempts):
                result = func(*args, **kwargs)
                
                # If it worked, return immediately
                if isinstance(result, dict) and result.get("status") == "ok":
                    return result
                
                # If it's a retryable error and not the last attempt
                if attempt < max_attempts - 1:
                    error = result.get("error", "") if isinstance(result, dict) else str(result)
                    if "rate limit" in error.lower() or "timeout" in error.lower():
                        delay = base_delay * (2 ** attempt)  # Exponential backoff
                        print(f"Retrying in {delay}s... (attempt {attempt + 1}/{max_attempts})")
                        time.sleep(delay)
                        continue
                
                # Non-retryable error or final attempt
                return result
            
            return result
        return wrapper
    return decorator

@with_retry(max_attempts=3, base_delay=2)
def search_web(query: str) -> dict:
    # Same implementation as above
    ...

Apply this decorator to any tool that calls an external service. Most transient failures resolve in a retry or two.

Human-in-the-Loop Checkpoints

For agents with real consequences (sending emails, posting publicly, modifying files, making purchases), you want a human approval step before the irreversible action.

The simplest implementation: a tool that pauses the agent and waits for input:


def request_approval(action: str, details: str) -> str:
    """Ask a human to approve an action before executing it."""
    print(f"\n[APPROVAL REQUIRED]")
    print(f"Action: {action}")
    print(f"Details: {details}")
    
    while True:
        response = input("Approve? (yes/no/abort): ").strip().lower()
        if response == "yes":
            return "APPROVED"
        elif response == "no":
            return "DENIED: Human rejected this action"
        elif response == "abort":
            raise SystemExit("Agent aborted by human")
        else:
            print("Please enter yes, no, or abort")

Add it to your system prompt: "Before sending any email or posting any content, call request_approval with the action and full details. Only proceed if the result is APPROVED."

For async agents that run unattended (scheduled jobs, background processes), the checkpoint becomes a queue. The agent writes a pending action to a queue, sends you a notification, and pauses. You review it in a web UI or via Slack command, approve or reject, and the agent continues.

This is overkill for low-stakes tasks. It's not overkill for anything that touches customer communications, financial systems, or public content.

Putting It Together: A Production-Ready Agent Template


class Agent:
    def __init__(self, name: str, system_prompt: str, tools: list, memory_path: str = None):
        self.name = name
        self.system_prompt = system_prompt
        self.tools = tools
        self.memory_path = memory_path
        self.memory = load_memory(memory_path) if memory_path else {}
        self.session_log = []
        self.client = anthropic.Anthropic()
    
    def run(self, task: str, max_steps: int = 20) -> str:
        # Build prompt with memory
        prompt = build_system_prompt_with_memory(self.system_prompt, self.memory)
        
        messages = [{"role": "user", "content": task}]
        
        for step in range(max_steps):
            try:
                response = self.client.messages.create(
                    model="claude-opus-4-5",
                    max_tokens=4096,
                    system=prompt,
                    tools=self.tools,
                    messages=messages
                )
            except anthropic.RateLimitError:
                print("Rate limited. Waiting 60 seconds...")
                time.sleep(60)
                continue
            except anthropic.APIError as e:
                self.session_log.append(f"API error at step {step}: {e}")
                break
            
            messages.append({"role": "assistant", "content": response.content})
            
            if response.stop_reason == "end_turn":
                final_text = next(
                    (b.text for b in response.content if hasattr(b, "text")), ""
                )
                self.session_log.append(f"Completed: {final_text[:100]}...")
                break
            
            # Handle tool calls
            tool_results = self._dispatch_tools(response.content)
            messages.append({"role": "user", "content": tool_results})
        
        else:
            self.session_log.append(f"Stopped: max steps ({max_steps}) reached")
        
        # Save memory at the end of the session
        if self.memory_path:
            self.memory.setdefault("sessions", []).append(
                summarize_session(self.memory, self.session_log)
            )
            save_memory(self.memory_path, self.memory)
        
        return final_text

This template handles rate limits, caps the loop at a sane step count, logs the session, and saves memory on completion. It's not production hardened for every case, but it's a solid starting point.

Exercise: Add File-Based Memory to Your Agent

Take the agent from Lesson 2 (website monitor) and add persistent memory.

Your memory file should store:

A history of every check: timestamp, URL, result (changed/unchanged)
A summary of the most recent change found for each URL
Any notes the agent has added about patterns it's noticed

Then update your system prompt to tell the agent to:

Read the memory at startup to know what's happened previously
Consider past changes when evaluating whether the current change is significant (a price that changes every week is less alarming than one that's never changed before)
Update the memory with today's check result

Run it 5 times on the same URL. After the 5th run, read the memory file. The agent should have built up a meaningful log of what's changed.

Then add one reliability feature: retry on timeout or rate limit, with 3 attempts and exponential backoff.

When both are working, you have an agent that's meaningfully closer to production-ready.

Key Takeaway

Memory is code. Reliability is code. Neither one is optional if you want agents that work in production and not just demos. Build both in from the start - they're much harder to retrofit later.

What's Next

You've built real agents. Course 4 shifts to the business layer: the economics, market strategy, and launch tactics that turn what you can build into something that generates revenue.

Multi-Agent SystemsPrevious Back to Learn →