Agent Failure Handling and Recovery

Agent workflows do not fail only by crashing. They can time out, pick the wrong tool, lose context, repeat the same step, produce malformed output, or confidently report that work is done when the evidence is missing. A production design has to account for all of those cases.

The practical question is not “how do we prevent every failure?” That is impossible. The better question is: when the workflow fails, can we see it, recover from it, and keep the damage contained?

Why Agent Failure Feels Different

Traditional software usually fails in visible ways: an exception, a failed health check, a rejected database write, or an invalid response code. Agent workflows can fail while still producing polished text.

That makes three failure modes especially important:

• False success: the agent says the work is complete, but the file, record, test result, or external proof does not exist.
• Silent degradation: each step technically completes, but context grows, assumptions drift, and output quality gets worse without a clear error.
• Compounded mistakes: one agent produces an unsupported claim and a later step treats it as source material.

These failures are not edge cases. They are normal operating risks when models, tools, memory, and human handoffs are combined.

Common Failure Types

Use this taxonomy during design review.

Model Failures

The model call itself fails or becomes unavailable.

Examples:

• rate limit
• timeout
• provider outage
• safety refusal
• model fallback to a lower-quality route

Design response: retry with limits, record the model actually used, and stop if the fallback would change quality expectations.

Tool Failures

The model runs, but the tool it needs fails.

Examples:

• API permission denied
• database timeout
• file not found
• search result unavailable
• stale session or expired token

Design response: distinguish retryable failures from permission or data-boundary failures. Retrying a permission problem wastes time and can hide the real issue.

Context Failures

The workflow has too much, too little, or stale context.

Examples:

• important source material was not loaded
• a summary dropped a requirement
• the workflow used an old artifact version
• the context window is full and early instructions are no longer attended to well

Design response: use named artifacts, source references, and checkpoints. Do not rely on one long conversation transcript as the system of record.

Loop Failures

The agent repeats a step or correction cycle without getting closer to done.

Examples:

• generator and reviewer bounce the same draft back and forth
• the agent retries an impossible command
• the workflow keeps asking for missing information that is not available

Design response: every loop needs a maximum iteration count and an escalation rule. A loop without a limit is a cost and reliability incident waiting to happen.

Output Failures

The output exists, but it cannot safely be used.

Examples:

• malformed JSON
• missing required sections
• unverifiable claims
• invented citations
• incomplete action items
• content written for the wrong audience

Design response: validate output against concrete criteria before it crosses a trust boundary.

Fail Closed, Fail Open, or Degrade Gracefully

Not every failure needs the same response.

Fail Closed

Fail closed when a wrong action would be costly, external, hard to reverse, or harmful.

Use fail-closed behavior for:

• sending email or messages
• creating or updating tickets
• changing production systems
• publishing public content
• handling sensitive or regulated data
• producing formal customer deliverables

Fail closed means the workflow stops, records why, and asks for human review.

Fail Open

Fail open only when the consequence of being wrong is low and reversible.

Use fail-open behavior for:

• personal drafts
• brainstorm lists
• optional recommendations
• local formatting suggestions
• internal notes clearly labeled as unverified

Fail open does not mean “ignore the error.” It means the system may return best-effort output with clear uncertainty.

Degrade Gracefully

Some workflows can return a partial result that is still useful.

For example:

• A document summary can return sections already verified and list skipped sections.
• A search task can report which sources were unavailable.
• A batch job can process completed records and leave failed records in a retry queue.

Graceful degradation requires honest reporting. Do not say “processed all items” when only 8 of 10 completed.

Verification Before Completion

The most important design rule is simple: claims are not evidence.

An agent should not be allowed to mark work complete merely because it generated a completion message. It needs proof from the system of record.

Good verification checks include:

• file exists at the expected path
• test command passed
• database record was created
• API returned the expected status
• generated page renders
• link resolves
• artifact hash or version matches the expected output
• reviewer approved the output

For content work, completion might require:

1. the lesson file exists
2. frontmatter validates
3. the course page links to it
4. the page renders in the built site
5. tests cover the new route
6. source-boundary review finds no restricted material

That is slower than a claim. It is also the difference between finished work and a persuasive status update.

Retry, Checkpoint, Resume

Retries are useful for transient failures, but retries are not a strategy by themselves.

A safer pattern is:

1. Retry: for temporary model or network failures, with a maximum attempt count.
2. Checkpoint: after each completed unit of work, write what happened to durable state.
3. Resume: if the run stops, continue from the last verified checkpoint instead of starting over or guessing.

This matters most for batch work. A workflow that processes 100 records should know which 73 are done, which 2 failed, and which 25 remain. Without that state tracker, a restart can duplicate work, skip work, or claim completion without proof.

Escalation Paths

An agent should know when to stop.

Escalate when:

• required source material is missing
• the same failure repeats after the retry limit
• the workflow would need broader permissions
• the output affects a person, customer, or external system
• confidence is low but the result would be consequential
• validation fails twice in a row
• the agent discovers scope that was not approved

Escalation is not failure theater. It is how a system avoids pretending to know what it does not know.

Worked Example: Research Packet Builder

Imagine a workflow that builds a research packet from a list of ten public web sources.

Weak design:

• The agent reads whatever it can find.
• It writes a summary.
• It says the packet is complete.

Stronger design:

• The workflow stores a source list before it starts.
• Each source has a status: pending, fetched, summarized, failed, or skipped.
• The agent records the source URL, fetch result, and summary artifact for each item.
• If a source fails twice, it is marked failed with the reason.
• The final packet includes only verified summaries and a short “not included” section.
• A human reviews the packet before it is published.

The second version is not more sophisticated because it uses more AI. It is better because it makes state visible and failure recoverable.

Compact Exercise

Pick one workflow that uses AI to create or update something useful.

Answer these questions:

• What are the three most likely failure types?
• Which failures should retry?
• Which failures should stop the workflow?
• What evidence proves the workflow is actually complete?
• What checkpoint would let the work resume after a dropped connection?
• What output, if any, requires human approval before use?

Keep the answers short. If you cannot answer them, the workflow is not ready for autonomy.