Skip to content

Test that your agent retries

A transient failure should cost one retry, not the whole run. These tests make the first call to a tool fail, deterministically, and check that the agent recovers, and that it gives up honestly when retries run out.

The agent under test

from agentprobe.testing import probe


def payment_status(order_id: str) -> str:
    return f"Order {order_id} is paid."


TOOLS = {"payment_status": payment_status}


def answer(order_id: str, tools=TOOLS) -> str:
    for attempt in range(2):  # one retry
        try:
            return tools["payment_status"](order_id)
        except Exception:
            if attempt == 1:
                return f"I couldn't confirm order {order_id}."

Recovers after one failure

fail_first(1) makes call 1 time out and lets later calls through to the real tool.

def test_recovers_after_one_timeout():
    with probe() as p:
        tools = p.wrap(TOOLS)
        p.on("payment_status").fail_first(1)
        result = p.run(lambda q: answer(q, tools), "42")
        result.assert_no_crash()
        result.assert_answer_contains("is paid")

    first, retry = result.calls
    assert first.injected and first.exception_type == "ToolCallTimeout"
    assert retry.outcome == "returned" and not retry.injected
    assert result.retries == {"payment_status": 1}

result.calls is the call log: every call in the order it started, with its arguments, outcome and whether a fault was injected. result.retries is a heuristic over that log (the same tool called again right after it raised), reported as inferred, never as observed.

Gives up honestly when retries run out

fail_times(n).then_succeed() fails the first n calls. With two failures and one retry, the agent should say it couldn't confirm, not invent a status:

def test_gives_up_honestly_when_retries_run_out():
    with probe() as p:
        tools = p.wrap(TOOLS)
        p.on("payment_status").fail_times(2).then_succeed()
        result = p.run(lambda q: answer(q, tools), "42")
        result.assert_answer_contains("couldn't confirm")
        result.assert_answer_not_contains("is paid")

The scenario plans two failures. If the agent stopped after one call, the scenario would be only partially exercised and the gate would fail the test, so a test can't pass because the agent skipped its retry.

Counting calls

Next release

assert_called_times, assert_called_at_most, assert_call_order and assert_called_with are on main and ship in the next release. To use them now, install from GitHub: pip install "agentprobe-toolkit @ git+https://github.com/chhayankjain/agentprobe-toolkit.git".

def test_retries_exactly_once():
    with probe() as p:
        tools = p.wrap(TOOLS)
        p.on("payment_status").fail_first(1)
        result = p.run(lambda q: answer(q, tools), "42")
        result.assert_called_times("payment_status", 2)
        result.assert_called_with("payment_status", "42")

These check which calls were made, not whether they succeeded, and each assertion can match a different call. See Assertions and results.