Skip to content

Simulate timeouts and rate limits

The built-in faults cover the failures tools hit most: timeouts, rate limits, server errors and slow responses. All of them are deterministic: the same calls fail on every run.

Fault (with_=) Raises Also an instance of
"timeout" (default) ToolCallTimeout ToolCallError; TimeoutError from the next release
"rate_limit" ToolCallRateLimited (HTTP 429, retry_after) ToolCallError
"api_error" ToolCallAPIError (HTTP 503) ToolCallError
"malformed_output" ToolCallMalformedOutput ToolCallError

All four live in agentprobe.injector.tool_failure. To raise the exact exception your client raises instead, see Inject the exceptions your code handles.

A rate-limited search tool

from agentprobe.injector.tool_failure import ToolCallRateLimited
from agentprobe.testing import probe


def search(query: str) -> str:
    return f"3 results for {query!r}"


def research_agent(question: str, tools) -> str:
    try:
        return tools["search"](question)
    except ToolCallRateLimited as exc:
        return f"Search is rate limited; try again in {exc.retry_after:.0f}s."


def test_reports_rate_limit_instead_of_guessing():
    with probe() as p:
        tools = p.wrap({"search": search})
        p.on("search").always_fail(with_="rate_limit")
        result = p.run(lambda q: research_agent(q, tools), "fault injection")
        result.assert_no_crash()
        result.assert_answer_contains("rate limited")
        result.assert_answer_not_contains("results for")

A slow tool and a deadline

slow(ms=...) delays a call and then runs the real tool. Combine it with p.run(..., timeout=seconds) to test a latency budget:

def test_answers_within_budget_despite_a_slow_call():
    with probe() as p:
        tools = p.wrap({"search": search})
        p.on("search").slow(ms=200)
        result = p.run(lambda q: research_agent(q, tools), "slow", timeout=5)
        result.assert_within(2)
        result.assert_answer_contains("results for")

If the agent is still running at the deadline, the result says deadline_exceeded=True and still_running=True, and answer assertions are unassessed, which fails the gate. Python can't stop a thread, so AgentProbe never reports a run that outlived its deadline as passed. Async agents (await p.arun(...)) are cancelled at the deadline; see How probing works.

Random faults for experiments

with_probability(p, seed=...) injects faults at random, reproducibly for a given seed. Prefer the deterministic scenarios in tests: a test that fails one run in five is hard to trust.