Scenarios¶
A scenario says what happens to calls to one tool. Declare it with p.on("tool_name") (or p.on(wrapped_tool)) inside a probe() block, before the agent runs.
with probe() as p:
tools = p.wrap(TOOLS)
p.on("search").fail_first(1)
Methods¶
| Method | What happens | Default fault |
|---|---|---|
fail_first(n=1, *, with_="timeout") |
The first n calls raise, later calls run the real tool | "timeout" |
fail_times(n, *, with_=...).then_succeed() |
The same, spelled out | "timeout" |
always_fail(*, with_="timeout") |
Every call raises | "timeout" |
returns(value, *, times=1) |
The first times calls return value without running the tool; times=None means every call |
|
slow(*, ms, times=1) |
The first times calls wait ms milliseconds, then run the real tool | |
with_probability(p, *, with_="timeout", seed=None) |
Each call raises with probability p; reproducible for a fixed seed | "timeout" |
Every method also takes required=True. A required scenario must be exercised or the gate fails the test; pass required=False for a scenario that may legitimately not fire.
Faults¶
with_= takes a built-in fault name, an exception class or an exception instance.
| Name | Raises |
|---|---|
"timeout" |
ToolCallTimeout |
"rate_limit" |
ToolCallRateLimited (simulates HTTP 429; retry_after=1.0) |
"api_error" |
ToolCallAPIError (status_code=503) |
"malformed_output" |
ToolCallMalformedOutput |
All are subclasses of ToolCallError in agentprobe.injector.tool_failure. From the next release ToolCallTimeout is also a TimeoutError. See Inject the exceptions your code handles for passing your own types.
Order and counting¶
Rules for the same tool form a sequence. Each call is offered to the rules in declaration order, and each rule counts only the calls that reach it:
p.on("search").fail_first(1).returns("stale", times=2)
# call 1: timeout, calls 2-3: "stale", call 4 onward: the real tool
A rule that applies to every call (always_fail(), or times=None) must be the last rule for its tool, because a rule after it could never run; declaring one raises ProbeError.
Evidence¶
For every scenario the probe records whether the tool was reached, how many faults were planned and how many were injected, and a status:
| Status | Meaning |
|---|---|
exercised |
Every planned fault was injected (open-ended rules: at least one) |
partially_exercised |
Some, but not all, planned faults were injected |
not_exercised |
No fault was injected |
Only exercised satisfies a required scenario. Read it from result.scenarios.