Inject the exceptions your code handles¶
Code that handles real failures catches specific types: TimeoutError, ConnectionError, an HTTP client's errors, LangChain's ToolException. If a fault test raises a different type, the handler never runs and the test reports a crash the real failure wouldn't cause. Pass the exception your code expects with with_=.
Any exception class or instance¶
from langchain_core.tools import ToolException
from agentprobe.testing import probe
def fetch_rates(currency: str) -> float:
return 1.08
def convert(amount: float, tools) -> str:
try:
rate = tools["fetch_rates"]("EUR")
except (ConnectionError, ToolException):
return "Exchange rates are unavailable right now."
return f"{amount * rate:.2f} USD"
def test_connection_refused():
with probe() as p:
tools = p.wrap({"fetch_rates": fetch_rates})
p.on("fetch_rates").fail_first(1, with_=ConnectionError("connection refused"))
result = p.run(lambda a: convert(a, tools), 100)
result.assert_answer_contains("unavailable")
def test_tool_exception_class():
with probe() as p:
tools = p.wrap({"fetch_rates": fetch_rates})
p.on("fetch_rates").always_fail(with_=ToolException)
result = p.run(lambda a: convert(a, tools), 100)
result.assert_answer_contains("unavailable")
Each call raises a fresh exception: a shallow copy of an instance, or a new instance of a class, created with the message "
Common choices¶
| Your code catches | Pass |
|---|---|
TimeoutError |
with_=TimeoutError("timed out"), or "timeout" from the next release |
ConnectionError |
with_=ConnectionError("connection refused") |
LangChain ToolException |
with_=ToolException("service unavailable") |
httpx timeouts |
with_=httpx.ReadTimeout("read timed out") |
| your own error type | with_=PaymentGatewayError(...) |
Next release
From the next release, the built-in "timeout" fault is also a TimeoutError, so except TimeoutError catches it without with_= (#114). The other built-in faults are subclasses of ToolCallError only.
For bad data rather than an exception, use returns(...).