Built specifically for async def coroutines. True latency percentiles (p50-p99), concurrency load testing, task leak detection, and statistically sound regression tests.
Works directly inside pytest-asyncio test coroutines or standard sync test functions. Zero boilerplate, zero loop hackery.
How it works
Times every individual call instead of batch averages. Measures min, max, mean, median, stddev, and true p50, p75, p90, p95, p99.
Latency metrics
Benchmark with N calls in flight at once. Reports throughput (ops/sec), latency percentiles, timeouts, errors, and event-loop lag.
Concurrency benchmarks
Run per-iteration setup and cleanup for database transactions or mocks, completely excluded from measured execution time.
Pedantic mode
Automatically detects orphan background tasks spawned by unawaited coroutines or leaky client libraries before they corrupt tests.
Leak detection
Compare against saved baselines. Fails CI only on statistically verified slowdowns using the non-parametric Mann-Whitney U test.
CI regression guard
Quick Example
Install pytest-async-benchmark and benchmark your coroutines with standard pytest fixtures:
import asyncio
import pytest
async def fetch_user_data(user_id: int):
await asyncio.sleep(0.005) # simulated network I/O
return {"id": user_id, "status": "active"}
@pytest.mark.asyncio
async def test_fetch_latency(async_benchmark):
# Benchmark the coroutine function with auto-calibrated runs
result = await async_benchmark(fetch_user_data, 42)
# Assert on real p95/p99 percentiles or throughput
assert result["p95"] < 0.010
assert result["ops"] > 100
Concurrency Benchmarking
Sequential benchmarks tell you how fast code runs in isolation; concurrency benchmarks tell you how it behaves under load.
@pytest.mark.asyncio
async def test_api_concurrency(async_benchmark):
# Run 100 total requests with 20 requests in flight simultaneously
stats = await async_benchmark.concurrent(
fetch_user_data,
42,
concurrency=20,
total=100,
leak_check=True,
)
print(f"Throughput: {stats.ops:.1f} ops/sec")
print(f"p99 Latency: {stats.p99 * 1000:.2f} ms")
print(f"Loop Lag: {stats.loop_lag * 1000:.2f} ms")
CI Regression Guard
Save historical baselines and prevent silent slowdowns in CI. To eliminate false alarms from noisy cloud runners, pytest-async-benchmark applies a one-sided Mann-Whitney U test across individual call samples, only failing your pipeline when a regression is both substantive and statistically significant ($p < 0.05$).
# Compare against baseline and fail if median latency regresses by >10%
pytest --async-bench-compare --async-bench-compare-fail=median:10%
Replaces fragile fixed cutoffs with non-parametric Mann-Whitney U testing, ignoring transient VM neighbor spikes.
--async-bench-autosave records numbered runs in .async_benchmarks/ with Git commit SHA and host hardware metadata.
Export all raw percentiles and per-call sample arrays via --async-bench-json for dashboard ingestion and PR comments.
From the Author
Articles written by Mihai Farcas on building performant Python architectures, AI agents, and type systems:
Why measuring coroutines with synchronous benchmark harnesses yields false medians, how per-call timer calibration prevents event-loop distortion, and what true percentiles reveal under load.
Read the article →mihai.ltdWhy SQL, a generated contract and a type checker let an agent change something everywhere without missing a place.
Read the article →mihai.ltdWhy AI coding agents routinely introduce data leaks in application-layer authorization, and how PostgreSQL RLS turns authorization into an auditable data-layer barrier.
Read the article →Written by Mihai Farcas, author and maintainer of pytest-async-benchmark. Follow along on GitHub.
Latest Updates