Docker
a multi-stage imageRuns as a non-root user, read-only root filesystem, and the extras it installs chosen at build time so an image carries only the provider SDKs it needs.
effgen serve · 1.0.0
The same agents, behind an API someone else calls
effgen serve puts an OpenAI-compatible server in front of everything on this site, and it is never unauthenticated by default. What it adds beyond the protocol is the part that decides whether you can run it: who called, what they were allowed to do, what it cost, how close the error budget is to being spent, and what happens when a provider stops answering.
9
Instruments on /metrics
5
Roles the server ships
12
Error categories, each with a next step
4
Deployment targets, one application
GET /metrics
$ curl -s http://127.0.0.1:8000/metrics# HELP python_gc_objects_collected_total Objects collected during gc# TYPE python_gc_objects_collected_total counterpython_gc_objects_collected_total{generation="0"} 48690.0python_gc_objects_collected_total{generation="1"} 5868.0python_gc_objects_collected_total{generation="2"} 1984.0# HELP python_gc_objects_uncollectable_total Uncollectable objects found during GC# TYPE python_gc_objects_uncollectable_total counterpython_gc_objects_uncollectable_total{generation="0"} 0.0python_gc_objects_uncollectable_total{generation="1"} 0.0python_gc_objects_uncollectable_total{generation="2"} 0.0# HELP python_gc_collections_total Number of times this generation was collected# TYPE python_gc_collections_total counterpython_gc_collections_total{generation="0"} 1201.0python_gc_collections_total{generation="1"} 109.0python_gc_collections_total{generation="2"} 8.0# HELP python_info Python platform information# TYPE python_info gaugepython_info{implementation="CPython",major="3",minor="11",patchlevel="15",version="3.11.15"} 1.0# HELP process_virtual_memory_bytes Virtual memory size in bytes.# TYPE process_virtual_memory_bytes gaugeprocess_virtual_memory_bytes 2.071785472e+010# HELP process_resident_memory_bytes Resident memory size in bytes.# TYPE process_resident_memory_bytes gaugeprocess_resident_memory_bytes 4.210688e+08# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.# TYPE process_start_time_seconds gaugeprocess_start_time_seconds 1.78739272692e+09# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.# TYPE process_cpu_seconds_total counterprocess_cpu_seconds_total 10.950000000000001# HELP process_open_fds Number of open file descriptors.# TYPE process_open_fds gaugeprocess_open_fds 25.0# HELP process_max_fds Maximum number of open file descriptors.# TYPE process_max_fds gaugeprocess_max_fds 1.048576e+06 # HELP effgen_model_call_latency_seconds Latency of model (LLM) calls in seconds# TYPE effgen_model_call_latency_seconds histogrameffgen_model_call_latency_seconds_bucket{model="gpt-5-nano",outcome="ok",provider="openai",le="0.05"} 0effgen_model_call_latency_seconds_bucket{model="gpt-5-nano",outcome="ok",provider="openai",le="0.1"} 0effgen_model_call_latency_seconds_bucket{model="gpt-5-nano",outcome="ok",provider="openai",le="0.25"} 0effgen_model_call_latency_seconds_bucket{model="gpt-5-nano",outcome="ok",provider="openai",le="0.5"} 0effgen_model_call_latency_seconds_bucket{model="gpt-5-nano",outcome="ok",provider="openai",le="1.0"} 0effgen_model_call_latency_seconds_bucket{model="gpt-5-nano",outcome="ok",provider="openai",le="2.5"} 3effgen_model_call_latency_seconds_bucket{model="gpt-5-nano",outcome="ok",provider="openai",le="5.0"} 3effgen_model_call_latency_seconds_bucket{model="gpt-5-nano",outcome="ok",provider="openai",le="10.0"} 3effgen_model_call_latency_seconds_bucket{model="gpt-5-nano",outcome="ok",provider="openai",le="20.0"} 3effgen_model_call_latency_seconds_bucket{model="gpt-5-nano",outcome="ok",provider="openai",le="30.0"} 3effgen_model_call_latency_seconds_bucket{model="gpt-5-nano",outcome="ok",provider="openai",le="60.0"} 3effgen_model_call_latency_seconds_bucket{model="gpt-5-nano",outcome="ok",provider="openai",le="+Inf"} 3effgen_model_call_latency_seconds_sum{model="gpt-5-nano",outcome="ok",provider="openai"} 5.076216697692871effgen_model_call_latency_seconds_count{model="gpt-5-nano",outcome="ok",provider="openai"} 3effgen_model_call_latency_seconds_bucket{model="gemini-3.1-flash-lite",outcome="ok",provider="gemini",le="0.05"} 0effgen_model_call_latency_seconds_bucket{model="gemini-3.1-flash-lite",outcome="ok",provider="gemini",le="0.1"} 0effgen_model_call_latency_seconds_bucket{model="gemini-3.1-flash-lite",outcome="ok",provider="gemini",le="0.25"} 0effgen_model_call_latency_seconds_bucket{model="gemini-3.1-flash-lite",outcome="ok",provider="gemini",le="0.5"} 0effgen_model_call_latency_seconds_bucket{model="gemini-3.1-flash-lite",outcome="ok",provider="gemini",le="1.0"} 1effgen_model_call_latency_seconds_bucket{model="gemini-3.1-flash-lite",outcome="ok",provider="gemini",le="2.5"} 1effgen_model_call_latency_seconds_bucket{model="gemini-3.1-flash-lite",outcome="ok",provider="gemini",le="5.0"} 1effgen_model_call_latency_seconds_bucket{model="gemini-3.1-flash-lite",outcome="ok",provider="gemini",le="10.0"} 1effgen_model_call_latency_seconds_bucket{model="gemini-3.1-flash-lite",outcome="ok",provider="gemini",le="20.0"} 1effgen_model_call_latency_seconds_bucket{model="gemini-3.1-flash-lite",outcome="ok",provider="gemini",le="30.0"} 1effgen_model_call_latency_seconds_bucket{model="gemini-3.1-flash-lite",outcome="ok",provider="gemini",le="60.0"} 1effgen_model_call_latency_seconds_bucket{model="gemini-3.1-flash-lite",outcome="ok",provider="gemini",le="+Inf"} 1effgen_model_call_latency_seconds_sum{model="gemini-3.1-flash-lite",outcome="ok",provider="gemini"} 0.6873214244842529effgen_model_call_latency_seconds_count{model="gemini-3.1-flash-lite",outcome="ok",provider="gemini"} 1 # HELP effgen_tool_call_latency_seconds Latency of tool calls in seconds# TYPE effgen_tool_call_latency_seconds histogram # HELP effgen_agent_iteration_latency_seconds Latency of a single agent iteration in seconds# TYPE effgen_agent_iteration_latency_seconds histogram # HELP effgen_tokens_total Total tokens consumed, by provider/model/kind# TYPE effgen_tokens_total countereffgen_tokens_total{kind="input",model="gpt-5-nano",provider="openai"} 72.0effgen_tokens_total{kind="output",model="gpt-5-nano",provider="openai"} 798.0effgen_tokens_total{kind="input",model="gemini-3.1-flash-lite",provider="gemini"} 18.0effgen_tokens_total{kind="output",model="gemini-3.1-flash-lite",provider="gemini"} 2.0 # HELP effgen_http_requests_total Total HTTP requests to the server, by route/method/status# TYPE effgen_http_requests_total countereffgen_http_requests_total{method="GET",route="/health",status="200"} 1.0effgen_http_requests_total{method="GET",route="/v1/models",status="401"} 2.0effgen_http_requests_total{method="POST",route="/v1/chat/completions",status="200"} 4.0effgen_http_requests_total{method="POST",route="/v1/chat/completions",status="404"} 1.0effgen_http_requests_total{method="GET",route="/v1/models",status="200"} 3.0effgen_http_requests_total{method="GET",route="/v1/models",status="429"} 18.0 # HELP effgen_circuit_breaker_state Circuit breaker state per provider (0=closed, 1=half_open, 2=open)# TYPE effgen_circuit_breaker_state gauge # HELP effgen_bulkhead_active Active in-flight calls held by the bulkhead, per provider# TYPE effgen_bulkhead_active gauge # HELP effgen_bulkhead_queued Calls waiting for a bulkhead permit, per provider# TYPE effgen_bulkhead_queued gauge # HELP effgen_bulkhead_utilization_pct Bulkhead active/max_concurrency utilization percentage, per provider# TYPE effgen_bulkhead_utilization_pct gauge # HELP effgen_requests_total Total number of agent run requests# TYPE effgen_requests_total countereffgen_requests_total{agent_name="api:openai:gpt-5-nano"} 3.0effgen_requests_total{agent_name="api:gemini:gemini-3.1-flash-lite"} 1.0 # HELP effgen_tool_calls_total Total number of tool calls# TYPE effgen_tool_calls_total counter # HELP effgen_tool_failures_total Total number of failed tool calls# TYPE effgen_tool_failures_total counter # HELP effgen_errors_total Total number of errors# TYPE effgen_errors_total counter # HELP effgen_fallbacks_total Total number of fallback invocations# TYPE effgen_fallbacks_total counter # HELP effgen_circuit_breaker_trips_total Total number of circuit breaker trips# TYPE effgen_circuit_breaker_trips_total counter # HELP effgen_tokens_used_total Total tokens consumed# TYPE effgen_tokens_used_total countereffgen_tokens_used_total{agent_name="api:openai:gpt-5-nano"} 870.0effgen_tokens_used_total{agent_name="api:gemini:gemini-3.1-flash-lite"} 20.0 # HELP effgen_response_latency_seconds Agent response latency in seconds# TYPE effgen_response_latency_seconds histogrameffgen_response_latency_seconds_sum{agent_name="api:openai:gpt-5-nano"} 5.076216697692871effgen_response_latency_seconds_count{agent_name="api:openai:gpt-5-nano"} 3effgen_response_latency_seconds_sum{agent_name="api:gemini:gemini-3.1-flash-lite"} 0.6873214244842529effgen_response_latency_seconds_count{agent_name="api:gemini:gemini-3.1-flash-lite"} 1 # HELP effgen_token_usage Token usage per request# TYPE effgen_token_usage histogrameffgen_token_usage_sum{agent_name="api:openai:gpt-5-nano"} 870.0effgen_token_usage_count{agent_name="api:openai:gpt-5-nano"} 3effgen_token_usage_sum{agent_name="api:gemini:gemini-3.1-flash-lite"} 20.0effgen_token_usage_count{agent_name="api:gemini:gemini-3.1-flash-lite"} 1 # HELP effgen_tool_execution_seconds Tool execution duration in seconds# TYPE effgen_tool_execution_seconds histogram # HELP effgen_active_agents Number of currently active agents# TYPE effgen_active_agents gaugeeffgen_active_agents{agent_name="api:openai:gpt-5-nano"} 0.0effgen_active_agents{agent_name="api:gemini:gemini-3.1-flash-lite"} 0.0 # HELP effgen_gpu_memory_used_bytes GPU memory used in bytes# TYPE effgen_gpu_memory_used_bytes gauge
A real scrape of a server this page drove traffic through: four completions across two providers, one more naming a model that does not exist, two requests rejected for carrying no credential, and eighteen refused by the rate limit. Latency is a histogram per provider and model, tokens are counted by provider, model and direction, and the request counter carries the route and the status — so a rise in errors can be attributed to a route and a backend before anyone opens a log.
The server
An OpenAI-compatible API, so your existing client already works
/v1/chat/completions, /v1/completions, /v1/models and /v1/embeddings, spoken the way every OpenAI client expects. Point the official SDK at it with a base_url and nothing else in your code changes — the model id carries the provider, so one endpoint serves every backend effGen reaches.
POST /v1/chat/completions
$ curl -s http://127.0.0.1:8000/v1/chat/completions -H "Authorization: Bearer $EFFGEN_API_KEY" -H 'Content-Type: application/json' -d '{"model":"openai:gpt-5-nano","messages":[{"role":"user","content":"Reply with the single word ok."}]}'{"id":"chatcmpl-c4b15234c0b04ed890fadfa6","object":"chat.completion","created":1787392732,"model":"openai:gpt-5-nano","choices":[{"index":0,"message":{"role":"assistant","content":"ok"},"finish_reason":"stop","logprobs":null}],"usage":{"prompt_tokens":26,"completion_tokens":202,"total_tokens":228},"effgen":{"requested_model":"openai:gpt-5-nano","resolved_model":"openai:gpt-5-nano","alias_applied":false,"cost_usd":0.0000821,"run_id":"7b7260803879"}}
The response is the shape a client expects, with one addition: an effgen block carrying the model that was requested, the model that actually served it, whether an alias was applied, what the call cost and the run id — which is the same id the trace spans and the run history carry, so a bill, a trace and a stored run all join up.
One worker by default
effgen serve binds to loopback and runs a single worker. For more, run the application factory under uvicorn or gunicorn — uvicorn effgen.server.app:create_app --factory --workers 4 — which is the same object the container image and the Lambda handler use. --host 0.0.0.0 exposes it, and the help text says to set a key first.
What serve takes
| Flag | Type | Description |
|---|---|---|
--host HOST | — | Host to bind to (default 127.0.0.1, loopback-only). Pass --host 0.0.0.0 to expose on all interfaces (set EFFGEN_API_KEY first). |
-p PORT, --port PORT | — | Port to bind to |
--rate-limit N | — | Requests/minute per client IP (overrides EFFGEN_RATE_LIMIT; 0 disables). Health probes are always exempt. |
--trust-proxy | — | Trust the first X-Forwarded-For hop as the rate-limit client IP (overrides EFFGEN_TRUST_PROXY). Enable only behind a reverse proxy that sets/overwrites this header — otherwise any caller can spoof it to bypass the rate limit. |
effgen serve --help · effGen 1.0.0
And what it reads from the environment
| Variable | Type | Description |
|---|---|---|
EFFGEN_API_KEY | — | static API key (Bearer or X-API-Key). If unset and not in dev mode, an ephemeral key is minted and printed once — never unauthenticated. |
EFFGEN_DEV_MODE=1 | — | disable auth (loud warning; local dev only). |
EFFGEN_RATE_LIMIT | — | requests/minute per client IP (0 disables; health probes are always exempt). Or use --rate-limit. |
EFFGEN_TRUST_PROXY=1 | — | trust the first X-Forwarded-For hop as the rate-limit client IP (default: off — the raw socket peer is used, since a caller can set that header to anything). Enable only behind a reverse proxy that sets/overwrites it. |
EFFGEN_CORS_ORIGINS | — | comma-separated allowed origins (default: none; cross-origin is fail-closed for a backend API). |
EFFGEN_OIDC_ISSUER / EFFGEN_OIDC_CLIENT_ID | — | enable OIDC/JWT auth instead of a static key. |
EFFGEN_PUBLIC_METRICS=1 | — | serve /metrics without auth (default: auth). |
EFFGEN_PUBLIC_DASHBOARD=1 | — | serve /dashboard/data.json + /dashboard/spans without auth, for local viewing (default: auth; the /dashboard page itself always loads, but its data calls 401 without this or an API key). |
EFFGEN_MODEL_POOL_SIZE | — | loaded models kept warm (default 4). |
EFFGEN_NO_DOTENV=1 | — | skip the .env filesystem search entirely, so only environment variables the orchestrator set are visible (EFFGEN_DOTENV=none is equivalent). |
The operational settings the command documents itself
Auth, roles, limits and the audit log
Never unauthenticated, even when you forget to configure it
With no OIDC issuer, no static key and no development flag set, effgen serve mints an ephemeral key at startup and prints it once. There is no state in which the server is running and open, and the only way to turn auth off is a flag that prints a warning while it does so.
no credential
$ curl -s -o /dev/null -w 'HTTP %{http_code}\n' http://127.0.0.1:8000/v1/models ; curl -s http://127.0.0.1:8000/v1/modelsHTTP 401{"error": {"message": "Missing API key (send 'Authorization: Bearer <key>' or 'X-API-Key: <key>')", "type": "invalid_request_error", "param": null, "code": "invalid_api_key"}}
A rejected request comes back in the same error envelope as everything else, so a client has one shape to parse. Bearer tokens from any OIDC provider and a static X-API-Key are both accepted; the static key is compared in constant time.
What answers without a credential
/docs, /health, /healthz, /livez, /openapi.json, /ready, /readyz, /redoc, /slo — liveness and readiness probes, the aggregate SLO status, and the API schema. None of them carries a request body, a cost or anyone’s data.
Everything else needs credentials, including every /v1 route and the data endpoints behind the dashboard and the playground. The two pages themselves always load, so they can prompt for a key. /metrics is protected by default.
The roles it ships with
| Role | Type | Description |
|---|---|---|
admin | all tools | no daily spend cap |
researcher | all tools | $50.00 a day |
limited_user | all tools | $5.00 a day |
viewer | no tools | $5.00 a day |
reader | no tools | $1.00 a day |
The policy registry in effGen 1.0.0. Read at run time, so a custom policy file replaces them.
A principal’s effective policy is the union of its roles, and the most permissive one wins — so adding a role can only widen access, never narrow it. A principal whose roles are all read-only is permitted no tools at all, and one with no recognised role falls back to read-only rather than failing open. Spend is counted per principal per UTC day; past the cap the call is refused rather than charged.
And the rate limit, per client address
the rate limit
$ for i in $(seq 1 20); do curl -s -o /dev/null -w '%{http_code} ' http://127.0.0.1:8000/v1/models -H "Authorization: Bearer $EFFGEN_API_KEY"; done # the server was started with EFFGEN_RATE_LIMIT=10200 200 200 200 200 200 200 200 200 200 429 429 429 429 429 429 429 429 429 429 {"error": {"message": "Rate limit exceeded. Please retry later.", "type": "rate_limit_exceeded", "param": null, "code": "rate_limit_exceeded"}}
Health probes are always exempt, so a limit cannot make an orchestrator think the service is down. The client address is the raw socket peer unless --trust-proxy is set, because any caller can put whatever they like in an X-Forwarded-For header.
Every request, on one line
the audit log
$ tail -4 "$EFFGEN_AUDIT_DIR/$(date -u +%F).jsonl"{"ts": "2026-08-22T10:04:38.714243+00:00", "principal": "anonymous", "roles": [], "endpoint": "GET /v1/models", "request_summary": "GET /v1/models", "response_summary": "HTTP 401 (application/json)", "outcome": "denied", "request_id": "", "duration_ms": 0.1, "extra": {}}{"ts": "2026-08-22T10:04:42.576906+00:00", "principal": "api-key", "roles": ["admin"], "endpoint": "POST /v1/chat/completions", "request_summary": "POST /v1/chat/completions", "response_summary": "HTTP 200 (application/json)", "outcome": "ok", "request_id": "", "duration_ms": 3856.09, "extra": {}}{"ts": "2026-08-22T10:04:42.588433+00:00", "principal": "api-key", "roles": ["admin"], "endpoint": "POST /v1/chat/completions", "request_summary": "POST /v1/chat/completions", "response_summary": "HTTP 404 (application/json)", "outcome": "error", "request_id": "", "duration_ms": 2.61, "extra": {}}
One JSON line per request/response pair, in a file per UTC day. The three lines above are the completion that succeeded, the request with no credential, and the one naming a model that does not exist — recorded as ok, denied and error.
| Field | Type | Description |
|---|---|---|
ts | — | ISO-8601 UTC timestamp |
principal | — | the caller — a JWT subject, an API key, or anonymous |
roles | — | the roles that caller held at the time of the request |
endpoint | — | method and path |
request_summary | — | method, path and the query string with secrets scrubbed |
response_summary | — | the status and the content type |
outcome | — | ok, error, or denied |
request_id | — | the X-Request-ID header, so a record joins to a trace |
duration_ms | — | handler wall-clock time |
What a record carries
And what it deliberately leaves out
- — request and response bodies
- — the Authorization header
- — API keys in a query string, which are replaced with [REDACTED]
An audit log that carries prompts is a second copy of your users’ data with a different retention policy. This one records who did what, when, and how it ended.
Metrics, traces, SLOs and alerts
Instrumented where it matters, not where it was easy
The three latencies that decide whether a service is usable — the model call, the tool call and one turn of the agent loop — are histograms with the labels you need to attribute a regression. Everything below is on /metrics in Prometheus format, with no exporter to configure.
| Metric | Type | Description |
|---|---|---|
effgen_agent_iteration_latency_seconds | histogram | Latency of a single agent iteration in seconds. Labels: preset. |
effgen_bulkhead_active | gauge | Active in-flight calls held by the bulkhead, per provider |
effgen_bulkhead_queued | gauge | Calls waiting for a bulkhead permit, per provider |
effgen_bulkhead_utilization_pct | gauge | Bulkhead active/max_concurrency utilization percentage, per provider |
effgen_circuit_breaker_state | gauge | Circuit breaker state per provider (0=closed, 1=half_open, 2=open) |
effgen_http_requests_total | counter | Total HTTP requests to the server, by route/method/status |
effgen_model_call_latency_seconds | histogram | Latency of model (LLM) calls in seconds. Labels: provider, model, outcome. |
effgen_tokens_total | counter | Total tokens consumed, by provider/model/kind |
effgen_tool_call_latency_seconds | histogram | Latency of tool calls in seconds. Labels: tool, outcome. |
Every instrument the observability module registers, effGen 1.0.0
An objective, and how fast its budget is burning
python
from effgen.observability.slo import SLO, get_tracker
tracker = get_tracker()
tracker.register(SLO(name="model_call_success", target_pct=99.0, window_seconds=3600))
for i in range(100):
tracker.record("model_call_success", ok=(i % 50 != 0))
slo = tracker.get_slo("model_call_success")
print("target: ", slo.target_pct, "%")
print("error budget:", slo.error_budget_fraction)
print("events: ", tracker.total_count("model_call_success"),
"· bad:", tracker.bad_count("model_call_success"))
print("burn rate: %.1fx" % tracker.burn_rate("model_call_success"))
print("fast burn: ", tracker.burn_rate("model_call_success") > 14.4)
print("status: ", tracker.status("model_call_success"))what that printed
target: 99.0 %
error budget: 0.01
events: 100 · bad: 2
burn rate: 2.0x
fast burn: False
status: {'name': 'model_call_success', 'target_pct': 99.0, 'window_seconds': 3600, 'query': '', 'total_events': 100, 'good_events': 98, 'bad_events': 2, 'good_ratio': 0.98, 'bad_ratio': 0.02, 'burn_rate': 2.0, 'within_budget': False}A burn rate of 1.0 is exactly on budget; two failures in a hundred against a 99% objective is twice the budget, which is the number above. The rolling window evicts old events as it is read, so the figure is about now rather than about the whole life of the process.
The alert pack that goes with it
python
from pathlib import Path
import yaml
from effgen import validate_alert_rules_yaml
rules = Path("docs/observability/alert_rules.yaml")
ok, errors = validate_alert_rules_yaml(rules)
print("valid:", ok, "· errors:", errors)
for group in yaml.safe_load(rules.read_text())["groups"]:
for rule in group["rules"]:
print(f"{rule['alert']:22} {rule['labels']['severity']:9} "
f"for {rule.get('for', 'instant')}")what that printed
valid: True · errors: [] HighErrorRate critical for 10m HighP95Latency warning for 5m CostBurnHigh warning for 0m SLOFastBurn critical for 0m SLOSlowBurn warning for 60m CircuitBreakerOpen warning for 1m
An Alertmanager rule file is in the repository at docs/observability/alert_rules.yaml — the path above is relative to a clone, not to an installed package — and validate_alert_rules_yaml will tell you it is well-formed before Prometheus does. AlertWebhook posts an alert to Slack or Discord and works out which from the URL.
Every hot path emits a span
the span tree
effgen.agent.run # entire agent.run() call effgen.agent.iteration # one ReAct iteration effgen.model.call # model inference effgen.tool.call [calculator] # tool 1 effgen.tool.call [web_search] # tool 2 effgen.tool.call [wikipedia] # tool 3 effgen.router.decision # policy-based routing (if enabled)
OpenTelemetry, so it goes to Jaeger, Zipkin, Tempo or anything else that speaks OTLP. The agent loop, the model adapters, the tools and the router already emit spans — setup_tracing(service_name=..., sampler=...) at startup is the whole integration. Five samplers ship, and a parent-based one over a ratio sampler is the one to reach for: a trace is either sampled whole or not at all.
What it does under load
effgen loadtest
$ effgen loadtest --url http://127.0.0.1:8000 --model openai:gpt-5-nano --concurrency 4 --duration 20Server mode: url=http://127.0.0.1:8214 model=openai:gpt-5-nanoStarting load test scenario=fixed concurrency=4 duration=20.0s──────────────────────────────────────────────────── Load Test Report — fixed──────────────────────────────────────────────────── Concurrency : 4 Duration : 20.0s Total requests: 18865 Successful : 9 Failed : 18856 Error rate : 99.95% Error types : rate_limited=18856 Throughput : 942.89 req/s Latency p50 : 3.5ms Latency p95 : 4.1ms Latency p99 : 4.4ms Latency mean : 4.2ms Latency stdev : 37.2ms────────────────────────────────────────────────────
Pointed at a URL rather than at a provider, the load test goes through the whole middleware stack — auth, the rate limit, the audit log — and not just the adapter. It reports the tail, not only the mean, and says how long it spent draining requests that were still in flight when the window closed.
Reliability and errors
Every failure is classified before anything decides to retry it
Retrying an authentication failure wastes a minute and fixes nothing; refusing to retry a transient blip turns a hiccup into an outage. So an exception is put in a category first, and the category decides. An unrecognised error is treated as retryable, so a genuine blip is not turned into a hard failure by a class name nobody had seen before.
Timeouts
effgen.reliability.timeouts.pyAn explicit wall-clock limit on every I/O call — the model call, the tool call, the HTTP request, the agent loop and the queue wait each have their own. A missing limit is caught by a test rather than discovered in production.
Retries
effgen.reliability.retry.pyExponential backoff with jitter, retrying only what could plausibly succeed on a second attempt, and honouring a Retry-After header when the provider sends one.
Circuit breaker
effgen.reliability.circuit.pyA per-provider state machine that stops calling a backend that keeps failing, lets one request through after a cooldown, and closes again on a success.
Bulkhead
effgen.reliability.bulkhead.pyA concurrency cap and a bounded queue per provider, so one slow backend cannot consume every worker and take the rest of the service down with it.
python
from effgen import RateLimitExceeded
from effgen.models.errors import (BackendUnreachableError, InvalidRequestError,
ModelAuthError, ModelNotFoundError,
ModelRefusalError, ModelTimeoutError,
ProviderTransientError,
classify_provider_error)
for exc in (ModelAuthError("x"), ModelNotFoundError("x"), RateLimitExceeded("x"),
ModelTimeoutError("x"), ProviderTransientError("x"),
ModelRefusalError("x"), InvalidRequestError("x"),
BackendUnreachableError("x"), ValueError("something unrecognised")):
classified = classify_provider_error(exc)
print(f"{type(exc).__name__:26} {classified.category:16} "
f"retry={classified.should_retry}")what that printed
ModelAuthError auth retry=False ModelNotFoundError not_found retry=False RateLimitExceeded rate_limited retry=True ModelTimeoutError timeout retry=True ProviderTransientError transient retry=True ModelRefusalError refusal retry=False InvalidRequestError invalid_request retry=False BackendUnreachableError unreachable retry=True ValueError unknown retry=True
The classifier recognises effGen’s own typed errors first, then falls back to the SDK exception name, the HTTP status and the message — so a raw provider exception still lands in one of these categories. 12 categories each carry a next step, which is what turns an error message into one that names the fix.
python
import time
from effgen import CircuitBreaker
breaker = CircuitBreaker(failure_threshold=3, cooldown_seconds=1)
for attempt in range(1, 5):
if not breaker.is_available("web_search"):
print(f"attempt {attempt}: not called — {breaker.get_state('web_search')}")
continue
breaker.record_failure("web_search")
print(f"attempt {attempt}: called, failed — {breaker.get_state('web_search')}")
time.sleep(1.1)
print("after the cooldown: ", breaker.get_state("web_search"))
print("would it be called now? ", breaker.is_available("web_search"))
breaker.record_success("web_search")
print("after that call succeeds:", breaker.get_state("web_search"))what that printed
attempt 1: called, failed — CircuitState.CLOSED attempt 2: called, failed — CircuitState.CLOSED attempt 3: called, failed — CircuitState.OPEN attempt 4: not called — CircuitState.OPEN after the cooldown: CircuitState.OPEN would it be called now? True after that call succeeds: CircuitState.CLOSED
Why an unreachable backend is its own case
Every category above describes something that answered. A connection refused, a host that does not resolve and a route that does not exist describe a backend that never did — and a batch that quietly completes against nothing looks healthy in the summary. So BackendUnreachableError is raised whatever raise_on_error says. It is the one error with no opt-out, and one of 1.0.0’s three breaking changes.
Guardrails, redaction and the sandbox
Checks over the text, and a wall around the code
A guardrail sits at one of 4 positions — input, output, tool_input, tool_output — and can pass, block or rewrite what goes through it. Four chains ship configured; the phi one is the health-record shape.
python
from effgen import get_guardrail_preset
for name in ("minimal", "standard", "strict", "phi"):
chain = get_guardrail_preset(name)
print(f"{name:9} {len(chain.guardrails)} guardrails: "
f"{[g.name for g in chain.guardrails]}")what that printed
minimal 2 guardrails: ['LengthGuardrail', 'PromptInjectionGuardrail'] standard 5 guardrails: ['LengthGuardrail', 'PromptInjectionGuardrail', 'PIIGuardrail', 'ToolInputGuardrail', 'ToolOutputGuardrail'] strict 7 guardrails: ['LengthGuardrail', 'PromptInjectionGuardrail', 'SystemPromptLeakGuardrail', 'ToxicityGuardrail', 'PIIGuardrail', 'ToolInputGuardrail', 'ToolOutputGuardrail'] phi 6 guardrails: ['LengthGuardrail', 'PromptInjectionGuardrail', 'SystemPromptLeakGuardrail', 'PIIGuardrail', 'ToolInputGuardrail', 'ToolOutputGuardrail']
python
from effgen import PIIGuardrail
guard = PIIGuardrail(action="redact")
record = ("Patient: Marta Reyes\nMRN: 55-2213\nDOB: January 5, 1980\n"
"Contact marta@example.com or 555-0142. Card 4111 1111 1111 1111.")
result = guard.check(record)
print("passed:", result.passed)
print(result.modified_content)what that printed
passed: True Patient: [NAME REDACTED] MRN: [MRN REDACTED] DOB: [DOB REDACTED] Contact [EMAIL REDACTED] or 555-0142. Card [CC REDACTED].
What redaction keeps, and what it admits
The label survives and the value goes, so a redacted record is still readable as a record. Social security numbers, emails, phone numbers, Luhn-checked card numbers and IP addresses are matched by shape; API keys and private-key headers are matched too, so a leaked credential is treated as sensitive rather than as ordinary text; and the label-anchored fields a health or insurance record carries — name, date of birth, medical record number, member and policy ids — are matched from their labels.
That last group is where the limit is: a name with no label in front of it is not matched. The framework documents that boundary rather than implying there is none.
Code the model wrote, run where it cannot reach anything
python
import asyncio
import os
from effgen.security.sandbox import SandboxConfig, get_sandbox
CODE = ("import os\n"
"print('processes visible: ', len([p for p in os.listdir('/proc') if p.isdigit()]))\n"
"print('~/.ssh contains: ', os.listdir(os.path.expanduser('~/.ssh')))\n"
"open('scratch.txt', 'w').write('allowed')\n"
"print('wrote in the scratch space')\n"
"try:\n"
" open(os.path.expanduser('~/escaped.txt'), 'w')\n"
"except OSError as e:\n"
" print('outside it: ', type(e).__name__, e.strerror)\n")
async def main() -> None:
sandbox = await get_sandbox()
result = await sandbox.run(CODE, "python", SandboxConfig())
print(result.stdout.rstrip())
print()
print("backend: ", result.backend_used)
print("network_isolated: ", result.network_isolated)
print("filesystem_confined: ", result.filesystem_confined)
print("process_table_isolated: ", result.process_table_isolated)
print("credential_reads_masked:", result.credential_reads_masked)
print()
print("on the host, for comparison:")
print(" processes:", len([p for p in os.listdir("/proc") if p.isdigit()]))
print(" ~/.ssh: ", os.listdir(os.path.expanduser("~/.ssh")))
asyncio.run(main())what that printed on this machine
processes visible: 1 ~/.ssh contains: [] wrote in the scratch space outside it: OSError Read-only file system backend: subprocess network_isolated: True filesystem_confined: True process_table_isolated: True credential_reads_masked: True on the host, for comparison: processes: 1603 ~/.ssh: ['known_hosts.old', 'known_hosts']
What that run actually enforced
This is the subprocess backend, which is what runs when there is no Docker daemon. The network namespace has no interfaces, so nothing gets out. Every mount is remounted read-only except one scratch directory — the same directory the file and shell tools use — so generated code can write the files the agent just made and nothing else.
Reads are not confined, and the framework says so rather than implying they are. What it does instead is mask the credential stores: ~/.ssh, ~/.aws, ~/.gnupg, ~/.kube, ~/.docker, ~/.azure, the gcloud configuration, the credential files beside them, /etc/shadow and mounted secrets are each covered by an empty directory or by /dev/null, so a read succeeds and returns nothing. Use the Docker backend when reads must be confined rather than masked.
The run also gets its own PID namespace, which is why one process is visible above against the host’s. Both are reported on the result as credential_reads_masked and process_table_isolated — a host that cannot create the namespace says so on the result instead of pretending.
Evaluation and CI
A prompt change is a code change, so give it an exit code
effgen eval runs a suite and exits non-zero when accuracy falls under a threshold you set, which is all a CI job needs. --compare-baseline goes further: a regression against a stored baseline fails the job whatever the threshold says, because the question is not whether the number is good but whether it got worse.
the gate passing
$ effgen eval --suite math -m openai:gpt-5-nano --max-cases 5 --fail-under 0.8Limited to first 5 casesLoading model openai:gpt-5-nano...Running math suite (5 cases, scoring=contains)... Evaluation Results: mathMetric Value------------- ------------Accuracy 100.0% (5/5)Avg Latency 1.6301sTotal Tokens 1222Tool Accuracy 0.0% By Difficulty: easy : 100.0% (5/5) Exit gate: PASS — accuracy 100.0% >= --fail-under 80%exit 0
the gate failing
$ effgen eval --suite math -m openai:gpt-5-nano --max-cases 5 --fail-under 1.01Limited to first 5 casesLoading model openai:gpt-5-nano...Running math suite (5 cases, scoring=contains)... Evaluation Results: mathMetric Value------------- ------------Accuracy 100.0% (5/5)Avg Latency 1.3205sTotal Tokens 905Tool Accuracy 0.0% By Difficulty: easy : 100.0% (5/5) Exit gate: FAIL — accuracy 100.0% < --fail-under 101%exit 1
Same suite, same result, different threshold — and the exit code is the only thing a CI job has to read. Five scoring modes ship, from exact match to a model as judge, and --temperature 0 makes the run reproducible wherever the provider supports it.
| Flag | Type | Description |
|---|---|---|
--suite SUITE | — | Built-in suite name (math, tool_use, reasoning, safety, conversation) OR a path to your own .jsonl/.json test cases |
-m MODEL, --model MODEL | — | Model to use |
--provider PROVIDER | — | Provider for a bare model id (e.g. openai, groq, cerebras, gemini, together, fireworks, replicate, anthropic, hf). Equivalent to the "provider:model" prefix. |
--preset {coding,general,math,media,minimal,multimodal,notify,rag,research} | — | Use a preset agent configuration |
--scoring {exact_match,contains,regex,semantic_similarity,llm_judge} | — | Scoring mode (default: contains) |
--threshold THRESHOLD | — | Per-case pass score for continuous scoring modes (semantic_similarity, llm_judge); has no effect on exact_match/contains/regex, whose scores are already binary (0 or 1) (default: 0.5). Use --fail-under to gate the exit code on suite accuracy. |
--fail-under ACCURACY | — | Minimum suite accuracy required for a zero exit code (default: 0.5). This is the CI gate; a --compare- baseline regression always fails regardless of this value. |
--temperature TEMPERATURE | — | Sampling temperature for the evaluated agent (0 for deterministic, reproducible scoring where the provider supports it; default: the model/preset default) |
--save-baseline | — | Save results as regression baseline |
--compare-baseline | — | Compare results against stored baseline |
--baseline-dir DIR | — | Directory for --save-baseline/--compare-baseline files (default: ./.effgen/baselines under the current directory, created if missing). A baseline saved under the installed package tree by an older effGen version is still read. |
-o OUTPUT, --output OUTPUT | — | Output file for results. The extension chooses the format: .html renders the shareable report, .md writes Markdown, anything else writes JSON. |
--report PATH.html | — | Write a self-contained HTML report to PATH — pass rate, exit gate, by-difficulty breakdown, and every case. The file opens offline with no external references. |
--difficulty {easy,medium,hard} | — | Filter test cases by difficulty |
--max-cases MAX_CASES | — | Only run the first N cases (quick subsample) |
--json | — | Emit the results object as JSON to stdout (for CI gating) |
--no-animation | — | Disable the live progress bar (plain output) |
effgen eval --help · effGen 1.0.0
against the stored baseline
$ effgen eval --suite math -m openai:gpt-5-nano --max-cases 5 --compare-baselineLimited to first 5 casesLoading model openai:gpt-5-nano...Running math suite (5 cases, scoring=contains)... Evaluation Results: mathMetric Value------------- ------------Accuracy 100.0% (5/5)Avg Latency 1.5968sTotal Tokens 1222Tool Accuracy 0.0% By Difficulty: easy : 100.0% (5/5) # Regression Report: math **Baseline:** 1.0.0 **Current:** 1.0.0 **Status:** PASS ## Metrics | Metric | Baseline | Current | Change ||--------|----------|---------|--------|| accuracy | 1.0000 | 1.0000 | +0.0% || avg_latency | 1.3822 | 1.5968 | +15.5% || total_tokens | 1030.0000 | 1222.0000 | +18.6% || avg_tool_accuracy | 0.0000 | 0.0000 | N/A | Exit gate: PASS — accuracy 100.0% >= --fail-under 50%
The regression report is Markdown, so it goes straight into a pull-request comment. It carries the version each side was measured at, which is what stops a comparison between two different releases being read as a behaviour change.
And the same run as a page you can send someone
--report out.html writes one self-contained file: the gate’s verdict, the headline figures, accuracy by difficulty and every case with what it expected and what it got. No external reference of any kind, so it opens from disk with the network off and survives being attached to an email. compare, cost and loadtest take the same flag.effgen eval --suite math -m openai:gpt-5-nano --max-cases 5 --fail-under 0.8 --report math-eval.htmlCost and budgets
What it spent, per model, against a cap
Spend is recorded per provider and per model as it happens, and a daily budget is a number rather than an alert someone reads later. A scaffolded project gets a $1.00 daily cap when none is set, so the first thing a new project cannot do is spend without a limit.
effgen cost
$ effgen cost set-budget 5.00 ; effgen cost today✓ Daily budget set to $5.0000 USD effGen Cost Summary — Last 24 hours--------------------------------------------------------------------------------Provider Model Reqs Cost (USD)--------------------------------------------------------------------------------openai gpt-5-nano 32 $0.002249gemini gemini-3.1-flash-lite 1 $0.000008--------------------------------------------------------------------------------TOTAL 33 $ 0.002256 Daily budget: $0.0023 / $5.0000 (0%)
Two caps, at two layers
The one above is the local one, kept in a database beside the session and run history, and it covers everything run from this machine. The server has its own: each role carries a daily cap in dollars, spend is counted per principal for the UTC day, and a call past the cap is refused with a 429 rather than charged and reported afterwards.
A price effGen does not have is never printed as $0
A model with no published rate — a fine-tune, an uncatalogued id, your own server — reports no cost and is counted as unpriced. A fabricated zero is the one answer that makes a spend total wrong without looking wrong, which is why the totals here can be smaller than the traffic and say so.
The catalog and its pricesWhere it is written down
Cost, sessions, run history and the rate-limit counters all live under $EFFGEN_HOME (~/.effgen by default), which is the one directory a container has to mount to keep any of it. Every store has its own variable when one directory is not what you want.
Deployment and hardware
One application, 4 places to put it
The object every one of these serves is the same application factory, effgen.server.app:create_app. Nothing is re-implemented per target, so auth, RBAC, the audit log and the /v1 routes behave the same wherever it runs.
Kubernetes
a Helm chartTwo replicas by default with horizontal autoscaling, liveness and readiness probes on the endpoints that answer without a credential, and configuration through values rather than a rebuilt image.
AWS Lambda
the same app under MangumThe FastAPI application that runs under uvicorn locally is deployed unchanged behind an API Gateway HTTP API. Auth, RBAC, the audit log and the /v1 routes behave identically.
Cloudflare
an edge proxy WorkerIn front of any of the three: CORS, a structural JWT and expiry check at the edge, per-IP and per-token rate limiting through KV, and security headers — forwarding to whichever backend you run.
A container that can prove it is up
bash
docker run --rm -p 8080:8080 \
-e EFFGEN_OIDC_ISSUER=https://your-issuer.example.com \
-e EFFGEN_OIDC_CLIENT_ID=your-client-id \
-v ~/.effgen:/home/effgen/.effgen:ro \
--read-only --tmpfs /tmp \
effgen:1.0.0
curl http://localhost:8080/healthRead-only root, a non-root user, and the state directory mounted rather than baked in. /health answers without a credential, which is what a probe needs and why the readiness endpoints are on the public list above.
If you are serving the weights yourself
The CUDA trap, and what effGen does about it
A PyTorch wheel is built against one CUDA runtime and a driver is forward-compatible only, so a CUDA-13 wheel on a 12.4 driver leaves you with torch.cuda.is_available() == False and everything running slowly on the CPU with no error. The framework detects that at runtime: when it sees physical NVIDIA cards that torch cannot use, it prints one warning naming the torch build and the driver’s version instead of running silently.
The second half of the trap is a later pip install quietly upgrading torch back. A constraints file per CUDA line pins torch and its companions, so an extras install cannot move them.
Or do not serve them from the agent at all
Loading weights in the agent’s process means one copy per process, no batching across callers and a GPU tied to the run’s lifetime. Running vLLM, SGLang or TGI separately and pointing effGen at it with a base_url fixes all three, and is the same code path this server exposes.
The reference for running it
The server’s endpoints and its error envelope, OIDC and the static key, the role model and its union semantics, the audit record, every metric and span attribute, the SLO maths, the reliability primitives, the sandbox threat model, and a page per deployment target.
docs/observability/slos.md — error budgets and burn rates