Run a small model on your own hardware, or reach 9 providers and any OpenAI-compatible server through the same agent. Tools, memory, RAG, guardrails, evaluation and a production server come with it.
Agents on small language models, wherever you run them
A Python framework for building agents that reason, call tools and finish work — on a model you load yourself, on a server you already run, or on a hosted provider. The same agent, the same tools, the same result object in all three, with the server, the operations surface and the command line around them.
In your own process
The weights load where your code runs. Four local engines — transformers, vllm, gguf, mlx — and no key, no network and no provider between the agent and the model.
python
from effgen importAgent,AgentConfig
agent =Agent(AgentConfig(model="Qwen/Qwen2.5-1.5B-Instruct"))print(agent.run("Name the three primary colours of light, comma separated.").text)
output
Red, Green, Blue
On a server you already run
One base_url points effGen at anything that speaks the OpenAI protocol — vLLM, SGLang, TGI, llama.cpp, Ollama, LM Studio, LiteLLM, a company gateway. The weights load once and every caller shares them.
python
import os
from effgen importAgent,AgentConfig
agent =Agent(AgentConfig(
model="openai:gpt-5-nano",
base_url="http://127.0.0.1:8000/v1",
api_key=os.environ["EFFGEN_API_KEY"],))print(agent.run("Reply with the single word: ready").text)
output
ready
Or a hosted provider
10 provider adapters, 9 of them carrying a bundled catalog of 417 priced models. Changing provider is changing one string; the agent, the tools and the results object do not move.
python
from effgen importAgent,AgentConfig
agent =Agent(AgentConfig(model="openai:gpt-5-nano"))print(agent.run("Name the three primary colours of light, comma separated.").text)
output
red, green, blue
version
1.0.0version
built-in tools
66built-in tools
presets
9presets
provider adapters
10provider adapters
catalogued models
417catalogued models
CLI commands
29CLI commands
public names
223public names
python
3.11–3.14python
Every figure on this page is read from the installed package when the site is built, not written into it.
Quick start
Your first three commands
From the shell, from Python, or behind an API. Everything below was run before it was written down, and the output under each step is what that run printed.
Nothing to write. Install it, check what your keys reach, and run an agent from the shell.
01
Install
One package. Extras only when you want a particular local engine or provider client.
bash
pip install -U effgen
effgen--version
output
effGen 1.0.0
02
See what your keys reach
doctor reads the keys in your shell and your project .env and reports what each provider can serve. It goes on to print a system section and a check of what the coding agent needs; the provider table is the part reproduced here.
A task and a model. The result line says how long it took, how many tokens it used and what it cost.
bash
effgen run "What is the capital of France? Answer in one word." \
-m openai:gpt-5-nano
output
effGen v1.0.0 - Running Task
Initializing agent: cli-agent
Model: openai:gpt-5-nano
Tools: 1 available
Sub-agents: enabled
Task: What is the capital of France? Answer in one word.
Thinking...
Response
╭─────────────────────────────── Agent Response ───────────────────────────────╮
│ Paris │
╰──────────────────────────────────────────────────────────────────────────────╯
✓ Done in 3.1s · 294 tokens · $0.000041
04
Give it a tool
-t names the tools the run may use. The result line counts the tool steps, and --trace prints their timeline.
bash
effgen run "Use the calculator tool to work out 24344 * 334." \
-m gemini:gemini-3.1-flash-lite-t calculator
output
effGen v1.0.0 - Running Task
Loading tools: calculator
✓ Loaded tool: calculator
Initializing agent: cli-agent
Model: gemini:gemini-3.1-flash-lite
Tools: 1 available
Sub-agents: enabled
Task: Use the calculator tool to work out 24344 * 334.
Thinking...
Response
╭─────────────────────────────── Agent Response ───────────────────────────────╮
│ 8130896 │
╰──────────────────────────────────────────────────────────────────────────────╯
✓ Done in 13.7s · 1 tool · 186 tokens · $0.000075
1 tool step — run with --trace to see the timeline
The first stable release, shown rather than listed
Five of the changes in 1.0.0, each with the output it produces beside it. Every sample here was run before it was written down.
01
Point it at any OpenAI-compatible server
A base_url is the whole instruction. vLLM, SGLang, TGI, llama.cpp, Ollama, LM Studio, LiteLLM or a gateway — effGen drives the model you are already serving instead of loading a second copy of the weights.
The ids come from the server, so no catalog is consulted and no price is invented: a call through your own endpoint reports no cost rather than $0. Ask it what it serves with list_served_models(), and when nothing is listening you get BackendUnreachableError naming the endpoint it tried.
import os
from effgen importAgent,AgentConfig
agent =Agent(AgentConfig(
model="openai:gpt-5-nano",
base_url="http://127.0.0.1:8000/v1",
api_key=os.environ["EFFGEN_API_KEY"],))print(agent.run("Reply with the single word: ready").text)
output
ready
02
A run tells you which calls it made
AgentResponse.tool_calls is the list of calls the run actually made, not a count. Each carries its name and iteration, plus the arguments, result, duration and error the provider reported.
.failed narrows it to the calls that went wrong, .by_name() to one tool, and .total to the count — so code written against the old integer still reads. How much each call carries depends on the provider: some return the arguments and the result, others report only that the call happened.
from effgen importAgent,AgentConfigfrom effgen.tools.builtin importCalculator
agent =Agent(AgentConfig(
model="gemini:gemini-3.1-flash-lite",
tools=[Calculator()],))
r = agent.run("Use the calculator tool to work out 24344 * 334.")print(r.text)print(r.tool_calls.total,"tool call")for call in r.tool_calls:print(call.name, call.arguments,"->", call.result)
effgen code reads the repository, proposes a change, shows it as a unified diff, applies it, runs the result and fixes what fails. Four permission modes decide how much of that happens without you.
--plan proposes and writes nothing, the default asks before each change, --auto-edit applies writes but still asks before a shell command, and -y applies everything inside the workspace. --undo reverses the last edits, --review runs with no tool that can write, and --session-id continues where you left off.
$ effgen code -p "Create slugify.py …" -m gemini:gemini-3.1-flash-lite -ynew file slugify.py (+18/-0)--- a/slugify.py+++ b/slugify.py@@ -0,0 +1,18 @@+import re++def slugify(text):+ # Lowercase the text+ text = text.lower()+ # Replace runs of non-alphanumeric characters with a single hyphen+ text = re.sub(r'[^a-z0-9]+', '-', text)+ # Strip leading and trailing hyphens+ text = text.strip('-')+ return text++if __name__ == "__main__":+ assert slugify("Hello World") == "hello-world"+ assert slugify(" Hello World ") == "hello-world"+ assert slugify("Hello-World!") == "hello-world"+ assert slugify("---Hello---World---") == "hello-world"+ assert slugify("123 456") == "123-456"+ print("All tests passed!")✓ Done in 21.5s · 2 tools · 5,390 tokens · $0.0017Tool calling: hybrid — provider tool API first, falling back to the textFiles written in the workspace: slugify.pyslugify.py | All tests passed!
04
Workflows resume where they stopped
Hand a WorkflowDAG a checkpoint store and a run id. A pipeline that died half way through does not start again from the top — running the same line picks up at the node that had not finished.
A completed node is restored and its output flows downstream without a model call; a failed node is retried, which is usually why you are resuming; a node that never started runs normally. There is no separate resume call — a run id the store has not seen starts from the beginning, one it knows continues. Progress is written after each level of the graph, atomically.
from effgen importAgent,AgentConfig,FileCheckpointStore,WorkflowDAG,WorkflowNode
store =FileCheckpointStore("./checkpoints")
dag =WorkflowDAG("briefing")
dag.add_node(WorkflowNode(id="research", agent=Agent(AgentConfig(
model="gemini:gemini-3.1-flash-lite",
system_prompt="Answer in three short bullet points. Nothing else.",))))
dag.add_node(WorkflowNode(id="draft", agent=Agent(AgentConfig(
model="Qwen/Qwen2.5-7B-Instruct",
base_url="http://127.0.0.1:9/v1",# a server that is not up
require_model=False,))))
dag.connect("research","draft")
result = dag.run("Why run an agent on a small model?",
checkpoint=store, run_id="briefing-1")for node in result.node_results:print(f"{node['id']:9} {node['status']:9} {node['execution_time']:6.2f}s")
output
# first run
research completed 3.08s
draft failed 13.68s
# the same line again, in a new process
research completed 0.00s
draft failed 13.46s
05
Surfaces to watch it through
effgen serve brings up the OpenAI-compatible API and, on the same port, a dashboard, a playground, a model browser and a topology graph. All of it is served from the package — no CDN, nothing fetched at view time.
In the terminal, effgen top reads that server and shows runs, traffic, spend and GPUs live; effgen battle races several models on one prompt; compare, eval, cost and loadtest each write a shareable HTML report, and any saved result renders again with effgen report.
$ effgen serve --port 8000effGen v1.0.0 - API Server✓ Auth: static API key (EFFGEN_API_KEY)Starting server on 127.0.0.1:8000 OpenAI-compatible API : http://127.0.0.1:8000/v1 Interactive docs : http://127.0.0.1:8000/docs Dashboard : http://127.0.0.1:8000/dashboard (data requires an API key; set EFFGEN_PUBLIC_DASHBOARD=1 for local viewing) Playground : http://127.0.0.1:8000/playground (paste an API key, or set EFFGEN_PUBLIC_PLAYGROUND=1 for local viewing) Both pages: Cmd/Ctrl-K opens the command palette, ? lists shortcuts.
Three things change when you upgrade
Everything else in 1.0.0 is additive — nothing was removed or renamed.
Breaking 01
Python 3.10 is no longer supported
The floor is 3.11, and the package is tested through 3.14.
Breaking 02
raise_on_error defaults to True
A failed run raises instead of returning a result with success=False. Pass raise_on_error=False for the old behaviour.
Breaking 03
An unreachable backend always raises
BackendUnreachableError is raised whatever raise_on_error is set to, because there is no result to return.
A model, 66 tools it can be given, and a loop that keeps going until there is an answer — or until it runs out of turns and says so.
Steps 02–05 repeat until the model answers or max_iterations is reached (the general preset stops at 10)
Step 03 — how a call is recognised
Not every model has a function-calling API, and the ones that do do not all get it right. tool_calling_mode decides how a call is read out of the model’s reply — the default, "auto", picks per model.
native
The model's own function-calling API
Tools become JSON-schema function definitions and the model returns a structured call. Nothing is parsed out of prose, so nothing can be mis-parsed.
react
Thought, Action, Action Input, in text
The call is written in the model's output and read back out with a parser. It works on any model, including a small local one with no tool API at all.
hybrid
Structured first, text as the fallback
Try the provider's tool API; when the call does not come back parseable, fall back to reading it out of the text. This is what effgen code runs on.
When a step fails
A tool that raised, a server that never answered and a run that stopped early are three different things, and effGen reports them as three different things. The output below is what each one prints.
The tool fails
The call comes back with success=False and a message. The run does not stop: the error becomes the next observation, and the call is kept on the response so you can see it afterwards.
python
r = agent.run("Work out 1/0 with the calculator, then tell me what happened.")print("success:", r.success)for call in r.tool_calls:print(call.name,"error:", call.error)print("failed calls:",len(r.tool_calls.failed))
output
success: True
calculator error: Error executing tool 'calculator': Tool execution failed: Calculation failed: division by zero. Check the expression for balanced brackets, a supported function and no stray characters.
failed calls: 1
Nothing is listening
A refused connection, a host that does not resolve and a route that does not exist are reported as unreachable — separately from a server that answered badly — and the error names the endpoint it tried. This one raises whatever raise_on_error is set to, because there is no result to return.
BackendUnreachableError -> openai did not answer (model='Qwen/Qwen2.5-7B-Instruct'): OpenAI generation failed [will_retry]: Connection error.. Nothing answered at that endpoint — check the server is running and the base_url, host and port are right. The call was sent to http://127.0.0.1:9/v1.
The run says how it ended
Every result records the stage it finished at, so a run that hit the iteration cap, or ended on a tool result rather than on something the model wrote, is distinguishable from one that answered. Anything produced before it stopped is on the response as partial output.
python
agent =Agent(AgentConfig(
model="gemini:gemini-3.1-flash-lite",
tools=[Calculator()],
max_iterations=1,))
r = agent.run("With the calculator: work out 24344 * 334, then multiply that by 7, ""then subtract 19, then divide by 3.")print("success:", r.success)print("calls:", r.tool_calls.total)print("partial_output:",repr(r.metadata.get("partial_output")))print("reason:", r.metadata.get("reason"))
Every one is registered under the name below and called the same way — awaited, with keyword arguments, returning a ToolResult. Open one for its parameters and the line that runs it, search for the one you need, or show the whole set.
The 2 each category leans on most — 16 of 66
What comes back
A ToolResult, whatever happened. Its fields are success, output, error, execution_time, metadata and timestamp. It is not a dictionary and it has no data field, so read result.output after checking result.success. A tool that could not do its job says why in result.error — a missing key names the environment variable, a missing file names the path.
Agent presets
9 presets, one line each
A preset is a tool set, a system prompt, a temperature and an iteration cap under one name. Each card carries what that preset costs you on every request: the tools it wires in, and the tokens their schemas take up.
python
from effgen.presets import create_agent
agent =create_agent("math","gemini:gemini-3.1-flash-lite")
response = agent.run("What is 24344 * 334?")print(response.text)print("tool calls:", response.tool_calls.total)
output
8130896
tool calls: 1
The answer came back from the calculator the math preset wired in, not from the model doing arithmetic in its head — which is what the call count is there to tell you.
Models and providers
Any model, anywhere it runs
10 provider adapters are registered. 9 of them ship a bundled catalog of 417 models with context windows, prices and capabilities. The tenth is the one that matters if you already serve a model yourself.
10
Provider adapters
9
Ship a bundled catalog
417
Catalogued models
4
Local engines
The 9 provider adapters that ship a bundled catalog, with the models each carries.
Provider
Models
Default
Tool calling
Vision
Largest window
Catalog checked
anthropic
17
—
17
17
1M
2026-06-08
cerebras
2
gpt-oss-120b
1
—
66k
2026-06-08
fireworks
16
accounts/fireworks/models/gpt-oss-120b
14
5
1.0M
2026-08-07
gemini
8
gemini-3.1-flash-lite
6
6
2M
2026-08-13
groq
15
llama-3.1-8b-instant
8
1
131k
2026-08-07
hf
124
Qwen/Qwen2.5-7B-Instruct
83
31
1.0M
2026-06-08
openai
30
gpt-5.4-nano
30
23
1.0M
2026-06-17
replicate
37
meta/meta-llama-3-8b-instruct
24
—
1M
2026-08-07
together
168
Qwen/Qwen3.5-9B
66
2
10.5M
2026-08-07
Model counts and capabilities are the bundled catalog’s; effgen models refresh updates it against each provider’s live API, and the last column is when that was last done. 249 of 417 catalogued models call tools, 85 take images and 6 take audio. A model the catalog has never seen is still callable — it simply reports no price rather than a made-up one.
pip install effgen, then one line to build an agent. Run it against a model on your own hardware, a server you already have, or any of the providers effGen ships an adapter for.