Run your agent | StateWeave Docs
Tasks and runs
Run your agent
Every run gives your agent fresh copies of a sandbox's apps, made from its latest version. Your agent runs where it always runs and reaches them over HTTPS; StateWeave records every call.
| Run | For | Judged |
|---|---|---|
| Twin run | Developing: run your agent, look, reset, run it again. | By you, for example with a task's checks |
| Test run | A test you uploaded: every run is graded and kept. | By the test, when your harness finishes the run |
Twin runs
A twin run is fresh copies of a sandbox's apps with no test. Open one, run your agent on it, judge it, and reset it to try again: each reset puts the apps back as the version holds them. A twin run is never graded and injects no faults.
From a shell, sw run with a sandbox's id opens one around your command, judges it with a task's checks when you give them, and stops it when the command exits:
Shell
sw
run p-7d21c4
--agent
my-agent@dev
--prompt
"$(cat kickoff/instruction.md)"
--checks
kickoff -- python my_agent.py
Your command gets the same variables as for a test ( below ), with the request you gave. See sw run for its options and output. In Python, the same with resets in between:
Python
import
json
import
stateweave client = stateweave.
Client
() checks = json.
load
(
open
(
"kickoff/tests/checks.json"
))
with
client.
open_twins
(
"p-7d21c4"
)
as
twins:
for
attempt
in
range
(
3
):
my_agent
(
open
(
"kickoff/instruction.md"
).
read
(), twins.apps) result = twins.
check
(checks)
(attempt +
1
,
"passed"
if
result[
"passed"
]
else
"failed"
)
if
result[
"passed"
]:
break
twins.
reset
()
# the next attempt starts from the version again
twins.apps holds, per app, its base_url , endpoints , credentials and MCP server, as in the run file . twins.env holds the same as environment variables, ready to pass to a subprocess: STATEWEAVE_<APP>_URL , STATEWEAVE_<APP>_MCP_URL and STATEWEAVE_<APP>_<KIND>_TOKEN .
| Method | What it does |
|---|---|
| check(checks, answer=None) | Judges the current attempt by a task's checks, as a graded run would: {"passed", "results": [{"index", "name", "ok", "detail", "actual"}]} . See Checks . |
| call(app, method, path, query=, body=, headers=, credential=) | One request to an app, as a checker: it is not counted as your agent's. credential is a label or a kind such as "user" ; Shopify takes its token in headers . |
| reset(apps=None) | Puts the apps back as they opened. The calls after it count as the next attempt. |
| trace(attempt=None) | Every call your agent made, whole, and the faults that acted. |
| download(directory) | Writes the whole record to a folder: trace.jsonl , changes.json , task.json , faults.json . |
| stop() | Stops the apps. Leaving the with block does it for you. |
A twin run stays up for an hour unless you stop it ( ttl= , in seconds, up to a day), and ends with status ended , keeping its final state. Each API key can keep three going at once; a fourth answers HTTP 429. persona= picks whose credentials you get, by handle. Over HTTP, start one with POST /api/v1/runs and {"sandbox": "p-7d21c4"} ; see Twin runs .
Test runs
A test run is a run of a test in your workspace. It hands your agent three things: the request, the apps' addresses and credentials, and a place to put its answer. When your harness finishes it, StateWeave grades it by the test's checks and keeps it with every call.
Start. Your harness starts a run of a test. StateWeave copies the test's apps from the sandbox's latest version, starts them, and answers with the request and each app's connection details. This takes a second or two.
Work. Your agent calls the apps. Every call is recorded on the run.
Finish. Your harness hands in the answer (nothing, for an action test). StateWeave grades the run, stops the apps and returns the result.
A run that is never finished ends as timed_out when its time limit passes, and one your harness gives up ends as cancelled . Neither is graded.
With sw run
sw run does the start and finish for you around any command. The command can be written in any language; it only needs to read environment variables and, for an answer test, write a file.
Shell
sw
run t-bde537
--agent
my-agent@0.1 -- python my_agent.py
Everything after -- is your command, run as is (no shell, unless you call one). It gets these variables on top of your own environment:
| Variable | What it holds |
|---|---|
| STATEWEAVE_PROMPT | The request, as a person in the company would put it. Give it to your agent as the task. |
| STATEWEAVE_KIND | answer (find something out) or action (change the apps). |
| STATEWEAVE_ANSWER_FORMAT | For an answer test, an example of the JSON value to answer with. Empty for an action test. |
| STATEWEAVE_RUN_FILE | Path to the whole run as JSON, including each app's base_url , endpoints , credentials and MCP server. See below . |
| STATEWEAVE_MCP_CONFIG | Path to {"mcpServers": {…}} for the apps that have an MCP server, in the format Claude Code and most MCP clients take. |
| STATEWEAVE_<APP>_URL , STATEWEAVE_<APP>_MCP_URL , STATEWEAVE_<APP>_<KIND>_TOKEN | Each app's address, its MCP server and the persona's tokens, such as STATEWEAVE_SLACK_BOT_TOKEN : map them onto the names your code already reads. See Apps and credentials . |
| STATEWEAVE_ANSWER_FILE | Path your agent writes its answer to: JSON, or plain text (handed in as a string). Leave it alone for an action test. |
| STATEWEAVE_TRANSCRIPT_FILE | Path your agent may write its own transcript to; it is handed in with the answer. See below . |
| STATEWEAVE_RUN | The run's id, such as r-de238618 . |
The run is graded when your command exits, whatever its exit code: an agent that did the work and then crashed is still graded on what it did. Press Ctrl + C to give the run up instead. sw run itself exits with 0 when the run passed and 1 otherwise. A command that doesn't exist is refused before any run starts.
A minimal agent
The skeleton below reads what a run hands over and answers. Replace the middle with your agent.
my_agent.py
import
json
import
os run = json.
load
(
open
(os.environ[
"STATEWEAVE_RUN_FILE"
])) prompt = os.environ[
"STATEWEAVE_PROMPT"
]
for
name, app
in
run[
"apps"
].
items
():
(name, app[
"base_url"
],
list
(app[
"credentials"
])) answer = ...
# your agent works here: call the apps, decide, act
if
os.environ[
"STATEWEAVE_KIND"
] ==
"answer"
:
with
open
(os.environ[
"STATEWEAVE_ANSWER_FILE"
],
"w"
)
as
f: json.
dump
(answer, f)
The run file
The run file holds the run as the API returns it. The parts your agent needs:
run.json (shortened)
{
"id"
:
"r-de238618"
,
"test"
:
"t-bde537"
,
"kind"
:
"answer"
,
"prompt"
:
"Ethan Smith looks like he's been buying from us under two different email addresses. Can you work out how much he has actually paid us in total across both of his accounts? …"
,
"answer_format"
:
"{\"total_paid_usd\": 123.45, \"orders\": [\"NG-1234\", \"NG-1235\"]}"
,
"persona"
: {
"name"
:
"Priya Castellano"
,
"handle"
:
"priya.castellano"
,
"title"
:
"Customer Operations Coordinator"
},
"expires_at"
:
1793670877.43
,
"apps"
: {
"shopify"
: {
"base_url"
:
"https://t-q8m2xk.app.stateweave.net"
,
"endpoints"
: {
"admin_graphql"
:
"https://t-q8m2xk.app.stateweave.net/admin/api/2026-10/graphql.json"
,
"token_endpoint"
:
"https://t-q8m2xk.app.stateweave.net/admin/oauth/access\_token"
,
"mcp"
:
"https://t-q8m2xk.app.stateweave.net/mcp"
},
"credentials"
: {
"app:Northstar Ops Coordinator"
:
"shpat_…"
,
"setup:priya.castellano"
:
"setup_…"
},
"mcp"
: {
"url"
:
"https://t-q8m2xk.app.stateweave.net/mcp"
,
"headers"
: {
"Authorization"
:
"Bearer setup_…"
} } } } }
| Field | Description |
|---|---|
| prompt | The request. Pass it to your agent word for word. |
| kind | answer or action . |
| answer_format | For an answer test, an example value in the expected shape. Its values are placeholders, never the answer. |
| persona | Whom the agent acts for: name , handle and job title . The user: credentials are this person's. |
| apps.<app>.base_url | The app's own address for this run. Only this run uses it. |
| apps.<app>.endpoints | The app's API roots and other addresses by name. See Apps and credentials for each app's. |
| apps.<app>.credentials | Tokens by label, kind:name . See Credentials . |
| apps.<app>.mcp | {"url", "headers"} of the app's MCP server, or null when it has none. |
| expires_at | When the run times out, in Unix seconds. |
Credentials
Credential labels are kind:name . The kind says whose token it is and what it opens:
| Kind | What it is | Apps |
|---|---|---|
| user:<handle> | The persona's own OAuth or user token. | Slack ( xoxp- ), Gmail, Google Calendar, Sheets and Drive ( ya29. ), Salesforce, Zendesk, WhatsApp ( EAA… ), Microsoft Teams (a delegated Graph token), Stripe (the secret key, sk_test_… ), HubSpot (in a generated company, an OAuth token), GitHub (a classic personal access token, ghp_… ) |
| bot:<name> | The company's bot token. | Slack ( xoxb- ), WhatsApp (a system user's token), Microsoft Teams (the bot's Bot Connector token) |
| app:<name> | An app's own credential. | Shopify Admin API token ( shpat_ ), Slack app-level token ( xapp- ), Salesforce connected app's client id, WhatsApp app access token, Microsoft Teams' public client id and its bot's <app id>:<client secret> , HubSpot's Service Key ( pat-… ) |
| mcp:<handle> | The persona's token for the app's MCP server. | Salesforce, Microsoft Teams, Stripe (a restricted key, rk_test_… ), HubSpot (an MCP connector token) |
| setup:<handle> | The persona's token for the app's MCP server. | Shopify |
A run never hands over a colleague's mailbox or a refresh token, and no client secret except inside WhatsApp's app access token ( <app id>|<app secret> , as Meta writes it) and Microsoft Teams' bot credential ( <app id>:<client secret> , which a bot built on the Teams SDK signs in with): your agent gets what a deployed agent acting for the persona would get. With the Python SDK, run.credential("slack", "user") returns the first credential of a kind.
Answers
For an answer test, answer with a JSON value in the shape answer_format shows: an object as an object, numbers as numbers, the same field names. Plain text is accepted too and handed in as a string, so a test that asks for an object fails it.
For an action test there is nothing to hand in. The result is what your agent changed in the apps, and what it left alone: a test can fail because the agent changed something it shouldn't have.
From Python
Inside a Python harness, use the SDK instead of sw run . The with block gives the run up if your agent raises or the block ends without finish() , so no run is left waiting for its time limit.
Python
import
stateweave client = stateweave.
Client
()
# STATEWEAVE_URL and STATEWEAVE_API_KEY, or what sw login kept
with
client.
start
(
"t-bde537"
, agent=
, meta={
"model"
:
"my-model"
,
"prompt"
:
"v7"
})
as
run: token = run.
credential
(
"shopify"
,
"app"
) graphql = run.apps[
"shopify"
][
"endpoints"
][
"admin_graphql"
] answer =
my_agent
(run.prompt, graphql, token) result = run.
finish
(answer)
(result.status, result.reasons, result.url)
See the Python SDK reference .
From any language, over HTTP
The same flow is three calls with your API key. See the REST API reference .
# start: the response is the run file above
curl
-s
https://acme.app.stateweave.net/api/v1/runs \
-H
"Authorization: Bearer $STATEWEAVE_API_KEY"
-H
"Content-Type: application/json"
\
-d
'{"test": "t-bde537", "agent": "my-agent@0.1"}'
# … your agent works …
# finish: hand in the answer and get the result
curl
-s
https://acme.app.stateweave.net/api/v1/runs/r-de238618/finish \
-H
"Authorization: Bearer $STATEWEAVE_API_KEY"
-H
"Content-Type: application/json"
\
-d
'{"answer": {"total_paid_usd": 349.10, "orders": ["NG-4824", "NG-4872", "NG-4881"]}}'
Time limits
A run has 30 minutes by default. Set another limit, from 60 seconds to 24 hours, with --timeout (in seconds) or timeout= in Python. A run still going at its limit ends as timed_out and its apps stop.
Labels and metadata
--agent ( agent= in Python) is the label your runs are compared by on the Runs board. Use one per version of your agent, such as support-bot@0.4 , up to 80 characters. Add meta , a JSON object of up to 4,000 characters, to record anything else with the run: the model, the prompt version, a git commit. sw run records your command line in it.
Many tests in a row
Run a sandbox's tests one after another from a shell, or from Python:
for
t in $(sw task ls
--env
p-041ba8 | cut
-d
' '
-f1
);
do
sw run
"$t"
--agent
my-agent@0.2 -- python my_agent.py
done
import
stateweave client = stateweave.
Client
()
for
test
in
client.
tests
(environment=
"p-041ba8"
):
with
client.
start
(test[
"id"
], agent=
)
as
run: result = run.
finish
(
my_agent
(run.prompt, run.apps))
(test[
"id"
], result.status)
Each run gets its own apps, so you can also run tests in parallel from separate processes, up to your workspace's limit on runs going at once. Past it, starting a run answers HTTP 429 until one finishes.
The result
Finishing a run returns its result. sw run prints it; in Python it is a Result :
| Field | Description |
|---|---|
| status | passed , failed , timed_out , cancelled , interrupted or error . See Run statuses . |
| passed | true or false once graded, null otherwise. |
| reasons | Why, in a few short lines: what was expected and what was found. |
| calls | How many calls your agent made to the apps. |
| url | The run's page in your workspace. |
Output
failed 9 calls https://acme.app.stateweave.net/\#/runs/r-aee3438d total 1.00 != expected 349.10 orders ['NG-1'] != expected ['NG-4824', 'NG-4872', 'NG-4881'] (missing ['NG-4824', 'NG-4872', 'NG-4881'], extra ['NG-1'])
The run's record from the API also has state , per app how many records were added, changed and removed, and the agent label, meta and timing.
The run's page
Verdict. Passed or failed, and the reasons.
Calls. Every call your agent made, in order, across all the apps: method, path, status, and the full request and response.
Faults. Each fault the test injected that acted, on the call it acted on.
Final state. How many records each app gained, changed and lost from the start, and a button that opens the end state in the app's web client to look around as it was left.
Verdicts of your own. When your own judge kept a verdict on the run ( run.verdict() in Python, or POST /api/v1/runs/<id>/verdict ), its verdict, criteria, reasons and evidence, beside the test's own result, which they don't change.
Transcript. When the harness handed in the agent's transcript, what the agent said and did, folded, one entry per line for JSON lines.
Hand in your agent's transcript
The calls show what your agent did to the apps; its transcript shows why. Hand it in with the answer and it is kept with the run, in whatever format your agent writes:
sw
run t-0e4243
--agent
claude-code -- sh
-c
'claude -p "$STATEWEAVE_PROMPT" \ --mcp-config "$STATEWEAVE_MCP_CONFIG" --strict-mcp-config \ --output-format stream-json --verbose > "$STATEWEAVE_TRANSCRIPT_FILE"'
result = run.
finish
(answer, transcript=messages, transcript_format=
"claude-agent-sdk"
)
sw run sends whatever the command wrote to STATEWEAVE_TRANSCRIPT_FILE . See Hand in a transcript for the limits.
Compare agent versions
The Runs board shows every test against every agent label you've run, so a new version's wins and regressions are one look away. To compare fairly:
Give each version its own --agent label, such as support-bot@0.4 .
Put what changed in meta : the model, the prompt version, a commit.
Run each test more than once. Agents vary from run to run, and so do tests with random faults.
Run again from the browser
When a run fails, you often want to fix the agent and try the same test again right away. Leave a worker running on your machine:
Shell
sw
worker
--agent
my-agent@0.3 -- python my_agent.py
Then press Run again on any run's page. The run is queued, your worker takes it, runs your command with it exactly as sw run does, and hands in the answer. Your command starts afresh for every run, so an edit to your agent counts from the next run on, without restarting the worker.
The run takes the worker's --agent label. Its meta records the command, the machine's name and, in a git checkout, the commit (with a note when there are uncommitted changes).
With several workers connected, the button lets you pick one. A worker counts as connected for a minute after it last asked for work.
A queued run that no worker takes within 10 minutes is cancelled.
--once drives one run and exits, with sw run 's exit code. Ctrl-C stops the worker.
Run in CI
sw run exits non-zero when a run doesn't pass, so a test is a CI step. Store the key as a secret and pass the address and key in the environment:
Shell
export
STATEWEAVE_URL=https://acme.app.stateweave.net
export
STATEWEAVE_API_KEY=
$STATEWEAVE_KEY_SECRET
# from your CI's secret store
pip
install stateweave-sdk status=0
for
t in t-bde537 t-0e4243 t-753b29;
do
sw run
"$t"
--agent
"my-agent@$GIT_SHA"
-- python my_agent.py || status=1
done
exit
$status
Use a key made for CI, so you can revoke it on its own.
Give up a run
A run your harness can't finish should be given up rather than left to time out, so its apps stop at once:
sw
cancel r-de238618
run.
cancel
()
# the with block does this for you when your agent raises
You can also cancel a running or queued run on its page.