Run your agent | StateWeave Docs

Tasks and runs

Run your agent

Every run gives your agent fresh copies of a sandbox's apps, made from its latest version. Your agent runs where it always runs and reaches them over HTTPS; StateWeave records every call.

Run For Judged
Twin run Developing: run your agent, look, reset, run it again. By you, for example with a task's checks
Test run A test you uploaded: every run is graded and kept. By the test, when your harness finishes the run

Twin runs

A twin run is fresh copies of a sandbox's apps with no test. Open one, run your agent on it, judge it, and reset it to try again: each reset puts the apps back as the version holds them. A twin run is never graded and injects no faults.

From a shell, sw run with a sandbox's id opens one around your command, judges it with a task's checks when you give them, and stops it when the command exits:

Shell

sw

run p-7d21c4

--agent

my-agent@dev

--prompt

"$(cat kickoff/instruction.md)"

--checks

kickoff -- python my_agent.py

Your command gets the same variables as for a test ( below ), with the request you gave. See sw run for its options and output. In Python, the same with resets in between:

Python

import

json

import

stateweave client = stateweave.

Client

() checks = json.

load

(

open

(

"kickoff/tests/checks.json"

))

with

client.

open_twins

(

"p-7d21c4"

)

as

twins:

for

attempt

in

range

(

3

):

my_agent

(

open

(

"kickoff/instruction.md"

).

read

(), twins.apps) result = twins.

check

(checks)

print

(attempt +

1

,

"passed"

if

result[

"passed"

]

else

"failed"

)

if

result[

"passed"

]:

break

twins.

reset

()

# the next attempt starts from the version again

twins.apps holds, per app, its base_url , endpoints , credentials and MCP server, as in the run file . twins.env holds the same as environment variables, ready to pass to a subprocess: STATEWEAVE_<APP>_URL , STATEWEAVE_<APP>_MCP_URL and STATEWEAVE_<APP>_<KIND>_TOKEN .

Method What it does
check(checks, answer=None) Judges the current attempt by a task's checks, as a graded run would: {"passed", "results": [{"index", "name", "ok", "detail", "actual"}]} . See Checks .
call(app, method, path, query=, body=, headers=, credential=) One request to an app, as a checker: it is not counted as your agent's. credential is a label or a kind such as "user" ; Shopify takes its token in headers .
reset(apps=None) Puts the apps back as they opened. The calls after it count as the next attempt.
trace(attempt=None) Every call your agent made, whole, and the faults that acted.
download(directory) Writes the whole record to a folder: trace.jsonl , changes.json , task.json , faults.json .
stop() Stops the apps. Leaving the with block does it for you.

A twin run stays up for an hour unless you stop it ( ttl= , in seconds, up to a day), and ends with status ended , keeping its final state. Each API key can keep three going at once; a fourth answers HTTP 429. persona= picks whose credentials you get, by handle. Over HTTP, start one with POST /api/v1/runs and {"sandbox": "p-7d21c4"} ; see Twin runs .

Test runs

A test run is a run of a test in your workspace. It hands your agent three things: the request, the apps' addresses and credentials, and a place to put its answer. When your harness finishes it, StateWeave grades it by the test's checks and keeps it with every call.

Start. Your harness starts a run of a test. StateWeave copies the test's apps from the sandbox's latest version, starts them, and answers with the request and each app's connection details. This takes a second or two.

Work. Your agent calls the apps. Every call is recorded on the run.

Finish. Your harness hands in the answer (nothing, for an action test). StateWeave grades the run, stops the apps and returns the result.

A run that is never finished ends as timed_out when its time limit passes, and one your harness gives up ends as cancelled . Neither is graded.

With sw run

sw run does the start and finish for you around any command. The command can be written in any language; it only needs to read environment variables and, for an answer test, write a file.

Shell

sw

run t-bde537

--agent

my-agent@0.1 -- python my_agent.py

Everything after -- is your command, run as is (no shell, unless you call one). It gets these variables on top of your own environment:

Variable What it holds
STATEWEAVE_PROMPT The request, as a person in the company would put it. Give it to your agent as the task.
STATEWEAVE_KIND answer (find something out) or action (change the apps).
STATEWEAVE_ANSWER_FORMAT For an answer test, an example of the JSON value to answer with. Empty for an action test.
STATEWEAVE_RUN_FILE Path to the whole run as JSON, including each app's base_url , endpoints , credentials and MCP server. See below .
STATEWEAVE_MCP_CONFIG Path to {"mcpServers": {…}} for the apps that have an MCP server, in the format Claude Code and most MCP clients take.
STATEWEAVE_<APP>_URL , STATEWEAVE_<APP>_MCP_URL , STATEWEAVE_<APP>_<KIND>_TOKEN Each app's address, its MCP server and the persona's tokens, such as STATEWEAVE_SLACK_BOT_TOKEN : map them onto the names your code already reads. See Apps and credentials .
STATEWEAVE_ANSWER_FILE Path your agent writes its answer to: JSON, or plain text (handed in as a string). Leave it alone for an action test.
STATEWEAVE_TRANSCRIPT_FILE Path your agent may write its own transcript to; it is handed in with the answer. See below .
STATEWEAVE_RUN The run's id, such as r-de238618 .

The run is graded when your command exits, whatever its exit code: an agent that did the work and then crashed is still graded on what it did. Press Ctrl + C to give the run up instead. sw run itself exits with 0 when the run passed and 1 otherwise. A command that doesn't exist is refused before any run starts.

A minimal agent

The skeleton below reads what a run hands over and answers. Replace the middle with your agent.

my_agent.py

import

json

import

os run = json.

load

(

open

(os.environ[

"STATEWEAVE_RUN_FILE"

])) prompt = os.environ[

"STATEWEAVE_PROMPT"

]

for

name, app

in

run[

"apps"

].

items

():

print

(name, app[

"base_url"

],

list

(app[

"credentials"

])) answer = ...

# your agent works here: call the apps, decide, act

if

os.environ[

"STATEWEAVE_KIND"

] ==

"answer"

:

with

open

(os.environ[

"STATEWEAVE_ANSWER_FILE"

],

"w"

)

as

f: json.

dump

(answer, f)

The run file

The run file holds the run as the API returns it. The parts your agent needs:

run.json (shortened)

{

"id"

:

"r-de238618"

,

"test"

:

"t-bde537"

,

"kind"

:

"answer"

,

"prompt"

:

"Ethan Smith looks like he's been buying from us under two different email addresses. Can you work out how much he has actually paid us in total across both of his accounts? …"

,

"answer_format"

:

"{\"total_paid_usd\": 123.45, \"orders\": [\"NG-1234\", \"NG-1235\"]}"

,

"persona"

: {

"name"

:

"Priya Castellano"

,

"handle"

:

"priya.castellano"

,

"title"

:

"Customer Operations Coordinator"

},

"expires_at"

:

1793670877.43

,

"apps"

: {

"shopify"

: {

"base_url"

:

"https://t-q8m2xk.app.stateweave.net"

,

"endpoints"

: {

"admin_graphql"

:

"https://t-q8m2xk.app.stateweave.net/admin/api/2026-10/graphql.json"

,

"token_endpoint"

:

"https://t-q8m2xk.app.stateweave.net/admin/oauth/access\_token"

,

"mcp"

:

"https://t-q8m2xk.app.stateweave.net/mcp"

},

"credentials"

: {

"app:Northstar Ops Coordinator"

:

"shpat_…"

,

"setup:priya.castellano"

:

"setup_…"

},

"mcp"

: {

"url"

:

"https://t-q8m2xk.app.stateweave.net/mcp"

,

"headers"

: {

"Authorization"

:

"Bearer setup_…"

} } } } }

Field Description
prompt The request. Pass it to your agent word for word.
kind answer or action .
answer_format For an answer test, an example value in the expected shape. Its values are placeholders, never the answer.
persona Whom the agent acts for: name , handle and job title . The user: credentials are this person's.
apps.<app>.base_url The app's own address for this run. Only this run uses it.
apps.<app>.endpoints The app's API roots and other addresses by name. See Apps and credentials for each app's.
apps.<app>.credentials Tokens by label, kind:name . See Credentials .
apps.<app>.mcp {"url", "headers"} of the app's MCP server, or null when it has none.
expires_at When the run times out, in Unix seconds.

Credentials

Credential labels are kind:name . The kind says whose token it is and what it opens:

Kind What it is Apps
user:<handle> The persona's own OAuth or user token. Slack ( xoxp- ), Gmail, Google Calendar, Sheets and Drive ( ya29. ), Salesforce, Zendesk, WhatsApp ( EAA… ), Microsoft Teams (a delegated Graph token), Stripe (the secret key, sk_test_… ), HubSpot (in a generated company, an OAuth token), GitHub (a classic personal access token, ghp_… )
bot:<name> The company's bot token. Slack ( xoxb- ), WhatsApp (a system user's token), Microsoft Teams (the bot's Bot Connector token)
app:<name> An app's own credential. Shopify Admin API token ( shpat_ ), Slack app-level token ( xapp- ), Salesforce connected app's client id, WhatsApp app access token, Microsoft Teams' public client id and its bot's <app id>:<client secret> , HubSpot's Service Key ( pat-… )
mcp:<handle> The persona's token for the app's MCP server. Salesforce, Microsoft Teams, Stripe (a restricted key, rk_test_… ), HubSpot (an MCP connector token)
setup:<handle> The persona's token for the app's MCP server. Shopify

A run never hands over a colleague's mailbox or a refresh token, and no client secret except inside WhatsApp's app access token ( <app id>|<app secret> , as Meta writes it) and Microsoft Teams' bot credential ( <app id>:<client secret> , which a bot built on the Teams SDK signs in with): your agent gets what a deployed agent acting for the persona would get. With the Python SDK, run.credential("slack", "user") returns the first credential of a kind.

Answers

For an answer test, answer with a JSON value in the shape answer_format shows: an object as an object, numbers as numbers, the same field names. Plain text is accepted too and handed in as a string, so a test that asks for an object fails it.

For an action test there is nothing to hand in. The result is what your agent changed in the apps, and what it left alone: a test can fail because the agent changed something it shouldn't have.

From Python

Inside a Python harness, use the SDK instead of sw run . The with block gives the run up if your agent raises or the block ends without finish() , so no run is left waiting for its time limit.

Python

import

stateweave client = stateweave.

Client

()

# STATEWEAVE_URL and STATEWEAVE_API_KEY, or what sw login kept

with

client.

start

(

"t-bde537"

, agent=

"my-agent@0.1"

, meta={

"model"

:

"my-model"

,

"prompt"

:

"v7"

})

as

run: token = run.

credential

(

"shopify"

,

"app"

) graphql = run.apps[

"shopify"

][

"endpoints"

][

"admin_graphql"

] answer =

my_agent

(run.prompt, graphql, token) result = run.

finish

(answer)

print

(result.status, result.reasons, result.url)

See the Python SDK reference .

From any language, over HTTP

The same flow is three calls with your API key. See the REST API reference .

# start: the response is the run file above

curl

-s

https://acme.app.stateweave.net/api/v1/runs \

-H

"Authorization: Bearer $STATEWEAVE_API_KEY"

-H

"Content-Type: application/json"

\

-d

'{"test": "t-bde537", "agent": "my-agent@0.1"}'

# … your agent works …

# finish: hand in the answer and get the result

curl

-s

https://acme.app.stateweave.net/api/v1/runs/r-de238618/finish \

-H

"Authorization: Bearer $STATEWEAVE_API_KEY"

-H

"Content-Type: application/json"

\

-d

'{"answer": {"total_paid_usd": 349.10, "orders": ["NG-4824", "NG-4872", "NG-4881"]}}'

Time limits

A run has 30 minutes by default. Set another limit, from 60 seconds to 24 hours, with --timeout (in seconds) or timeout= in Python. A run still going at its limit ends as timed_out and its apps stop.

Labels and metadata

--agent ( agent= in Python) is the label your runs are compared by on the Runs board. Use one per version of your agent, such as support-bot@0.4 , up to 80 characters. Add meta , a JSON object of up to 4,000 characters, to record anything else with the run: the model, the prompt version, a git commit. sw run records your command line in it.

Many tests in a row

Run a sandbox's tests one after another from a shell, or from Python:

for

t in $(sw task ls

--env

p-041ba8 | cut

-d

' '

-f1

);

do

sw run

"$t"

--agent

my-agent@0.2 -- python my_agent.py

done

import

stateweave client = stateweave.

Client

()

for

test

in

client.

tests

(environment=

"p-041ba8"

):

with

client.

start

(test[

"id"

], agent=

"my-agent@0.2"

)

as

run: result = run.

finish

(

my_agent

(run.prompt, run.apps))

print

(test[

"id"

], result.status)

Each run gets its own apps, so you can also run tests in parallel from separate processes, up to your workspace's limit on runs going at once. Past it, starting a run answers HTTP 429 until one finishes.

The result

Finishing a run returns its result. sw run prints it; in Python it is a Result :

Field Description
status passed , failed , timed_out , cancelled , interrupted or error . See Run statuses .
passed true or false once graded, null otherwise.
reasons Why, in a few short lines: what was expected and what was found.
calls How many calls your agent made to the apps.
url The run's page in your workspace.

Output

failed 9 calls https://acme.app.stateweave.net/\#/runs/r-aee3438d total 1.00 != expected 349.10 orders ['NG-1'] != expected ['NG-4824', 'NG-4872', 'NG-4881'] (missing ['NG-4824', 'NG-4872', 'NG-4881'], extra ['NG-1'])

The run's record from the API also has state , per app how many records were added, changed and removed, and the agent label, meta and timing.

The run's page

Verdict. Passed or failed, and the reasons.

Calls. Every call your agent made, in order, across all the apps: method, path, status, and the full request and response.

Faults. Each fault the test injected that acted, on the call it acted on.

Final state. How many records each app gained, changed and lost from the start, and a button that opens the end state in the app's web client to look around as it was left.

Verdicts of your own. When your own judge kept a verdict on the run ( run.verdict() in Python, or POST /api/v1/runs/<id>/verdict ), its verdict, criteria, reasons and evidence, beside the test's own result, which they don't change.

Transcript. When the harness handed in the agent's transcript, what the agent said and did, folded, one entry per line for JSON lines.

Hand in your agent's transcript

The calls show what your agent did to the apps; its transcript shows why. Hand it in with the answer and it is kept with the run, in whatever format your agent writes:

sw

run t-0e4243

--agent

claude-code -- sh

-c

'claude -p "$STATEWEAVE_PROMPT" \ --mcp-config "$STATEWEAVE_MCP_CONFIG" --strict-mcp-config \ --output-format stream-json --verbose > "$STATEWEAVE_TRANSCRIPT_FILE"'

result = run.

finish

(answer, transcript=messages, transcript_format=

"claude-agent-sdk"

)

sw run sends whatever the command wrote to STATEWEAVE_TRANSCRIPT_FILE . See Hand in a transcript for the limits.

Compare agent versions

The Runs board shows every test against every agent label you've run, so a new version's wins and regressions are one look away. To compare fairly:

Give each version its own --agent label, such as support-bot@0.4 .

Put what changed in meta : the model, the prompt version, a commit.

Run each test more than once. Agents vary from run to run, and so do tests with random faults.

Run again from the browser

When a run fails, you often want to fix the agent and try the same test again right away. Leave a worker running on your machine:

Shell

sw

worker

--agent

my-agent@0.3 -- python my_agent.py

Then press Run again on any run's page. The run is queued, your worker takes it, runs your command with it exactly as sw run does, and hands in the answer. Your command starts afresh for every run, so an edit to your agent counts from the next run on, without restarting the worker.

The run takes the worker's --agent label. Its meta records the command, the machine's name and, in a git checkout, the commit (with a note when there are uncommitted changes).

With several workers connected, the button lets you pick one. A worker counts as connected for a minute after it last asked for work.

A queued run that no worker takes within 10 minutes is cancelled.

--once drives one run and exits, with sw run 's exit code. Ctrl-C stops the worker.

Run in CI

sw run exits non-zero when a run doesn't pass, so a test is a CI step. Store the key as a secret and pass the address and key in the environment:

Shell

export

STATEWEAVE_URL=https://acme.app.stateweave.net

export

STATEWEAVE_API_KEY=

$STATEWEAVE_KEY_SECRET

# from your CI's secret store

pip

install stateweave-sdk status=0

for

t in t-bde537 t-0e4243 t-753b29;

do

sw run

"$t"

--agent

"my-agent@$GIT_SHA"

-- python my_agent.py || status=1

done

exit

$status

Use a key made for CI, so you can revoke it on its own.

Give up a run

A run your harness can't finish should be given up rather than left to time out, so its apps stop at once:

sw

cancel r-de238618

run.

cancel

()

# the with block does this for you when your agent raises

You can also cancel a running or queued run on its page.

Twin run

Test run

Checks

See below

Apps and credentials

See below

Apps and credentials

Credentials

Run statuses

below

sw run

Twin runs

Python SDK reference

REST API reference

Hand in a transcript