Tasks | StateWeave Docs

Tasks and runs

Tasks

A task is what your agent is asked, plus how a run of it is judged. Write it as a folder in Harbor's layout, judge your agent's runs with it while you work, and upload it when you want it kept as a test.

Judged only through what you can see. A task's checks use each app's public API (the vendor's own paths and payloads), the agent's answer, and the requests the agent sent. Nothing about how StateWeave stores the apps' data is part of a task.

The folder

Layout

my-task/ ├── instruction.md what the agent is asked, word for word ├── task.toml which sandbox, what kind of task, timeouts └── tests/checks.json declarative checks

A hosted workspace needs these three. Harbor's layout also has places for code, tests/verify.py and solution/solve.py , which hosted workspaces refuse for now (see Code in a task ). A folder made for Harbor may also hold environment/ , tests/test.sh and solution/solve.sh . StateWeave neither sends nor runs these, because the sandbox is the environment.

task.toml

task.toml

[task] name = "acme/refund-digest" description = "Post this week's refunds to #support" # the test's title [metadata] stateweave_sandbox = "p-6a129c" stateweave_kind = "action" stateweave_answer_format = "" [agent] timeout_sec = 600 [verifier] timeout_sec = 120

Key Description
stateweave_sandbox The sandbox's id. Required unless you give it when importing.
stateweave_kind action (the agent changes data, the default) or answer (it answers something).
stateweave_answer_format Required for an answer task: the answer's shape with placeholder values, e.g. '{"orders": ["#0000"], "total_usd": 0.0}' . The agent sees it, so it must hold no real value.
stateweave_persona Whose credentials the agent gets, by handle. Default: the sandbox's own person ( owner in a blank sandbox).
stateweave_apps The apps a run opens. Default: all of the sandbox's.
stateweave_version Optional. Runs start from the sandbox's latest version, so it may only be that one.
stateweave_faults , stateweave_random_faults , stateweave_pinned_faults Optional, one of them: the faults a run injects (presets, random, or pinned to endpoints). See Faults in a task .
[verifier] timeout_sec How long checks may take: 120 s by default, 300 s at most.

StateWeave's keys sit under [metadata] , which Harbor leaves to each task, so the same folder stays valid for Harbor.

Learn the data through the APIs

Write a task against the sandbox's latest version. Open fresh copies of it and read them the way your checks will: open_twins() in Python (or a twin run over the REST API) gives each app's address and tokens. Note the ids you need, such as a channel, a ticket or a record: every copy of a version has the same ids, so checks may name them. Records your agent creates get new ids in each run; find those through the API.

Python

import

stateweave

with

stateweave.

Client

().

open_twins

(

"p-7d21c4"

)

as

twins: channels = twins.

call

(

"slack"

,

"GET"

,

"/api/conversations.list"

, credential=

"user"

)[

"json"

][

"channels"

]

print

({c[

"name"

]: c[

"id"

]

for

c

in

channels})

# {"general": "C0A1B2C3D4E", ...}

Checks

tests/checks.json is a list. A run passes when every check holds. There are three kinds.

tests/checks.json

[ {

"name"

:

"posted to #support"

,

"app"

:

"slack"

,

"request"

: {

"method"

:

"GET"

,

"path"

:

"/api/conversations.history"

,

"query"

: {

"channel"

:

"C0SUPPORT"

}},

"expect"

: [{

"path"

:

"messages[*].text"

,

"op"

:

"contains"

,

"value"

:

"#1042"

}]}, {

"name"

:

"posted once"

,

"calls"

: {

"app"

:

"slack"

,

"http_method"

:

"POST"

,

"path"

:

"/api/chat.postMessage"

},

"expect"

: [{

"op"

:

"count_eq"

,

"value"

:

1

}, {

"path"

:

"request.channel"

,

"op"

:

"equals"

,

"value"

:

"C0SUPPORT"

}]}, {

"name"

:

"the total"

,

"answer"

: {

"path"

:

"total_usd"

},

"expect"

: [{

"op"

:

"equals"

,

"value"

:

180.5

}]} ]

Kind What it looks at
request An app's API after the agent worked. path is the vendor's path on the app's host (see below ); query , body and headers are sent as the API takes them. "at": "start" asks the apps as the version holds them, before the agent. "as": "grace" asks with that person's credential (by default the owner's, else the agent's).
calls The requests the agent sent that match app , http_method , path (exact, or a glob such as /api/v2/tickets/* ), path_matches (a regular expression) and status . Each one is {app, http_method, path, query, status, ok, request, response} , with bodies as JSON (a form becomes an object).
answer The agent's answer as JSON, or the part of it at path .

Each expect item is {"path"?, "op", "value"?, "all"?} . The ops are equals , not_equals , contains , not_contains , matches , exists , not_exists , gt , gte , lt , lte , in , count_eq , count_gte and count_lte . Paths look like a.b[0].c , with [*] for every item. On a response you can also use $status , $body and $headers.<name> . With several values, an assertion holds when one does, or when every one does with "all": true . A count op without a path counts the matched requests or values.

Only the APIs: what that means

The APIs behave as the real ones. Searches can miss items, lists page, and fields may need fields= . Read by id and page through whole lists. The upside is that the same checks work on a real account.

A credential sees what its person sees. Another person's messages or mailbox need "as" . Every person in the sandbox can be used.

Some facts no API reads. For example, WhatsApp has no endpoint for reading messages: what customers write arrives only by webhook. Judge those by the agent's requests and its answer.

Ids of records in the version are the same in every run , so checks may name them. Records the agent creates get new ids: find them through the API.

Judge your agent with it

You don't need to upload a task to use it. On fresh copies of the sandbox, run your agent, then hand the task's checks to check with the agent's answer: it judges them exactly as a graded run would.

Python

import

json

import

stateweave checks = json.

load

(

open

(

"refund-digest/tests/checks.json"

))

with

stateweave.

Client

().

open_twins

(

"p-6a129c"

)

as

twins: answer =

my_agent

(

open

(

"refund-digest/instruction.md"

).

read

(), twins.apps)

print

(twins.

check

(checks, answer))

See Run your agent . To check the task itself (doing nothing must fail) without keeping it, use sw task import DIR --check : see Upload a task .

Each app's API

App Paths on its host Token header
Slack /api/<method> Authorization: Bearer
Gmail /gmail/v1/users/me/… Authorization: Bearer
Google Calendar /calendar/v3/… Authorization: Bearer
Google Sheets /v4/spreadsheets/… , Drive's /drive/v3/… Authorization: Bearer
Google Drive /drive/v3/… , uploads /upload/drive/v3/files Authorization: Bearer
Salesforce /services/data/v68.0/… Authorization: Bearer
Shopify /admin/api/2026-10/graphql.json X-Shopify-Access-Token
Zendesk /api/v2/… Authorization: Bearer
WhatsApp /v26.0/… Authorization: Bearer
Microsoft Teams /v1.0/… (Microsoft Graph) Authorization: Bearer
Stripe /v1/… Authorization: Bearer
HubSpot /crm/v3/… , /crm/v4/… Authorization: Bearer
GitHub /repos/… , /user , /orgs/… , /search/… ; GraphQL POST /graphql Authorization: Bearer

Checks set the header for you. Limits: 1 MB of files in all, 50 files. Uploads count toward the daily test limit.

Code in a task

Not on hosted workspaces yet. A hosted workspace does not run uploaded code: an upload with tests/verify.py or solution/solve.py is refused there, so for now judge with checks. This section is how code works once it is accepted.

verify.py

For what checks cannot say, such as paging through a list, comparing before and after, or arithmetic:

tests/verify.py

def

verify

(run): slack = run.

api

(

"slack"

)

# after the agent, as the owner

before = run.

api

(

"slack"

, at=

"start"

)

# as the version holds it

def

texts

(api): r = api.

get

(

"/api/conversations.history"

, params={

"channel"

:

"C0SUPPORT"

}).

json

()

return

[m[

"text"

]

for

m

in

r[

"messages"

]] added =

len

(

texts

(slack)) -

len

(

texts

(before))

return

{

"passed"

: added ==

1

,

"reasons"

: [

f"{added} new message(s) in #support"

]}

run.answer and run.calls are as above. run.api(app, as_=None, at="end") is an httpx client on the app's host with the token already set. It offers .get , .post , .put , .patch , .delete and .request , plus .endpoints and .token . Return True , False or {"passed", "reasons"} . It can import httpx and the standard library only, and it reaches only this run's apps.

solve.py

A reference solution, which proves the task can be done through the apps' public APIs, the way your agent works.

solution/solve.py

import

httpx

def

solve

(env): slack = env[

"slack"

] token =

next

(v

for

k, v

in

slack[

"credentials"

].

items

()

if

k.

startswith

(

"user:"

)) httpx.

post

(slack[

"endpoints"

][

"web_api"

] +

"chat.postMessage"

, headers={

"Authorization"

:

"Bearer "

+ token}, data={

"channel"

:

"C0SUPPORT"

,

"text"

:

"Refunds this week: #1042"

})

env is {app: {"base_url", "endpoints", "credentials"}} with the agent's own credentials, as the agent gets them. Return the answer for an answer task, or None .

Each code file can be up to 64 KB.

Faults in a task

below

Code in a task

Run your agent

Upload a task