Complex scenarios, as one workflow.
Probe runs workflows defined in YAML. The definition looks like GitHub Actions, and the actions come built in: HTTP, DB, Shell, SSH, gRPC, SMTP, IMAP, Browser. The result of an action can be checked, so the same file serves for testing and monitoring too.
An example sign-up scenario
Create account posts the form and publishes the id that comes back. Database and Mailbox both depend on it, so Probe runs them at the same time — one asks SQLite for the row, the other opens the mailbox over IMAP. Activate waits for both before following the confirmation link.
1name: Sign-up Flow2 3vars:4 api: https://api.example.com5 mail_user: ada@example.com6 mail_pass: "{{MAIL_PASS}}"7 8jobs:9- name: Create account10 id: signup11 defaults:12 http:13 url: "{{vars.api}}"14 steps:15 - name: Post the form16 id: create17 uses: http18 with:19 post: /signup20 test: res.code == 201 && res.body.status == "pending"21 outputs:22 user_id: res.body.id23 24- name: Database25 id: db26 needs: [signup]27 defaults:28 db:29 dsn: file:./app.db30 steps:31 - name: Row is pending32 uses: db33 with:34 query: SELECT status FROM users WHERE id = ?35 params: ["{{outputs.create.user_id}}"]36 test: res.code == 0 && res.rows[0].status == "pending"37 38- name: Mailbox39 id: mailbox40 needs: [signup]41 defaults:42 imap:43 host: imap.example.com44 username: "{{vars.mail_user}}"45 password: "{{vars.mail_pass}}"46 steps:47 - name: Confirmation arrived48 uses: imap49 with:50 commands:51 - name: select52 mailbox: INBOX53 - name: fetch54 sequence: "1"55 dataitem: ALL56 test: |57 res.code == 0 &&58 res.data.fetch.messages[0].subject == "Confirm your address" &&59 res.data.fetch.messages[0].to == vars.mail_user60 61- name: Activate62 needs: [db, mailbox]63 defaults:64 http:65 url: "{{vars.api}}"66 steps:67 - name: Follow the link68 uses: http69 with:70 get: "/users/{{outputs.create.user_id}}"71 test: res.code == 200 && res.body.status == "active"Sign-up Flow ⏺ Create account (Completed in 0.11s) ⎿ 0. ✓ Post the form ⏺ Database (Completed in 0.02s) ⎿ 0. ✓ Row is pending ⏺ Mailbox (Completed in 0.03s) ⎿ 0. ✓ Confirmation arrived ⏺ Activate (Completed in 0.01s) ⎿ 0. ✓ Follow the link Total workflow time: 0.15s ✓ All jobs succeeded
What a workflow is made of
A workflow is a set of jobs, and a job is a sequence of steps. Steps run one after another, in the order you wrote them. Jobs run in parallel by default; only the ones with needs wait for what they depend on. Add repeat to a job and the same file runs on an interval and reports a success rate.
Supported actions
A step names an action in uses, and Probe starts that action as its own process and talks to it over gRPC. The built-in actions go through exactly the same handshake an action you write does, so none of them is a special case. What Probe cannot reach yet, you can add.
Shared steps, in a file of their own
A scenario usually starts by logging in, and every scenario after the first repeats it. Put those steps in a job file and any workflow runs it with uses: embedded, handing it vars and reading its outputs back. The job's steps appear nested under the step that called it, so a failure still says which one broke.
1name: Orders2 3vars:4 password: "{{PASSWORD}}"5 6jobs:7- name: Order history8 defaults:9 http:10 url: https://api.example.com11 12 steps:13 - name: Log in14 id: auth15 uses: embedded16 with:17 path: ./login-job.yml18 vars:19 username: ada20 password: "{{vars.password}}"21 test: res.code == 022 outputs:23 token: res.outputs.token24 25 - name: List orders26 uses: http27 with:28 get: /orders29 headers:30 authorization: "Bearer {{outputs.auth.token}}"31 test: res.code == 2001name: Log in2 3defaults:4 http:5 url: https://api.example.com6 headers:7 content-type: application/json8 9steps:10- name: Post credentials11 id: login12 uses: http13 with:14 post: /login15 body:16 username: "{{vars.username}}"17 password: "{{vars.password}}"18 test: res.code == 20019 outputs:20 token: res.body.access_tokenWhat a step can say
Whatever action a step names, these fields are available to it.
test
A step passes when the expression is true. Everything about the response is in scope.
1test: res.code == 200 && res.body.status == "ok"outputs
Name a value once, then read it from any later step or job.
1outputs:2 token: res.body.access_tokeniteration
Run one step once per set of variables, without copying it.
1iteration:2- {name: Alice, role: admin}3- {name: Bob, role: user}retry
Give a flaky step another go, after a pause.
1retry:2 max_attempts: 33 interval: 5sskipif
Leave the step out when the condition holds.
1skipif: vars.env == "local"wait
Pause before the step runs, for whatever has to settle first.
1wait: 5sBuilt for coding agents
When an agent writes the code and the tests, something has to check both. Probe teaches the agent how to use the version it runs, finds mistakes in a workflow before it runs, judges responses by your spec rather than by the agent's own test, and refuses writes, and connections to hosts you did not allow, before they are sent.
1name: Profile2 3jobs:4- name: Profile5 defaults:6 http:7 url: https://api.example.com8 steps:9 - name: Log in10 id: login11 uses: http12 with:13 post: /login14 bdoy:15 username: ada16 test: res.code == 20017 outputs:18 token: res.body.token19 20 - name: Read profile21 uses: http22 with:23 get: /me24 headers:25 authorization: "Bearer {{outputs.login.tokn}}"26 tset: res.code == 200agent-workflow.yml:14: error: job 0 "Profile", step 0 "Log in": with: unknown key "bdoy" for the http action; did you mean "body"?agent-workflow.yml:20: warning: job 0 "Profile", step 1 "Read profile": nothing checks this step: it has no testagent-workflow.yml:25: error: job 0 "Profile", step 1 "Read profile": with: outputs.login.tokn is not published: step "login" publishes tokenagent-workflow.yml:26: error: job 0 "Profile", step 1 "Read profile": unknown key "tset"; did you mean "test"? 3 errors, 1 warning
probe check reads a workflow without running it. A run ignores a key it does not know, in a step or in with, so the step can pass for the wrong reason, and finds a wrong output name only when the step reaches it. Here each one is an error with its line before anything is sent, and a step that nothing checks is a warning.
skill
Install a skill that teaches the agent to write, run and debug workflows. It reads probe guide, the reference built into the binary, so what it writes fits the version it runs.
1probe skill install2probe guide httpopenapi
Check each request and response against your OpenAPI document. The spec decides what is right, not the test the agent wrote, so a wrong test cannot agree with a wrong response.
1uses: http2with:3 get: /users/14 openapi:5 spec: ./openapi.ymlproto
The same for gRPC, from your .proto files, with the rules of buf.validate and google.api.field_behavior.
1uses: grpc2with:3 method: GetUser4 proto:5 files: [./proto/user/v1/user.proto]coverage
Tell which operations, responses and methods the run checked, and which ones no step reached yet.
1probe --report json workflow.yml2probe coverage openapi.yml \3 probe-report.json--read-only
Run what the agent wrote without letting it write: anything but reads, and any host not listed, is refused before it is sent.
1probe --read-only \2 --allow-host api.staging.example.com \3 workflow.ymlexit code
The exit code says what to look at: a failed test or broken contract, the workflow itself, or a target that did not answer.
1probe workflow.yml2echo $? # 1 test, 2 workflow, 3 actionInstall
Probe is a single Go binary with no runtime dependencies.
$ go install github.com/linyows/probe/cmd/probe@latestOr build from source
git clone https://github.com/linyows/probe.git cd probe go build -o probe ./cmd/probe