Proof that AI-written code works.
When an AI coding agent changes a service, something other than the code and tests it wrote has to show that the change works. Probe runs a YAML workflow, written like GitHub Actions, that drives the running service through a browser, an iOS app, HTTP, gRPC, databases, mail, SSH and shell, and checks each result. The agent writes the workflow, Probe checks it before it runs, and runs it without writes or connections you did not allow.
An example sign-up scenario
Sign up fills in the form in Chrome. API, Database and Mailbox each check what it left behind, at the same time: the API refuses to sign in an address not yet confirmed, SQLite holds the row as pending, and the confirmation mail is in the mailbox over IMAP. Activate waits for all three and opens the link from the mail in the browser. Then two jobs use the account at once: iOS app signs in on an iPhone simulator, and First visit hands the web app to an AI agent playing Ada, who has never seen it, and judges whether she reaches her goal — what a test written step by step cannot tell you.
1name: Sign-up Flow2 3actions:4 browser: github.com/mozership/probe-browser@634baa8dff2b273ed326996fec4d91c2dba9d38b # v0.2.05 ios: github.com/mozership/probe-ios@aed5436143e9536754e2123b953831f60cd635eb # v0.2.06 agent: github.com/mozership/probe-agent@4d0d6ccee6165624283b22fa0541b02d17e50ac5 # v0.2.07 8secrets:9- ANTHROPIC_API_KEY10 11vars:12 web: https://app.example.com13 api: https://api.example.com14 mail_user: ada@example.com15 mail_pass: "{{MAIL_PASS}}"16 password: "{{PASSWORD}}"17 18jobs:19- name: Sign up20 id: signup21 steps:22 - name: Fill in the form23 uses: browser24 with:25 timeout: 15s26 actions:27 - name: navigate28 url: "{{vars.web}}/signup"29 - name: type30 selector: "#email"31 value: "{{vars.mail_user}}"32 - name: type33 selector: "#password"34 value: "{{vars.password}}"35 - name: click36 selector: button[type=submit]37 - name: wait_text38 id: notice39 selector: "#notice"40 test: res.code == 0 && res.results.notice startsWith "Check your inbox"41 42- name: API43 id: api44 needs: [signup]45 defaults:46 http:47 url: "{{vars.api}}"48 headers:49 content-type: application/json50 steps:51 - name: Sign-in is refused52 uses: http53 with:54 post: /login55 body:56 email: "{{vars.mail_user}}"57 password: "{{vars.password}}"58 test: res.code == 403 && res.body.error == "confirm your address first"59 60- name: Database61 id: db62 needs: [signup]63 defaults:64 db:65 dsn: file:./app.db66 steps:67 - name: Row is pending68 uses: db69 with:70 query: SELECT status FROM users WHERE email = ?71 params: ["{{vars.mail_user}}"]72 test: res.code == 0 && res.rows[0].status == "pending"73 74- name: Mailbox75 id: mailbox76 needs: [signup]77 defaults:78 imap:79 host: imap.example.com80 username: "{{vars.mail_user}}"81 password: "{{vars.mail_pass}}"82 steps:83 - name: Confirmation arrived84 id: mail85 uses: imap86 with:87 commands:88 - name: select89 mailbox: INBOX90 - name: fetch91 sequence: "1"92 dataitem: ENVELOPE BODY[TEXT]93 test: |94 res.code == 0 &&95 res.data.fetch.messages[0].subject == "Confirm your address" &&96 res.data.fetch.messages[0].to == vars.mail_user97 outputs:98 link: last(split(trim(res.data.fetch.messages[0].body), "\r\n"))99 100- name: Activate101 id: activate102 needs: [api, db, mailbox]103 steps:104 - name: Follow the link105 uses: browser106 with:107 timeout: 15s108 actions:109 - name: navigate110 url: "{{outputs.mail.link}}"111 - name: text112 id: heading113 selector: h1114 test: res.code == 0 && res.results.heading == "Your account is active"115 116- name: iOS app117 id: ios118 needs: [activate]119 steps:120 - name: Sign in on iPhone121 uses: ios122 timeout: 10m123 with:124 app: com.example.acme125 actions:126 - name: launch127 args: ["-api", "{{vars.api}}"]128 - name: type129 selector: email130 value: "{{vars.mail_user}}"131 - name: type132 selector: type=securetextfield identifier=password133 value: "{{vars.password}}"134 - name: tap135 selector: signin136 - name: text137 id: greeting138 selector: greeting139 test: res.code == 0 && res.results.greeting == "Welcome, ada@example.com"140 141- name: First visit142 id: visit143 needs: [activate]144 steps:145 - name: Ada names herself146 uses: agent147 with:148 url: "{{vars.web}}/signin"149 persona: |150 Ada, who has just confirmed her address and has151 never used the app. She signs in with {secret:EMAIL}152 and the password {secret:PASSWORD}.153 goal: |154 Sign in, and set the display name of the account155 to "Ada Lovelace".156 secrets:157 EMAIL: "{{vars.mail_user}}"158 PASSWORD: "{{vars.password}}"159 evidence_dir: out160 test: res.verdict == "pass"Sign-up Flow ⏺ Sign up (Completed in 3.25s) ⎿ 0. ✓ Fill in the form ⏺ API (Completed in 0.03s) ⎿ 0. ✓ Sign-in is refused ⏺ Database (Completed in 0.04s) ⎿ 0. ✓ Row is pending ⏺ Mailbox (Completed in 0.05s) ⎿ 0. ✓ Confirmation arrived ⏺ Activate (Completed in 1.02s) ⎿ 0. ✓ Follow the link ⏺ iOS app (Completed in 53.90s) ⎿ 0. ✓ Sign in on iPhone ⏺ First visit (Completed in 29.86s) ⎿ 0. ✓ Ada names herself ✅ pass: Signing in led to /home, so the persona got into the app. On the Account page she typed "Ada Lovelace" and saved, and the page showed "Saved." at /account?saved=1. The tester then opened /account again as a new page load. The page's form data shows the Display name field holding value="Ada Lovelace", and the screenshot shows the same name, so the change was kept. No server errors or slow requests turned up. The only failed request was the site icon (favicon.ico, 404), which the persona would not notice. evidence: "Home · Account Your projects You have no projects yet." evidence: "Saved." evidence: "textbox \"Display name\" value=\"Ada Lovelace\" [ref_7]" 0 findings 21 steps, stopped: done, 28s report: out/probe-agent-1662083251/report.md Total workflow time: 58.23s ✓ All jobs succeeded
Structure
What a workflow is made of
A workflow is a set of jobs, and a job is a sequence of steps. Steps run one after another, in the order you wrote them. Jobs run in parallel by default; only the ones with needs wait for what they depend on. Add repeat to a job and the same file runs on an interval and reports a success rate.
Architecture
Supported actions
A step names an action in uses, and Probe starts that action as its own process and talks to it over gRPC. The built-in actions go through exactly the same handshake an action you write does, so none of them is a special case. What Probe cannot reach yet, you can add.
Reuse
Shared steps, in a file of their own
A scenario usually starts by logging in, and every scenario after the first repeats it. Put those steps in a job file and any workflow runs it with uses: embedded, handing it vars and reading its outputs back. The job's steps appear nested under the step that called it, so a failure still says which one broke.
1name: Orders2 3vars:4 password: "{{PASSWORD}}"5 6jobs:7- name: Order history8 defaults:9 http:10 url: https://api.example.com11 12 steps:13 - name: Log in14 id: auth15 uses: embedded16 with:17 path: ./login-job.yml18 vars:19 username: ada20 password: "{{vars.password}}"21 test: res.code == 022 outputs:23 token: res.outputs.token24 25 - name: List orders26 uses: http27 with:28 get: /orders29 headers:30 authorization: "Bearer {{outputs.auth.token}}"31 test: res.code == 2001name: Log in2 3defaults:4 http:5 url: https://api.example.com6 headers:7 content-type: application/json8 9steps:10- name: Post credentials11 id: login12 uses: http13 with:14 post: /login15 body:16 username: "{{vars.username}}"17 password: "{{vars.password}}"18 test: res.code == 20019 outputs:20 token: res.body.access_tokenSyntax
What a step can say
Whatever action a step names, these fields are available to it.
test
A step passes when the expression is true. Everything about the response is in scope.
1test: res.code == 200 && res.body.status == "ok"outputs
Name a value once, then read it from any later step or job.
1outputs:2 token: res.body.access_tokeniteration
Run one step once per set of variables, without copying it.
1iteration:2- {name: Alice, role: admin}3- {name: Bob, role: user}retry
Give a flaky step another go, after a pause.
1retry:2 max_attempts: 33 interval: 5sskipif
Leave the step out when the condition holds.
1skipif: vars.env == "local"wait
Pause before the step runs, for whatever has to settle first.
1wait: 5sQuality
What deterministic tests leave out
The steps above check what can be stated exactly, and they are what should gate a merge. Whether a first-time user gets through, or whether the application still holds up once data and settings pile up on staging, cannot be written as an expression. The agent action takes it in natural language, as a persona and a goal: a model plays the persona in a real Chrome, and another call of the model reviews the run, keeping only what the pages and the requests bear out. Run it with repeat against staging, and it keeps using the application as people do.
Works with
- Claude
- OpenAI
- Gemini
- xAI
- Sakura AI Engine
1name: First note2 3actions:4 agent: github.com/mozership/probe-agent@4d0d6ccee6165624283b22fa0541b02d17e50ac5 # v0.2.05 6vars:7 password: "{{PASSWORD}}"8 9jobs:10- name: A first-time visitor11 steps:12 - name: Writes a first note13 uses: agent14 with:15 url: https://staging.example.com/16 persona: |17 Carol, who has never used a notes app and is18 not technical. She signs up with {secret:EMAIL}19 and the password {secret:PASSWORD}.20 goal: |21 Create an account, and write a first note22 that says "Buy milk".23 secrets:24 EMAIL: "carol+{{random_str(8)}}@example.com"25 PASSWORD: "{{vars.password}}"26 evidence_dir: out27 test: res.verdict != "blocked"First note ⏺ A first-time visitor (Completed in 48.49s) ⎿ 0. ✓ Writes a first note ❌ fail: The account was created: the welcome page greeted Carol and said the account was ready. No note was ever written, though. The only way to write one is the "Write your first note" link to /notes/new, and that page returned a 500 "Internal Server Error" all three times it was loaded. Nothing in the record shows a "Buy milk" note being saved. evidence: "Welcome to Acme Notes, Carol!" evidence: "Your account {secret:EMAIL} is ready." evidence: "link \"Write your first note\" href=\"/notes/new\"" evidence: "Internal Server Error" 2 findings 🔴 high: The new-note page (/notes/new) fails with a server error (500) 🟠 medium: After sign-up, nothing shows Carol is signed in and there is no other way to reach notes 24 steps, stopped: done, 47s report: out/probe-agent-3220800333/report.md Total workflow time: 48.49s ✓ All jobs succeeded
The first finding is a server error, which an http step would have caught had someone written one. The second is in no test: after signing up, nothing tells Carol she is signed in, nor how to get to her notes. The test of the step asks only that the review ran; the verdict and the findings are the result to read.
Read the agent action referenceCoding agents
Built for coding agents
When an agent writes the code and the tests, something has to check both. Probe teaches the agent how to use the version it runs, finds mistakes in a workflow before it runs, judges responses by your spec rather than by the agent's own test, and refuses writes, and connections to hosts you did not allow, before they are sent.
1name: Profile2 3jobs:4- name: Profile5 defaults:6 http:7 url: https://api.example.com8 steps:9 - name: Log in10 id: login11 uses: http12 with:13 post: /login14 bdoy:15 username: ada16 test: res.code == 20017 outputs:18 token: res.body.token19 20 - name: Read profile21 uses: http22 with:23 get: /me24 headers:25 authorization: "Bearer {{outputs.login.tokn}}"26 tset: res.code == 200agent-workflow.yml:14: error: job 0 "Profile", step 0 "Log in": with: unknown key "bdoy" for the http action; did you mean "body"?agent-workflow.yml:20: warning: job 0 "Profile", step 1 "Read profile": nothing checks this step: it has no testagent-workflow.yml:25: error: job 0 "Profile", step 1 "Read profile": with: outputs.login.tokn is not published: step "login" publishes tokenagent-workflow.yml:26: error: job 0 "Profile", step 1 "Read profile": unknown key "tset"; did you mean "test"? 3 errors, 1 warning
probe check reads a workflow without running it. A run ignores a key it does not know, in a step or in with, so the step can pass for the wrong reason, and finds a wrong output name only when the step reaches it. Here each one is an error with its line before anything is sent, and a step that nothing checks is a warning.
skill
Install a skill that teaches the agent to write, run and debug workflows. It reads probe guide, the reference built into the binary, so what it writes fits the version it runs.
1probe skill install2probe guide httpopenapi
Check each request and response against your OpenAPI document. The spec decides what is right, not the test the agent wrote, so a wrong test cannot agree with a wrong response.
1uses: http2with:3 get: /users/14 openapi:5 spec: ./openapi.ymlproto
The same for gRPC, from your .proto files, with the rules of buf.validate and google.api.field_behavior.
1uses: grpc2with:3 method: GetUser4 proto:5 files: [./proto/user/v1/user.proto]coverage
Tell which operations, responses and methods the run checked, and which ones no step reached yet.
1probe --report json workflow.yml2probe coverage openapi.yml \3 probe-report.json--read-only
Run what the agent wrote without letting it write: anything but reads, and any host not listed, is refused before it is sent.
1probe --read-only \2 --allow-host api.staging.example.com \3 workflow.ymlexit code
The exit code says what to look at: a failed test or broken contract, the workflow itself, or a target that did not answer.
1probe workflow.yml2echo $? # 1 test, 2 workflow, 3 actionGet started
Install
Probe is a single binary with no runtime dependencies. Homebrew installs it with one command.
$ brew install mozership/tap/probeOr install with Go
go install github.com/mozership/probe/cmd/probe@latest
FAQ
Questions you may have
Do I need an AI coding agent to use Probe?
No. A workflow is plain YAML, and Probe runs it the same whoever wrote it. What Probe does for an agent, such as finding the mistakes in a workflow before it runs, it does for a person writing one by hand too.
Read the quickstart →How is it different from runn, Hurl or k6?
Probe drives a browser, an iOS app, HTTP, databases and mail from one YAML file, runs its jobs in parallel where needs allows, and is a single binary. It does not put load on a system, which is what k6 is for, and for HTTP alone Hurl is the lighter tool.
Compare it with similar tools →Can it run in CI?
Yes. On GitHub Actions, mozership/probe-action downloads Probe, verifies its checksum, caches it and runs your workflows. The exit code says what to look at: 1 for a failed test, 2 for the workflow itself, 3 for a target that did not answer. The iOS action needs a macOS runner, where Probe is installed with go install.
Set it up in GitHub Actions →Can a workflow an agent wrote change a system it should not?
Not when you run it under the guard. --read-only refuses what writes, and --allow-host refuses connecting to a host you did not name, before anything is sent; nothing in a workflow can loosen either. An action that cannot tell what it will do, such as the browser or the agent action, is refused under the guard unless --allow-action names it.
Read about the guard →What is the agent action for, when its result is not deterministic?
For what a deterministic step cannot state. A status, a row or a mail can be checked exactly, and those steps should gate a merge. Whether a first-time user gets through a page, or whether the application still holds up on staging once data and settings pile up, cannot be written as an expected value. The agent action leaves that to a model using the application through a real Chrome, and adds to the deterministic steps rather than replacing them.
See what deterministic tests leave out →The agent action asks a model. Does its result change from run to run?
It can. The judge quotes what the pages showed and marks whether each quote was found there, and blocked, a test that could not be judged, is kept apart from fail. A scenario test can keep the steps of a run that passed and replay them without the model; only the judge is asked again.
Read the agent action →What does testing an iOS or Android app need?
The iOS action runs on macOS with Xcode, against a booted simulator with the app installed. The Android action runs on Linux and macOS through adb, with an emulator or a device, and nothing but the app is installed on it. Neither boots the device or installs the app: do that in a shell step before.
Read the iOS action →What if the protocol I need is not supported?
Write an action. Probe talks to an action over gRPC, so it can be written in any language, and the built-in actions go through the same handshake. A workflow pins an external action to a full commit SHA, and Probe runs only the executable whose SHA-256 the action.yml at that commit names.
Read about actions →Is it free?
Yes. Probe is open source under the MIT License. The agent action calls the model provider you choose with your own API key, and the provider bills that use.
See it on GitHub →