If you build agents for software, you have been handed a gift you probably do not think about. Your world is a closed one. Every claim your agent makes can be checked by a deterministic, objective test that is not the agent.
The code compiles or it doesn’t. The test passes or it fails. The type checker is not persuaded by a confident explanation, which is more than can be said for models trained on human approval (Sharma et al., 2023). The whole modern practice of harnessing an agent, the evals, the reward models, the verifier loops, rests on one quiet assumption: somewhere there is an oracle, and it is cheap to ask. The field even has a law for it: AI conquers whatever is easy to verify first (Wei, 2025).
We build a duty manager for short-term rental operators. It works the night shift. It handles the cancelled cleaner at 21:40, the guest who cannot get in at 23:12, the third lockout this month at the same back door. And in that world there is no oracle at all.
The compiler has no feelings. The guest has little else.
A hand-drawn sheet. The coding harness: act, verify, improve; the same test every run, one right answer, checking is free and instant. Versus the real-world operation harness: one cleaned bathroom where the towels, mirror, corner and floor each take two honest answers, and five dials (speed, mood, updates, guest, review) move per owner. A badge reads: the deadline, 15:00, a guest walks in.
same test, every run: one right answer
- The code compiles, or it does not.
- Tests pass or fail in seconds, for free.
- The same checks rerun on every attempt.
- A failure costs nothing: revert, run again.
same room, different nights: never the same expectations
owners disagree the mirrorspotless to you,
streaky to the guest the cornerthe photo never shows it! the floormopped, or still wet at check-in?
one operator’s definition of done. the next operator disagrees.
One of these loops closes by itself. The other never does, which is the whole problem and most of the work.
No oracle for real-life tasks
Take one job: a cleaner cancels, and something has to happen before a guest arrives at three. In a coding harness that is a task with a definite end state. Here, the obvious questions fall apart in your hands.
Did it work? The status of the task is not observable when you need it. The backup cleaner said yes on WhatsApp at 21:58. That is not the same as the flat being clean. She might find the previous guest still there, or do a worse job than Maria, and the guest might only notice a week later, in a review that costs four hundred dollars of ranking. The outcome is real and consequential, and it resolves over days, not milliseconds.
Who says so? What you get are observations, not proofs. There is no build server. The evidence is a photo of a bathroom, a message that says all good, and a five-star review that lands Thursday and might be about the location. Every signal is partial, delayed, and produced by a human with their own reasons.
Would it happen the same way twice? No. The environment never holds still. The same cancellation on a Tuesday in February and a Saturday in July are different problems: one has three backups available, one has none. Agent researchers met a milder version and invented a metric for it, pass^k, reliability across repeated trials (Yao et al., 2024). Our k is one.
Correct according to whom? The actors are people, each with their own definition of done. This one took us longest to accept. Dana runs fifty cabins for thirty owners and wants to be asked about anything over twenty-five dollars. Quyen runs sixty listings from her phone and would rather it just handled it. Neither is wrong. The policy is a property of the operator, not of the domain, and alignment research is arriving at the same place: one aligned answer per question does not survive plural users (Sorensen et al., 2024).
The compiler vs. the human
Compiler, type checker, test suite. Free to run, runs instantly.
There isn't one. The nearest thing is a human who is asleep.
Deterministic and binary. Passed or failed.
Partial, delayed and contested. A photo, a text, a review four days later.
Milliseconds to minutes.
Hours to weeks. The review that grades tonight lands on Thursday.
Same input, same output. Rerun it as often as you like.
You get one attempt. There is no second Tuesday night.
Already digital. The repo is the world, and the world is in the repo.
Mostly in someone's head, or in a WhatsApp group, or in a note on a fridge.
Objective. One right answer per test.
Per-operator. Two customers, two correct answers, same situation.
A failed CI run. Revert and move on.
A one-star review, a refund, a guest standing in the rain at midnight.
Trivial. Git has your back.
Nonexistent. You cannot un-send a message to a guest.
Thousands of cases, cheap to generate.
Every case costs a real night of a real business.
What we own in a code harness versus what the operator owns in ours: spec versus playbook, threshold versus guardrails, proof versus the rule that fired, release gate versus a week in shadow.
Four things a normal harness keeps in-house, and what stands in their place when there is no oracle. Left is ours; right is the operator’s.
So how do we grade?
You stop trying to manufacture an oracle and you start designing around its absence. Five things have carried us further than anything else.
1. Make the operator the oracle, but only where it counts
If the ground truth lives in a person, the honest move is to route to that person deliberately. Money, liability and anything that touches a guest relationship stop and wait; everything else proceeds. The interesting engineering is not can the model decide this but where is the line, and who gets to move it.
That line is per-customer, so it lives in their configuration, not in our weights. And only the operator can move it: messages and photos can change what the system believes about the night, never what it is allowed to do. That structural split prevents the oldest failure in the safety literature, an agent optimising an objective nobody quite meant (Amodei et al., 2016). Below the line everything proceeds and is recorded, because an employee who asks permission for everything is not much of an employee.
2. Treat the policy as data, written by the user
We do not try to learn the universally correct handling of a lockout. We ask the operator to write down what they would tell a new hire, and we run that. Not a flowchart. Sentences.
The variance that makes the domain unlearnable is exactly what the customer is best placed to specify, and they did this work once already, training a human. The playbook is the eval set, and the customer wrote it.
Those sentences compile into tiered rules the runtime enforces, and the compiled version rides every run: open a night from March and you see which edition it was graded against. Elsewhere, this step is a manual onboarding project: a person reads your preferences and configures a system by hand. Ours is a compiler, because at small-business prices there is no room for a person in that loop.
Every operator brings their own definition of correct. Ask for it.
3. Treat a yes as a claim, never a state
The backup cleaner’s yes at 21:58 is where most automation quietly lies to you. In every adjacent agent platform we studied, the human is where the conversation goes to die: hand off, mark resolved, move on. In physical operations the human is not the end of the story. They are the hands. So work handed to a person becomes a commitment with a lifecycle: requested, accepted, claimed, verified. The deadline sits on a timer, the chase is automatic, and the state changes only on evidence.
“All good” is a claim. The photo is evidence. Each claim declares up front what counts as proof, because room is clean and guest got in accept different kinds. And the judge reading the evidence is never the model that did the work: models grading their own reasoning reliably flatter it (Huang et al., 2023).
One commitment in four states: requested, accepted, claimed, verified. Claimed is a claim, not a state; only verification changes what the system believes.
One commitment, left to right. Only the last arrow changes what the system believes. Step through it to see what is true at each point, and what we still will not do.
You asked, at 21:47. Nothing is true yet except the ask, and the deadline is already a timer: if nobody accepts before it fires, the chase starts without waking you.
Marta said yes at 21:58. The room is not clean; a person has taken it on. The clock is hers now, and the system can answer the only question that matters: who owes what.
“All good” arrives at 23:10. It is recorded, and it is not believed: the room stays unknown, and the morning briefing says so rather than quietly rounding up to done.
The photo lands and a separate judge reads it against what this claim accepts as proof. Only now does the state change, and the checker is never the thing that did the work.
4. Log the reasoning, not just the outcome
When you cannot verify the result, verify the process, the same move the reasoning literature made when outcomes proved too coarse to grade (Lightman et al., 2023). Every action carries the rule that produced it, so the morning briefing reads opened by Repeat faults, rule 1, not a claim that something was handled well.
That is what replaces the passing test: a legible chain from the operator’s instruction to the thing that happened, five seconds to audit and one edit to fix.
5. Earn autonomy in shadow, per customer
New operators run us in shadow: we draft what we would have said and hold all of it, and they read a week of our judgment against their real nights before anything goes out under their name. Slow, and the only honest way: let the customer run the eval, on their own data, before they take the risk.
# a coding harness run(task) -> patch -> verify() # cheap, instant, objective -> reward -> improve # ours run(night) -> action -> verify()? # no oracle, days of latency, # and it disagrees per customer # so the loop moves policy = operator.playbook # they write the spec guard = operator.thresholds # they draw the line proof = action.rule_that_fired # we show our working done = evidence.verified # a "yes" is a claim, not a state trust = shadow_until(operator.says_ok)
Our solution: the Three Rooms
Put the five moves together and an architecture falls out: three rooms with strict doors. A studio where the operator’s sentences become the standard. A runtime that works the nights inside it. A learning loop that reads every run and writes proposals, never production. No room can reach into another’s files, and that one constraint carries most of the safety story.
Three rooms with strict doors. Business events wake the Supervisor runtime. The Playbook studio turns the operator’s sentences into versioned rules. The runtime holds run, commitment, gate and evidence; Quality and learning sends back one thing, an improved playbook for a human to approve. Notes record the invariant and the shadow week.
Notes
- No plane writes into another’s scope. The run cannot edit its rules; nothing learned applies itself.
- rules@version rides every run: every action names the rule that fired.
- New operators run a week in shadow before anything goes out under their name.
The oversimplified whole: the standard flows down, the record flows out, and the only path back to the rules runs through a human. Pick a plane to see what it may write.
The studio is the only place the standard can change. It compiles the operator’s sentences into tiered rules, stamps a version, and releases. It never touches a live night.
The runtime works the nights inside the rules it was handed, and writes only the work: runs, commitments, evidence. It cannot edit the standard it is being judged against.
The learning plane reads every run and writes one thing: proposals into the studio’s inbox. It cannot take an action, and nothing it learns applies itself.
The open problems
None of this closes the gap. It organizes it. Every problem below is still open:
- Evidence stays partial and late. A photo can flatter, a review grades Tuesday on Thursday, and a fail-closed judge will sometimes hold work a human would have waved through. Where to put that dial is a nightly argument.
- People accept and go quiet. The chase that gets the bathroom cleaned without burning out the cleaner is a judgment call we re-tune constantly.
- Every new operator is a cold start: their runbook, their thresholds, their definition of done, on day one. The compiler has to be good enough that a stranger’s business teaches the system faster than the system annoys the stranger.
- The learning loop improves playbooks with no test suite to regress against. Replaying proposals over recorded nights is the closest thing we have, and a recorded night never quite repeats; the benchmark literature is making the same critique of itself (Kapoor et al., 2024).
If those sound like reasons to stay away, fair enough. They are the reason we are here: problems this open, in a real business with real consequences, are the most fun we have had in our careers.
Every SMB adopting AI ends up here
Almost every job still done by people has this shape. Field service, clinics, logistics, property, care: consequential, hard to verify, wildly variable between operators, mostly undocumented. Agents landed in code first not because code is the most valuable work, but because code came with a free oracle.
The rest of the economy did not. So the interesting problem is not making the model better at deciding. It is the scaffolding that lets a specific business hand over a specific decision, watch it happen, and take it back.
Strip everything above down and five primitives remain. We think any small operation runs on the same five questions, whatever the industry.
| The primitive | The question it answers | On the cleaner’s night |
|---|---|---|
| The standard | What should happen? | The playbook the owner wrote, at the version that ran |
| The run | What is happening right now? | Tonight’s cancellation, parked and woken on timers |
| The commitment | Who owes what? | The backup’s accepted 3pm clean, chase included |
| The gate | What is allowed? | The ask-me-above-$25 line held; the rest proceeds, recorded |
| The evidence | What can we legitimately conclude? | The photo, never the “all good” |
That is a product problem far more than a model problem, and it is most of what we do.
References
- Towards Understanding Sycophancy in Language Models. Sharma et al., Anthropic. ICLR 2024.
- Asymmetry of Verification and Verifier's Law. Jason Wei, 2025.
- τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. Yao, Shinn, Razavi, Narasimhan. Sierra, 2024.
- A Roadmap to Pluralistic Alignment. Sorensen et al. ICML 2024.
- Concrete Problems in AI Safety. Amodei, Olah, Steinhardt, Christiano, Schulman, Mané, 2016.
- Large Language Models Cannot Self-Correct Reasoning Yet. Huang et al. ICLR 2024.
- Let's Verify Step by Step. Lightman et al., OpenAI, 2023.
- AI Agents That Matter. Kapoor, Stroebl, Siegel, Nadgir, Narayanan. Princeton, 2024.
If you run an operation like this and want to argue with any of the above, we would like that. Write to us.