Case study · Clara Building Company
An operating system for a construction company
Sole architect and engineer. Deal sourcing, AI estimation, project management, and financial reconciliation, in one system built and operated by one person. The interesting part is not the AI. It is the machinery that keeps a human accountable for what the AI does.
- Next.js 16 · React 19
- ~41 Python services
- GCP · Firestore · BigQuery
- Sole engineer
- lines of TypeScript across 1,660 files
- 308Klines of TypeScript across 1,660 files
- API routes behind one auth gateway
- 349API routes behind one auth gateway
- Python services on Cloud Run
- ~41Python services on Cloud Run
- frontend test cases
- 3,939frontend test cases
The problem
Construction runs on memory
A residential build is a few hundred dependent tasks, a dozen subcontractors who do not use software, a budget that moves daily, and a project manager whose actual system of record is their own recollection of a phone call. Nothing is missing because nobody wrote it down. It is missing because writing it down costs more than the person has.
Clara is an attempt to lower that cost to nearly zero. The PM texts a contractor the way they already do. A camera photographs the site the way it already does. Invoices arrive in an inbox the way they already do. Clara's job is to turn those existing, unstructured human behaviors into structured state (schedule, budget, progress) without asking anyone to change how they work.
That framing is what makes it an AI problem. It is also what makes it an accountability problem, which is the part most of this case study is about.
Architecture
One gateway, no side doors
A Next.js 16 application acts as the backend-for-frontend, sitting in front of roughly forty Python services on Cloud Run, wired together with Pub/Sub, Eventarc, and about twenty scheduled jobs. Firestore is the operational system of record; BigQuery holds analytics, MLS data, and the product catalog.
The shape that matters is the second row. Every channel, whether the web app, an inbound SMS, a vendor invoice, or a bank webhook, resolves to an identity at the same gateway before it can touch data. When I added the SMS bot, I did not add a new trust boundary. I reused the one that already existed.
The core idea
The confidence threshold is a governance decision
"Human in the loop" asserts that a person is present. It says nothing about what that person can actually do. So I use a RACI lens instead, because it forces a harder question: for this specific task, who does the work, and who answers for it when it is wrong.
Here is the shipped version of the task. Every evening Clara texts the project manager a check-in. They reply in plain English, drywall wrapped today, painter starts Monday, and the schedule moves.
Task: Update the schedule from the PM's daily check-in reply
| Role | Who | Why |
|---|---|---|
| Responsible | The extraction pipeline | It does the work: reads the reply, decides which milestones it refers to, and produces the change. |
| Accountable | The project manager | Their schedule, their client, their money. One person, and it cannot be delegated to software. |
| Consulted | The PM, when the model asks a clarifying question | Two-way. The bot texts back rather than guessing, and the answer feeds a fresh extraction. |
| Informed | The PM, by receipt, after an auto-apply | One-way. Told after the fact, with the ability to reverse it. |
Assigning the roles that way makes the confidence threshold do something visible. It is the line between the system acting and the system withholding.
Above 0.85, the pipeline applies the change and texts the PM a receipt. Below it, nothing is written at all. The proposal is recorded with status: 'proposed' and parked for review, and the schedule is untouched.
Which makes that number something other than a tuning parameter. Most teams arrive at a threshold by feel and then treat it like a retry count. It is not a retry count. It decides which class of decisions a machine is allowed to make on a person's behalf without asking, and moving it two points moves that line. I would rather write that down than discover it later.
The genuinely two-way path is separate from the number. When the model cannot tell what a reply refers to, it does not guess and it does not silently drop it. It texts back a clarifying question, and the answer feeds a fresh extraction. That is where a human is actually Consulted, and it runs off the model saying it does not know, not off a confidence score. A low confidence value and an honest "I cannot tell" are different signals, and only one of them should start a conversation.
def extract_with_vote(extract_fn, *, k: int = VOTE_RUNS):
with ThreadPoolExecutor(max_workers=k) as pool:
runs = list(pool.map(lambda _: extract_fn(), range(k)))
majority = k // 2 + 1
for (pid, mid, action), props in buckets.items():
if len(props) < majority:
continue # no consensus, drop it
voted.append({
...,
"median_confidence": statistics.median(p.confidence for p in props),
})Being merely Informed is only a safe role if reacting is cheap. So the rest of the design is about making the veto cost almost nothing. Every applied change captures the prior state inside the same Firestore transaction as the write:
before_state = {
"status": milestone.get("status"),
"inProgressAt": milestone.get("inProgressAt"),
"completedAt": milestone.get("completedAt"),
}
# written to the append-only extraction_log in the SAME transaction
# as the milestone update, so rollback is an exact inverse, not a guess.That one decision is what makes undo trustworthy. Because the log row holds the exact prior state, reversing a change is deterministic. There is no reconstructing history, no inferring what it must have been. And so the PM can simply text back:
_UNDO_RE = re.compile(r"^undo(?:\s+(all|[\d\s,\-]+))?$")
# "undo" → revert the whole last applied batch
# "undo 1 3" → revert only lines 1 and 3 of the receipt the PM saw
# "undo 1-3" → revert a rangeThe undo path is regex, not a model. A person trying to reverse a mistake made by an AI should not have to get past another AI to do it. Reversing restores the captured before-state, reverses the counters, and soft-marks the log row rolled_back. It never deletes. The record that the model was wrong is itself part of the audit trail, and both the SMS undo and the button in the admin review UI restore from that same recorded state.
Designed, not shipped
The table is telling me the product is not finished
Go back and look at who is in that table. The project manager is Accountable, Consulted, and Informed. They are three of the four rows. The only party who is not them is the software.
That is a degenerate RACI, and it is a real finding rather than a presentation problem. It means the system is currently an excellent note-taker for one person. Clara is still asking the PM what happened, which means the PM still has to know, which means the expensive thing has not actually been removed. It has been made cheaper to record.
The thing that collapses the cost is the subcontractor. If the drywall crew texts when the drywall is done, the roles finally separate: the sub becomes the source, the PM stays Accountable without having to be present, and Clara stops being a transcription service and starts being the place the project actually lives.
I designed that flow and did not ship it, and I want to be precise about why, because the reason is not schedule.
That distinction fell out of the RACI exercise. The framework is not decoration on top of the engineering. It is what told me the next feature is not a bigger contact list, it is a second trust tier, and it is why I built the confidence gate and the audit log the way I did rather than as a schedule of if-statements.
When the model was confidently wrong
A guard that overrules the model at any confidence
The system marked a live, in-progress Permit Review milestone as not-applicable. It was not a low-confidence guess that slipped through the gate; the model was confident, and the vote agreed with itself. Consensus and confidence are correlated with correctness, but they do not bound it, and a threshold cannot save you from a failure mode that all three runs share.
So the fix was not a higher threshold. It was a deterministic rule sitting above the model, in code, that refuses to mark an in_progressmilestone not-applicable no matter what the model says or how sure it is. Some transitions are simply not the model's to make.
I think this is the most useful thing in the whole pipeline, and it generalizes: probabilistic components need deterministic guardrails at the boundaries where the damage is real. You do not tune your way out of a class of error. You take the decision away.
Security
The habits of someone who used to break these
Before this I was a penetration tester and product security engineer. That shows up less in any single control than in a set of defaults I did not have to be talked into.
export async function requireProjectAccess({ projectId, auth, requestHeaders }) {
// Check access BEFORE existence, so a 403 vs 404 can't be used
// to probe which project IDs are real.
if (auth.kind === 'user') {
if (!canAccessProject(auth.user, projectId, requestHeaders ?? new Headers())) {
throw new ProjectAccessError('Access denied', 403)
}
}
const project = await projectServiceAdmin.getById(projectId)
if (!project) throw new ProjectAccessError('Project not found', 404)
return project
}Authorization itself is a matrix of about eighty permission keys across nine roles, compiled once into sets at module load so a check is a hash lookup rather than a scan. The live-update stream is filtered by the same matrix per entity: a project reader gets milestones and roster pushed to them in real time and never receives a room, because the server never sends one, not because the client hides it.
And the audit layer re-verifies the assertion independently of the auth layer. Double verification is nearly free with HMAC, and it means a forged header cannot corrupt the record of what happened. The audit trail has to be trustworthy precisely when the rest of the system is not.
Cost
Bounding the model's appetite
An always-on vision pipeline over jobsite cameras is a great way to spend unbounded money. Two bounds keep it cheap. At the camera edge, a Lambda takes the first, middle, and last photo of the day rather than everything the camera saw. Then the analyzer makes exactly one model call per project per night, over that day's photos together, rather than one call per image.
Taking the ends and the middle rather than the best is a concession to the hardware. These are consumer cameras, and they do not give you the kind of control that would let you select intelligently at the edge. So the sampling is dumb on purpose, and the constraint is honest: I am bounding cost with the crudest tool that the camera actually supports.
The reduction still happens before the expensive thing runs, which is where it belongs. It is the same instinct as the SQLite work in Argus: decide what you actually need before you pay for it.
What I'd take from this
Building it alone meant every architectural shortcut came back to me personally, usually within a month. The habits that survived that feedback loop are the ones I would bring anywhere: put every channel behind one trust boundary; make the audit trail append-only and the undo deterministic; give probabilistic components deterministic guardrails at the point where damage becomes real; and keep the human's veto cheaper than their acquiescence.
The last one is the whole argument, and it is the subject of a longer essay.