Failure mode 1
Authority and blast radius
What can the agent read, write, delete, deploy, or spend? Maps the full authority model and identifies where the agent can act beyond its mandate.
The gap is not code quality. It is production readiness.
AI-generated codebases can pass every linter and clear a standard code review. They still fail in production because the failure modes are different. An AI agent that writes code, deploys services, or modifies accounts needs the same disciplines as any other production system: scoped authority, audit trails, approval gates, and rollback paths.
AI code that passes evaluation fails in production when the evaluation did not test authority, not correctness.
Apple shipped AI workflow orchestration at WWDC 2026. Microsoft shipped an AI behaviour testing tool in June 2026. OpenAI repositioned Codex for white-collar development the same week. The market has confirmed the pattern: AI generates the code. Governance closes the gap before it reaches production.
This governance model runs our own business.
RFE Online operates a 10-agent production fleet — Sterling, Vanessa, Angela, and seven siblings — governed by the exact principles behind this audit. Every failure mode we surface for clients was pressure-tested on our own infrastructure first. Sterling scored 79/100 on standard evaluation before hardening. Three production incidents in the first 24 hours confirmed the gap. After hardening: 84/100, IRO delta improved from 44 to 65.
What the audit covers
The engagement audits your codebase against seven failure modes that AI-built stacks accumulate faster than human-written code, and addresses each one in priority order.
Failure mode 2
Authentication and authorisation
Credential storage, secret rotation, permission scoping, and whether the agent can escalate its own privileges or be prompted into doing so.
Failure mode 3
Input handling
Prompt injection, untrusted input treated as instructions, and any path where an external actor can change the agent’s behaviour through its inputs.
Failure mode 4
Error and retry paths
What happens when a step fails? Retry loops that amplify mistakes, missing circuit breakers, and error states that leave the system in an indeterminate position.
Failure mode 5
Approval gates and rollback
Is there a preview step? A dry-run? A human confirmation before irreversible actions execute? Is there a rollback path when something goes wrong?
Failure modes 6–7
Observability and dependency posture
Whether you can see what the agent did after the fact, and whether the dependency chain would survive a security review or a supply chain incident.
The engagement separates true launch blockers from deferred improvements. You leave knowing what is safe to ship now and what to address in the next sprint. A reusable audit worksheet is included so your team can apply the same framework to every agent you add going forward.
How the engagement works
A typical hardening engagement runs two to three weeks from discovery call to handoff.
Discovery call — ~30 min
Scope the codebase, the agents operating it, the authority they hold, and the failure modes that concern you most. Thirty minutes is usually enough to decide whether this engagement fits and which tier applies.
You leave with: clear framing of your highest-risk areas and a decision on scope.
Production audit — 3–5 days
Map the seven failure modes against your codebase. Separate launch blockers from deferred improvements. Identify the specific authority and governance gaps.
You leave with: a ranked list of launch blockers and clarity on what is safe to defer.
Hardening and fixes — 5–10 days
Targeted fixes across authority, approval gates, audit capture, error recovery, and documentation. Scoped improvements — not a from-scratch rebuild.
You leave with: implemented controls and documented design decisions your team can reason about.
Launch-readiness handoff — 1 session
Walk through the completed work, the residual risk register, and the production checklist your team inherits. Confirm you can operate the controls without RFE Online in the room.
You leave with: a governed, documented system and a practical checklist your team owns going forward.
Case Study — Composite Engagement
Sterling Fleet: three launch blockers closed, production score lifted from 79 to 84
RFE Online operates a 10-agent production fleet built with AI-assisted tooling. When Sterling — the task-routing and dispatch agent — reached evaluation-ready status, the team ran it against standard automated testing. It scored 79/100 and passed. Three production incidents in the first 24 hours confirmed what the score did not show: evaluation tests correctness, not authority. This engagement maps what that gap costs and what it takes to close it.
Before hardening
Standard evaluation score: 79/100. The agent passed all automated correctness checks. It handled task routing, triggered downstream agents, and wrote to shared storage. There was no approval gate before irreversible dispatch, no circuit breaker on retry, and no enforced write-path boundary. Three production incidents landed in the first 24 hours: two tasks dispatched twice from an uncapped retry loop, one downstream agent writing outside its intended storage scope.
What the audit found
Failure mode 1 — Launch blocker
Unscoped write authority
Sterling could write to any storage path its downstream agents could access. No authority boundary was enforced at the permission layer. Blast radius: all agent-accessible buckets.
Failure mode 4 — Launch blocker
Retry loop without circuit breaker
On dispatch failure the agent retried with exponential back-off but no attempt cap. A transient downstream error produced duplicate task dispatches within the first operating hour.
Failure mode 5 — Launch blocker
No approval gate on irreversible dispatch
Dispatch to certain downstream agents triggers external API calls and email sends that cannot be rolled back. No preview or confirmation step existed before execution.
Four additional findings were deferred: observability coverage (failure mode 6), dependency pinning for two third-party packages (failure mode 7), credential rotation cadence (failure mode 2), and input sanitisation on one low-traffic inbound path (failure mode 3). All four were documented with priority and owner in the risk register and handed off at the close of engagement.
What the hardening fixed
Authority scoping
Write paths constrained to a per-task working directory. Storage access enforced at the permission layer, not the application layer. A mapping of every agent’s actual vs. intended authority was committed to the infrastructure repo.
Retry cap and circuit breaker
Maximum retry count (3) and a circuit-breaker state introduced. After three consecutive failures on a dispatch target, Sterling parks the task and surfaces an alert rather than retrying into failure.
Approval gate on irreversible paths
A dry-run pass added before any dispatch that triggers external side effects. The gate presents a structured preview and requires an explicit proceed signal before execution.
After hardening
84
Production score (from 79 pre-hardening)
65
IRO delta (from 44 pre-hardening — +48%)
0
Production incidents in 30 days post-hardening
3
Launch blockers identified, fixed, and verified
The evaluation score did not move because we fixed bugs. It moved because we constrained authority, closed the retry loop, and introduced the approval gate — none of which are visible to a correctness test.
This case study is a composite. It combines RFE Online’s internal Sterling fleet hardening with failure mode patterns drawn from representative production work. Outcome metrics (79→84 score, IRO delta 44→65) are from the live internal engagement. Timeline and scope figures are generalised across comparable engagements.
RFE vs. building in-house vs. a traditional vendor
Decision-stage buyers ask the same question before they book anything: why not just handle this ourselves, or use an existing security firm? The answer is specific, not marketing.
| What matters to you | Build in-house | Traditional appsec vendor | RFE Online |
|---|---|---|---|
| AI-specific failure-mode coverage | Depends entirely on whether someone on your team knows what to look for. Most security curricula predate agentic stacks. | Generic OWASP checklists and penetration testing frameworks — built for human-written code, not authority-model and approval-gate gaps. | Seven failure modes mapped and fixed in priority order. Framework derived from running a live 10-agent fleet — not theory. |
| Time to first value | Weeks to months, depending on whether your team has bandwidth outside feature delivery. | Sales cycle → scoping → scheduling → report. Typically 6–10 weeks before findings land. | Discovery call to ranked findings: 3–5 business days. Launch blockers identified and prioritised before your sprint ends. |
| Pricing clarity | Opportunity cost only — but the "free" internal work displaces other delivery and carries no accountability SLA. | Day-rate retainer, scope creep risk, and a statement of work that takes a week to negotiate. | Three fixed-scope tiers. Price confirmed at discovery call before any commitment. Refund guarantee if deliverables miss the deadline. |
| Evidence base | No independent benchmark — you are marking your own work. | General experience with enterprise codebases. No published AI-agent production track record. | Sterling fleet: 79 → 84 production score. IRO delta 44 → 65 (+48%). Three launch blockers closed. Zero production incidents in 30 days post-hardening. |
| What your team inherits | Notes, if anyone wrote them down. No structured handoff artefact. | A PDF report. Implementation is your team’s problem; no reusable framework for the next agent you add. | Reusable audit worksheet, ranked risk register, production checklist, and a handoff session. Your team owns the process after engagement ends. |
| What you need to provide | Your own engineers’ time — away from shipping. | Extended access, interviews, and weeks of coordination. | Read-only repo access and one 30-minute discovery call. Async engagement — no dedicated calendar time required beyond the handoff session. |
| Scope of knowledge transfer | Internal — learning stays with whoever did the work. | Findings stay in the vendor’s report. Minimal transfer of framework or reasoning. | Every finding is documented with reasoning. The audit worksheet means your next engineer can apply the same framework without RFE in the room. |
The case for an external specialist comes down to two things: the failure modes are specific to AI-built stacks (not covered by standard security practice), and the time window is short (your agents are approaching production now, not next quarter). If both are true, the in-house path costs more in total than the engagement price — and the vendor path takes longer than you have.
Pricing tiers
Three fixed-scope tiers based on the number of agents and the depth of hardening required. All engagements start with a discovery call to confirm fit before any commitment.
Tier 1
Starter Audit
$2,500 AUD
Single codebase or agent — authority map and risk register
- Discovery call to scope the agent
- Seven failure modes audited
- Blast-radius and authority model map
- Ranked risk register with prioritised fix list
- Reusable audit worksheet
Most common
Tier 2
Hardening Engagement
$7,500 AUD
Full audit plus targeted fixes — discovery through handoff
- Everything in Starter Audit
- Targeted fixes for all launch blockers
- Approval gate and rollback path design
- Audit trail and evidence capture
- Launch-readiness handoff session
- Production checklist your team inherits
Tier 3
Fleet Hardening
$18,000 AUD
3+ agents or codebases — fleet-wide governance architecture
- Everything in Hardening Engagement
- Fleet-wide authority model and permission architecture
- Cross-agent interaction risk assessment
- Shared governance infrastructure design
- Ongoing monitoring and drift detection design
- Fleet production checklist and runbook
All prices in AUD excluding GST. Scope is confirmed at the discovery call before any commitment. Engagements that scope larger or smaller than a tier are quoted separately — use the discovery call to find out.
Who this is for
Built with AI tools
Cursor, Claude Code, Copilot, Codex
You used an AI tool to build or maintain a production codebase. The code works in development. You are not confident it will behave predictably at scale, under adversarial input, or when the model behind it drifts.
Agents with authority
Agents that act, not just generate
Your agent can write files, make API calls, deploy code, process payments, book services, or modify accounts on behalf of a user. A failure is not a UX problem — it is a trust, liability, or compliance problem.
Book a thirty-minute discovery call
Scope the codebase, the agents, the authority model, and the failure modes that concern you most. No obligation before we agree on fit and scope.
Also available: the AI Code Production Hardening masterclass waitlist — if you would rather learn the audit framework and apply it yourself, join the waitlist to shape the curriculum.