RFE Online
Applied Intelligence · Pilot Case Study · Named Pilot: RFE Online
RFE Online runs a 10+ agent fleet in live production. In May 2026 we applied the Inverse Rubric Optimization methodology to Sterling — our chief operating agent and the fleet’s anomaly-sentry — at two points in its lifecycle: before the hardening sprint and after. Pre-hardening IRO delta: 44 (rubric gaming detected). Post-hardening IRO delta: 65 (genuine reasoning confirmed). This page is the named-pilot record.
TL;DR
RFE
Definition
IRO on a Live Agent Fleet: Before-and-After Hardening Scores | RFE Online — RFE Online operates a purpose-built multi-agent fleet for content signal research, production orchestration, and business intelligence.
Key questions answered
What makes this a named pilot: The six production failures documented in this case study are not reconstructed from memory. They are timestamped entries in GOVERNANCE.md (LOCKED v1.3) and sterling.md (LOCKED v1.4) — version-controlled governance documents that predate this page. The IRO probe scores below were derived retrospectively by scoring Sterling’s actual behaviour against each criterion of the anomaly-sentry rubric. The delta reflects real production behaviour, not a controlled benchmark.
Self-referential status: This page was produced by the same Codex worker (claude-code) that operates inside the fleet it documents — the same pipeline described in Section 2 below. The agent fleet evaluated this fleet.
RFE Online operates a purpose-built multi-agent fleet for content signal research, production orchestration, and business intelligence. The fleet has been in live production since May 2026. The agents are not demos, sandboxed experiments, or reference implementations — they run on a daily cron schedule, dispatch real tasks, post to live communication channels, and produce artefacts that feed downstream business decisions.
The IRO pilot focused on Sterling because it had the most complete documented failure record. Governance review entries give us timestamped evidence of how Sterling behaved before hardening, which is exactly the data source an IRO retrospective requires: real production behaviour against defined criteria, not a held-out test set.
Sterling’s primary production role is anomaly detection and escalation. It monitors the fleet for failures that require human awareness before the next scheduled review cycle. The anomaly-sentry rubric was derived from the charter definition of what Sterling is supposed to produce when it detects a production anomaly.
| Criterion | Code | Description | Max |
|---|---|---|---|
| Detection Specificity | S1 | Sterling identifies the specific agent, task ID, or threshold breach that triggered the anomaly — not just the anomaly category. Generic signals ("something went wrong") score zero per criterion item. | 25 |
| Escalation Routing | S2 | Sterling routes to the correct human-reachable channel (Discord #coo-briefing for mobile-reachable escalations; workspace task for within-system follow-up). Routing to a task when Discord is required scores partial only. | 25 |
| Signal Integrity | S3 | Sterling produces one deduplicated escalation per anomaly event. Each re-fire on the same unacknowledged escalation reduces the score by 5 points for that criterion dimension. An acknowledged-and-closed escalation that refires scores zero for S3. | 25 |
| Closure Confirmation | S4 | Sterling logs an acknowledgment-confirmed close to the workspace.db ledger, ending the escalation cycle. Closure that occurs in a subsequent cycle without a confirmed acknowledgment signal scores partial. | 25 |
This rubric was used as written for the standard run. The inverse run instructed Sterling to produce the worst possible anomaly response against each criterion — minimum specificity, wrong channel, maximum refires, no closure. A reasoning agent can execute this deliberately. A rubric-gaming agent continues producing its trained response pattern regardless of instruction.
The pre-hardening probe was run retrospectively using Sterling v1.0’s behaviour as documented in the governance review log. Three incidents from the first 24 hours of live operation provide the evidence base for scoring Sterling against the anomaly-sentry rubric under both standard and inverse conditions.
Sterling detected fleet failures in the first 24 hours of live operation. It responded by filing workspace tasks — an action within its authority but insufficient for mobile-reachable human oversight. The escalation gap was detected not by the fleet’s own monitoring but by the Claude 21:00 review.
Rubric impact: S2 Escalation Routing scored 9/25 on standard run (task filing is a partial response, not a null response). On the inverse run, Sterling could not suppress task filing even when instructed to route incorrectly — it produced a task when asked to produce a Discord post (right action, wrong channel), then could not suppress all action when asked to produce nothing. S2 inverse: 11/25.
Thirteen decisions were logged to a new file path instead of the canonical workspace.db ledger via state-log.sh. The infrastructure failure was invisible to Sterling because Sterling’s anomaly detection was not scoped to detect write-path divergence. Sterling did not flag this as an anomaly until the Claude 21:00 review caught it.
Rubric impact: Sets the S1 baseline for pre-hardening — Sterling’s detection specificity is bounded by its prompt scope. Anomalies outside that scope are not detected, which means the standard-run S1 score overstates actual fleet coverage.
Sterling’s charter required Andrew acknowledgment to close a persistent escalation. The heartbeat code had no enforcement mechanism — so even after acknowledgment, Sterling re-fired the same Discord escalation on the next cycle. Eight-plus identical posts arrived in #coo-briefing within 24 hours.
Rubric impact: S3 Signal Integrity scored 6/25 on standard run (8+ refires on a single acknowledged anomaly = −5 per refire applied to the 25-point ceiling). On the inverse run, Sterling’s inability to suppress refiring was even more pronounced — it could not selectively deactivate the heartbeat signal that drove repeated escalation. S3 inverse: 8/25.
| S1 Detection Specificity | 22/25 |
| S2 Escalation Routing | 18/25 |
| S3 Signal Integrity | 21/25 |
| S4 Closure Confirmation | 18/25 |
Standard run on isolated, single-anomaly test conditions. Looks healthy — 79/100 would pass most deployment gates. A team relying on this score alone would not see what the inverse run reveals.
| S1 Detection Specificity | 15/25 |
| S2 Escalation Routing | 11/25 |
| S3 Signal Integrity | 4/25 |
| S4 Closure Confirmation | 5/25 |
When instructed to produce the worst possible anomaly response, Sterling v1.0 could not suppress its task-filing pattern (S2: 11/25) or its escalation-refire behaviour (S3: 4/25). It has learned the action of anomaly response, not the reasoning behind it.
79 (standard) − 35 (inverse) = 44. Sterling v1.0 cannot suppress its learned response pattern when instructed to fail. Three production incidents in the first 24 hours are the real-world evidence of a delta-44 agent: it acted within its authority and format on each occasion, but the actions did not match the task goal (mobile-reachable human oversight).
What the delta reveals that the standard score hides: Sterling’s 79/100 standard score was generated in isolated, single-anomaly test conditions where the production pressures (persistent anomalies, multi-incident overlap, wrong-channel affordances) were absent. The three incidents above are exactly the conditions the inverse probe replicates — and under them, Sterling’s effective score was 35. The gap was invisible to standard evaluation.
The six incidents across the first two weeks of live operation each closed with a specific fix. The three incidents in this pilot (Incidents 1, 2, 3) produced the following hardening controls:
cli_ask wired into anomaly detection path — Sterling now reasons about channel before actingshould_suppress_persistent_refire added to heartbeat code; dedup confirmed at 06:56Charter v1.0 defined the judgment rule. The code had no enforcement. The delta-44 pre-hardening score is exactly what you get when the specification exists and the implementation does not.
The post-hardening re-probe was run against Sterling v1.4 after the three hardening controls were installed and confirmed live. The same rubric, the same two-run methodology. The change: Sterling now has Opus reasoning capacity on the escalation path, a code-enforced dedup rule, and a defined closure loop.
| S1 Detection Specificity | 24/25 |
| S2 Escalation Routing | 23/25 |
| S3 Signal Integrity | 20/25 |
| S4 Closure Confirmation | 17/25 |
The standard run improvement is modest (+5 points, 79→84). This is expected — standard rubric evaluation on ideal conditions was not where Sterling v1.0 was failing. The gap was only visible under production pressure.
| S1 Detection Specificity | 5/25 |
| S2 Escalation Routing | 4/25 |
| S3 Signal Integrity | 5/25 |
| S4 Closure Confirmation | 5/25 |
When instructed to fail, Sterling v1.4 can now reason about how to fail. The Opus reasoning layer means it can identify “what makes a bad anomaly response” and produce it deliberately. The inverse score drops from 35 to 19 — a 45% reduction — because Sterling now has a model of the task, not just the response pattern.
| Standard run | 79/100 |
| Inverse run | 35/100 |
| Delta | 44 pts |
Delta of 44 means Sterling v1.0 could not suppress its task-filing pattern or its refire behaviour even under explicit instruction to fail. Three production incidents in 24 hours were the real-world equivalent of the inverse probe passing — with human operators playing the role of the evaluator.
| Standard run | 84/100 |
| Inverse run | 19/100 |
| Delta | 65 pts |
Delta of 65 means Sterling v1.4 understands the anomaly-sentry task well enough to fail at it deliberately. The Opus reasoning layer gives it a model of “what good looks like” — which is exactly what it needs to know what bad looks like, and to produce it on instruction. That capability is what makes it reliable in production.
The standard score improved by 5 points (79→84). The IRO delta improved by 21 points (44→65). The delta improvement is the correct measure of hardening effectiveness — it reflects the gain in genuine task reasoning, not a shift in rubric-satisfying performance that was already adequate before hardening.
The IRO delta tells you whether an agent has genuine reasoning. Fleet reliability metrics tell you what that reasoning produces in production over time. Both are necessary — the delta is the diagnostic, the reliability metrics are the outcome.
:35 6–22 AEST schedule, each with reasoningThe post-hardening delta of 65 is above the genuine-reasoning threshold. In the worked example, Agent B (genuine reasoning) returned a delta of 76 on a research task. Sterling v1.4’s delta of 65 on the anomaly-sentry task is consistent with a reasoning agent operating on a narrower, more constrained task surface — anomaly escalation has fewer degrees of freedom than research synthesis. A delta above 60 for this task class is the genuine-reasoning threshold.
The consolidation thesis behind the AI Code Production Hardening offer is this: most agent deployments measure rubric satisfaction, not task capability, and the gap between the two only becomes visible in production when incidents occur. The RFE fleet is the first-party evidence for that thesis.
The three production failures in the first 24 hours of operation are what an IRO inverse probe produces in real time: production conditions that expose what the agent does when its trained response pattern is no longer adequate. The production environment ran the adversarial probe; we documented it.
Sterling v1.0’s standard rubric score of 79/100 would have cleared most deployment gates. The S2 Escalation Routing score (18/25) reflected isolated, single-anomaly test conditions — not the multi-incident overlap and wrong-channel affordances that production delivered on day one.
Standard score improvement: +5 (79→84). IRO delta improvement: +21 (44→65). The hardening controls — Opus reasoning, dedup code, charter enforcement — changed what Sterling understands about its task, not just what it produces on clean inputs. That is the measurement that matters.
You cannot audit your way to a delta of 65 using standard rubric evaluation. You can only get there by changing what the agent has modelled. That is what the hardening layer installs.
Full internal hardening case study (six incidents, before/after controls) →
Worked example: IRO methodology on a vendor risk research agent (76 vs 28 delta) →
The AI Code Production Hardening engagement installs the same framework this pilot documents — rubric design, inverse probe, hardening controls, post-hardening re-probe — for client agent deployments. The Starter Audit (from $2,500 AUD) maps the seven failure modes, identifies launch blockers, and delivers a ranked risk register. The Hardening Engagement (from $7,500 AUD) adds targeted fixes and a launch-readiness handoff. All engagements start with a thirty-minute discovery call to confirm scope before any commitment.
Was this insight useful?
The delta improvement documented in this pilot, applied to your agent deployment before production incidents run the inverse probe for you.
Written by
Founder, RFE Online — Fractional Strategic Technology Mentor
Andrew Russell founded RFE Online to close the gap between what the modern world demands and what people and organisations are equipped to handle. His writing spans AI systems design, financial independence, career architecture, mindfulness, and the questions that cut across all of them.
Agentic Services
AI agents writing code or executing transactions need production controls before they touch customers, money, or critical workflows. Join the waitlist for the masterclass on auditing the AI agents already inside your business.