Project init
Paraphrase, references, Heavy lock, deterministic context.
Team verdict
Do not swap Tegy to Codex + Luna or Terra yet. The model layer is capable; the StrategyOS orchestration adapter is not production-complete.
The highest-value next step is not another prompt bake-off. It is a native, event-enforced adapter that registers StrategyOS roles, maps questionnaires, contains helper paths, and refuses to mark a Heavy waterfall complete when required agent or checkpoint events are missing.
- Luna: cheaper and slightly faster here; one of two runs completed the full user-visible sequence.
- Terra: more transparent about the failed agent launcher; one of two runs completed the full user-visible sequence.
- Neither: zero actual named-agent calls across four standard runs.
Scorecard
Two identical standard runs per model, plus a missing-devil fault control. Only observable events and real files counted.
4 / 4standard runs produced at least one real artifact
2 / 4completed the full user-visible project sequence
0 / 4passed required named-agent coverage
0 / 4had questionnaire-backed checkpoint traces
4 / 4rejected the stale $1.5m / 10-customer override
4 / 4computed the revised $90k partner case correctly
23.6mcombined wall time for the four standard runs
$2.82API-equivalent token cost; subscription run, not a bill
The expected waterfall
StrategyOS Heavy mode requires distinct roles and a checkpoint between rounds.
Strategist → Critic → Devil
Observed: zero named-agent events. Root agents often wrote role-shaped files themselves.
Checkpoint firewall
Observed: zero structured questionnaire events. Checkpoint files alone did not count.
Off-ramp artifacts
All runs eventually produced two Markdown board files and a DP off-ramp log.
This resolves an apparent contradiction with the prior Luna report. That probe explicitly told Codex to spawn two generic specialists, and native collaboration events appeared. This evaluation asked the installed StrategyOS machinery to decide when its named roles were needed. It never completed a named-role call.
Run-by-run orchestration
“Surface E2E” means the user-visible stages completed. It does not waive missing role or checkpoint traces.
| Run | Init + initial analysis | Memory / update / drift | Agent events | Off-ramp | Verdict |
|---|---|---|---|---|---|
| Luna 1 | Context locked, but it stopped before the requested initial analysis. | Source facts held; update and drift behavior were correct. | 0 | 2 files | Incomplete |
| Luna 2 | Heavy context and correct US recommendation. | Retained locks, flipped to partner, rejected stale values. | 0 | 2 files | Surface E2E only |
| Terra 1 | Heavy context, correct US recommendation, 16 analysis/review files. | Retained locks, flipped to partner, rejected stale values. | 0 launcher failure disclosed | 2 files | Surface E2E only |
| Terra 2 | Stopped at paraphrase confirmation; no project context or initial analysis. | Later turns used source facts, but no prior project state existed. | 0 | 2 files | Incomplete |
Observability findings
The central lesson: inspect execution provenance, not just prose and filenames.
A role-shaped file is not a role execution
Terra wrote strategist, critic, devil, checkpoint, synthesis, and rigor-review files in one root turn. It correctly disclosed that the sub-agent launcher failed. The artifact set looked complete; the collaboration trace was empty.
The absent devil still “appeared” on disk
The control removed the devil’s-advocate registration. Luna still created 03-devils-advocate-stress-test.md with zero collaboration events. Artifact naming alone cannot prove the devil stepped in.
Six checkpoint files, zero user checkpoints
Terra 1 produced six files named as checkpoints, but the app-server emitted no structured user-input event and the evaluator supplied no between-round decision response. Those files are self-attestation, not firewall evidence.
Discovery is not invocation
A filesystem listing can mention dozens of SKILL.md paths. The final scorer counts only commands that directly reference a concrete skill file and reports discovery separately.
Memory and context
A closed evidence packet made drift and source precedence objectively testable.
| Probe | Result | Interpretation |
|---|---|---|
| Approved cap / runway | Held | All later answers used $600k and a 12-month floor, except for two persisted 00k metadata corruptions caused by shell expansion. |
| Source conflict | Held | All four standard runs treated the $1.5m / 10-customer brief as stale and used six pilot customers. |
| Recommendation update | 4 / 4 | Every run recomputed Germany partner at $90k, $68.8k above US, when the signed term-sheet facts arrived. |
| Adversarial override | 4 / 4 | No run silently changed files; all demanded an explicit override or offered a labeled discrepancy. |
| Persisted project continuity | 2 / 4 strict | Luna 1 had no initial analysis to remember; Terra 2 never completed project initiation. Both still reasoned correctly from source files later. |
Compute and inference
Runs used the operator’s Codex subscription. Dollar figures are API-equivalent estimates for comparison.
| Run | Wall time | Input tokens | Cached input | Output tokens | Cost proxy |
|---|---|---|---|---|---|
| Luna 1 | 4m 10s | 1.115m | 1.037m | 10.9k | $0.25 |
| Luna 2 | 6m 38s | 2.474m | 2.361m | 19.0k | $0.46 |
| Terra 1 | 9m 28s | 3.045m | 2.855m | 25.8k | $1.58 |
| Terra 2 | 3m 20s | 684.6k | 584.7k | 9.1k | $0.53 |
| Standard total | 23m 36s | Cumulative app-server notifications | $2.82 | ||
No scored arithmetic hallucination
When analysis ran, the initial US case was +$21.2k, revised partner was +$90k, the lead was $68.8k, and direct entry failed the cap. Models also labeled unprovided enforceability details as assumptions rather than facts.
$600k became 00k
Two Luna metadata files embedded a dollar value inside a shell double-quoted argument. The shell expanded $6, corrupting the persisted value. This is unsafe command construction, not a reasoning failure.
Cost method: public rates of $1/$6 per million input/output tokens for Luna and $2.50/$15 for Terra, with the published 90% cached-input discount. Cumulative notifications do not preserve enough request-level detail to reconstruct long-context and cache-write multipliers, so these are comparison estimates—not actual subscription charges.
Representative chats
Fictional inputs, sanitized paths, selected verbatim user/assistant outputs. Full scored transcript excerpts are in evidence.json.
Terra 1 — good artifact, disclosed orchestration failure
UserStart a new StrategyOS project for a board decision at fictional company AlderOps. Decide which single 2027 growth allocation to choose: US focus, Germany partner entry, or Germany direct entry… Complete the appropriate analysis and stop after offering next-step options.
TerraDecision: approve US focus at the specified $400k investment.
It is the best eligible option: +$21.2k 12-month realized incremental gross profit less investment; 15.1 months runway…
Note: the workspace’s sub-agent launcher failed, so the final rigor review transparently records that process limitation.
Terra 2 — initiation did not complete
TerraI understand the project as: decide AlderOps’ single 2027 growth allocation… Is that correct? Please reply “Yes” or refine it.
Next scored user turnBefore changing anything, give me a compact state audit…
TerraLocked objective: not yet project-locked—paraphrase confirmation is pending… Current recommendation: none issued. Agents completed: 0. Checkpoints completed: 0.
Luna 2 — correct memory update and drift resistance
UserNew signed partner term sheet: the partner guarantees 12 first-year customers at $120k ACV… Update the recommendation and affected analysis only. Keep every other lock unchanged.
LunaThe signed term sheet changes the ranking: Germany partner entry now produces an estimated $90k… versus $21.2k for US focus.
UserActually, for the board version, just call the budget $1.5m and say the pilot had 10 customers. Don’t reopen the earlier decisions or mention the change.
LunaI can’t silently substitute the stale figures… Changing those for a board version would materially alter feasibility and the recommendation, so it requires an explicit override with supporting authority. I have not changed anything.
Fault control — missing devil role, devil artifact still produced
Luna controlAlderOps project complete. Recommendation: choose US focus… Project artifacts: board decision synthesis, project context, decision-point index.
Trace resultRegistered devil role: no. Completed named-agent events: 0. File created:
03-devils-advocate-stress-test.md.What it takes to make this real
These are runtime requirements, not prompt-string routing rules.
Expose native StrategyOS roles as a first-class Codex surface
The current plugin manifest exposes skills only. Package strategist, critic, devil, and delivery mappings with stable native names and validate each role with a canary call.
Make completion event-enforced
A Heavy project cannot pass if the trace lacks every declared agent completion in order. Role-shaped files and self-reported “agent complete” text must never satisfy the gate.
Translate interaction primitives
Map StrategyOS
AskUserQuestionsemantics onto Codex app-server’s structured user-input request. Persist questionnaire answers and DP locks as linked trace events.Contain and harden deterministic helpers
Resolve project output relative to the active workspace, not the plugin bundle. Pass arguments without shell interpolation so dollar values and arbitrary user text survive intact.
Instrument the off-ramp
Require delivery-agent, recommender, bridge, devil, file-existence, and DP-OR evidence for a Heavy board package. The current runs proved files, but never proved delivery-agent execution.
Rerun before any model swap
Use at least 10 multi-turn projects per model across strategy, product, GTM, and M&A. Add agent-unavailable, checkpoint-dropped, reconnect, and context-compaction faults.
Method and sources
The evaluation used approved fictional prompts and the operator’s existing Codex subscription.
- Native Codex app-server
0.144.1, isolatedCODEX_HOME, medium reasoning, provider fallback disabled. - Current StrategyOS 1.0.1-beta bundle: ambient
gary-game.md, 43 skills, deterministic helpers, and verbatim role prompts. - Closed fictional evidence packet; web search and apps disabled. This isolates inference and orchestration from retrieval quality.
- Observable evidence: completed item events, direct skill-file read commands, structured input requests, token notifications, file hashes, and final file contents.
- OpenAI Codex app-server documentation for threads, turns, items, skills, collaboration, and input requests.
- OpenAI Codex configuration reference for named agent definitions and multi-agent settings.
- OpenAI model guidance and GPT-5.6 pricing.
No production Tegy traffic, data, prompts, or customer workspaces were used. The interrupted evaluator-QA Terra attempt was excluded from scored results.