BUG CLASS (doc: BUGCLASS_INDETERMINATE_OUTCOME_20260713.md): an operation with an external side effect has THREE outcomes — NOT_ATTEMPTED, ATTEMPTED_REFUSED, ATTEMPTED_INDETERMINATE — and rollback is sound only for the first two. Collapsing the third into 'failed' is what orphaned 6 live SHORTs: a post-ack TypeError reached rust_backend's 'except Exception -> synthetic REJECTED -> FSM rollback', which asserted 'no order exists' about an order that was already filled. Telemetry never had a veto; it hijacked the failure channel. FIXES (no new seams, no re-architecture): - venue.py: VenuePostAckError — typed channel meaning THE EFFECT EXISTS. Carries receipt. - bingx_venue submit/submit_async: point-of-no-return fence. Post-ack bookkeeping failures raise VenuePostAckError instead of a bare exception. - rust_backend (BOTH submit paths): catch VenuePostAckError FIRST -> no synthetic REJECT, no rollback. Slot stays working; E-feed FULL_FILL / reconcile settles the truth. LOSSLESS TELEMETRY (HJ: 'DITAv2 exists precisely because seams dropped 40% of inputs'): drop-oldest is data loss and is GONE. Exec path appends O(1) to an unbounded queue and returns. A SEPARATE spiller thread (which never touches the plane, so a wedged plane cannot starve it) parks the backlog above HWM into a durable append-only spool; the publisher replays the spool when the plane recovers. Proven: wedged-forever plane + 200k records -> 0 dropped, 195903 durable on disk, 4096 in memory, 7.4 us/call on the exec path. Lossless AND memory-bounded. Healthy plane: 2000/2000. STILL BROKEN, flagged to codex: the pre-ack branch rolls back on TIMEOUT — but a timeout is the definition of INDETERMINATE (the order may have filled). Same bug class, older, live. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
6.9 KiB
Bug class: indeterminate-outcome collapse
"not done" is NOT the same as "tried and failed" — and neither is "tried, outcome unknown"
Status: doctrine. Codebase-wide sweep assigned to codex. Discovered by: HJ, from the 2026-07-13 orphan incident.
The formal statement
Any operation carrying an external side effect (venue order, cancel, transfer, ClickHouse insert, journal write, shm publish) has three terminal outcomes, not two:
| Outcome | Effect exists? | Is rollback sound? |
|---|---|---|
| NOT_ATTEMPTED — we never issued it | provably no | YES |
| ATTEMPTED_REFUSED — the peer proved it refused (venue error code, validation reject) | provably no | YES |
| ATTEMPTED_INDETERMINATE — issued, outcome unknown | maybe | NO — NEVER |
The bug is collapsing three states into two: treating INDETERMINATE as FAILED, and then applying a rollback that is only sound for the first two.
The invariant that gets violated: the effect (the venue order) and the record (the kernel FSM) are not atomic. Any failure between them leaves them divergent. This is the classic dual-write / write-then-record problem, and it has exactly one correct answer: when you cannot prove the effect did not happen, you must not assert that it didn't. You must reconcile against the authoritative source (the venue).
Unknown is not flat. Unknown is not failed. Unknown is UNKNOWN.
How it presented on 2026-07-13
# bingx_venue.submit_async()
receipt = await backend.submit_intent(legacy) # <-- POINT OF NO RETURN: order is LIVE
events = self._events_from_submit(...) # bookkeeping
self._publish_telemetry(..., order_id=...) # <-- TypeError (signature skew)
return events # never reached
# rust_backend, the caller
try:
emitted_events = await submit_async(intent)
except Exception: # <-- ONE channel for BOTH meanings
-> synthetic REJECTED -> FSM rollback to IDLE
submit_async signals "the venue never took the order" by raising, and signals
"a bug fired somewhere inside me" by raising. Same channel. The caller cannot tell
them apart, so it takes the interpretation written for the timeout case and asserts a
fact it cannot know: no order exists.
The venue kept 6 SHORT positions. The kernel believed it was flat. No SL, TP, MAX_HOLD or
ADVSL can protect a position the kernel does not know it holds — which is why execution
truth (tier B) outranks capital preservation (tier C) in UV_EXEC_PRIORITY_LADDER.
Telemetry never had a "veto". It hijacked the failure channel. It was the bullet;
except Exception -> assume not-done is the gun. Any post-ack line — a log call, a dict
build, an events-builder edge case — would have produced the identical 6 orphans.
THE SECOND INSTANCE — older, still live, and worse
The "safe" pre-ack branch is itself an instance of the same bug:
except Exception as _submit_exc:
# venue.submit() failed (e.g. BingX timeout) ... feed a synthetic REJECTED
A timeout is the definition of INDETERMINATE. The request may have reached BingX and filled. Rolling back to IDLE on a timeout asserts "no order exists" on exactly the evidence that cannot support it. This has been in the code far longer than tonight's skew and produces orphans on every venue timeout.
Only a proven refusal may roll back:
- connection refused / DNS failure before the request was sent → NOT_ATTEMPTED ✔
- venue returned an explicit rejection code (validation, insufficient margin) → REFUSED ✔
- timeout, connection reset, 5xx, unparseable response → INDETERMINATE ✘ must reconcile
The fix pattern (minimally invasive — no new seams, no re-architecture)
- Mark the point of no return. Everything after the side-effect call is fenced; it may not reach the caller through the bare-exception channel.
- Type the channel.
VenuePostAckError(added,venue.py) carries the receipt and means the effect exists. Pre-ack failures keep raising normally. - Callers branch on it.
rust_backendcatchesVenuePostAckErrorfirst: no synthetic REJECT, no rollback. Leave the slot in its working state and let execution truth settle it — the E-feed deliversFULL_FILLfrom the account stream independently. - Non-critical code cannot use the failure channel at all. Telemetry is off the exec path entirely (bounded-memory, lossless spill lane).
- Indeterminate ⇒ reconcile. Never rollback. Query the venue; it is the authority.
No architectural change. No new seam. One typed exception, one fence, one branch.
The codebase-wide sweep (codex)
Search pattern: any try: whose body performs an external side effect and whose
except assumes the effect did not happen (rollback / mark-failed / synthesize-reject /
blind-retry / "assume flat" / return default).
Known/suspect sites:
bingx_venue.submit / submit_async✔ fixedbingx_venue.cancel / cancel_async— same shape, unauditedrust_backendpre-ack timeout branch — BUG, live (see above)EKBridge.mirror_future,e_feed._publish_converted— failure of the E→K mirrorch_writer/ journal lane inserts — "insert failed" vs "insert maybe landed" (duplicates)ExecutionRoutermaker/taker requote + amend paths- promotion bridge (
uv.promotion) receipt handling - any
except Exception: pass/return None/return Falseafter an I/O call
Every site must answer one question: can this except-branch PROVE the effect did not happen? If no → it must not assert absence. Reconcile or escalate to UNKNOWN.
Tests demanded (nuclear grade)
- Fault-injection matrix. For each side-effecting call: inject an exception at every statement after the effect (parametrize over injection index) and assert the kernel never concludes the effect is absent. This is the test that would have caught tonight's bug at any of the post-ack lines, not just the telemetry one.
- Timeout is indeterminate. Venue times out after the exchange filled → assert NO rollback, assert reconcile is triggered, assert the position is adopted, not orphaned.
- Property/fuzz. Random interleavings of {ack, no-ack, timeout, post-ack raise,
partial fill, duplicate ack}; invariant:
kernel_believes_flat ⇒ venue_is_flat. This invariant is the whole ballgame; it must never be violated for any interleaving. - Signature-skew guard.
inspect.signature().bind()over every call site of every telemetry/journal helper — a pure binding test that survives re-vendoring. - Losslessness. Telemetry/journal lanes: wedge the sink, flood N records, assert
dropped == 0andon_disk + in_memory == N, and assert memory stays bounded. - Mutation litmus. Remove each fence → the corresponding test must go RED. If removing the guard keeps the suite green, the test is theatre.