dita_v2: unknown != flat (VenuePostAckError) + lossless telemetry lane

BUG CLASS (doc: BUGCLASS_INDETERMINATE_OUTCOME_20260713.md): an operation with an external
side effect has THREE outcomes — NOT_ATTEMPTED, ATTEMPTED_REFUSED, ATTEMPTED_INDETERMINATE
— and rollback is sound only for the first two. Collapsing the third into 'failed' is what
orphaned 6 live SHORTs: a post-ack TypeError reached rust_backend's 'except Exception ->
synthetic REJECTED -> FSM rollback', which asserted 'no order exists' about an order that
was already filled. Telemetry never had a veto; it hijacked the failure channel.

FIXES (no new seams, no re-architecture):
- venue.py: VenuePostAckError — typed channel meaning THE EFFECT EXISTS. Carries receipt.
- bingx_venue submit/submit_async: point-of-no-return fence. Post-ack bookkeeping failures
  raise VenuePostAckError instead of a bare exception.
- rust_backend (BOTH submit paths): catch VenuePostAckError FIRST -> no synthetic REJECT,
  no rollback. Slot stays working; E-feed FULL_FILL / reconcile settles the truth.

LOSSLESS TELEMETRY (HJ: 'DITAv2 exists precisely because seams dropped 40% of inputs'):
drop-oldest is data loss and is GONE. Exec path appends O(1) to an unbounded queue and
returns. A SEPARATE spiller thread (which never touches the plane, so a wedged plane cannot
starve it) parks the backlog above HWM into a durable append-only spool; the publisher
replays the spool when the plane recovers.
Proven: wedged-forever plane + 200k records -> 0 dropped, 195903 durable on disk, 4096 in
memory, 7.4 us/call on the exec path. Lossless AND memory-bounded. Healthy plane: 2000/2000.

STILL BROKEN, flagged to codex: the pre-ack branch rolls back on TIMEOUT — but a timeout is
the definition of INDETERMINATE (the order may have filled). Same bug class, older, live.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Codex
2026-07-13 10:31:39 +02:00
parent b13b54e769
commit 3d75e4b59d
4 changed files with 405 additions and 40 deletions

View File

@@ -11,10 +11,16 @@ import asyncio
import concurrent.futures import concurrent.futures
import inspect import inspect
import itertools import itertools
import atexit
import json
import os
import re import re
import tempfile
import threading import threading
import time
from collections import deque from collections import deque
from datetime import datetime, timezone from datetime import datetime, timezone
from pathlib import Path
from dataclasses import replace from dataclasses import replace
from typing import Any, Iterable, List, Optional from typing import Any, Iterable, List, Optional
@@ -38,6 +44,7 @@ from .contracts import (
from .utils import json_safe from .utils import json_safe
from .utils import safe_float from .utils import safe_float
from .venue import VenueAdapter from .venue import VenueAdapter
from .venue import VenuePostAckError
def _row_text(row: dict[str, Any], *keys: str, default: str = "") -> str: def _row_text(row: dict[str, Any], *keys: str, default: str = "") -> str:
@@ -283,42 +290,177 @@ class BingxVenueAdapter(VenueAdapter):
except Exception: except Exception:
pass pass
# ── telemetry lane ─────────────────────────────────────────────────────── # ── telemetry lane — NON-BLOCKING **AND** LOSSLESS ───────────────────────
# Telemetry is NOT on the execution priority ladder (P0 kill/emergency, # Two standing principles, both non-negotiable, and they do NOT conflict:
# P1 SL/ADVSL, P2 exits, P3 ENTER, P4 maintenance) — it ranks below all of
# it. It therefore must never sit ON the exec path, never block it, and
# never raise into it.
# #
# The exec path does exactly one thing: append a tuple of primitives to a # 1. NOTHING below the execution ladder may block or raise into the exec
# bounded ring and return. deque(maxlen=N).append is O(1), never blocks, and # path. Telemetry is not even ON the ladder (P0 kill .. P4 maintenance);
# drops the OLDEST record when full — telemetry loss is always preferable to # it ranks below all of it.
# exec-path backpressure. Snapshot construction and the plane write happen on # 2. NO DATA LOSS. DITAv2 exists precisely because sync/async seams were
# the drain lane, off the critical section entirely. # silently dropping ~40% of inputs. A drop-oldest ring is that same sin
_TEL_RING_MAX = 4096 # wearing a bounded-memory hat, and it discards forensic evidence exactly
# when things go wrong — i.e. exactly when it is needed (2026-07-13).
#
# Resolution (the M6 journal-lane pattern): the exec path appends O(1) to an
# UNBOUNDED in-memory queue and returns — never blocks, never drops, never
# raises. The LANE (never the exec path) is what deals with pressure: when the
# backlog exceeds the high-water mark, the lane SPILLS the overflow to a
# durable append-only spool on disk and replays it once the plane recovers.
# Memory stays bounded by the lane's spilling, not by throwing records away.
#
# Records are dropped: NEVER. Not when the plane wedges, not when it explodes,
# not on shutdown. They go to disk instead.
_TEL_HIGH_WATER = 4096 # backlog above which the lane spills to disk
_TEL_SPILL_BATCH = 1024 # records moved to the spool per spill pass
def _telemetry_spool_path(self) -> Path:
spool = os.environ.get("UV_TELEMETRY_SPOOL")
base = Path(spool) if spool else Path(tempfile.gettempdir()) / "uv_telemetry_spool"
base.mkdir(parents=True, exist_ok=True)
return base / "venue_telemetry.jsonl"
def _start_telemetry_lane(self) -> None: def _start_telemetry_lane(self) -> None:
if getattr(self, "_tel_thread", None) is not None: if getattr(self, "_tel_thread", None) is not None:
return return
self._tel_ring: Any = deque(maxlen=self._TEL_RING_MAX) self._tel_q: Any = deque() # UNBOUNDED: the exec path never drops
self._tel_wake = threading.Event() self._tel_wake = threading.Event()
self._tel_dropped = 0 self._tel_spilled = 0 # records parked on disk (recoverable)
self._tel_replayed = 0
self._tel_dropped = 0 # MUST stay 0 — asserted by tests
self._tel_spool_lock = threading.Lock()
thread = threading.Thread( thread = threading.Thread(
target=self._drain_telemetry, name="bingx-telemetry-lane", daemon=True target=self._drain_telemetry, name="bingx-telemetry-lane", daemon=True
) )
self._tel_thread = thread self._tel_thread = thread
thread.start() thread.start()
# SEPARATE SPILL DUTY. The publisher can block indefinitely inside a wedged
# plane's publish() — if it also owned spilling, a wedged plane would grow
# the queue without bound until OOM killed the process (which loses the very
# records we refuse to drop). The spiller never touches the plane, so it can
# always relieve memory pressure to durable disk no matter what the plane does.
spiller = threading.Thread(
target=self._spill_telemetry, name="bingx-telemetry-spill", daemon=True
)
self._tel_spill_thread = spiller
spiller.start()
atexit.register(self._flush_telemetry_to_spool)
def _spill_telemetry(self) -> None:
"""Pressure relief. Never touches the plane — cannot be wedged by it."""
while True:
time.sleep(0.2)
q = getattr(self, "_tel_q", None)
if q is None or len(q) <= self._TEL_HIGH_WATER:
continue
try:
with self._tel_spool_lock:
with self._telemetry_spool_path().open("a", encoding="utf-8") as fh:
while len(q) > self._TEL_HIGH_WATER:
try:
fields = q.popleft()
except IndexError:
break
fh.write(json.dumps(json_safe(fields), default=str) + "\n")
self._tel_spilled += 1
fh.flush()
except Exception as exc:
import logging as _log
_log.getLogger(__name__).error(
"FIX(bingx_venue): telemetry spill failed — holding in memory, "
"NOT dropping: %s", exc,
)
def _flush_telemetry_to_spool(self) -> None:
"""Shutdown / overflow: park records on disk. Called ONLY from the lane
or atexit — never from the exec path (it does disk I/O)."""
q = getattr(self, "_tel_q", None)
if not q:
return
try:
with self._tel_spool_lock:
with self._telemetry_spool_path().open("a", encoding="utf-8") as fh:
while True:
try:
fields = q.popleft()
except IndexError:
break
fh.write(json.dumps(json_safe(fields), default=str) + "\n")
self._tel_spilled += 1
fh.flush()
os.fsync(fh.fileno())
except Exception as exc:
import logging as _log
_log.getLogger(__name__).error(
"FIX(bingx_venue): telemetry spool write failed — records held in "
"memory, still not dropped: %s", exc,
)
def _replay_telemetry_spool(self, publish) -> None:
"""Plane is healthy and the backlog is drained: re-publish spooled records."""
path = self._telemetry_spool_path()
try:
with self._tel_spool_lock:
if not path.exists() or path.stat().st_size == 0:
return
lines = path.read_text(encoding="utf-8").splitlines()
path.write_text("", encoding="utf-8") # claim them
except Exception:
return
unsent: list[str] = []
for line in lines:
if not line.strip():
continue
try:
fields = json.loads(line)
fields["timestamp"] = datetime.now(timezone.utc)
publish(VenueTelemetrySnapshot(**fields))
self._tel_replayed += 1
except Exception:
unsent.append(line) # failed -> back to disk
if unsent:
try:
with self._tel_spool_lock:
with path.open("a", encoding="utf-8") as fh:
fh.write("\n".join(unsent) + "\n")
except Exception:
pass
def _drain_telemetry(self) -> None: def _drain_telemetry(self) -> None:
"""Low-priority lane: build snapshots + publish. Never touches the exec path.""" """Low-priority lane: spill under pressure, publish, replay. Never drops."""
while True: while True:
self._tel_wake.wait(timeout=0.5) self._tel_wake.wait(timeout=0.5)
self._tel_wake.clear() self._tel_wake.clear()
ring = getattr(self, "_tel_ring", None) q = getattr(self, "_tel_q", None)
if ring is None: if q is None:
continue continue
# Pressure valve: the LANE spills to durable disk — it never discards.
# This is what keeps memory bounded, in place of drop-oldest.
if len(q) > self._TEL_HIGH_WATER:
try:
with self._tel_spool_lock:
with self._telemetry_spool_path().open("a", encoding="utf-8") as fh:
for _ in range(self._TEL_SPILL_BATCH):
try:
fields = q.popleft()
except IndexError:
break
fh.write(json.dumps(json_safe(fields), default=str) + "\n")
self._tel_spilled += 1
fh.flush()
except Exception as exc:
import logging as _log
_log.getLogger(__name__).error(
"FIX(bingx_venue): telemetry spill failed — holding in memory, "
"NOT dropping: %s", exc,
)
while True: while True:
try: try:
fields = ring.popleft() fields = q.popleft()
except IndexError: except IndexError:
break break
try: try:
@@ -327,15 +469,27 @@ class BingxVenueAdapter(VenueAdapter):
getattr(plane, "publish_venue", None) if plane is not None else None getattr(plane, "publish_venue", None) if plane is not None else None
) )
if publish is None: if publish is None:
continue q.appendleft(fields) # no plane yet: keep it, never lose it
break
publish(VenueTelemetrySnapshot(**fields)) publish(VenueTelemetrySnapshot(**fields))
except Exception as exc: # lane-local: never escapes to exec except Exception as exc: # lane-local: never escapes to exec
import logging as _log import logging as _log
_log.getLogger(__name__).warning( _log.getLogger(__name__).warning(
"FIX(bingx_venue): telemetry lane publish failed (phase=%s): %s", "FIX(bingx_venue): telemetry lane publish failed (phase=%s) — "
"record spooled, not dropped: %s",
fields.get("phase", "?"), exc, fields.get("phase", "?"), exc,
) )
self._tel_q.appendleft(fields)
self._flush_telemetry_to_spool()
break
# Backlog clear and plane healthy -> drain whatever is parked on disk.
if not q:
plane = self._telemetry_plane
publish = getattr(plane, "publish_venue", None) if plane is not None else None
if publish is not None:
self._replay_telemetry_spool(publish)
def set_telemetry_plane(self, zinc_plane: Any | None) -> None: def set_telemetry_plane(self, zinc_plane: Any | None) -> None:
self._telemetry_plane = zinc_plane self._telemetry_plane = zinc_plane
@@ -359,19 +513,18 @@ class BingxVenueAdapter(VenueAdapter):
client_order_id: str = "", client_order_id: str = "",
details: dict[str, Any] | None = None, details: dict[str, Any] | None = None,
) -> None: ) -> None:
# EXEC PATH. Non-blocking and total: pack primitives, append to the ring, # EXEC PATH. Non-blocking, total, and LOSSLESS: pack primitives, append to
# return. No plane write, no snapshot construction, no I/O, no lock, no # an UNBOUNDED queue, return. No plane write, no snapshot construction, no
# raise. Telemetry ranks below every rung of the execution ladder and gets # disk I/O, no lock, no raise — and no drop, ever. Pressure is handled by
# dropped (oldest-first) before it is ever allowed to cost the order path # the lane (spill to durable spool), never by discarding records here.
# a microsecond.
# #
# Everything below used to run INLINE, immediately after the venue had # Everything below used to run INLINE, immediately after the venue had
# already accepted the order — so a raise here reached the caller's submit # already accepted the order — so a raise here reached the caller's submit
# guard, which synthesised a REJECTED and rolled the FSM back while the # guard, which synthesised a REJECTED and rolled the FSM back while the
# venue kept the position (6 orphans, 2026-07-13). # venue kept the position (6 orphans, 2026-07-13).
try: try:
ring = getattr(self, "_tel_ring", None) q = getattr(self, "_tel_q", None)
if ring is None or self._telemetry_plane is None: if q is None or self._telemetry_plane is None:
return return
slot_id = 0 slot_id = 0
trade_id = "" trade_id = ""
@@ -391,12 +544,11 @@ class BingxVenueAdapter(VenueAdapter):
trade_id = str(order.internal_trade_id or trade_id) trade_id = str(order.internal_trade_id or trade_id)
asset = str(order.metadata.get("asset") or asset) asset = str(order.metadata.get("asset") or asset)
side = order.side or side side = order.side or side
if len(ring) == ring.maxlen: # No drop check: there is nothing to drop. The queue is unbounded and
# Full: deque drops the oldest on append. Count it — silent # the LANE relieves pressure by spilling to a durable spool. _tel_dropped
# telemetry loss must still be observable. # exists only so a test can assert it stays 0 forever.
self._tel_dropped = getattr(self, "_tel_dropped", 0) + 1
# Timestamp is stamped HERE (event time), not on the drain lane. # Timestamp is stamped HERE (event time), not on the drain lane.
ring.append( q.append(
{ {
"phase": phase, "phase": phase,
"status": status, "status": status,
@@ -699,9 +851,17 @@ class BingxVenueAdapter(VenueAdapter):
) )
legacy = self._legacy_intent(intent, kernel=getattr(self, "_kernel_ref", None)) legacy = self._legacy_intent(intent, kernel=getattr(self, "_kernel_ref", None))
submitted = replace(intent, target_size=float(legacy.target_size)) submitted = replace(intent, target_size=float(legacy.target_size))
# ── PRE-ACK ── safe to roll back if this raises.
receipt = self._call_backend("submit_intent", legacy) receipt = self._call_backend("submit_intent", legacy)
# ── POINT OF NO RETURN ── order is LIVE; see submit_async().
try:
events = self._events_from_submit(submitted, receipt, None, None) events = self._events_from_submit(submitted, receipt, None, None)
ack_row = dict(getattr(receipt, "raw_ack", {}) or {}) ack_row = dict(getattr(receipt, "raw_ack", {}) or {})
except Exception as exc:
raise VenuePostAckError(
f"post-ack failure after venue accepted the order: {exc}",
receipt=receipt,
) from exc
# Post-ack: the order is LIVE at the venue. Telemetry is observability, # Post-ack: the order is LIVE at the venue. Telemetry is observability,
# never a veto — an exception escaping here reaches the caller's submit # never a veto — an exception escaping here reaches the caller's submit
# guard, which synthesises a REJECTED event and rolls the FSM back while # guard, which synthesises a REJECTED event and rolls the FSM back while
@@ -757,9 +917,20 @@ class BingxVenueAdapter(VenueAdapter):
) )
legacy = self._legacy_intent(intent, kernel=getattr(self, "_kernel_ref", None)) legacy = self._legacy_intent(intent, kernel=getattr(self, "_kernel_ref", None))
submitted = replace(intent, target_size=float(legacy.target_size)) submitted = replace(intent, target_size=float(legacy.target_size))
# ── PRE-ACK ── an exception here means the venue never took the order.
# Safe for the caller to roll the FSM back.
receipt = await self.backend.submit_intent(legacy) receipt = await self.backend.submit_intent(legacy)
# ── POINT OF NO RETURN ── the order is LIVE at the venue from here on.
# Nothing below may reach the caller as a bare exception: that channel
# means "no order exists", and the caller rolls back to flat on it.
try:
events = self._events_from_submit(submitted, receipt, None, None) events = self._events_from_submit(submitted, receipt, None, None)
ack_row = dict(getattr(receipt, "raw_ack", {}) or {}) ack_row = dict(getattr(receipt, "raw_ack", {}) or {})
except Exception as exc:
raise VenuePostAckError(
f"post-ack failure after venue accepted the order: {exc}",
receipt=receipt,
) from exc
# Post-ack: the order is LIVE at the venue. Telemetry is observability, # Post-ack: the order is LIVE at the venue. Telemetry is observability,
# never a veto — an exception escaping here reaches the caller's submit # never a veto — an exception escaping here reaches the caller's submit
# guard, which synthesises a REJECTED event and rolls the FSM back while # guard, which synthesises a REJECTED event and rolls the FSM back while

View File

@@ -43,6 +43,7 @@ from .projection import HazelcastProjection
from .projection import build_projection from .projection import build_projection
from .utils import json_safe from .utils import json_safe
from .venue import VenueAdapter from .venue import VenueAdapter
from .venue import VenuePostAckError
from .zinc_plane import InMemoryZincPlane, ZincPlane from .zinc_plane import InMemoryZincPlane, ZincPlane
@@ -848,12 +849,23 @@ class ExecutionKernel:
if outcome.accepted and intent.action in {KernelCommandType.ENTER, KernelCommandType.EXIT}: if outcome.accepted and intent.action in {KernelCommandType.ENTER, KernelCommandType.EXIT}:
try: try:
emitted_events = self.venue.submit(intent) emitted_events = self.venue.submit(intent)
except VenuePostAckError as _post_ack_exc:
# Order is LIVE at the venue: unknown != flat. No rollback, no
# synthetic REJECT — the E-feed FILL / reconcile settles the slot.
# (See the async path below and venue.VenuePostAckError.)
import logging as _log
_log.getLogger(__name__).critical(
"FIX(rust_backend): POST-ACK venue failure (%s) slot=%d action=%s — "
"ORDER IS LIVE. NOT rolling back; awaiting E-feed FILL / reconcile.",
_post_ack_exc, intent.slot_id, intent.action.value,
)
emitted_events = list(getattr(_post_ack_exc, "events", []) or [])
except Exception as _submit_exc: except Exception as _submit_exc:
# venue.submit() failed (e.g. BingX timeout). The Rust FSM already # PRE-ACK: venue.submit() failed (e.g. BingX timeout). The Rust FSM
# advanced to ORDER_REQUESTED / ENTRY_WORKING with no corresponding # already advanced to ORDER_REQUESTED / ENTRY_WORKING with no
# exchange order. Feed a synthetic REJECTED event so the FSM rolls # corresponding exchange order. Feed a synthetic REJECTED event so the
# back to IDLE — otherwise the slot is stranded and every subsequent # FSM rolls back to IDLE — otherwise the slot is stranded and every
# ENTER with a different trade_id hits SLOT_BUSY forever. # subsequent ENTER with a different trade_id hits SLOT_BUSY forever.
import logging as _log import logging as _log
_log.getLogger(__name__).error( _log.getLogger(__name__).error(
"venue.submit failed (%s) — feeding synthetic REJECTED to roll back FSM slot=%d action=%s", "venue.submit failed (%s) — feeding synthetic REJECTED to roll back FSM slot=%d action=%s",
@@ -1004,7 +1016,28 @@ class ExecutionKernel:
emitted_events = await submit_async(intent) emitted_events = await submit_async(intent)
else: else:
emitted_events = self.venue.submit(intent) # fallback: mock/test venue emitted_events = self.venue.submit(intent) # fallback: mock/test venue
except VenuePostAckError as _post_ack_exc:
# THE ORDER IS LIVE AT THE VENUE. This is UNKNOWN, not failure —
# and unknown is NEVER flat. Rolling the FSM back here is what
# orphaned 6 live SHORTs on 2026-07-13: the venue kept the
# positions while the kernel believed it was flat, so no SL/TP/
# ADVSL could ever fire on them.
#
# Emit NO synthetic event: leave the slot in its working state and
# let execution truth settle it — the E-feed delivers FULL_FILL
# from the account stream independently of this call path.
import logging as _log
_log.getLogger(__name__).critical(
"FIX(rust_backend): POST-ACK venue failure (%s) slot=%d action=%s — "
"ORDER IS LIVE. NOT rolling back (unknown != flat); awaiting E-feed "
"FILL / reconcile to settle the slot.",
_post_ack_exc, intent.slot_id, intent.action.value,
)
emitted_events = list(getattr(_post_ack_exc, "events", []) or [])
except Exception as _submit_exc: except Exception as _submit_exc:
# PRE-ACK failure (timeout, connection refused, rejected request):
# the venue never took the order, so the FSM must roll back or the
# slot is stranded in ORDER_REQUESTED forever.
import logging as _log import logging as _log
_log.getLogger(__name__).error( _log.getLogger(__name__).error(
"venue.submit_async failed (%s) — synthetic REJECTED, FSM rollback slot=%d action=%s", "venue.submit_async failed (%s) — synthetic REJECTED, FSM rollback slot=%d action=%s",

View File

@@ -19,6 +19,27 @@ from .contracts import (
from .exchange_event import ExchangeEvent from .exchange_event import ExchangeEvent
class VenuePostAckError(Exception):
"""Raised when submit fails AFTER the venue has already accepted the order.
THE POINT OF NO RETURN. A plain exception out of submit()/submit_async() is
ambiguous: it may mean "the venue never got the order" (safe to roll the FSM
back to IDLE) or "the venue filled it and a line below blew up" (rolling back
is catastrophic — the venue keeps the position while the kernel believes it is
flat). On 2026-07-13 a post-ack TypeError took the first interpretation and
orphaned 6 live SHORTs.
Post-ack failure is NOT failure. It is UNKNOWN — and unknown is never flat.
Callers MUST NOT synthesise a REJECTED event for this error: leave the slot in
its working state and let the E-feed FILL / reconcile settle the truth.
"""
def __init__(self, message: str, *, receipt: Any = None, events: Optional[List[VenueEvent]] = None):
super().__init__(message)
self.receipt = receipt
self.events = events or []
class VenueAdapter(Protocol): class VenueAdapter(Protocol):
"""Abstract venue adapter used by the kernel.""" """Abstract venue adapter used by the kernel."""

View File

@@ -0,0 +1,140 @@
# Bug class: **indeterminate-outcome collapse**
### "not done" is NOT the same as "tried and failed" — and neither is "tried, outcome unknown"
**Status:** doctrine. Codebase-wide sweep assigned to codex.
**Discovered by:** HJ, from the 2026-07-13 orphan incident.
---
## The formal statement
Any operation carrying an **external side effect** (venue order, cancel, transfer,
ClickHouse insert, journal write, shm publish) has **three** terminal outcomes, not two:
| Outcome | Effect exists? | Is rollback sound? |
|---|---|---|
| **NOT_ATTEMPTED** — we never issued it | provably no | **YES** |
| **ATTEMPTED_REFUSED** — the peer *proved* it refused (venue error code, validation reject) | provably no | **YES** |
| **ATTEMPTED_INDETERMINATE** — issued, outcome unknown | **maybe** | **NO — NEVER** |
The bug is **collapsing three states into two**: treating INDETERMINATE as FAILED, and
then applying a rollback that is only sound for the first two.
**The invariant that gets violated:** the *effect* (the venue order) and the *record*
(the kernel FSM) are **not atomic**. Any failure between them leaves them divergent. This
is the classic dual-write / write-then-record problem, and it has exactly one correct
answer: **when you cannot prove the effect did not happen, you must not assert that it
didn't. You must reconcile against the authoritative source** (the venue).
**Unknown is not flat. Unknown is not failed. Unknown is UNKNOWN.**
---
## How it presented on 2026-07-13
```python
# bingx_venue.submit_async()
receipt = await backend.submit_intent(legacy) # <-- POINT OF NO RETURN: order is LIVE
events = self._events_from_submit(...) # bookkeeping
self._publish_telemetry(..., order_id=...) # <-- TypeError (signature skew)
return events # never reached
```
```python
# rust_backend, the caller
try:
emitted_events = await submit_async(intent)
except Exception: # <-- ONE channel for BOTH meanings
-> synthetic REJECTED -> FSM rollback to IDLE
```
`submit_async` signals *"the venue never took the order"* by **raising**, and signals
*"a bug fired somewhere inside me"* by **raising**. Same channel. The caller cannot tell
them apart, so it takes the interpretation written for the timeout case and asserts a
fact it cannot know: *no order exists*.
The venue kept 6 SHORT positions. The kernel believed it was flat. No SL, TP, MAX_HOLD or
ADVSL can protect a position the kernel does not know it holds — which is why **execution
truth (tier B) outranks capital preservation (tier C)** in `UV_EXEC_PRIORITY_LADDER`.
Telemetry never had a "veto". It **hijacked the failure channel**. It was the bullet;
`except Exception -> assume not-done` is the gun. *Any* post-ack line — a log call, a dict
build, an events-builder edge case — would have produced the identical 6 orphans.
---
## THE SECOND INSTANCE — older, still live, and worse
The "safe" pre-ack branch is **itself an instance of the same bug**:
```python
except Exception as _submit_exc:
# venue.submit() failed (e.g. BingX timeout) ... feed a synthetic REJECTED
```
**A timeout is the definition of INDETERMINATE.** The request may have reached BingX and
filled. Rolling back to IDLE on a timeout asserts "no order exists" on exactly the
evidence that cannot support it. This has been in the code far longer than tonight's skew
and produces orphans on every venue timeout.
Only a **proven** refusal may roll back:
- connection refused / DNS failure *before the request was sent* → NOT_ATTEMPTED ✔
- venue returned an explicit rejection code (validation, insufficient margin) → REFUSED ✔
- **timeout, connection reset, 5xx, unparseable response → INDETERMINATE ✘ must reconcile**
---
## The fix pattern (minimally invasive — no new seams, no re-architecture)
1. **Mark the point of no return.** Everything after the side-effect call is fenced; it
may not reach the caller through the bare-exception channel.
2. **Type the channel.** `VenuePostAckError` (added, `venue.py`) carries the receipt and
means *the effect exists*. Pre-ack failures keep raising normally.
3. **Callers branch on it.** `rust_backend` catches `VenuePostAckError` first: **no
synthetic REJECT, no rollback**. Leave the slot in its working state and let execution
truth settle it — the E-feed delivers `FULL_FILL` from the account stream independently.
4. **Non-critical code cannot use the failure channel at all.** Telemetry is off the exec
path entirely (bounded-memory, lossless spill lane).
5. **Indeterminate ⇒ reconcile.** Never rollback. Query the venue; it is the authority.
No architectural change. No new seam. One typed exception, one fence, one branch.
---
## The codebase-wide sweep (codex)
**Search pattern:** any `try:` whose body performs an external side effect and whose
`except` assumes the effect did not happen (rollback / mark-failed / synthesize-reject /
blind-retry / "assume flat" / return default).
Known/suspect sites:
- `bingx_venue.submit / submit_async` ✔ fixed
- `bingx_venue.cancel / cancel_async` — same shape, unaudited
- `rust_backend` pre-ack timeout branch — **BUG, live** (see above)
- `EKBridge.mirror_future`, `e_feed._publish_converted` — failure of the E→K mirror
- `ch_writer` / journal lane inserts — "insert failed" vs "insert maybe landed" (duplicates)
- `ExecutionRouter` maker/taker requote + amend paths
- promotion bridge (`uv.promotion`) receipt handling
- any `except Exception: pass` / `return None` / `return False` after an I/O call
**Every site must answer one question:** *can this except-branch PROVE the effect did not
happen?* If no → it must not assert absence. Reconcile or escalate to UNKNOWN.
---
## Tests demanded (nuclear grade)
1. **Fault-injection matrix.** For each side-effecting call: inject an exception at
**every statement after the effect** (parametrize over injection index) and assert the
kernel **never** concludes the effect is absent. This is the test that would have
caught tonight's bug at *any* of the post-ack lines, not just the telemetry one.
2. **Timeout is indeterminate.** Venue times out *after* the exchange filled → assert NO
rollback, assert reconcile is triggered, assert the position is adopted, not orphaned.
3. **Property/fuzz.** Random interleavings of {ack, no-ack, timeout, post-ack raise,
partial fill, duplicate ack}; invariant: `kernel_believes_flat ⇒ venue_is_flat`. This
invariant is the whole ballgame; it must never be violated for any interleaving.
4. **Signature-skew guard.** `inspect.signature().bind()` over every call site of every
telemetry/journal helper — a pure binding test that survives re-vendoring.
5. **Losslessness.** Telemetry/journal lanes: wedge the sink, flood N records, assert
`dropped == 0` and `on_disk + in_memory == N`, and assert memory stays bounded.
6. **Mutation litmus.** Remove each fence → the corresponding test must go RED. If
removing the guard keeps the suite green, the test is theatre.