diff --git a/prod/docs/COMPREHENSIVE_UV_EDGE_CASE_THEORETICALS.md b/prod/docs/COMPREHENSIVE_UV_EDGE_CASE_THEORETICALS.md new file mode 100644 index 0000000..9baa202 --- /dev/null +++ b/prod/docs/COMPREHENSIVE_UV_EDGE_CASE_THEORETICALS.md @@ -0,0 +1,465 @@ +# COMPREHENSIVE UV EDGE CASE THEORETICALS +## State Machine Edge Cases, Undesirable States, and Mitigations for DITAv2 + +**Document Status:** Reference for operators and engineers +**Last Updated:** 2026-07-13 (Fable commit: INDETERMINATE submit fix) +**Scope:** DITAv2 execution kernel, BingX venue adapter, `bingx_direct` adapter, Rust-backed FSM + +--- + +## 1. EXECUTIVE SUMMARY + +The 2026-07-13 Fable commit (INDETERMINATE submit fix) revealed a **fundamental state machine flaw**: treating an **INDETERMINATE** HTTP outcome (request sent, answer lost) as a **confident REJECTED** with `fill_qty=0`. This caused the kernel FSM to roll back to `IDLE` (flat) while the venue kept the position live — creating **orphan positions** that had no SL/TP/ADVSL protection. + +This document catalogs: +1. **The root flaw** (INDETERMINATE ≡ REJECTED misclassification) +2. **All seams** where this class of bug can manifest +3. **Foreseeable & unforeseeable state transitions** that are undesirable +4. **Missing states/transitions** that should exist +5. **Mitigations** (implemented, partial, and proposed) + +--- + +## 2. THE ROOT FLAW: INDETERMINATE ≠ REJECTED + +### 2.1 What Happened +```python +# OLD bingx_direct.py (pre-2026-07-13) +except BingxHttpError as exc: + status = "REJECTED" # ← WRONG: treated timeout/5xx as confident rejection + ack_row = {"status": "REJECTED", "msg": str(exc), ...} + fill_qty = 0.0 # ← kernel rolls back to IDLE, believes flat +``` + +### 2.2 The Correction (2026-07-13) +```python +# NEW bingx_direct.py (post-fix) +except BingxHttpError as exc: + effect = getattr(exc, "effect", BingxHttpError.INDETERMINATE) + if effect == BingxHttpError.INDETERMINATE and not _is_rate_limited_error(exc): + # THE ORDER MAY BE LIVE. We do NOT guess. We ask the venue. + found = await self._lookup_own_order_by_client_id(...) + if isinstance(found, dict): + # Venue confirms: order exists → adopt its state + status = ack_row.get("status") or "ACKED" + elif found == "ABSENT": + # Venue says "no such order" → rollback IS sound + status = "REJECTED" + else: + # TRUTH NOT ESTABLISHED. We REFUSE to claim rejection. + status = "INDETERMINATE" + # Venue layer escalates to VenueIndeterminateError +``` + +### 2.3 The Three-Way Classification (NOW ENFORCED) + +| Classification | HTTP Examples | What It Proves | Rollback Sound? | +|----------------|---------------|----------------|-----------------| +| **NOT_ATTEMPTED** | `ConnectError`, `ConnectTimeout`, DNS failure | Request never left the box | ✅ YES | +| **REFUSED** | 4xx (non-429), "insufficient margin" | Venue received and explicitly refused | ✅ YES | +| **INDETERMINATE** | `ReadTimeout`, `ConnectReset`, 5xx, transport death | **Request was sent, answer lost** — order MAY be live | ❌ **NEVER** | + +--- + +## 3. THE SEAMS: WHERE THIS BUG CLASS MANIFESTS + +### 3.1 Seam 1: `bingx_direct.submit_intent()` — The Origin + +**File:** `/mnt/dolphinng5_predict/prod/clean_arch/adapters/bingx_direct.py` +**Method:** `BingxDirectExecutionAdapter.submit_intent()` (lines ~850-1100) + +**Critical Sections:** +- **Lines ~950-1020**: `try/except BingxHttpError` block — the classification logic +- **Lines ~970-1010**: `_lookup_own_order_by_client_id()` — the point lookup that reconciles truth +- **Lines ~1010-1050**: Status assignment (`REJECTED` vs `INDETERMINATE` vs `ABSENT`) + +**Enumerated Failure Modes Here:** +| Failure | Classification | Root Cause | +|---------|----------------|------------| +| `httpx.ReadTimeout` | INDETERMINATE | Request sent, response lost | +| `httpx.ConnectTimeout` | NOT_ATTEMPTED | Never left the box | +| `httpx.ConnectError("refused")` | NOT_ATTEMPTED | Connection refused | +| 500/502/503/504 | INDETERMINATE | Server error, request may have been processed | +| `httpx.RemoteProtocolError` | INDETERMINATE | Protocol violation mid-stream | +| `httpx.RemoteProtocolError` (partial response) | INDETERMINATE | May have been accepted | +| TLS handshake failure | NOT_ATTEMPTED | Never established secure channel | +| `BingxHttpError` with unknown code | INDETERMINATE (default) | Conservative: assume may be live | + +**Mitigation (Implemented):** +- Lines ~950-1020: Three-way classification with `_lookup_own_order_by_client_id()` reconciliation +- Lines ~1010-1050: `status = "INDETERMINATE"` path that does NOT set `REJECTED` + +--- + +### 3.2 Seam 2: `BingxVenueAdapter.submit()` / `submit_async()` — The Venue Layer + +**File:** `/mnt/dolphinng5_predict/prod/clean_arch/dita_v2/bingx_venue.py` +**Methods:** `submit()` (sync, line ~800) and `submit_async()` (async, line ~950) + +**Critical Logic:** +```python +# Both submit() and submit_async() share this pattern: +receipt = self._call_backend("submit_intent", legacy) # or await backend.submit_intent_async() + +# ── UNKNOWN OUTCOME ── +if str(getattr(receipt, "status", "") or "").upper() == "INDETERMINATE": + raise VenueIndeterminateError( + f"submit outcome UNKNOWN for intent={intent.intent_id} asset={intent.asset} " + f"— order may be live at the venue; refusing to claim rejection", + receipt=receipt, + ) +``` + +**Error Hierarchy:** +``` +VenuePostAckError (base) + └── VenueIndeterminateError ← raised for INDETERMINATE status +``` + +**Mitigation (Implemented):** +- Both sync and async paths check `receipt.status == "INDETERMINATE"` +- Raises `VenueIndeterminateError` (subclass of `VenuePostAckError`) +- Kernel's "no-rollback fences" catch `VenuePostAckError` (see Seam 3) + +--- + +### 3.3 Seam 3: `ExecutionKernel.process_intent()` / `process_intent_async()` — The Kernel Boundary + +**File:** `/mnt/dolphinng5_predict/prod/clean_arch/dita_v2/rust_backend.py` +**Methods:** `process_intent()` (sync, line ~880) and `process_intent_async()` (async, line ~1050) + +**Critical Logic (sync):** +```python +try: + emitted_events = self.venue.submit(intent) +except VenuePostAckError as _post_ack_exc: + # ORDER IS LIVE AT THE VENUE. Unknown != flat. + # NO rollback, NO synthetic REJECT. + logger.critical("FIX(rust_backend): POST-ACK venue failure — ORDER IS LIVE. NOT rolling back.") + emitted_events = list(getattr(_post_ack_exc, "events", []) or []) +except Exception as _submit_exc: + # PRE-ACK failure: venue never took the order. Safe to roll back. + logger.error("venue.submit failed — synthetic REJECTED to roll back FSM") + emitted_events = [synthetic_REJECTED_event] +``` + +**Critical Logic (async):** +```python +submit_async = getattr(self.venue, "submit_async", None) +try: + if submit_async is not None: + emitted_events = await submit_async(intent) + else: + emitted_events = self.venue.submit(intent) +except VenuePostAckError as _post_ack_exc: + # SAME LOGIC: ORDER IS LIVE. NO ROLLBACK. + logger.critical("FIX(rust_backend): POST-ACK venue failure — ORDER IS LIVE. NOT rolling back.") + emitted_events = list(getattr(_post_ack_exc, "events", []) or []) +except Exception as _submit_exc: + # PRE-ACK: synthetic REJECT to roll back FSM + emitted_events = [synthetic_REJECTED_event] +``` + +**The "No-Rollback Fence" — THE KEY MITIGATION:** +```python +# In on_venue_event (and process_intent), kernel checks: +if isinstance(exc, VenuePostAckError): + # NO ROLLBACK — slot stays in ENTRY_WORKING / EXIT_WORKING + # Let E-feed FILL / reconcile settle it +``` + +--- + +### 3.4 Seam 4: `on_venue_event()` — The Event Ingestion Path + +**File:** `/mnt/dolphinng5_predict/prod/clean_arch/dita_v2/rust_backend.py` +**Method:** `on_venue_event()` (line ~1200) + +**Logic:** +```python +def on_venue_event(self, event: VenueEvent) -> KernelOutcome: + # Processes all venue events through the Rust FSM + # If event.status == REJECTED and slot is in ENTRY_WORKING/EXIT_WORKING: + # → FSM transitions to IDLE (flat) + # If event.status == INDETERMINATE: + # → Venue layer NEVER emits this; it raises VenueIndeterminateError instead +``` + +**Key Point:** The venue adapter **never emits an `INDETERMINATE` event** — it raises `VenueIndeterminateError` instead. The kernel only sees events with `status` ∈ {ACKED, REJECTED, FILLED, PARTIAL_FILL, CANCELED, CANCEL_REJECTED, RATE_LIMITED}. + +--- + +### 3.5 Seam 5: Rust FSM (Hidden in `rust_backend.so`) — The Core State Machine + +**Location:** `/mnt/dolphinng5_predict/prod/clean_arch/dita_v2/_rust_kernel/` (built into `libdita_v2_kernel.so`) + +**State Machine (Simplified):** +``` +IDLE + ──ENTER_intent──→ ORDER_REQUESTED + ──venue.submit() succeeds (ACKED)──→ ENTRY_WORKING + ──FILL event──→ OPEN + ──REJECTED event──→ IDLE (rollback) + ──INDETERMINATE (VenuePostAckError)──→ STAYS ENTRY_WORKING (no rollback) + ──venue.submit() fails (PRE-ACK)──→ synthetic REJECTED → IDLE + ──EXIT_intent──→ ORDER_REQUESTED (exit) + ──venue.submit() succeeds──→ EXIT_WORKING + ──FILL event──→ IDLE + ──REJECTED event──→ IDLE + ──INDETERMINATE──→ STAYS EXIT_WORKING +``` + +**Rust FSM Responsibility:** +- Manages `TradeSlot.fsm_state` (IDLE, ORDER_REQUESTED, ENTRY_WORKING, OPEN, EXIT_WORKING, etc.) +- Transitions are deterministic given event `(kind, status)` +- **Does NOT know about INDETERMINATE** — it only sees events the Python layer emits +- The Python layer is the **semantic firewall**: it decides whether to emit a `REJECTED` event or suppress it + +--- + +### 3.6 Seam 6: `on_venue_event()` — The Feedback Loop + +**File:** `/mnt/dolphinng5_predict/prod/clean_arch/dita_v2/rust_backend.py` +**Method:** `on_venue_event()` (line ~1200) + +```python +def on_venue_event(self, event: VenueEvent) -> KernelOutcome: + # All venue events flow here → Rust FSM → state transitions + # If event.status == REJECTED: + # kernel.fsm_state → IDLE (flat) + # If event.status == FILLED/PARTIAL_FILL: + # kernel.fsm_state → OPEN / IDLE +``` + +**Critical Invariant:** The kernel **only rolls back to IDLE on explicit `REJECTED` events**. It has no concept of "unknown". + +--- + +## 4. ENUMERATED UNDESIRABLE STATES & TRANSITIONS + +### 4.1 States That SHOULD Exist But Don't + +| Missing State | Description | Why It Matters | +|---------------|-------------|----------------| +| `ENTRY_INDETERMINATE` | Order sent, answer lost — may be live | Distinguishes "we don't know" from "working normally" | +| `EXIT_INDETERMINATE` | Exit sent, answer lost — may be live | Same for exits | +| `ORDER_TRUTH_UNKNOWN` | Lookup failed after N attempts | Explicit "truth unknown" state vs silent failure | +| `VENUE_LOOKUP_PENDING` | Background reconcile in progress | Prevents duplicate reconcile attempts | +| `RECONCILE_IN_PROGRESS` | Full reconcile running | Prevents concurrent reconciles | + +### 4.2 Transitions That Should Not Exist (But Do) + +| Transition | Why It's Bad | Where It Happens | +|------------|--------------|------------------| +| `ENTRY_WORKING` → `IDLE` on `INDETERMINATE` | Orphans live position | `bingx_direct` pre-2026-07-13 (FIXED) | +| `EXIT_WORKING` → `IDLE` on `INDETERMINATE` | Orphans live position, no SL/TP | Same as above | +| `ENTRY_WORKING` → `IDLE` on `VenuePostAckError` (broad) | Over-broad catch rolls back live orders | `rust_backend.py` line ~900 (MITIGATED: catches `VenuePostAckError` specifically) | +| `ORDER_REQUESTED` → `IDLE` without event | Silent state loss | Never directly, but possible via exception path | + +### 4.3 States That Should Be Unreachable But Are Reachable + +| State | How It's Reached | Why It's Bad | +|-------|------------------|--------------| +| `SLOT_BUSY` with no active order | FSM advanced but order never sent | `process_intent` exception before `venue.submit` | +| `ENTRY_WORKING` with `active_entry_order = None` | `attach_entry_order` not called | `venue.submit` exception before `attach_entry_order` | +| `OPEN` with `position = None` | Phantom fill event | E-feed race, duplicate fill | +| `IDLE` with `trade_id` set | Incomplete cleanup | `reset()` not called on all fields | + +### 4.4 Foreseeable "Phantom" States (Unforeseeable in Practice) + +| Scenario | Resulting State | Detection | +|----------|-----------------|-----------| +| BingX accepts order, sends ACK, TCP dies before ACK arrives | Venue: LIVE / Kernel: ENTRY_WORKING (no ACK event) | `_lookup_own_order_by_client_id` reconciliation | +| BingX fills order, sends FILL, WS dies before delivery | Venue: FILLED / Kernel: ENTRY_WORKING | E-feed FILL event / reconcile | +| BingX rejects order, sends REJECTED, TCP dies | Venue: REJECTED / Kernel: ENTRY_WORKING | Reconcile finds REJECTED | +| Network partition: client can't reach BingX, but BingX can fill | Venue: FILLED / Kernel: INDETERMINATE | Reconcile + E-feed | +| Duplicate `clientOrderId` sent (retries) → BingX returns original order | Venue: FILLED / Kernel: sees duplicate | `clientOrderId` deduplication in `_lookup` | + +--- + +## 5. MITIGATIONS — IMPLEMENTED, PARTIAL, PROPOSED + +### 5.1 Implemented (Post-2026-07-13) + +| Mitigation | Location | What It Prevents | +|------------|----------|------------------| +| Three-way HTTP classification | `bingx_direct.py` lines ~950-1020 | Misclassification of INDETERMINATE as REJECTED | +| `_lookup_own_order_by_client_id()` | `bingx_direct.py` lines ~1030-1100 | Reconciling truth from venue after INDETERMINATE | +| `VenueIndeterminateError` exception | `bingx_venue.py` line ~859/937 | Semantic escalation: INDETERMINATE ≠ REJECTED | +| `VenuePostAckError` catch (no rollback) | `rust_backend.py` lines ~900, ~1080 | Kernel no-rollback fence for INDETERMINATE | +| `VenueIndeterminateError` inherits `VenuePostAckError` | `venue.py:43` | Existing no-rollback fence catches it | +| `_lookup_own_order_by_client_id` bounded retries | `bingx_direct.py` line ~1070 | Bounded reconciliation, then gives up with `None` | +| `VenuePostAckError` catch in async path too | `rust_backend.py` line ~1080 | Async path same protection as sync | +| No synthetic event on INDETERMINATE | `rust_backend.py` lines ~910, ~1090 | No synthetic REJECT pollutes event log | + +### 5.2 Partial / Incomplete + +| Gap | Location | Risk | +|-----|----------|------| +| No `ENTRY_INDETERMINATE` / `EXIT_INDETERMINATE` FSM states | Rust FSM | Can't distinguish "working normally" from "truth unknown" in state | +| No `ORDER_TRUTH_UNKNOWN` explicit state | `bingx_direct._lookup_own_order` | Returns `None` but no state reflects this | +| No background reconcile task for INDETERMINATE orders | `bingx_direct._s2_tasks` | Only MARKET fills trigger background refresh | +| No metrics/alerting on INDETERMINATE frequency | None | Can't detect systemic issues | +| No alert on `VenueIndeterminateError` rate | Logging only | Operators don't know it's happening | + +### 5.3 Proposed (For Future Hardening) + +| Proposal | Effort | Impact | +|----------|--------|--------| +| Add `ENTRY_INDETERMINATE` / `EXIT_INDETERMINATE` FSM states | Rust FSM + Python bindings | Explicit "truth unknown" state visible in TUI/monitoring | +| Add `OrderTruth` enum: `KNOWN_LIVE`, `KNOWN_FLAT`, `UNKNOWN` | `bingx_direct` + kernel | Explicit truth tracking per order | +| Background reconcile task for INDETERMINATE orders | New async task in `DolphinLiveTrader` | Proactive truth discovery | +| `INDETERMINATE` metric + alert | Prometheus/Grafana | Operator visibility | +| `OrderTruth` persisted in Zinc/CH | Zinc plane + CH writer | Survives restarts | +| `ReconcileManager` service | New module | Centralized truth discovery | +| `ORDER_TRUTH_UNKNOWN` event type | Contracts + kernel | Explicit event for "we don't know" | + +--- + +## 6. ENUMERATED INSTANCES IN CODEBASE + +### 6.1 Where INDETERMINATE Classification Happens + +| File | Line | Function | Purpose | +|------|------|----------|---------| +| `prod/bingx/http.py` | ~50 | `BingxHttpError.effect` | Classification property | +| `prod/bingx/http.py` | ~60 | `_effect_of_httpx_error()` | Maps `httpx` exceptions | +| `prod/bingx/http.py` | ~80 | `_effect_of_status()` | Maps HTTP status codes | +| `prod/clean_arch/adapters/bingx_direct.py` | ~950 | `submit_intent()` exception handler | Main classification | +| `prod/clean_arch/adapters/bingx_direct.py` | ~1030 | `_lookup_own_order_by_client_id()` | Truth reconciliation | +| `prod/clean_arch/dita_v2/bingx_venue.py` | ~859 | `submit()` INDETERMINATE check | Sync venue | +| `prod/clean_arch/dita_v2/bingx_venue.py` | ~937 | `submit_async()` INDETERMINATE check | Async venue | + +### 6.2 Where INDETERMINATE Is Caught / Escalated + +| File | Line | Context | +|------|------|---------| +| `prod/clean_arch/dita_v2/bingx_venue.py` | ~859 | `raise VenueIndeterminateError(...)` (sync) | +| `prod/clean_arch/dita_v2/bingx_venue.py` | ~937 | `raise VenueIndeterminateError(...)` (async) | +| `prod/clean_arch/dita_v2/rust_backend.py` | ~900 | `except VenuePostAckError:` (sync) | +| `prod/clean_arch/dita_v2/rust_backend.py` | ~1080 | `except VenuePostAckError:` (async) | +| `prod/clean_arch/dita_v2/venue.py` | 43 | `class VenueIndeterminateError(VenuePostAckError)` | + +### 6.3 Where No-Rollback Fence Exists + +| File | Line | Protection | +|------|------|------------| +| `prod/clean_arch/dita_v2/rust_backend.py` | ~900 | `except VenuePostAckError:` in sync `process_intent` | +| `prod/clean_arch/dita_v2/rust_backend.py` | ~1080 | `except VenuePostAckError:` in async `process_intent_async` | + +### 6.4 Where Synthetic REJECTED Is Emitted (PRE-ACK Only) + +| File | Line | Context | +|------|------|---------| +| `prod/clean_arch/dita_v2/rust_backend.py` | ~920 | Sync: `synthetic_REJECTED_event` on generic `Exception` | +| `prod/clean_arch/dita_v2/rust_backend.py` | ~1110 | Async: `synthetic_REJECTED_event` on generic `Exception` | + +--- + +## 7. TEST VERIFICATION MATRIX + +**File:** `/mnt/dolphinng5_predict/prod/clean_arch/dita_v2/test_indeterminate_submit.py` + +| Test | What It Verifies | +|------|------------------| +| `test_read_timeout_is_indeterminate_not_rejection` | `ReadTimeout` → INDETERMINATE | +| `test_connect_failures_are_not_attempted` | `ConnectError`/`ConnectTimeout` → NOT_ATTEMPTED | +| `test_5xx_is_indeterminate_and_4xx_is_refused` | 5xx → INDETERMINATE, 4xx → REFUSED | +| `test_unknown_failure_defaults_to_indeterminate` | Unknown error → INDETERMINATE (conservative) | +| `test_lookup_finds_the_order_and_never_posts` | Reconcile finds live order, never re-POSTs | +| `test_lookup_reports_absent_when_venue_has_no_such_order` | Venue says "no order" → ABSENT (rollback sound) | +| `test_lookup_treats_venue_refusal_as_absent` | Venue REFUSES lookup → ABSENT | +| `test_lookup_returns_none_when_truth_cannot_be_established` | Timeout on lookup → None (not ABSENT) | +| `test_sync_venue_escalates_indeterminate_receipt` | `VenueIndeterminateError` raised | +| `test_async_venue_escalates_indeterminate_receipt` | Async path same | +| `test_indeterminate_is_a_post_ack_error_so_existing_fences_catch_it` | Subclass check | +| `test_kernel_does_not_roll_back_slot_on_indeterminate_submit` | Sync kernel no-rollback | +| `test_kernel_does_not_roll_back_slot_on_indeterminate_submit_async` | Async kernel no-rollback | +| `test_genuine_rejection_still_rolls_the_slot_back` | REFUSED still rolls back (contrast test) | + +--- + +## 8. DECISION LOG (Appendix) + +| Date | Decision | Rationale | +|------|----------|-----------| +| 2026-07-13 | Classify `ReadTimeout`/`5xx` as INDETERMINATE | Request sent, answer lost — order may be live | +| 2026-07-13 | Default unknown errors to INDETERMINATE | Conservative: never guess "rejected" | +| 2026-07-13 | `_lookup_own_order_by_client_id` bounded retries | Prevent infinite retry loops | +| 2026-07-13 | `VenueIndeterminateError` inherits `VenuePostAckError` | Reuses existing no-rollback fence | +| 2026-07-13 | NO synthetic event on INDETERMINATE | Don't pollute event log with guesses | +| 2026-07-13 | `VenuePostAckError` catch in BOTH sync/async | Both code paths protected | + +--- + +## 9. OPERATOR RUNBOOK: INDETERMINATE INCIDENT + +**If you see `VenueIndeterminateError` in logs:** + +1. **Do NOT manually flatten the slot** — the order may be live +2. Check BingX VST/LIVE for the `clientOrderId` (logged in the error) +3. If position exists at venue: let E-feed FILL settle it, or manually reconcile +4. If position NOT at venue: kernel will eventually reconcile or you can trigger manual reconcile +5. **Do NOT restart the kernel to "clear" the slot** — this loses the `ENTRY_WORKING`/`EXIT_WORKING` state + +**Log Signature:** +``` +CRITICAL FIX(bingx_direct): submit outcome INDETERMINATE (ReadTimeout("...")) +symbol=TRXUSDT clientOrderId=p-e-1q3k7m-ab4c — order MAY be live. Looking it up; +NOT assuming rejection. +``` + +**Resolution Paths:** +- `found == dict`: Order is LIVE → kernel adopts venue truth, slot stays OPEN/WORKING +- `found == "ABSENT"`: Order never existed → rollback IS sound, slot → IDLE +- `found == None`: Truth unknown → slot stays in ENTRY_WORKING/EXIT_WORKING, E-feed FILL settles it + +--- + +## 10. FUTURE WORK PRIORITY ORDER + +| Priority | Item | Effort | Blocker | +|----------|------|--------|---------| +| P0 | `ENTRY_INDETERMINATE` / `EXIT_INDETERMINATE` FSM states | Medium | Rust FSM changes + Python bindings | +| P0 | Background reconcile task for INDETERMINATE orders | Low | New async task in trader loop | +| P1 | `OrderTruth` enum + Zinc persistence | Medium | Zinc plane schema change | +| P1 | INDETERMINATE metrics + Grafana alert | Low | Prometheus exporter | +| P2 | `ReconcileManager` service | High | New module, cross-cutting | +| P2 | `ORDER_TRUTH_UNKNOWN` event type | Medium | Contracts + kernel + venue adapters | + +--- + +## 11. FABLE REVIEW (2026-07-13) — verdict + MISSING SEAMS (fleet work) + +**Truth audit:** Doc is faithful to the shipped fix (verified: `_is_rate_limited_error` +exists at `bingx_direct.py:127`; test matrix §7 matches the real 14-test suite; +FSM description matches kernel-invariant tests). Two factual errors corrected +in-place: `VenueIndeterminateError` lives at `venue.py:43` (not contracts.py); +fix date is 2026-07-13 (commits `bdc54fb` canonical / `54cd5d5` vendored). + +**Scope gap — THE assignment ("extrapolate the class across the codebase") is +unfinished.** §3 covers the SUBMIT seam deeply; the same not-done ≠ tried-and- +failed ≠ unknown triage is UNAUDITED at every other venue-side-effect seam: + +| # | Seam | Specific risk | +|---|------|---------------| +| M1 | `cancel()` / `cancel_async()` | cancel-INDETERMINATE = order MAY still be live → blind re-place = double fill. (Encoded in SPEC_UV_SMART_EXEC_MM §5; kernel path unaudited.) | +| M2 | leverage SET (`bingx_direct`) | On the money path (seen live 17:20 today). SET-INDETERMINATE → position sized at unknown leverage. | +| M3 | promotion bridge receipts (`uv.promotion`) | Bridge assumes receipt is truth; INDETERMINATE mid-bridge unhandled? | +| M4 | E-feed / EKBridge gap handling | WS death between ACK and FILL — partially covered by rehydrate; no triage taxonomy. | +| M5 | exit-leg placement (multi-leg exits) | Leg 1 fills, leg 2 INDETERMINATE → partial exit with unknown remainder. | +| M6 | ch_writer / journal lane | Covered for data (spool), but write-INDETERMINATE→retry = possible duplicate rows (dedup ratchet history). | +| M7 | every `except: pass` / broad `except Exception` after venue I/O | Grep-sweep + classify each; the b46ebd2 class hid here. | + +**DOCTRINE GUARDRAIL on §10 P0 "background reconcile task":** operator standing +order — reconcile is proto-PINK hell. Any background truth-discovery MUST stay +a bounded, read-only, own-clientOrderId point lookup (the §2.2 primitive), never +a scan-all-orders reconciler. Build the loop around the existing primitive. + +**Disposition:** M1–M7 are fleet-sized (audit + tests per seam, same pattern as +`test_indeterminate_submit.py`). Corrections done in-place are complete. + +--- + +**END OF DOCUMENT** +*This document is a living reference. Update on every INDETERMINATE-related change.* diff --git a/prod/docs/HANDOVER_20260713_FABLE_EOD.md b/prod/docs/HANDOVER_20260713_FABLE_EOD.md index d4a46f4..3b28c1f 100644 --- a/prod/docs/HANDOVER_20260713_FABLE_EOD.md +++ b/prod/docs/HANDOVER_20260713_FABLE_EOD.md @@ -73,10 +73,14 @@ journal lane auto-drains the spool or rows need manual JSONEachRow replay ExecutionRouter (maker/taker policy, 265 tests) ∪ BLUE SmartPlacer (OB-aware placement) vs prod/docs/BingX_FILL_CHARACTERIZATION_AND_ADVANTAGES.md. Router under 1s steel clock = maker orders actually managed. -8. **Indeterminate sweep codebase-wide** (belled to codex, napped): cancel/ - cancel_async, EKBridge, e_feed, ch_writer, ExecutionRouter, promotion receipts, - every `except: pass` after venue I/O — same NOT_ATTEMPTED/REFUSED/INDETERMINATE - triage as bdc54fb. +8. **Indeterminate sweep codebase-wide — NOW SPECCED FOR FLEET**: pi red-teamed + the class (`COMPREHENSIVE_UV_EDGE_CASE_THEORETICALS.md`); Fable reviewed + 2026-07-13 eve: doc FAITHFUL (2 factual errors fixed in-place: venue.py:43 + location, 07-13 date), submit seam fully covered, **7 missing seams + enumerated in its §11 (M1–M7)**: cancel, leverage SET, promotion receipts, + E-feed gaps, exit legs, ch_writer dups, broad-except sweep. §11 carries the + NO-RECONCILE guardrail for the P0 background task. Fleet-ready: one seam + per agent; test pattern = test_indeterminate_submit.py. 9. **TIER-B inversion** (open): asex_kernel_executor.py:319 enqueues account/fill at P4 below ENTER P3 — execution truth must outrank entries. 10. **Research studies** (logged POST_FINISH): dvol sweep × regime (PREREQ: persist