docs(review): pi's INDETERMINATE edge-case doc — verified faithful, 2 errors fixed, 7 missing seams M1-M7 enumerated for fleet
venue.py:43 location + 2026-07-13 date corrected in-place. §11 added: truth audit, missing-seam table (cancel, lev SET, promo receipts, e-feed, exit legs, ch_writer, broad-except), NO-RECONCILE guardrail on the P0 background task. Handover item 8 updated to fleet-ready. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
465
prod/docs/COMPREHENSIVE_UV_EDGE_CASE_THEORETICALS.md
Normal file
465
prod/docs/COMPREHENSIVE_UV_EDGE_CASE_THEORETICALS.md
Normal file
@@ -0,0 +1,465 @@
|
||||
# COMPREHENSIVE UV EDGE CASE THEORETICALS
|
||||
## State Machine Edge Cases, Undesirable States, and Mitigations for DITAv2
|
||||
|
||||
**Document Status:** Reference for operators and engineers
|
||||
**Last Updated:** 2026-07-13 (Fable commit: INDETERMINATE submit fix)
|
||||
**Scope:** DITAv2 execution kernel, BingX venue adapter, `bingx_direct` adapter, Rust-backed FSM
|
||||
|
||||
---
|
||||
|
||||
## 1. EXECUTIVE SUMMARY
|
||||
|
||||
The 2026-07-13 Fable commit (INDETERMINATE submit fix) revealed a **fundamental state machine flaw**: treating an **INDETERMINATE** HTTP outcome (request sent, answer lost) as a **confident REJECTED** with `fill_qty=0`. This caused the kernel FSM to roll back to `IDLE` (flat) while the venue kept the position live — creating **orphan positions** that had no SL/TP/ADVSL protection.
|
||||
|
||||
This document catalogs:
|
||||
1. **The root flaw** (INDETERMINATE ≡ REJECTED misclassification)
|
||||
2. **All seams** where this class of bug can manifest
|
||||
3. **Foreseeable & unforeseeable state transitions** that are undesirable
|
||||
4. **Missing states/transitions** that should exist
|
||||
5. **Mitigations** (implemented, partial, and proposed)
|
||||
|
||||
---
|
||||
|
||||
## 2. THE ROOT FLAW: INDETERMINATE ≠ REJECTED
|
||||
|
||||
### 2.1 What Happened
|
||||
```python
|
||||
# OLD bingx_direct.py (pre-2026-07-13)
|
||||
except BingxHttpError as exc:
|
||||
status = "REJECTED" # ← WRONG: treated timeout/5xx as confident rejection
|
||||
ack_row = {"status": "REJECTED", "msg": str(exc), ...}
|
||||
fill_qty = 0.0 # ← kernel rolls back to IDLE, believes flat
|
||||
```
|
||||
|
||||
### 2.2 The Correction (2026-07-13)
|
||||
```python
|
||||
# NEW bingx_direct.py (post-fix)
|
||||
except BingxHttpError as exc:
|
||||
effect = getattr(exc, "effect", BingxHttpError.INDETERMINATE)
|
||||
if effect == BingxHttpError.INDETERMINATE and not _is_rate_limited_error(exc):
|
||||
# THE ORDER MAY BE LIVE. We do NOT guess. We ask the venue.
|
||||
found = await self._lookup_own_order_by_client_id(...)
|
||||
if isinstance(found, dict):
|
||||
# Venue confirms: order exists → adopt its state
|
||||
status = ack_row.get("status") or "ACKED"
|
||||
elif found == "ABSENT":
|
||||
# Venue says "no such order" → rollback IS sound
|
||||
status = "REJECTED"
|
||||
else:
|
||||
# TRUTH NOT ESTABLISHED. We REFUSE to claim rejection.
|
||||
status = "INDETERMINATE"
|
||||
# Venue layer escalates to VenueIndeterminateError
|
||||
```
|
||||
|
||||
### 2.3 The Three-Way Classification (NOW ENFORCED)
|
||||
|
||||
| Classification | HTTP Examples | What It Proves | Rollback Sound? |
|
||||
|----------------|---------------|----------------|-----------------|
|
||||
| **NOT_ATTEMPTED** | `ConnectError`, `ConnectTimeout`, DNS failure | Request never left the box | ✅ YES |
|
||||
| **REFUSED** | 4xx (non-429), "insufficient margin" | Venue received and explicitly refused | ✅ YES |
|
||||
| **INDETERMINATE** | `ReadTimeout`, `ConnectReset`, 5xx, transport death | **Request was sent, answer lost** — order MAY be live | ❌ **NEVER** |
|
||||
|
||||
---
|
||||
|
||||
## 3. THE SEAMS: WHERE THIS BUG CLASS MANIFESTS
|
||||
|
||||
### 3.1 Seam 1: `bingx_direct.submit_intent()` — The Origin
|
||||
|
||||
**File:** `/mnt/dolphinng5_predict/prod/clean_arch/adapters/bingx_direct.py`
|
||||
**Method:** `BingxDirectExecutionAdapter.submit_intent()` (lines ~850-1100)
|
||||
|
||||
**Critical Sections:**
|
||||
- **Lines ~950-1020**: `try/except BingxHttpError` block — the classification logic
|
||||
- **Lines ~970-1010**: `_lookup_own_order_by_client_id()` — the point lookup that reconciles truth
|
||||
- **Lines ~1010-1050**: Status assignment (`REJECTED` vs `INDETERMINATE` vs `ABSENT`)
|
||||
|
||||
**Enumerated Failure Modes Here:**
|
||||
| Failure | Classification | Root Cause |
|
||||
|---------|----------------|------------|
|
||||
| `httpx.ReadTimeout` | INDETERMINATE | Request sent, response lost |
|
||||
| `httpx.ConnectTimeout` | NOT_ATTEMPTED | Never left the box |
|
||||
| `httpx.ConnectError("refused")` | NOT_ATTEMPTED | Connection refused |
|
||||
| 500/502/503/504 | INDETERMINATE | Server error, request may have been processed |
|
||||
| `httpx.RemoteProtocolError` | INDETERMINATE | Protocol violation mid-stream |
|
||||
| `httpx.RemoteProtocolError` (partial response) | INDETERMINATE | May have been accepted |
|
||||
| TLS handshake failure | NOT_ATTEMPTED | Never established secure channel |
|
||||
| `BingxHttpError` with unknown code | INDETERMINATE (default) | Conservative: assume may be live |
|
||||
|
||||
**Mitigation (Implemented):**
|
||||
- Lines ~950-1020: Three-way classification with `_lookup_own_order_by_client_id()` reconciliation
|
||||
- Lines ~1010-1050: `status = "INDETERMINATE"` path that does NOT set `REJECTED`
|
||||
|
||||
---
|
||||
|
||||
### 3.2 Seam 2: `BingxVenueAdapter.submit()` / `submit_async()` — The Venue Layer
|
||||
|
||||
**File:** `/mnt/dolphinng5_predict/prod/clean_arch/dita_v2/bingx_venue.py`
|
||||
**Methods:** `submit()` (sync, line ~800) and `submit_async()` (async, line ~950)
|
||||
|
||||
**Critical Logic:**
|
||||
```python
|
||||
# Both submit() and submit_async() share this pattern:
|
||||
receipt = self._call_backend("submit_intent", legacy) # or await backend.submit_intent_async()
|
||||
|
||||
# ── UNKNOWN OUTCOME ──
|
||||
if str(getattr(receipt, "status", "") or "").upper() == "INDETERMINATE":
|
||||
raise VenueIndeterminateError(
|
||||
f"submit outcome UNKNOWN for intent={intent.intent_id} asset={intent.asset} "
|
||||
f"— order may be live at the venue; refusing to claim rejection",
|
||||
receipt=receipt,
|
||||
)
|
||||
```
|
||||
|
||||
**Error Hierarchy:**
|
||||
```
|
||||
VenuePostAckError (base)
|
||||
└── VenueIndeterminateError ← raised for INDETERMINATE status
|
||||
```
|
||||
|
||||
**Mitigation (Implemented):**
|
||||
- Both sync and async paths check `receipt.status == "INDETERMINATE"`
|
||||
- Raises `VenueIndeterminateError` (subclass of `VenuePostAckError`)
|
||||
- Kernel's "no-rollback fences" catch `VenuePostAckError` (see Seam 3)
|
||||
|
||||
---
|
||||
|
||||
### 3.3 Seam 3: `ExecutionKernel.process_intent()` / `process_intent_async()` — The Kernel Boundary
|
||||
|
||||
**File:** `/mnt/dolphinng5_predict/prod/clean_arch/dita_v2/rust_backend.py`
|
||||
**Methods:** `process_intent()` (sync, line ~880) and `process_intent_async()` (async, line ~1050)
|
||||
|
||||
**Critical Logic (sync):**
|
||||
```python
|
||||
try:
|
||||
emitted_events = self.venue.submit(intent)
|
||||
except VenuePostAckError as _post_ack_exc:
|
||||
# ORDER IS LIVE AT THE VENUE. Unknown != flat.
|
||||
# NO rollback, NO synthetic REJECT.
|
||||
logger.critical("FIX(rust_backend): POST-ACK venue failure — ORDER IS LIVE. NOT rolling back.")
|
||||
emitted_events = list(getattr(_post_ack_exc, "events", []) or [])
|
||||
except Exception as _submit_exc:
|
||||
# PRE-ACK failure: venue never took the order. Safe to roll back.
|
||||
logger.error("venue.submit failed — synthetic REJECTED to roll back FSM")
|
||||
emitted_events = [synthetic_REJECTED_event]
|
||||
```
|
||||
|
||||
**Critical Logic (async):**
|
||||
```python
|
||||
submit_async = getattr(self.venue, "submit_async", None)
|
||||
try:
|
||||
if submit_async is not None:
|
||||
emitted_events = await submit_async(intent)
|
||||
else:
|
||||
emitted_events = self.venue.submit(intent)
|
||||
except VenuePostAckError as _post_ack_exc:
|
||||
# SAME LOGIC: ORDER IS LIVE. NO ROLLBACK.
|
||||
logger.critical("FIX(rust_backend): POST-ACK venue failure — ORDER IS LIVE. NOT rolling back.")
|
||||
emitted_events = list(getattr(_post_ack_exc, "events", []) or [])
|
||||
except Exception as _submit_exc:
|
||||
# PRE-ACK: synthetic REJECT to roll back FSM
|
||||
emitted_events = [synthetic_REJECTED_event]
|
||||
```
|
||||
|
||||
**The "No-Rollback Fence" — THE KEY MITIGATION:**
|
||||
```python
|
||||
# In on_venue_event (and process_intent), kernel checks:
|
||||
if isinstance(exc, VenuePostAckError):
|
||||
# NO ROLLBACK — slot stays in ENTRY_WORKING / EXIT_WORKING
|
||||
# Let E-feed FILL / reconcile settle it
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3.4 Seam 4: `on_venue_event()` — The Event Ingestion Path
|
||||
|
||||
**File:** `/mnt/dolphinng5_predict/prod/clean_arch/dita_v2/rust_backend.py`
|
||||
**Method:** `on_venue_event()` (line ~1200)
|
||||
|
||||
**Logic:**
|
||||
```python
|
||||
def on_venue_event(self, event: VenueEvent) -> KernelOutcome:
|
||||
# Processes all venue events through the Rust FSM
|
||||
# If event.status == REJECTED and slot is in ENTRY_WORKING/EXIT_WORKING:
|
||||
# → FSM transitions to IDLE (flat)
|
||||
# If event.status == INDETERMINATE:
|
||||
# → Venue layer NEVER emits this; it raises VenueIndeterminateError instead
|
||||
```
|
||||
|
||||
**Key Point:** The venue adapter **never emits an `INDETERMINATE` event** — it raises `VenueIndeterminateError` instead. The kernel only sees events with `status` ∈ {ACKED, REJECTED, FILLED, PARTIAL_FILL, CANCELED, CANCEL_REJECTED, RATE_LIMITED}.
|
||||
|
||||
---
|
||||
|
||||
### 3.5 Seam 5: Rust FSM (Hidden in `rust_backend.so`) — The Core State Machine
|
||||
|
||||
**Location:** `/mnt/dolphinng5_predict/prod/clean_arch/dita_v2/_rust_kernel/` (built into `libdita_v2_kernel.so`)
|
||||
|
||||
**State Machine (Simplified):**
|
||||
```
|
||||
IDLE
|
||||
──ENTER_intent──→ ORDER_REQUESTED
|
||||
──venue.submit() succeeds (ACKED)──→ ENTRY_WORKING
|
||||
──FILL event──→ OPEN
|
||||
──REJECTED event──→ IDLE (rollback)
|
||||
──INDETERMINATE (VenuePostAckError)──→ STAYS ENTRY_WORKING (no rollback)
|
||||
──venue.submit() fails (PRE-ACK)──→ synthetic REJECTED → IDLE
|
||||
──EXIT_intent──→ ORDER_REQUESTED (exit)
|
||||
──venue.submit() succeeds──→ EXIT_WORKING
|
||||
──FILL event──→ IDLE
|
||||
──REJECTED event──→ IDLE
|
||||
──INDETERMINATE──→ STAYS EXIT_WORKING
|
||||
```
|
||||
|
||||
**Rust FSM Responsibility:**
|
||||
- Manages `TradeSlot.fsm_state` (IDLE, ORDER_REQUESTED, ENTRY_WORKING, OPEN, EXIT_WORKING, etc.)
|
||||
- Transitions are deterministic given event `(kind, status)`
|
||||
- **Does NOT know about INDETERMINATE** — it only sees events the Python layer emits
|
||||
- The Python layer is the **semantic firewall**: it decides whether to emit a `REJECTED` event or suppress it
|
||||
|
||||
---
|
||||
|
||||
### 3.6 Seam 6: `on_venue_event()` — The Feedback Loop
|
||||
|
||||
**File:** `/mnt/dolphinng5_predict/prod/clean_arch/dita_v2/rust_backend.py`
|
||||
**Method:** `on_venue_event()` (line ~1200)
|
||||
|
||||
```python
|
||||
def on_venue_event(self, event: VenueEvent) -> KernelOutcome:
|
||||
# All venue events flow here → Rust FSM → state transitions
|
||||
# If event.status == REJECTED:
|
||||
# kernel.fsm_state → IDLE (flat)
|
||||
# If event.status == FILLED/PARTIAL_FILL:
|
||||
# kernel.fsm_state → OPEN / IDLE
|
||||
```
|
||||
|
||||
**Critical Invariant:** The kernel **only rolls back to IDLE on explicit `REJECTED` events**. It has no concept of "unknown".
|
||||
|
||||
---
|
||||
|
||||
## 4. ENUMERATED UNDESIRABLE STATES & TRANSITIONS
|
||||
|
||||
### 4.1 States That SHOULD Exist But Don't
|
||||
|
||||
| Missing State | Description | Why It Matters |
|
||||
|---------------|-------------|----------------|
|
||||
| `ENTRY_INDETERMINATE` | Order sent, answer lost — may be live | Distinguishes "we don't know" from "working normally" |
|
||||
| `EXIT_INDETERMINATE` | Exit sent, answer lost — may be live | Same for exits |
|
||||
| `ORDER_TRUTH_UNKNOWN` | Lookup failed after N attempts | Explicit "truth unknown" state vs silent failure |
|
||||
| `VENUE_LOOKUP_PENDING` | Background reconcile in progress | Prevents duplicate reconcile attempts |
|
||||
| `RECONCILE_IN_PROGRESS` | Full reconcile running | Prevents concurrent reconciles |
|
||||
|
||||
### 4.2 Transitions That Should Not Exist (But Do)
|
||||
|
||||
| Transition | Why It's Bad | Where It Happens |
|
||||
|------------|--------------|------------------|
|
||||
| `ENTRY_WORKING` → `IDLE` on `INDETERMINATE` | Orphans live position | `bingx_direct` pre-2026-07-13 (FIXED) |
|
||||
| `EXIT_WORKING` → `IDLE` on `INDETERMINATE` | Orphans live position, no SL/TP | Same as above |
|
||||
| `ENTRY_WORKING` → `IDLE` on `VenuePostAckError` (broad) | Over-broad catch rolls back live orders | `rust_backend.py` line ~900 (MITIGATED: catches `VenuePostAckError` specifically) |
|
||||
| `ORDER_REQUESTED` → `IDLE` without event | Silent state loss | Never directly, but possible via exception path |
|
||||
|
||||
### 4.3 States That Should Be Unreachable But Are Reachable
|
||||
|
||||
| State | How It's Reached | Why It's Bad |
|
||||
|-------|------------------|--------------|
|
||||
| `SLOT_BUSY` with no active order | FSM advanced but order never sent | `process_intent` exception before `venue.submit` |
|
||||
| `ENTRY_WORKING` with `active_entry_order = None` | `attach_entry_order` not called | `venue.submit` exception before `attach_entry_order` |
|
||||
| `OPEN` with `position = None` | Phantom fill event | E-feed race, duplicate fill |
|
||||
| `IDLE` with `trade_id` set | Incomplete cleanup | `reset()` not called on all fields |
|
||||
|
||||
### 4.4 Foreseeable "Phantom" States (Unforeseeable in Practice)
|
||||
|
||||
| Scenario | Resulting State | Detection |
|
||||
|----------|-----------------|-----------|
|
||||
| BingX accepts order, sends ACK, TCP dies before ACK arrives | Venue: LIVE / Kernel: ENTRY_WORKING (no ACK event) | `_lookup_own_order_by_client_id` reconciliation |
|
||||
| BingX fills order, sends FILL, WS dies before delivery | Venue: FILLED / Kernel: ENTRY_WORKING | E-feed FILL event / reconcile |
|
||||
| BingX rejects order, sends REJECTED, TCP dies | Venue: REJECTED / Kernel: ENTRY_WORKING | Reconcile finds REJECTED |
|
||||
| Network partition: client can't reach BingX, but BingX can fill | Venue: FILLED / Kernel: INDETERMINATE | Reconcile + E-feed |
|
||||
| Duplicate `clientOrderId` sent (retries) → BingX returns original order | Venue: FILLED / Kernel: sees duplicate | `clientOrderId` deduplication in `_lookup` |
|
||||
|
||||
---
|
||||
|
||||
## 5. MITIGATIONS — IMPLEMENTED, PARTIAL, PROPOSED
|
||||
|
||||
### 5.1 Implemented (Post-2026-07-13)
|
||||
|
||||
| Mitigation | Location | What It Prevents |
|
||||
|------------|----------|------------------|
|
||||
| Three-way HTTP classification | `bingx_direct.py` lines ~950-1020 | Misclassification of INDETERMINATE as REJECTED |
|
||||
| `_lookup_own_order_by_client_id()` | `bingx_direct.py` lines ~1030-1100 | Reconciling truth from venue after INDETERMINATE |
|
||||
| `VenueIndeterminateError` exception | `bingx_venue.py` line ~859/937 | Semantic escalation: INDETERMINATE ≠ REJECTED |
|
||||
| `VenuePostAckError` catch (no rollback) | `rust_backend.py` lines ~900, ~1080 | Kernel no-rollback fence for INDETERMINATE |
|
||||
| `VenueIndeterminateError` inherits `VenuePostAckError` | `venue.py:43` | Existing no-rollback fence catches it |
|
||||
| `_lookup_own_order_by_client_id` bounded retries | `bingx_direct.py` line ~1070 | Bounded reconciliation, then gives up with `None` |
|
||||
| `VenuePostAckError` catch in async path too | `rust_backend.py` line ~1080 | Async path same protection as sync |
|
||||
| No synthetic event on INDETERMINATE | `rust_backend.py` lines ~910, ~1090 | No synthetic REJECT pollutes event log |
|
||||
|
||||
### 5.2 Partial / Incomplete
|
||||
|
||||
| Gap | Location | Risk |
|
||||
|-----|----------|------|
|
||||
| No `ENTRY_INDETERMINATE` / `EXIT_INDETERMINATE` FSM states | Rust FSM | Can't distinguish "working normally" from "truth unknown" in state |
|
||||
| No `ORDER_TRUTH_UNKNOWN` explicit state | `bingx_direct._lookup_own_order` | Returns `None` but no state reflects this |
|
||||
| No background reconcile task for INDETERMINATE orders | `bingx_direct._s2_tasks` | Only MARKET fills trigger background refresh |
|
||||
| No metrics/alerting on INDETERMINATE frequency | None | Can't detect systemic issues |
|
||||
| No alert on `VenueIndeterminateError` rate | Logging only | Operators don't know it's happening |
|
||||
|
||||
### 5.3 Proposed (For Future Hardening)
|
||||
|
||||
| Proposal | Effort | Impact |
|
||||
|----------|--------|--------|
|
||||
| Add `ENTRY_INDETERMINATE` / `EXIT_INDETERMINATE` FSM states | Rust FSM + Python bindings | Explicit "truth unknown" state visible in TUI/monitoring |
|
||||
| Add `OrderTruth` enum: `KNOWN_LIVE`, `KNOWN_FLAT`, `UNKNOWN` | `bingx_direct` + kernel | Explicit truth tracking per order |
|
||||
| Background reconcile task for INDETERMINATE orders | New async task in `DolphinLiveTrader` | Proactive truth discovery |
|
||||
| `INDETERMINATE` metric + alert | Prometheus/Grafana | Operator visibility |
|
||||
| `OrderTruth` persisted in Zinc/CH | Zinc plane + CH writer | Survives restarts |
|
||||
| `ReconcileManager` service | New module | Centralized truth discovery |
|
||||
| `ORDER_TRUTH_UNKNOWN` event type | Contracts + kernel | Explicit event for "we don't know" |
|
||||
|
||||
---
|
||||
|
||||
## 6. ENUMERATED INSTANCES IN CODEBASE
|
||||
|
||||
### 6.1 Where INDETERMINATE Classification Happens
|
||||
|
||||
| File | Line | Function | Purpose |
|
||||
|------|------|----------|---------|
|
||||
| `prod/bingx/http.py` | ~50 | `BingxHttpError.effect` | Classification property |
|
||||
| `prod/bingx/http.py` | ~60 | `_effect_of_httpx_error()` | Maps `httpx` exceptions |
|
||||
| `prod/bingx/http.py` | ~80 | `_effect_of_status()` | Maps HTTP status codes |
|
||||
| `prod/clean_arch/adapters/bingx_direct.py` | ~950 | `submit_intent()` exception handler | Main classification |
|
||||
| `prod/clean_arch/adapters/bingx_direct.py` | ~1030 | `_lookup_own_order_by_client_id()` | Truth reconciliation |
|
||||
| `prod/clean_arch/dita_v2/bingx_venue.py` | ~859 | `submit()` INDETERMINATE check | Sync venue |
|
||||
| `prod/clean_arch/dita_v2/bingx_venue.py` | ~937 | `submit_async()` INDETERMINATE check | Async venue |
|
||||
|
||||
### 6.2 Where INDETERMINATE Is Caught / Escalated
|
||||
|
||||
| File | Line | Context |
|
||||
|------|------|---------|
|
||||
| `prod/clean_arch/dita_v2/bingx_venue.py` | ~859 | `raise VenueIndeterminateError(...)` (sync) |
|
||||
| `prod/clean_arch/dita_v2/bingx_venue.py` | ~937 | `raise VenueIndeterminateError(...)` (async) |
|
||||
| `prod/clean_arch/dita_v2/rust_backend.py` | ~900 | `except VenuePostAckError:` (sync) |
|
||||
| `prod/clean_arch/dita_v2/rust_backend.py` | ~1080 | `except VenuePostAckError:` (async) |
|
||||
| `prod/clean_arch/dita_v2/venue.py` | 43 | `class VenueIndeterminateError(VenuePostAckError)` |
|
||||
|
||||
### 6.3 Where No-Rollback Fence Exists
|
||||
|
||||
| File | Line | Protection |
|
||||
|------|------|------------|
|
||||
| `prod/clean_arch/dita_v2/rust_backend.py` | ~900 | `except VenuePostAckError:` in sync `process_intent` |
|
||||
| `prod/clean_arch/dita_v2/rust_backend.py` | ~1080 | `except VenuePostAckError:` in async `process_intent_async` |
|
||||
|
||||
### 6.4 Where Synthetic REJECTED Is Emitted (PRE-ACK Only)
|
||||
|
||||
| File | Line | Context |
|
||||
|------|------|---------|
|
||||
| `prod/clean_arch/dita_v2/rust_backend.py` | ~920 | Sync: `synthetic_REJECTED_event` on generic `Exception` |
|
||||
| `prod/clean_arch/dita_v2/rust_backend.py` | ~1110 | Async: `synthetic_REJECTED_event` on generic `Exception` |
|
||||
|
||||
---
|
||||
|
||||
## 7. TEST VERIFICATION MATRIX
|
||||
|
||||
**File:** `/mnt/dolphinng5_predict/prod/clean_arch/dita_v2/test_indeterminate_submit.py`
|
||||
|
||||
| Test | What It Verifies |
|
||||
|------|------------------|
|
||||
| `test_read_timeout_is_indeterminate_not_rejection` | `ReadTimeout` → INDETERMINATE |
|
||||
| `test_connect_failures_are_not_attempted` | `ConnectError`/`ConnectTimeout` → NOT_ATTEMPTED |
|
||||
| `test_5xx_is_indeterminate_and_4xx_is_refused` | 5xx → INDETERMINATE, 4xx → REFUSED |
|
||||
| `test_unknown_failure_defaults_to_indeterminate` | Unknown error → INDETERMINATE (conservative) |
|
||||
| `test_lookup_finds_the_order_and_never_posts` | Reconcile finds live order, never re-POSTs |
|
||||
| `test_lookup_reports_absent_when_venue_has_no_such_order` | Venue says "no order" → ABSENT (rollback sound) |
|
||||
| `test_lookup_treats_venue_refusal_as_absent` | Venue REFUSES lookup → ABSENT |
|
||||
| `test_lookup_returns_none_when_truth_cannot_be_established` | Timeout on lookup → None (not ABSENT) |
|
||||
| `test_sync_venue_escalates_indeterminate_receipt` | `VenueIndeterminateError` raised |
|
||||
| `test_async_venue_escalates_indeterminate_receipt` | Async path same |
|
||||
| `test_indeterminate_is_a_post_ack_error_so_existing_fences_catch_it` | Subclass check |
|
||||
| `test_kernel_does_not_roll_back_slot_on_indeterminate_submit` | Sync kernel no-rollback |
|
||||
| `test_kernel_does_not_roll_back_slot_on_indeterminate_submit_async` | Async kernel no-rollback |
|
||||
| `test_genuine_rejection_still_rolls_the_slot_back` | REFUSED still rolls back (contrast test) |
|
||||
|
||||
---
|
||||
|
||||
## 8. DECISION LOG (Appendix)
|
||||
|
||||
| Date | Decision | Rationale |
|
||||
|------|----------|-----------|
|
||||
| 2026-07-13 | Classify `ReadTimeout`/`5xx` as INDETERMINATE | Request sent, answer lost — order may be live |
|
||||
| 2026-07-13 | Default unknown errors to INDETERMINATE | Conservative: never guess "rejected" |
|
||||
| 2026-07-13 | `_lookup_own_order_by_client_id` bounded retries | Prevent infinite retry loops |
|
||||
| 2026-07-13 | `VenueIndeterminateError` inherits `VenuePostAckError` | Reuses existing no-rollback fence |
|
||||
| 2026-07-13 | NO synthetic event on INDETERMINATE | Don't pollute event log with guesses |
|
||||
| 2026-07-13 | `VenuePostAckError` catch in BOTH sync/async | Both code paths protected |
|
||||
|
||||
---
|
||||
|
||||
## 9. OPERATOR RUNBOOK: INDETERMINATE INCIDENT
|
||||
|
||||
**If you see `VenueIndeterminateError` in logs:**
|
||||
|
||||
1. **Do NOT manually flatten the slot** — the order may be live
|
||||
2. Check BingX VST/LIVE for the `clientOrderId` (logged in the error)
|
||||
3. If position exists at venue: let E-feed FILL settle it, or manually reconcile
|
||||
4. If position NOT at venue: kernel will eventually reconcile or you can trigger manual reconcile
|
||||
5. **Do NOT restart the kernel to "clear" the slot** — this loses the `ENTRY_WORKING`/`EXIT_WORKING` state
|
||||
|
||||
**Log Signature:**
|
||||
```
|
||||
CRITICAL FIX(bingx_direct): submit outcome INDETERMINATE (ReadTimeout("..."))
|
||||
symbol=TRXUSDT clientOrderId=p-e-1q3k7m-ab4c — order MAY be live. Looking it up;
|
||||
NOT assuming rejection.
|
||||
```
|
||||
|
||||
**Resolution Paths:**
|
||||
- `found == dict`: Order is LIVE → kernel adopts venue truth, slot stays OPEN/WORKING
|
||||
- `found == "ABSENT"`: Order never existed → rollback IS sound, slot → IDLE
|
||||
- `found == None`: Truth unknown → slot stays in ENTRY_WORKING/EXIT_WORKING, E-feed FILL settles it
|
||||
|
||||
---
|
||||
|
||||
## 10. FUTURE WORK PRIORITY ORDER
|
||||
|
||||
| Priority | Item | Effort | Blocker |
|
||||
|----------|------|--------|---------|
|
||||
| P0 | `ENTRY_INDETERMINATE` / `EXIT_INDETERMINATE` FSM states | Medium | Rust FSM changes + Python bindings |
|
||||
| P0 | Background reconcile task for INDETERMINATE orders | Low | New async task in trader loop |
|
||||
| P1 | `OrderTruth` enum + Zinc persistence | Medium | Zinc plane schema change |
|
||||
| P1 | INDETERMINATE metrics + Grafana alert | Low | Prometheus exporter |
|
||||
| P2 | `ReconcileManager` service | High | New module, cross-cutting |
|
||||
| P2 | `ORDER_TRUTH_UNKNOWN` event type | Medium | Contracts + kernel + venue adapters |
|
||||
|
||||
---
|
||||
|
||||
## 11. FABLE REVIEW (2026-07-13) — verdict + MISSING SEAMS (fleet work)
|
||||
|
||||
**Truth audit:** Doc is faithful to the shipped fix (verified: `_is_rate_limited_error`
|
||||
exists at `bingx_direct.py:127`; test matrix §7 matches the real 14-test suite;
|
||||
FSM description matches kernel-invariant tests). Two factual errors corrected
|
||||
in-place: `VenueIndeterminateError` lives at `venue.py:43` (not contracts.py);
|
||||
fix date is 2026-07-13 (commits `bdc54fb` canonical / `54cd5d5` vendored).
|
||||
|
||||
**Scope gap — THE assignment ("extrapolate the class across the codebase") is
|
||||
unfinished.** §3 covers the SUBMIT seam deeply; the same not-done ≠ tried-and-
|
||||
failed ≠ unknown triage is UNAUDITED at every other venue-side-effect seam:
|
||||
|
||||
| # | Seam | Specific risk |
|
||||
|---|------|---------------|
|
||||
| M1 | `cancel()` / `cancel_async()` | cancel-INDETERMINATE = order MAY still be live → blind re-place = double fill. (Encoded in SPEC_UV_SMART_EXEC_MM §5; kernel path unaudited.) |
|
||||
| M2 | leverage SET (`bingx_direct`) | On the money path (seen live 17:20 today). SET-INDETERMINATE → position sized at unknown leverage. |
|
||||
| M3 | promotion bridge receipts (`uv.promotion`) | Bridge assumes receipt is truth; INDETERMINATE mid-bridge unhandled? |
|
||||
| M4 | E-feed / EKBridge gap handling | WS death between ACK and FILL — partially covered by rehydrate; no triage taxonomy. |
|
||||
| M5 | exit-leg placement (multi-leg exits) | Leg 1 fills, leg 2 INDETERMINATE → partial exit with unknown remainder. |
|
||||
| M6 | ch_writer / journal lane | Covered for data (spool), but write-INDETERMINATE→retry = possible duplicate rows (dedup ratchet history). |
|
||||
| M7 | every `except: pass` / broad `except Exception` after venue I/O | Grep-sweep + classify each; the b46ebd2 class hid here. |
|
||||
|
||||
**DOCTRINE GUARDRAIL on §10 P0 "background reconcile task":** operator standing
|
||||
order — reconcile is proto-PINK hell. Any background truth-discovery MUST stay
|
||||
a bounded, read-only, own-clientOrderId point lookup (the §2.2 primitive), never
|
||||
a scan-all-orders reconciler. Build the loop around the existing primitive.
|
||||
|
||||
**Disposition:** M1–M7 are fleet-sized (audit + tests per seam, same pattern as
|
||||
`test_indeterminate_submit.py`). Corrections done in-place are complete.
|
||||
|
||||
---
|
||||
|
||||
**END OF DOCUMENT**
|
||||
*This document is a living reference. Update on every INDETERMINATE-related change.*
|
||||
@@ -73,10 +73,14 @@ journal lane auto-drains the spool or rows need manual JSONEachRow replay
|
||||
ExecutionRouter (maker/taker policy, 265 tests) ∪ BLUE SmartPlacer (OB-aware
|
||||
placement) vs prod/docs/BingX_FILL_CHARACTERIZATION_AND_ADVANTAGES.md. Router
|
||||
under 1s steel clock = maker orders actually managed.
|
||||
8. **Indeterminate sweep codebase-wide** (belled to codex, napped): cancel/
|
||||
cancel_async, EKBridge, e_feed, ch_writer, ExecutionRouter, promotion receipts,
|
||||
every `except: pass` after venue I/O — same NOT_ATTEMPTED/REFUSED/INDETERMINATE
|
||||
triage as bdc54fb.
|
||||
8. **Indeterminate sweep codebase-wide — NOW SPECCED FOR FLEET**: pi red-teamed
|
||||
the class (`COMPREHENSIVE_UV_EDGE_CASE_THEORETICALS.md`); Fable reviewed
|
||||
2026-07-13 eve: doc FAITHFUL (2 factual errors fixed in-place: venue.py:43
|
||||
location, 07-13 date), submit seam fully covered, **7 missing seams
|
||||
enumerated in its §11 (M1–M7)**: cancel, leverage SET, promotion receipts,
|
||||
E-feed gaps, exit legs, ch_writer dups, broad-except sweep. §11 carries the
|
||||
NO-RECONCILE guardrail for the P0 background task. Fleet-ready: one seam
|
||||
per agent; test pattern = test_indeterminate_submit.py.
|
||||
9. **TIER-B inversion** (open): asex_kernel_executor.py:319 enqueues account/fill
|
||||
at P4 below ENTER P3 — execution truth must outrank entries.
|
||||
10. **Research studies** (logged POST_FINISH): dvol sweep × regime (PREREQ: persist
|
||||
|
||||
Reference in New Issue
Block a user