sentiment-engine/prod/docs/COMPREHENSIVE_UV_EDGE_CASE_THEORETICALS.md

COMPREHENSIVE UV EDGE CASE THEORETICALS

State Machine Edge Cases, Undesirable States, and Mitigations for DITAv2

Document Status: Reference for operators and engineers
Last Updated: 2026-07-13 (Fable commit: INDETERMINATE submit fix)
Scope: DITAv2 execution kernel, BingX venue adapter, bingx_direct adapter, Rust-backed FSM

---

1. EXECUTIVE SUMMARY

The 2026-07-13 Fable commit (INDETERMINATE submit fix) revealed a fundamental state machine flaw: treating an INDETERMINATE HTTP outcome (request sent, answer lost) as a confident REJECTED with fill_qty=0. This caused the kernel FSM to roll back to IDLE (flat) while the venue kept the position live — creating orphan positions that had no SL/TP/ADVSL protection.

This document catalogs:

1. The root flaw (INDETERMINATE ≡ REJECTED misclassification)
2. All seams where this class of bug can manifest
3. Foreseeable & unforeseeable state transitions that are undesirable
4. Missing states/transitions that should exist
5. Mitigations (implemented, partial, and proposed)

---

2. THE ROOT FLAW: INDETERMINATE ≠ REJECTED

2.1 What Happened

# OLD bingx_direct.py (pre-2026-07-13)
except BingxHttpError as exc:
    status = "REJECTED"  # ← WRONG: treated timeout/5xx as confident rejection
    ack_row = {"status": "REJECTED", "msg": str(exc), ...}
    fill_qty = 0.0  # ← kernel rolls back to IDLE, believes flat

2.2 The Correction (2026-07-13)

# NEW bingx_direct.py (post-fix)
except BingxHttpError as exc:
    effect = getattr(exc, "effect", BingxHttpError.INDETERMINATE)
    if effect == BingxHttpError.INDETERMINATE and not _is_rate_limited_error(exc):
        # THE ORDER MAY BE LIVE. We do NOT guess. We ask the venue.
        found = await self._lookup_own_order_by_client_id(...)
        if isinstance(found, dict):
            # Venue confirms: order exists → adopt its state
            status = ack_row.get("status") or "ACKED"
        elif found == "ABSENT":
            # Venue says "no such order" → rollback IS sound
            status = "REJECTED"
        else:
            # TRUTH NOT ESTABLISHED. We REFUSE to claim rejection.
            status = "INDETERMINATE"
            # Venue layer escalates to VenueIndeterminateError

2.3 The Three-Way Classification (NOW ENFORCED)

Classification	HTTP Examples	What It Proves	Rollback Sound?
NOT_ATTEMPTED	ConnectError, ConnectTimeout, DNS failure	Request never left the box	✅ YES
REFUSED	4xx (non-429), "insufficient margin"	Venue received and explicitly refused	✅ YES
INDETERMINATE	ReadTimeout, ConnectReset, 5xx, transport death	Request was sent, answer lost — order MAY be live	❌ NEVER

---

3. THE SEAMS: WHERE THIS BUG CLASS MANIFESTS

3.1 Seam 1: bingx_direct.submit_intent() — The Origin

File: /mnt/dolphinng5_predict/prod/clean_arch/adapters/bingx_direct.py
Method: BingxDirectExecutionAdapter.submit_intent() (lines ~850-1100)

Critical Sections:

• Lines ~950-1020: try/except BingxHttpError block — the classification logic
• Lines ~970-1010: _lookup_own_order_by_client_id() — the point lookup that reconciles truth
• Lines ~1010-1050: Status assignment (REJECTED vs INDETERMINATE vs ABSENT)

Enumerated Failure Modes Here:

Failure	Classification	Root Cause
httpx.ReadTimeout	INDETERMINATE	Request sent, response lost
httpx.ConnectTimeout	NOT_ATTEMPTED	Never left the box
httpx.ConnectError("refused")	NOT_ATTEMPTED	Connection refused
500/502/503/504	INDETERMINATE	Server error, request may have been processed
httpx.RemoteProtocolError	INDETERMINATE	Protocol violation mid-stream
httpx.RemoteProtocolError (partial response)	INDETERMINATE	May have been accepted
TLS handshake failure	NOT_ATTEMPTED	Never established secure channel
BingxHttpError with unknown code	INDETERMINATE (default)	Conservative: assume may be live

Mitigation (Implemented):

• Lines ~950-1020: Three-way classification with _lookup_own_order_by_client_id() reconciliation
• Lines ~1010-1050: status = "INDETERMINATE" path that does NOT set REJECTED

---

3.2 Seam 2: BingxVenueAdapter.submit() / submit_async() — The Venue Layer

File: /mnt/dolphinng5_predict/prod/clean_arch/dita_v2/bingx_venue.py
Methods: submit() (sync, line ~800) and submit_async() (async, line ~950)

Critical Logic:

# Both submit() and submit_async() share this pattern:
receipt = self._call_backend("submit_intent", legacy)  # or await backend.submit_intent_async()

# ── UNKNOWN OUTCOME ──
if str(getattr(receipt, "status", "") or "").upper() == "INDETERMINATE":
    raise VenueIndeterminateError(
        f"submit outcome UNKNOWN for intent={intent.intent_id} asset={intent.asset} "
        f"— order may be live at the venue; refusing to claim rejection",
        receipt=receipt,
    )

Error Hierarchy:

VenuePostAckError (base)
  └── VenueIndeterminateError  ← raised for INDETERMINATE status

Mitigation (Implemented):

• Both sync and async paths check receipt.status == "INDETERMINATE"
• Raises VenueIndeterminateError (subclass of VenuePostAckError)
• Kernel's "no-rollback fences" catch VenuePostAckError (see Seam 3)

---

3.3 Seam 3: ExecutionKernel.process_intent() / process_intent_async() — The Kernel Boundary

File: /mnt/dolphinng5_predict/prod/clean_arch/dita_v2/rust_backend.py
Methods: process_intent() (sync, line ~880) and process_intent_async() (async, line ~1050)

Critical Logic (sync):

try:
    emitted_events = self.venue.submit(intent)
except VenuePostAckError as _post_ack_exc:
    # ORDER IS LIVE AT THE VENUE. Unknown != flat.
    # NO rollback, NO synthetic REJECT.
    logger.critical("FIX(rust_backend): POST-ACK venue failure — ORDER IS LIVE. NOT rolling back.")
    emitted_events = list(getattr(_post_ack_exc, "events", []) or [])
except Exception as _submit_exc:
    # PRE-ACK failure: venue never took the order. Safe to roll back.
    logger.error("venue.submit failed — synthetic REJECTED to roll back FSM")
    emitted_events = [synthetic_REJECTED_event]

Critical Logic (async):

submit_async = getattr(self.venue, "submit_async", None)
try:
    if submit_async is not None:
        emitted_events = await submit_async(intent)
    else:
        emitted_events = self.venue.submit(intent)
except VenuePostAckError as _post_ack_exc:
    # SAME LOGIC: ORDER IS LIVE. NO ROLLBACK.
    logger.critical("FIX(rust_backend): POST-ACK venue failure — ORDER IS LIVE. NOT rolling back.")
    emitted_events = list(getattr(_post_ack_exc, "events", []) or [])
except Exception as _submit_exc:
    # PRE-ACK: synthetic REJECT to roll back FSM
    emitted_events = [synthetic_REJECTED_event]

The "No-Rollback Fence" — THE KEY MITIGATION:

# In on_venue_event (and process_intent), kernel checks:
if isinstance(exc, VenuePostAckError):
    # NO ROLLBACK — slot stays in ENTRY_WORKING / EXIT_WORKING
    # Let E-feed FILL / reconcile settle it

---

3.4 Seam 4: on_venue_event() — The Event Ingestion Path

File: /mnt/dolphinng5_predict/prod/clean_arch/dita_v2/rust_backend.py
Method: on_venue_event() (line ~1200)

Logic:

def on_venue_event(self, event: VenueEvent) -> KernelOutcome:
    # Processes all venue events through the Rust FSM
    # If event.status == REJECTED and slot is in ENTRY_WORKING/EXIT_WORKING:
    #   → FSM transitions to IDLE (flat)
    # If event.status == INDETERMINATE:
    #   → Venue layer NEVER emits this; it raises VenueIndeterminateError instead

Key Point: The venue adapter never emits an INDETERMINATE event — it raises VenueIndeterminateError instead. The kernel only sees events with status ∈ {ACKED, REJECTED, FILLED, PARTIAL_FILL, CANCELED, CANCEL_REJECTED, RATE_LIMITED}.

---

3.5 Seam 5: Rust FSM (Hidden in rust_backend.so) — The Core State Machine

Location: /mnt/dolphinng5_predict/prod/clean_arch/dita_v2/_rust_kernel/ (built into libdita_v2_kernel.so)

State Machine (Simplified):

IDLE
  ──ENTER_intent──→ ORDER_REQUESTED
    ──venue.submit() succeeds (ACKED)──→ ENTRY_WORKING
      ──FILL event──→ OPEN
      ──REJECTED event──→ IDLE (rollback)
      ──INDETERMINATE (VenuePostAckError)──→ STAYS ENTRY_WORKING (no rollback)
    ──venue.submit() fails (PRE-ACK)──→ synthetic REJECTED → IDLE
  ──EXIT_intent──→ ORDER_REQUESTED (exit)
    ──venue.submit() succeeds──→ EXIT_WORKING
      ──FILL event──→ IDLE
      ──REJECTED event──→ IDLE
      ──INDETERMINATE──→ STAYS EXIT_WORKING

Rust FSM Responsibility:

• Manages TradeSlot.fsm_state (IDLE, ORDER_REQUESTED, ENTRY_WORKING, OPEN, EXIT_WORKING, etc.)
• Transitions are deterministic given event (kind, status)
• Does NOT know about INDETERMINATE — it only sees events the Python layer emits
• The Python layer is the semantic firewall: it decides whether to emit a REJECTED event or suppress it

---

3.6 Seam 6: on_venue_event() — The Feedback Loop

File: /mnt/dolphinng5_predict/prod/clean_arch/dita_v2/rust_backend.py
Method: on_venue_event() (line ~1200)

def on_venue_event(self, event: VenueEvent) -> KernelOutcome:
    # All venue events flow here → Rust FSM → state transitions
    # If event.status == REJECTED:
    #   kernel.fsm_state → IDLE (flat)
    # If event.status == FILLED/PARTIAL_FILL:
    #   kernel.fsm_state → OPEN / IDLE

Critical Invariant: The kernel only rolls back to IDLE on explicit REJECTED events. It has no concept of "unknown".

---

4. ENUMERATED UNDESIRABLE STATES & TRANSITIONS

4.1 States That SHOULD Exist But Don't

Missing State	Description	Why It Matters
ENTRY_INDETERMINATE	Order sent, answer lost — may be live	Distinguishes "we don't know" from "working normally"
EXIT_INDETERMINATE	Exit sent, answer lost — may be live	Same for exits
ORDER_TRUTH_UNKNOWN	Lookup failed after N attempts	Explicit "truth unknown" state vs silent failure
VENUE_LOOKUP_PENDING	Background reconcile in progress	Prevents duplicate reconcile attempts
RECONCILE_IN_PROGRESS	Full reconcile running	Prevents concurrent reconciles

4.2 Transitions That Should Not Exist (But Do)

Transition	Why It's Bad	Where It Happens
ENTRY_WORKING → IDLE on INDETERMINATE	Orphans live position	bingx_direct pre-2026-07-13 (FIXED)
EXIT_WORKING → IDLE on INDETERMINATE	Orphans live position, no SL/TP	Same as above
ENTRY_WORKING → IDLE on VenuePostAckError (broad)	Over-broad catch rolls back live orders	rust_backend.py line ~900 (MITIGATED: catches VenuePostAckError specifically)
ORDER_REQUESTED → IDLE without event	Silent state loss	Never directly, but possible via exception path

4.3 States That Should Be Unreachable But Are Reachable

State	How It's Reached	Why It's Bad
SLOT_BUSY with no active order	FSM advanced but order never sent	process_intent exception before venue.submit
ENTRY_WORKING with active_entry_order = None	attach_entry_order not called	venue.submit exception before attach_entry_order
OPEN with position = None	Phantom fill event	E-feed race, duplicate fill
IDLE with trade_id set	Incomplete cleanup	reset() not called on all fields

4.4 Foreseeable "Phantom" States (Unforeseeable in Practice)

Scenario	Resulting State	Detection
BingX accepts order, sends ACK, TCP dies before ACK arrives	Venue: LIVE / Kernel: ENTRY_WORKING (no ACK event)	_lookup_own_order_by_client_id reconciliation
BingX fills order, sends FILL, WS dies before delivery	Venue: FILLED / Kernel: ENTRY_WORKING	E-feed FILL event / reconcile
BingX rejects order, sends REJECTED, TCP dies	Venue: REJECTED / Kernel: ENTRY_WORKING	Reconcile finds REJECTED
Network partition: client can't reach BingX, but BingX can fill	Venue: FILLED / Kernel: INDETERMINATE	Reconcile + E-feed
Duplicate clientOrderId sent (retries) → BingX returns original order	Venue: FILLED / Kernel: sees duplicate	clientOrderId deduplication in _lookup

---

5. MITIGATIONS — IMPLEMENTED, PARTIAL, PROPOSED

5.1 Implemented (Post-2026-07-13)

Mitigation	Location	What It Prevents
Three-way HTTP classification	bingx_direct.py lines ~950-1020	Misclassification of INDETERMINATE as REJECTED
_lookup_own_order_by_client_id()	bingx_direct.py lines ~1030-1100	Reconciling truth from venue after INDETERMINATE
VenueIndeterminateError exception	bingx_venue.py line ~859/937	Semantic escalation: INDETERMINATE ≠ REJECTED
VenuePostAckError catch (no rollback)	rust_backend.py lines ~900, ~1080	Kernel no-rollback fence for INDETERMINATE
VenueIndeterminateError inherits VenuePostAckError	venue.py:43	Existing no-rollback fence catches it
_lookup_own_order_by_client_id bounded retries	bingx_direct.py line ~1070	Bounded reconciliation, then gives up with None
VenuePostAckError catch in async path too	rust_backend.py line ~1080	Async path same protection as sync
No synthetic event on INDETERMINATE	rust_backend.py lines ~910, ~1090	No synthetic REJECT pollutes event log

5.2 Partial / Incomplete

Gap	Location	Risk
No ENTRY_INDETERMINATE / EXIT_INDETERMINATE FSM states	Rust FSM	Can't distinguish "working normally" from "truth unknown" in state
No ORDER_TRUTH_UNKNOWN explicit state	bingx_direct._lookup_own_order	Returns None but no state reflects this
No background reconcile task for INDETERMINATE orders	bingx_direct._s2_tasks	Only MARKET fills trigger background refresh
No metrics/alerting on INDETERMINATE frequency	None	Can't detect systemic issues
No alert on VenueIndeterminateError rate	Logging only	Operators don't know it's happening

5.3 Proposed (For Future Hardening)

Proposal	Effort	Impact
Add ENTRY_INDETERMINATE / EXIT_INDETERMINATE FSM states	Rust FSM + Python bindings	Explicit "truth unknown" state visible in TUI/monitoring
Add OrderTruth enum: KNOWN_LIVE, KNOWN_FLAT, UNKNOWN	bingx_direct + kernel	Explicit truth tracking per order
Background reconcile task for INDETERMINATE orders	New async task in DolphinLiveTrader	Proactive truth discovery
INDETERMINATE metric + alert	Prometheus/Grafana	Operator visibility
OrderTruth persisted in Zinc/CH	Zinc plane + CH writer	Survives restarts
ReconcileManager service	New module	Centralized truth discovery
ORDER_TRUTH_UNKNOWN event type	Contracts + kernel	Explicit event for "we don't know"

---

6. ENUMERATED INSTANCES IN CODEBASE

6.1 Where INDETERMINATE Classification Happens

File	Line	Function	Purpose
prod/bingx/http.py	~50	BingxHttpError.effect	Classification property
prod/bingx/http.py	~60	_effect_of_httpx_error()	Maps httpx exceptions
prod/bingx/http.py	~80	_effect_of_status()	Maps HTTP status codes
prod/clean_arch/adapters/bingx_direct.py	~950	submit_intent() exception handler	Main classification
prod/clean_arch/adapters/bingx_direct.py	~1030	_lookup_own_order_by_client_id()	Truth reconciliation
prod/clean_arch/dita_v2/bingx_venue.py	~859	submit() INDETERMINATE check	Sync venue
prod/clean_arch/dita_v2/bingx_venue.py	~937	submit_async() INDETERMINATE check	Async venue

6.2 Where INDETERMINATE Is Caught / Escalated

File	Line	Context
prod/clean_arch/dita_v2/bingx_venue.py	~859	raise VenueIndeterminateError(...) (sync)
prod/clean_arch/dita_v2/bingx_venue.py	~937	raise VenueIndeterminateError(...) (async)
prod/clean_arch/dita_v2/rust_backend.py	~900	except VenuePostAckError: (sync)
prod/clean_arch/dita_v2/rust_backend.py	~1080	except VenuePostAckError: (async)
prod/clean_arch/dita_v2/venue.py	43	class VenueIndeterminateError(VenuePostAckError)

6.3 Where No-Rollback Fence Exists

File	Line	Protection
prod/clean_arch/dita_v2/rust_backend.py	~900	except VenuePostAckError: in sync process_intent
prod/clean_arch/dita_v2/rust_backend.py	~1080	except VenuePostAckError: in async process_intent_async

6.4 Where Synthetic REJECTED Is Emitted (PRE-ACK Only)

File	Line	Context
prod/clean_arch/dita_v2/rust_backend.py	~920	Sync: synthetic_REJECTED_event on generic Exception
prod/clean_arch/dita_v2/rust_backend.py	~1110	Async: synthetic_REJECTED_event on generic Exception

---

7. TEST VERIFICATION MATRIX

File: /mnt/dolphinng5_predict/prod/clean_arch/dita_v2/test_indeterminate_submit.py

Test	What It Verifies
test_read_timeout_is_indeterminate_not_rejection	ReadTimeout → INDETERMINATE
test_connect_failures_are_not_attempted	ConnectError/ConnectTimeout → NOT_ATTEMPTED
test_5xx_is_indeterminate_and_4xx_is_refused	5xx → INDETERMINATE, 4xx → REFUSED
test_unknown_failure_defaults_to_indeterminate	Unknown error → INDETERMINATE (conservative)
test_lookup_finds_the_order_and_never_posts	Reconcile finds live order, never re-POSTs
test_lookup_reports_absent_when_venue_has_no_such_order	Venue says "no order" → ABSENT (rollback sound)
test_lookup_treats_venue_refusal_as_absent	Venue REFUSES lookup → ABSENT
test_lookup_returns_none_when_truth_cannot_be_established	Timeout on lookup → None (not ABSENT)
test_sync_venue_escalates_indeterminate_receipt	VenueIndeterminateError raised
test_async_venue_escalates_indeterminate_receipt	Async path same
test_indeterminate_is_a_post_ack_error_so_existing_fences_catch_it	Subclass check
test_kernel_does_not_roll_back_slot_on_indeterminate_submit	Sync kernel no-rollback
test_kernel_does_not_roll_back_slot_on_indeterminate_submit_async	Async kernel no-rollback
test_genuine_rejection_still_rolls_the_slot_back	REFUSED still rolls back (contrast test)

---

8. DECISION LOG (Appendix)

Date	Decision	Rationale
2026-07-13	Classify ReadTimeout/5xx as INDETERMINATE	Request sent, answer lost — order may be live
2026-07-13	Default unknown errors to INDETERMINATE	Conservative: never guess "rejected"
2026-07-13	_lookup_own_order_by_client_id bounded retries	Prevent infinite retry loops
2026-07-13	VenueIndeterminateError inherits VenuePostAckError	Reuses existing no-rollback fence
2026-07-13	NO synthetic event on INDETERMINATE	Don't pollute event log with guesses
2026-07-13	VenuePostAckError catch in BOTH sync/async	Both code paths protected

---

9. OPERATOR RUNBOOK: INDETERMINATE INCIDENT

If you see VenueIndeterminateError in logs:

1. Do NOT manually flatten the slot — the order may be live
2. Check BingX VST/LIVE for the clientOrderId (logged in the error)
3. If position exists at venue: let E-feed FILL settle it, or manually reconcile
4. If position NOT at venue: kernel will eventually reconcile or you can trigger manual reconcile
5. Do NOT restart the kernel to "clear" the slot — this loses the ENTRY_WORKING/EXIT_WORKING state

Log Signature:

CRITICAL FIX(bingx_direct): submit outcome INDETERMINATE (ReadTimeout("...")) 
symbol=TRXUSDT clientOrderId=p-e-1q3k7m-ab4c — order MAY be live. Looking it up; 
NOT assuming rejection.

Resolution Paths:

• found == dict: Order is LIVE → kernel adopts venue truth, slot stays OPEN/WORKING
• found == "ABSENT": Order never existed → rollback IS sound, slot → IDLE
• found == None: Truth unknown → slot stays in ENTRY_WORKING/EXIT_WORKING, E-feed FILL settles it

---

10. FUTURE WORK PRIORITY ORDER

Priority	Item	Effort	Blocker
P0	ENTRY_INDETERMINATE / EXIT_INDETERMINATE FSM states	Medium	Rust FSM changes + Python bindings
P0	Background reconcile task for INDETERMINATE orders	Low	New async task in trader loop
P1	OrderTruth enum + Zinc persistence	Medium	Zinc plane schema change
P1	INDETERMINATE metrics + Grafana alert	Low	Prometheus exporter
P2	ReconcileManager service	High	New module, cross-cutting
P2	ORDER_TRUTH_UNKNOWN event type	Medium	Contracts + kernel + venue adapters

---

11. FABLE REVIEW (2026-07-13) — verdict + MISSING SEAMS (fleet work)

Truth audit: Doc is faithful to the shipped fix (verified: _is_rate_limited_error exists at bingx_direct.py:127; test matrix §7 matches the real 14-test suite; FSM description matches kernel-invariant tests). Two factual errors corrected in-place: VenueIndeterminateError lives at venue.py:43 (not contracts.py); fix date is 2026-07-13 (commits bdc54fb canonical / 54cd5d5 vendored).

Scope gap — THE assignment ("extrapolate the class across the codebase") is unfinished. §3 covers the SUBMIT seam deeply; the same not-done ≠ tried-and- failed ≠ unknown triage is UNAUDITED at every other venue-side-effect seam:

#	Seam	Specific risk
M1	cancel() / cancel_async()	cancel-INDETERMINATE = order MAY still be live → blind re-place = double fill. (Encoded in SPEC_UV_SMART_EXEC_MM §5; kernel path unaudited.)
M2	leverage SET (bingx_direct)	On the money path (seen live 17:20 today). SET-INDETERMINATE → position sized at unknown leverage.
M3	promotion bridge receipts (uv.promotion)	Bridge assumes receipt is truth; INDETERMINATE mid-bridge unhandled?
M4	E-feed / EKBridge gap handling	WS death between ACK and FILL — partially covered by rehydrate; no triage taxonomy.
M5	exit-leg placement (multi-leg exits)	Leg 1 fills, leg 2 INDETERMINATE → partial exit with unknown remainder.
M6	ch_writer / journal lane	Covered for data (spool), but write-INDETERMINATE→retry = possible duplicate rows (dedup ratchet history).
M7	every except: pass / broad except Exception after venue I/O	Grep-sweep + classify each; the b46ebd2 class hid here.

DOCTRINE GUARDRAIL on §10 P0 "background reconcile task": operator standing order — reconcile is proto-PINK hell. Any background truth-discovery MUST stay a bounded, read-only, own-clientOrderId point lookup (the §2.2 primitive), never a scan-all-orders reconciler. Build the loop around the existing primitive.

Disposition: M1–M7 are fleet-sized (audit + tests per seam, same pattern as test_indeterminate_submit.py). Corrections done in-place are complete.

---

END OF DOCUMENT
This document is a living reference. Update on every INDETERMINATE-related change.