siloqy/docs/planning/full_conversation.md

# SILOQY Market Sensing System - Complete Conversation

## Project Overview

This conversation covers the development of SILOQY, a comprehensive crypto sensing, regime detection, and signaling system consisting of:

- **DOLPHIN**: Regime detection subsystem (500 crypto symbols analysis)
- **JERICHO**: State machine for signal generation and position management
- **SILOQY**: The overarching system combining all components

Project motto: "Through music, not war"

---

## Initial User Request

The user described a putative crypto sensing, regime detection, and signaling system, alias JERICHO. The "sensing" system is described in project knowledge documents, and the "regime detection subsystem" is described in Python code, ALIAS "DOLPHIN" (they lead the way of ships) NG.

JERICHO basically describes a *state machine*.

Key requirements:
- Study the JERICHO "state machine" and make sure it is coherent, complete, and that all paths and possibilities are clearly stated
- Produce a clear, complete and thorough Jericho system description
- Suggestions to help "approximate" human market psychology for buy, sell and hold signals
- Ability to mark with confidence turnarounds in asset direction
- Tested in Bitcoin 3m candles, yielding good results
- Parameters hand-tuned for Bitcoin during a two hour period between 11 and 1330 CET, on Monday

After JERICHO implementation, modify DOLPHIN code to provide:
- Current "distances" from current price to Bollinger Bands (percentage)
- A process loop ensuring DOLPHIN runs continuously
- Websocket interface for feeding SCAN information to JERICHO
- Simple TUI for output display

JERICHO requirements:
- Lightweight, DOLPHIN compatible server
- Query/listen to DOLPHIN and process accordingly
- Generate signals and output to TUI showing:
  - All status
  - Current DOLPHIN connection status
  - Last implemented "decision" (ie. "LONGING")
  - At most two succinct lines explaining logic
- Provide outgoing Websocket interfaces for signals to feed trading simulation system ("HARLEQUIN")

Code requirements:
- HFT-standard, best-of-breed
- Run on both Windows and Linux
- Data structures amenable to ML approaches

---

## Assistant's Initial Analysis

I provided a comprehensive analysis of the JERICHO state machine, identifying it as an orchestration layer that:
- Receives sensing data (SCAN) and DOLPHIN regime detections
- Aggregates multiple module outputs ("siloqies")
- Decides Buy/Sell/Hold/Exit with confidence scores
- Publishes signals to outgoing websockets for HARLEQUIN
- Manages execution lifecycle and safety limits

I proposed a canonical JERICHO state machine with 12 top-level states and detailed transition rules, but then the user corrected me...

---

## User Provides Actual JERICHO Specification

The user apologized and provided the actual JERICHO spec:

### THE SENSING SYSTEM - JERICHO

The REGIME DETECTION subsystem produces a continued SCAN of BULLISH vs. BEARISH total assets, expressed as percentage bullish and percentage bearish, and other information. This report is - by default - produced every 5 seconds. Most references to percentages below are to these percentages of bullish and/or bearish number of total market 15m candles per SCAN period. References to 6x or 9x are to a number of consecutive SCAN periods.

SENTIMENT and REGIME are "BULLISH" and "BEARISH" or "SIDEWAYS". SENTIMENT and REGIME are interchangeable. At any point, the REGIME DETECTION subsystem provides a "CURRENT MARKET REGIME".

A STREAK is a sequence of 9x reports of a given SENTIMENT. Any SCAN with the relevant (BULLISH/BEARISH) value <20% breaks a STREAK.

A SIDEWAYS sentiment "resets" the streak, except prior SENTIMENT values in the interrupted STREAK are "accounted for" for the ongoing (previous STREAK) as stated below.

**Default states:**
- Default sentiment is NEUTRAL
- Default attention state is NORMAL
- Default state POSITION is NONE (can also be LONG or SHORT)
- Default ACTION is "STANDING BY" (can be "LONGING" or "SHORTING")

**Key Rules:**

90% is the magic number.

Any decrease or increase (depending on the current sentiment vs. the opposite) >2%. Attention state becomes "WATCHING".
- ie. if currently BULLISH, and BULLISH % drops > 2% or BEARISH and bearish % drops >2%.

Any "jump" to 95% or greater in any value becomes "WATCHING".

**IMPORTANT:** Price (or index value, if market) at 1% proximity to an UPPER, MIDDLE (from both directions) or LOWER, BOLLINGER BAND enters "WATCHING" attention. Any slight % over (the upper) or under (the lower) BOLLINGER BAND becomes "WATCHING FOR BREAKOUT" and "WATCHING FOR BREAKDOWN" respectively. These are special states.

Two consecutive 10% drops (even while a given % in a given REGIME is still major) exits any position, if held, from a given starting point (ie. 80% vs. 70%/75% - the last two are ten percent drops relative to the first reading).
(These drops, however, do not change the SENTIMENT per. Regular rules apply).

2x SCANS with >99% or 98.5% for any given candle type (BULLISH/BEARISH) buy "into" the sentiment, whatever it might be (ie SHORT for BEARISH, LONG for BULLISH). If NEUTRAL, do not enter or exit any POSITION. POSITION becomes LONGING or SHORTING for the next SCAN period, LONG or SHORT for the remaining, until position is closed (ie. we assume entering or exiting a POSITION takes one SCAN PERIOD). Exiting a position yields a STANDING BY state.

**WATCHING FOR PRICE REVERSALS:**
- Sudden "turn" (BULL/BEAR BEAR/BULL) *without* SIDEWAYS, any SENTIMENT >90% confirms the SENTIMENT turnaround.
- Turns "via" a SIDEWAYS need 9x streaks confirming
- A SIDEWAYS raises attention to "WATCHING". 9x to CONFIRM state SWITCH. A new SIDEWAYS, resets count. Every 2x consecutive opposite regimes "between" SIDEWAYs count as 1x to CONFIRM. A CONTRARY sentiment "resets" state to last known sentiment.

---

## Assistant's Revised Analysis

After receiving the actual specification, I provided a detailed analysis mapping the prose rules into a precise finite state machine:

### Canonical JERICHO State Machine (Following Specification)

**Global Assumptions:**
- SCAN period = 5 seconds (default)
- References to "6x or 9x" mean consecutive SCAN periods
- REGIME detection reports pct_bullish, pct_bearish, and sentiment ∈ {BULLISH, BEARISH, SIDEWAYS, NEUTRAL}
- "90% is the magic number" - special rules reference 90%, 95%, 98.5% and 99% as thresholds
- Bollinger Band proximity measured as % distance to band

**Top-Level States:**
1. `STANDING_BY` (default)
2. `WATCHING` (elevated attention)
3. `WATCHING_FOR_BREAKOUT` (price crossing above upper BB)
4. `WATCHING_FOR_BREAKDOWN` (price crossing below lower BB)
5. `WATCHING_FOR_PRICE_REVERSAL` (possible reversal events)
6. `CONFIRMING` (transient state for streak confirmation)
7. `LONGING` (attempting to enter LONG position)
8. `SHORTING` (attempting to enter SHORT position)  
9. `LONG` (position active LONG)
10. `SHORT` (position active SHORT)
11. `EXITING` (exit from position occurring)
12. `ERROR/SAFE` (faults, disconnections)
13. `COOLDOWN` (optional debounce after exiting)

**Key Triggers and Guards:**

From STANDING_BY:
- WATCHING triggers: Δpct > 2%, any % ≥ 95%, BB proximity 1%
- Entry triggers: 2 consecutive SCANs ≥ 98.5%/99% AND sentiment non-NEUTRAL

WATCHING triggers:
- Any increase/decrease > 2% in relevant sentiment
- Any pct hitting ≥ 95%
- Bollinger proximity: price within 1% to upper, middle, or lower BB

Sideways, streak, and turn rules:
- STREAK = 9 consecutive SCANs of same sentiment
- SIDEWAYS resets streak count but previous counts are "accounted for"
- Sudden turn without SIDEWAYS + sentiment >90% = immediate confirmation
- Turns via SIDEWAYS need 9x streaks confirming

**JSON Message Schemas:**

DOLPHIN → JERICHO (example):
```json
{
  "type": "scan",
  "timestamp": "2025-08-12T09:03:00Z",
  "asset": "BTC",
  "bull_pct": 92.35,
  "bear_pct": 3.21,
  "sentiment": "BULLISH",
  "price": 57301.12,
  "bb": { "upper": 57700.00, "middle": 57200.00, "lower": 56700.00 },
  "volume": 1234.5
}
```

JERICHO → HARLEQUIN (signal):
```json
{
  "type": "jericho_signal",
  "id": "jericho-0001",
  "timestamp": "2025-08-12T09:04:00Z",
  "asset": "BTC",
  "signal": "LONG",
  "action_state": "LONGING",
  "confidence": 0.9235,
  "explain": [
    "Regime BULLISH @ 92.35% sustained 6 epochs",
    "BB distance: 1.23% from upper; WATCHING due to BTC near upper BB"
  ]
}
```

---

## Clarifications Needed

I identified several ambiguities requiring clarification:

1. **Two consecutive 10% drops**: Absolute percentage-point drops vs multiplicative relative percent?
2. **Threshold values**: 98.5% vs 99% vs 95% - confirm preferred values
3. **"Jump to 95%"**: Delta jump vs absolute threshold?
4. **NEUTRAL handling**: Absolute block on position entry confirmed?
5. **Bollinger Band middle**: Treating middle band as full "band" for proximity triggers

The user confirmed middle band should be treated as any other WATCHING circumstance - "delicate" but not implying reversal, hoping for breakthrough.

---

## DOLPHIN Code and Sample Outputs

The user provided the DOLPHIN main execution script and sample outputs showing:

**DOLPHIN Architecture:**
- Fetches up to 500 crypto symbols from Binance
- Analyzes 15m candles (300 units each, ~75 hours)
- Calculates correlations and up/down ratios
- Regime detection based on thresholds:
  - Bull market: ≥60% symbols up
  - Bear market: ≥55% symbols down
  - Transition: >15% change

**Sample Outputs:**
```json
{
  "regime": "BULL",
  "up_ratio": 0.7627118644067796,
  "down_ratio": 0.23728813559322035,
  "total_symbols": 405,
  "correlation_symbols": 405,
  "direction_symbols": 405,
  "timestamp": "2025-08-12T17:10:16.389625"
}
```

Range examples from samples:
- up_ratio: 0.04 to 0.89
- down_ratio: 0.11 to 0.96
- regime: "BULL" or "BEAR"

---

## Integration Requirements

**Missing for JERICHO Integration:**
- Bollinger Band distances (need to add to DOLPHIN)
- Price momentum indicators
- Volume confirmation
- Confidence metrics

**Architecture:**
```
DOLPHIN (500 assets) → Market regime signal
JERICHO FSM → Applies market signal to BTC-specific entry/exit
BB distance + momentum → Asset-specific confirmation
```

---

## Time Memory Feature Discussion

**User Question 1:** "Any advantage, vs just encoding timestamps, in (ie.) naming every minute and quantizing over those?"

**Assistant Response:** Yes, quantized named minutes have substantial advantages over raw timestamps for predictive power. Pattern recognition, market microstructure, regime persistence, and cross-asset synchronization all benefit from discrete temporal features.

**User Clarification:** "Not minute of day. Thinking 'minute of *year*' as a 'bucket'"

**Assistant Response:** Minute-of-year bucketing (1-525,600) provides superior predictive power by creating repeatable temporal contexts. Advantages include:
- Seasonal pattern learning
- Calendar effect encoding (tax deadlines, earnings seasons)
- Multi-year pattern recognition
- Sparse feature benefits for ML
- Regime memory at specific calendar positions

**User Question 2:** "RCDD over a *entire market* - or index of it?"

**Assistant Response:** Yes, market-wide RCDD provides massive advantages:
- Signal strength (500+ assets voting vs 1 asset)
- Early warning (portfolio effects precede individual breakdowns)
- Higher confidence (systemic vs idiosyncratic)
- DOLPHIN's 500-symbol analysis IS market-wide RCDD
- Market regime changes cascade to individual assets

The 500-asset DOLPHIN feeding BTC-specific JERICHO is the optimal approach.

---

## Key Insights and Recommendations

1. **JERICHO State Machine**: Needs clarification on ambiguous rules but core logic is sound
2. **Integration**: DOLPHIN provides market-wide regime detection, JERICHO applies to specific asset
3. **Time Memory**: Minute-of-year bucketing superior to timestamps for ML pattern recognition
4. **Market-Wide RCDD**: Existing DOLPHIN architecture correctly implements this principle
5. **Missing Components**: Need to add BB calculations, websocket interfaces, TUI implementation

---

## Next Steps

The conversation established the foundation for implementing the complete SILOQY system with:
- Clarified JERICHO state machine specification
- DOLPHIN integration requirements
- Time memory and market-wide RCDD approaches
- Message schemas and architectural patterns

Implementation would proceed with:
1. Resolve ambiguous rules in JERICHO spec
2. Add BB distance calculations to DOLPHIN
3. Implement websocket communication layer
4. Build TUI interfaces
5. Create HARLEQUIN trading simulation interface

---

*End of Conversation Record*