sentiment-engine/sentiment_engine/README.md

# Sentiment Analysis Engine v2.0.0

> **Real-time sentiment analysis engine for DOLPHIN NG5 trading system**

## Overview

The Sentiment Analysis Engine ingests news, social media, and structured text from 9 source categories and produces **parametrized sentiment outputs** at three hierarchical levels:

| Level | Outputs | Use Case |
|-------|---------|----------|
| **Per-Asset** | `fear_state`, `greed_state`, `pump_score`, `dump_score`, `hype_velocity`, `event_flags` | Entry veto, position sizing, exit timing |
| **Industry/Class** | Aggregated fear/greed, pump/dump risk, dominant events | Sector rotation, correlation analysis |
| **Market-Wide** | Sentiment index, aggregate pump/dump risk, hype velocity | ACB gating, regime detection, portfolio risk |

**Replaces** the single `fng` (Fear & Greed) indicator (r=-0.19, p=0.19, 5-day lag) with a real-time, multi-dimensional signal factory.

## Architecture

```
┌─────────────────────────────────────────────────────────────────────┐
│  SENTIMENT ANALYSIS ENGINE                                          │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  ┌──────────┐   ┌──────────────────┐   ┌────────────────────────┐   │
│  │ Ingestion│ → │ NLP Processing   │ → │ Event Detection &      │   │
│  │ Queue    │   │ Pipeline         │   │ Signal Extraction      │   │
│  └──────────┘   └──────────────────┘   └────────────────────────┘   │
│            │          │           │          │                      │
│            │  entity  │ sentiment │  event   │  per-asset events    │
│            │  + asset │ polarity  │  type    │  + polarity +        │
│            │  mapping │ + emo.    │  class   │  intensity            │
│            ▼          ▼           ▼          ▼                      │
│  ┌────────────────────────────────────────────────────────────────┐ │
│  │ Signal Processing Layer                                        │ │
│  │  • Event Strength Computation (credibility × sources × details)│ │
│  │  • Velocity Computation (hype_velocity, pub_velocity)          │ │
│  │  • Decay & Temporal Weighting                                  │ │
│  │  • Multi-source Signal Fusion                                  │ │
│  └────────────────────────────────────────────────────────────────┘ │
│                              │                                      │
│                              ▼                                      │
│  ┌────────────────────────────────────────────────────────────────┐ │
│  │ Scoring Engine                                                 │ │
│  │  • fear_state, greed_state (per asset, class, market)          │ │
│  │  • pump_score, dump_score (probability, per asset)             │ │
│  │  • event_flags catalog (0-100 strength per event)              │ │
│  └────────────────────────────────────────────────────────────────┘ │
│                              │                                      │
│                              ▼                                      │
│  ┌────────────────────────────────────────────────────────────────┐ │
│  │ Aggregation & Output                                           │ │
│  │  • Per-Asset → Industry/Class → Market                          │ │
│  │  • Output Schema (Section 8)                                    │ │
│  └────────────────────────────────────────────────────────────────┘ │
│                              │                                      │
│                              ▼                                      │
│  ┌────────────────────────────────────────────────────────────────┐ │
│  │ Sinks:                                                          │ │
│  │  • Hazelcast (hot path, <5ms latency) → nautilus_event_trader   │ │
│  │  • ClickHouse (analytical, backtests)                           │ │
│  │  • LatticeDB (graph: credibility propagation, co-occurrence)    │ │
│  └────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
```

## Source Categories

| Category | Examples | Cadence | Credibility |
|----------|----------|---------|-------------|
| Crypto-native news | CoinDesk, CoinTelegraph, The Block | 1-5 min RSS | 0.75-0.85 |
| Traditional finance | Bloomberg, Reuters, WSJ | 1-5 min RSS | 0.8-0.9 |
| Twitter/X | Firehose API | Real-time WS | 0.4 |
| Reddit | Pushshift/PRAW | 1-10 min | 0.3-0.35 |
| Discord/Telegram | Bot listeners | Real-time | 0.4 |
| Exchange announcements | Binance, Coinbase, Kraken | 1 min RSS | 0.85-0.9 |
| On-chain/DeFi | DeFi Llama, Nansen, governance | 5-30 min | 0.7-0.8 |
| Regulatory | SEC EDGAR, CFTC, Fed | Real-time RSS | 0.95 |
| Corporate | Earnings calls, filings | Daily batch | 0.7 |

## Key Features

### 1. Real-time NLP Pipeline
- **Entity Extraction**: Ticker detection, contract addresses, alias resolution (Vitalik→ETH, CZ→BNB)
- **Sentiment + Emotion**: FinBERT polarity + 6 emotions (joy, fear, anger, greed, sadness, intensity)
- **Event Classification**: 12 event types (listing, hack, regulatory, governance, upgrade, partnership, earnings, macro, liquidation, whale, manipulation)
- **Temporal Anchoring**: Immediate/near/medium/long horizons + breaking news detection
- **Credibility Scoring**: Source base + content quality + engagement authenticity + cross-source corroboration

### 2. Signal Processing
- **Event Strength**: Credibility-weighted, multi-source fused
- **Velocity**: Hype velocity (sentiment acceleration) + Publication velocity (source frequency)
- **Temporal Decay**: Exponential decay with parameter-specific half-lives (60-480 min)
- **Multi-source Fusion**: Weighted by recency and credibility

### 3. Trading Integration
- **ACB Signals**: `market_sentiment_state`, `aggregate_pump_risk`, `fear_state`, `greed_state`, `hype_velocity`
- **BookHealthGate**: Entry veto when `pump_score > 75`
- **AlphaExitEngineV7**: Exit context from `dump_score > 70`, `fear_state > 80`
- **Hazelcast Hot Path**: Sub-5ms latency for trading engine consumption

## Quick Start

### Prerequisites
- Python 3.12+
- Docker Compose (for NATS, ClickHouse, Hazelcast, Prefect)
- GPU (recommended for NLP models)

### Installation

```bash
# Clone and install
cd sentiment_engine
pip install -e ".[dev,gpu]"

# Copy environment template
cp .env.example .env
# Edit .env with your API keys

# Start infrastructure
docker-compose -f docker/docker-compose.yml up -d

# Build centroids (first run)
python scripts/build_centroids.py

# Run engine
python -m sentiment_engine.main
```

### Configuration

Main config: `config/settings.yaml`
- NATS, ClickHouse, Hazelcast connection details
- NLP model settings (device, batch sizes, quantization)
- Scoring parameters (half-lives, thresholds, centroid weights)
- Source connector configurations
- Trading integration thresholds

Asset mappings: `config/asset_aliases.yaml`, `config/known_entities.yaml`
Source credibility: `config/source_credibility.yaml`
Industry mapping: `config/asset_industry_map.yaml`

## Deployment

### Docker Compose (Recommended)
```bash
docker-compose -f docker/docker-compose.yml up -d
```

Services:
- `sentiment-engine`: Main engine (4 CPU, 8GB RAM)
- `nats`: JetStream message bus
- `clickhouse`: Analytical storage
- `hazelcast`: Hot cache
- `prefect`: Workflow orchestration
- `otel-collector`: OpenTelemetry
- `latticedb`: Graph layer (optional)

### Prefect Flows (Scheduled Connectors)
```bash
# Deploy flows
prefect deploy --all -p sentiment-engine

# Run manually
python -m prefect_flows.connectors.rss_ingest
python -m prefect_flows.connectors.api_ingest
python -m prefect_flows.connectors.web_crawl
```

## Output Schema

### Per-Asset (`AssetSentiment`)
```json
{
  "asset_id": "BTC",
  "fear_state": 20.0,
  "greed_state": 80.0,
  "sentiment_polarity": 60.0,
  "emotion_profile": {"joy": 0.8, "fear": 0.1, "anger": 0.05, "greed": 0.7, "sadness": 0.05, "intensity": 0.75},
  "pump_dump": {"pump_score": 75.0, "dump_score": 15.0, "pump_confidence": 0.8},
  "event_flags": [{"event_type": "listing", "strength": 60.0, "confidence": 0.7}],
  "velocity": {"hype_velocity": 0.7, "pub_velocity": 0.5, "direction": "accelerating"},
  "last_update_ts": 1724262305.0,
  "decay_factor": 0.95
}
```

### Market (`MarketSentiment`)
```json
{
  "fear_state": 25.0,
  "greed_state": 75.0,
  "sentiment_index": 50.0,
  "hype_velocity": 65.0,
  "pub_velocity": 55.0,
  "aggregate_pump_risk": 75.0,
  "aggregate_dump_risk": 20.0,
  "top_pump_assets": ["BTC", "ETH", "SOL"],
  "top_dump_assets": [],
  "last_update_ts": 1724262305.0
}
```

## Testing

```bash
# Unit tests
pytest tests/unit -v

# Integration tests
pytest tests/integration -v

# With coverage
pytest --cov=sentiment_engine tests/
```

## Monitoring

- **Prometheus**: `:9090/metrics`
- **OpenTelemetry**: `otel-collector:4317` → ClickHouse `sentiment_otel`
- **NATS Monitoring**: `:8222`
- **Hazelcast Management Center**: `:5701`

## Integration with DOLPHIN NG5

The engine publishes to Hazelcast map `exf_latest` with keys consumed by `nautilus_event_trader.py:on_exf_update()`:

```python
# ACB_KEYS enriched with:
"market_sentiment_state",    # -1 to 1
"aggregate_pump_risk",       # 0 to 1
"fear_state",                # 0 to 1
"greed_state",               # 0 to 1
"hype_velocity"              # 0 to 1
```

## License

Proprietary - DOLPHIN NG5 Project