sentiment-engine/sentiment_engine

Sentiment Analysis Engine v2.0.0

Real-time sentiment analysis engine for DOLPHIN NG5 trading system

Overview

The Sentiment Analysis Engine ingests news, social media, and structured text from 9 source categories and produces parametrized sentiment outputs at three hierarchical levels:

Level	Outputs	Use Case
Per-Asset	fear_state, greed_state, pump_score, dump_score, hype_velocity, event_flags	Entry veto, position sizing, exit timing
Industry/Class	Aggregated fear/greed, pump/dump risk, dominant events	Sector rotation, correlation analysis
Market-Wide	Sentiment index, aggregate pump/dump risk, hype velocity	ACB gating, regime detection, portfolio risk

Replaces the single fng (Fear & Greed) indicator (r=-0.19, p=0.19, 5-day lag) with a real-time, multi-dimensional signal factory.

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│  SENTIMENT ANALYSIS ENGINE                                          │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  ┌──────────┐   ┌──────────────────┐   ┌────────────────────────┐   │
│  │ Ingestion│ → │ NLP Processing   │ → │ Event Detection &      │   │
│  │ Queue    │   │ Pipeline         │   │ Signal Extraction      │   │
│  └──────────┘   └──────────────────┘   └────────────────────────┘   │
│            │          │           │          │                      │
│            │  entity  │ sentiment │  event   │  per-asset events    │
│            │  + asset │ polarity  │  type    │  + polarity +        │
│            │  mapping │ + emo.    │  class   │  intensity            │
│            ▼          ▼           ▼          ▼                      │
│  ┌────────────────────────────────────────────────────────────────┐ │
│  │ Signal Processing Layer                                        │ │
│  │  • Event Strength Computation (credibility × sources × details)│ │
│  │  • Velocity Computation (hype_velocity, pub_velocity)          │ │
│  │  • Decay & Temporal Weighting                                  │ │
│  │  • Multi-source Signal Fusion                                  │ │
│  └────────────────────────────────────────────────────────────────┘ │
│                              │                                      │
│                              ▼                                      │
│  ┌────────────────────────────────────────────────────────────────┐ │
│  │ Scoring Engine                                                 │ │
│  │  • fear_state, greed_state (per asset, class, market)          │ │
│  │  • pump_score, dump_score (probability, per asset)             │ │
│  │  • event_flags catalog (0-100 strength per event)              │ │
│  └────────────────────────────────────────────────────────────────┘ │
│                              │                                      │
│                              ▼                                      │
│  ┌────────────────────────────────────────────────────────────────┐ │
│  │ Aggregation & Output                                           │ │
│  │  • Per-Asset → Industry/Class → Market                          │ │
│  │  • Output Schema (Section 8)                                    │ │
│  └────────────────────────────────────────────────────────────────┘ │
│                              │                                      │
│                              ▼                                      │
│  ┌────────────────────────────────────────────────────────────────┐ │
│  │ Sinks:                                                          │ │
│  │  • Hazelcast (hot path, <5ms latency) → nautilus_event_trader   │ │
│  │  • ClickHouse (analytical, backtests)                           │ │
│  │  • LatticeDB (graph: credibility propagation, co-occurrence)    │ │
│  └────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘

Source Categories

Category	Examples	Cadence	Credibility
Crypto-native news	CoinDesk, CoinTelegraph, The Block	1-5 min RSS	0.75-0.85
Traditional finance	Bloomberg, Reuters, WSJ	1-5 min RSS	0.8-0.9
Twitter/X	Firehose API	Real-time WS	0.4
Reddit	Pushshift/PRAW	1-10 min	0.3-0.35
Discord/Telegram	Bot listeners	Real-time	0.4
Exchange announcements	Binance, Coinbase, Kraken	1 min RSS	0.85-0.9
On-chain/DeFi	DeFi Llama, Nansen, governance	5-30 min	0.7-0.8
Regulatory	SEC EDGAR, CFTC, Fed	Real-time RSS	0.95
Corporate	Earnings calls, filings	Daily batch	0.7

Key Features

1. Real-time NLP Pipeline

• Entity Extraction: Ticker detection, contract addresses, alias resolution (Vitalik→ETH, CZ→BNB)
• Sentiment + Emotion: FinBERT polarity + 6 emotions (joy, fear, anger, greed, sadness, intensity)
• Event Classification: 12 event types (listing, hack, regulatory, governance, upgrade, partnership, earnings, macro, liquidation, whale, manipulation)
• Temporal Anchoring: Immediate/near/medium/long horizons + breaking news detection
• Credibility Scoring: Source base + content quality + engagement authenticity + cross-source corroboration

2. Signal Processing

• Event Strength: Credibility-weighted, multi-source fused
• Velocity: Hype velocity (sentiment acceleration) + Publication velocity (source frequency)
• Temporal Decay: Exponential decay with parameter-specific half-lives (60-480 min)
• Multi-source Fusion: Weighted by recency and credibility

3. Trading Integration

• ACB Signals: market_sentiment_state, aggregate_pump_risk, fear_state, greed_state, hype_velocity
• BookHealthGate: Entry veto when pump_score > 75
• AlphaExitEngineV7: Exit context from dump_score > 70, fear_state > 80
• Hazelcast Hot Path: Sub-5ms latency for trading engine consumption

Quick Start

Prerequisites

• Python 3.12+
• Docker Compose (for NATS, ClickHouse, Hazelcast, Prefect)
• GPU (recommended for NLP models)

Installation

# Clone and install
cd sentiment_engine
pip install -e ".[dev,gpu]"

# Copy environment template
cp .env.example .env
# Edit .env with your API keys

# Start infrastructure
docker-compose -f docker/docker-compose.yml up -d

# Build centroids (first run)
python scripts/build_centroids.py

# Run engine
python -m sentiment_engine.main

Configuration

Main config: config/settings.yaml

• NATS, ClickHouse, Hazelcast connection details
• NLP model settings (device, batch sizes, quantization)
• Scoring parameters (half-lives, thresholds, centroid weights)
• Source connector configurations
• Trading integration thresholds

Asset mappings: config/asset_aliases.yaml, config/known_entities.yaml Source credibility: config/source_credibility.yaml Industry mapping: config/asset_industry_map.yaml

Deployment

Docker Compose (Recommended)

docker-compose -f docker/docker-compose.yml up -d

Services:

• sentiment-engine: Main engine (4 CPU, 8GB RAM)
• nats: JetStream message bus
• clickhouse: Analytical storage
• hazelcast: Hot cache
• prefect: Workflow orchestration
• otel-collector: OpenTelemetry
• latticedb: Graph layer (optional)

Prefect Flows (Scheduled Connectors)

# Deploy flows
prefect deploy --all -p sentiment-engine

# Run manually
python -m prefect_flows.connectors.rss_ingest
python -m prefect_flows.connectors.api_ingest
python -m prefect_flows.connectors.web_crawl

Output Schema

Per-Asset (AssetSentiment)

{
  "asset_id": "BTC",
  "fear_state": 20.0,
  "greed_state": 80.0,
  "sentiment_polarity": 60.0,
  "emotion_profile": {"joy": 0.8, "fear": 0.1, "anger": 0.05, "greed": 0.7, "sadness": 0.05, "intensity": 0.75},
  "pump_dump": {"pump_score": 75.0, "dump_score": 15.0, "pump_confidence": 0.8},
  "event_flags": [{"event_type": "listing", "strength": 60.0, "confidence": 0.7}],
  "velocity": {"hype_velocity": 0.7, "pub_velocity": 0.5, "direction": "accelerating"},
  "last_update_ts": 1724262305.0,
  "decay_factor": 0.95
}

Market (MarketSentiment)

{
  "fear_state": 25.0,
  "greed_state": 75.0,
  "sentiment_index": 50.0,
  "hype_velocity": 65.0,
  "pub_velocity": 55.0,
  "aggregate_pump_risk": 75.0,
  "aggregate_dump_risk": 20.0,
  "top_pump_assets": ["BTC", "ETH", "SOL"],
  "top_dump_assets": [],
  "last_update_ts": 1724262305.0
}

Testing

# Unit tests
pytest tests/unit -v

# Integration tests
pytest tests/integration -v

# With coverage
pytest --cov=sentiment_engine tests/

Monitoring

• Prometheus: :9090/metrics
• OpenTelemetry: otel-collector:4317 → ClickHouse sentiment_otel
• NATS Monitoring: :8222
• Hazelcast Management Center: :5701

Integration with DOLPHIN NG5

The engine publishes to Hazelcast map exf_latest with keys consumed by nautilus_event_trader.py:on_exf_update():

# ACB_KEYS enriched with:
"market_sentiment_state",    # -1 to 1
"aggregate_pump_risk",       # 0 to 1
"fear_state",                # 0 to 1
"greed_state",               # 0 to 1
"hype_velocity"              # 0 to 1

License

Proprietary - DOLPHIN NG5 Project