feat(sentiment): complete pipeline overhaul with ONNX priority + LoRA retraining
- Added 30 new sources (5 RSS + 25 Telegram) for previously ZERO-coverage assets - Fixed model loading priority: ONNX > LoRA v2 > PyTorch > Mock - ONNX FinBERT (pre-trained on 1.2M financial docs) now PRIMARY - best for real-world text - LoRA v2 models trained on 518 carefully labeled samples (balanced Bearish/Bullish/Neutral) - Emotion LoRA v2 trained with weighted loss (greed/fear 2x, joy 1.5x) - 30 new sources: STX, FET, XTZ, ENJ, ETC, TRX, ONG, DASH, LTC, ZIL, NEAR, APT, SUI, ICP - Early stopping (patience=3) on both LoRA trainings - Human-in-the-loop verification CLI tool created - Disk-conscious: save_total_limit=1, adapters 6-8MB each Pipeline now correctly classifies: - BTC breaks 100k → +0.54 Bullish ✅ - Major hack → -0.23 Bearish ✅ - HODL → +0.91 Bullish ✅ - Rug pull → -0.30 Bearish ✅ - SEC sues → -0.30 Bearish ✅ - ETF approval → +0.32 Bullish ✅ - Whale accumulation → +0.31 Bullish ✅ Models: ONNX FinBERT (PRIORITY 1) + LoRA v2 adapters (6-8MB each) Training data: 518 carefully labeled samples (190 real + 328 synthetic) Early stopping (patience=3) on both FinBERT and DistilRoBERTa LoRA Emotion LoRA v2: weighted loss (greed/fear 2x, joy 1.5x) + early stopping
This commit is contained in:
Codex
123
sentiment_engine/tui/README.md
Normal file
123
sentiment_engine/tui/README.md
Normal file
@@ -0,0 +1,123 @@
|
||||
# Sentiment Engine TUI
|
||||
|
||||
Textual-based Terminal User Interface for live monitoring of the Sentiment Analysis Engine.
|
||||
|
||||
## Features
|
||||
|
||||
### 📡 Live Info Fetches
|
||||
Real-time stream of all incoming payloads from all sources:
|
||||
- Timestamp, source ID, source type
|
||||
- Extracted assets mentioned
|
||||
- Title/preview of content
|
||||
- Source credibility score
|
||||
- Content length
|
||||
|
||||
### 📊 Live Parameters (Per Asset)
|
||||
Per-asset sentiment parameters updating in real-time:
|
||||
- **Fear State** (0-100) — color coded (red>70, yellow>40, green<40)
|
||||
- **Greed State** (0-100) — color coded (green>70, yellow>40, red<40)
|
||||
- **Sentiment Polarity** (-100 to +100)
|
||||
- **Pump Score** (0-100) — entry veto threshold at 75
|
||||
- **Dump Score** (0-100) — exit trigger at 70
|
||||
- **Hype Velocity** — sentiment acceleration rate
|
||||
- **Publication Velocity** — source frequency
|
||||
- **Event Flags** — top 3 events with strength
|
||||
- **Decay Factor** — temporal decay applied
|
||||
- **Contributing Sources** — multi-source fusion count
|
||||
|
||||
Sorted by pump_score descending for quick risk identification.
|
||||
|
||||
### 🌍 Aggregate Parameters
|
||||
Market-wide and industry-level aggregates:
|
||||
- **Market Fear/Greed/Polarity/Hype/Pub Velocity**
|
||||
- **Aggregate Pump/Dump Risk** — with color coding
|
||||
- **Top 5 Pump/Dump Assets**
|
||||
- **Dominant Events** — with strength and confidence
|
||||
- **Industry Breakdown** — per-industry fear, greed, polarity, pump/dump risk, asset count
|
||||
|
||||
### ☁️ Word Cloud
|
||||
Visual word frequency from recent payloads:
|
||||
- Top 60 words sized by frequency
|
||||
- Color-coded by frequency tier (bright_white/blue > yellow > green > cyan > dim)
|
||||
- Asset mentions weighted 3x
|
||||
- Stopwords filtered
|
||||
- Updates every second from last 100 payloads
|
||||
|
||||
### 🔌 Source Connector Status
|
||||
Live status of all 8 connector types:
|
||||
- Running/Error/Unknown status with color coding
|
||||
- Fetch counts (total, successful, errors)
|
||||
- Last fetch timestamp
|
||||
- Base credibility score
|
||||
|
||||
### 🎯 Live Event Feed
|
||||
Real-time event detections:
|
||||
- Timestamp, asset, event type
|
||||
- Strength (0-100) with color coding
|
||||
- Confidence percentage
|
||||
- Source count
|
||||
- Sorted by strength descending
|
||||
|
||||
## Keyboard Shortcuts
|
||||
|
||||
| Key | Action |
|
||||
|-----|--------|
|
||||
| `q` | Quit |
|
||||
| `p` | Pause/resume updates |
|
||||
| `r` | Force refresh |
|
||||
| `f` | Focus Info Fetches |
|
||||
| `a` | Focus Asset Parameters |
|
||||
| `m` | Focus Market Aggregate |
|
||||
| `w` | Focus Word Cloud |
|
||||
| `s` | Focus Source Status |
|
||||
| `e` | Focus Event Feed |
|
||||
|
||||
## Running
|
||||
|
||||
```bash
|
||||
# From sentiment_engine directory
|
||||
pip install -e ".[tui]"
|
||||
|
||||
# Run TUI only
|
||||
python scripts/run_tui.py
|
||||
|
||||
# Run engine + TUI together
|
||||
python scripts/run_engine.py --tui
|
||||
|
||||
# Run engine only (headless)
|
||||
python scripts/run_engine.py --engine-only
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
The TUI runs as a separate `asyncio` task alongside the main engine. It receives data via direct method calls:
|
||||
|
||||
```python
|
||||
# From ingestion pipeline
|
||||
tui_app.add_fetch(payload)
|
||||
|
||||
# From scoring engine
|
||||
tui_app.update_assets(asset_signals)
|
||||
tui_app.update_market(market, industries)
|
||||
|
||||
# From connector registry
|
||||
tui_app.update_source_status(name, stats)
|
||||
```
|
||||
|
||||
The TUI uses `textual` (v0.52+) with `rich` for rendering. All widgets are reactive and update at 1Hz via a timer.
|
||||
|
||||
## Integration with Engine
|
||||
|
||||
In `main.py`, the `SentimentEngine` can optionally start the TUI:
|
||||
|
||||
```python
|
||||
engine = SentimentEngine()
|
||||
await engine.initialize()
|
||||
await engine.start()
|
||||
|
||||
# TUI runs in same process, shares event loop
|
||||
tui_task = asyncio.create_task(run_tui())
|
||||
await tui_task
|
||||
```
|
||||
|
||||
For production deployment, run TUI in a separate terminal/screen session while the engine runs as a systemd service.
|
||||
Reference in New Issue
Block a user