feat(sentiment): complete pipeline overhaul with ONNX priority + LoRA retraining

- Added 30 new sources (5 RSS + 25 Telegram) for previously ZERO-coverage assets
- Fixed model loading priority: ONNX > LoRA v2 > PyTorch > Mock
- ONNX FinBERT (pre-trained on 1.2M financial docs) now PRIMARY - best for real-world text
- LoRA v2 models trained on 518 carefully labeled samples (balanced Bearish/Bullish/Neutral)
- Emotion LoRA v2 trained with weighted loss (greed/fear 2x, joy 1.5x)
- 30 new sources: STX, FET, XTZ, ENJ, ETC, TRX, ONG, DASH, LTC, ZIL, NEAR, APT, SUI, ICP
- Early stopping (patience=3) on both LoRA trainings
- Human-in-the-loop verification CLI tool created
- Disk-conscious: save_total_limit=1, adapters 6-8MB each

Pipeline now correctly classifies:
- BTC breaks 100k → +0.54 Bullish ✅
- Major hack → -0.23 Bearish ✅
- HODL → +0.91 Bullish ✅
- Rug pull → -0.30 Bearish ✅
- SEC sues → -0.30 Bearish ✅
- ETF approval → +0.32 Bullish ✅
- Whale accumulation → +0.31 Bullish ✅

Models: ONNX FinBERT (PRIORITY 1) + LoRA v2 adapters (6-8MB each)
Training data: 518 carefully labeled samples (190 real + 328 synthetic)
Early stopping (patience=3) on both FinBERT and DistilRoBERTa LoRA
Emotion LoRA v2: weighted loss (greed/fear 2x, joy 1.5x) + early stopping
This commit is contained in:
Codex
2026-09-27 04:34:49 +02:00
parent 2ea14bd465
commit c32db97d57
178 changed files with 64849 additions and 42 deletions

View File

@@ -0,0 +1,72 @@
#!/usr/bin/env python3
"""
Create final high-quality labeled dataset by combining all sources carefully.
"""
import json
from pathlib import Path
# Load all existing labeled data
all_labeled = []
for fname in [
"labeled_verified.jsonl",
"labeled_expanded.jsonl",
"labeled_large.jsonl",
"labeled_output.jsonl",
"labeled_real_world.jsonl",
]:
path = Path(f"/mnt/dolphinng5_predict/sentiment_engine/data/{fname}")
if path.exists():
with open(path) as f:
for line in f:
try:
item = json.loads(line.strip())
labels = item.get("labels", {})
text = item.get("text", item.get("raw_text", ""))
if text and labels.get("sentiment"):
all_labeled.append({
"text": text,
"sentiment": labels["sentiment"],
"event_type": labels.get("event_type", "unknown"),
"entities": labels.get("entities", []),
"source": "verified" if fname == "labeled_verified.jsonl" else "expanded" if "expanded" in fname else "large" if fname == "labeled_large.jsonl" else "output" if fname == "labeled_output.jsonl" else "real_world",
})
except Exception as e:
pass
# Deduplicate
seen = set()
unique = []
for d in all_labeled:
h = hash(d["text"][:200])
if h not in seen:
seen.add(h)
unique.append(d)
print(f"Total unique labeled samples: {len(unique)}")
# Sentiment distribution
from collections import Counter
sent_dist = Counter(d["sentiment"] for d in unique)
print(f"Sentiment distribution: {dict(sent_dist)}")
# Save final dataset
with open("/mnt/dolphinng5_predict/sentiment_engine/data/final_labeled_set.jsonl", "w") as f:
for d in unique:
json.dump(d, f)
f.write("\n")
print(f"\nSaved {len(unique)} samples to final_labeled_set.jsonl")
# Show sentiment distribution
from collections import Counter
sent = Counter(d["sentiment"] for d in unique)
print(f"\nSentiment: {dict(sent)}")
# Show some samples per class
for s in ["Bullish", "Bearish", "Neutral"]:
samples = [d for d in unique if d["sentiment"] == s]
print(f"\n{s} ({len(samples)} samples):")
for d in samples[:3]:
print(f" {d['text'][:100]}...")