99 lines
2.6 KiB
Markdown
99 lines
2.6 KiB
Markdown
|
# DOLPHIN NG HD Data Locations
|
||
|
|
|
||
|
|
## Production Data
|
||
|
|
|
||
|
|
**Location**: `C:\Users\Lenovo\Documents\- Dolphin NG HD (NG3)\correlation_arb512`
|
||
|
|
|
||
|
|
### Directory Structure
|
||
|
|
|
||
|
|
```
|
||
|
|
correlation_arb512/
|
||
|
|
├── matrices/
|
||
|
|
│ ├── 2025-12-26_SKIP/
|
||
|
|
│ ├── 2025-12-27_SKIP/
|
||
|
|
│ ├── ...
|
||
|
|
│ ├── 2025-12-31/
|
||
|
|
│ ├── 2026-01-01/
|
||
|
|
│ │ ├── scan_016875_w50_000003.arb512.pkl.zst
|
||
|
|
│ │ ├── scan_016875_w150_000003.arb512.pkl.zst
|
||
|
|
│ │ ├── scan_016875_w300_000003.arb512.pkl.zst
|
||
|
|
│ │ ├── scan_016875_w750_000003.arb512.pkl.zst
|
||
|
|
│ │ └── ...
|
||
|
|
│ ├── 2026-01-02/
|
||
|
|
│ ├── 2026-01-03/
|
||
|
|
│ └── 2026-01-04/
|
||
|
|
│
|
||
|
|
├── eigenvalues/
|
||
|
|
│ ├── 2025-12-26_SKIP/
|
||
|
|
│ ├── ...
|
||
|
|
│ ├── 2026-01-01/
|
||
|
|
│ │ ├── scan_016875_000003.json
|
||
|
|
│ │ ├── scan_016876_000014.json
|
||
|
|
│ │ └── ...
|
||
|
|
│ └── ...
|
||
|
|
│
|
||
|
|
├── eigenvectors/
|
||
|
|
│ └── [dated directories with eigenvector data]
|
||
|
|
│
|
||
|
|
└── metadata/
|
||
|
|
└── [dated directories with metadata]
|
||
|
|
```
|
||
|
|
|
||
|
|
### File Naming Convention
|
||
|
|
|
||
|
|
**Eigenvalue JSON**: `scan_NNNNNN_HHMMSS.json`
|
||
|
|
- `NNNNNN`: 6-digit scan number
|
||
|
|
- `HHMMSS`: Timestamp (HHMMSS format)
|
||
|
|
|
||
|
|
**Matrix ZST**: `scan_NNNNNN_wWWW_HHMMSS.arb512.pkl.zst`
|
||
|
|
- `NNNNNN`: 6-digit scan number (matches eigenvalue)
|
||
|
|
- `WWW`: Window size (50, 150, 300, 750)
|
||
|
|
- `HHMMSS`: Timestamp
|
||
|
|
- `.arb512.pkl.zst`: Blosc-compressed pickle with 512-bit arb precision
|
||
|
|
|
||
|
|
### SKIP Directories
|
||
|
|
|
||
|
|
Directories with `_SKIP` suffix should be excluded from processing.
|
||
|
|
These contain data that failed validation or is marked for exclusion.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Test Data (Current Project)
|
||
|
|
|
||
|
|
**Location**: `C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict`
|
||
|
|
|
||
|
|
Test data should mirror production structure with partial data:
|
||
|
|
```
|
||
|
|
- DOLPHIN NG HD HCM TSF Predict/
|
||
|
|
├── matrices/
|
||
|
|
│ ├── [root level files - legacy format]
|
||
|
|
│ └── 2026-01-03/
|
||
|
|
├── eigenvalues/
|
||
|
|
│ ├── 2026-01-01/
|
||
|
|
│ └── 2026-01-03/
|
||
|
|
└── ...
|
||
|
|
```
|
||
|
|
|
||
|
|
**Note**: Test data scan numbers may not match between directories.
|
||
|
|
Always verify pairing before running pipelines.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Quick Reference
|
||
|
|
|
||
|
|
| Environment | Path |
|
||
|
|
|-------------|------|
|
||
|
|
| **Production** | `C:\Users\Lenovo\Documents\- Dolphin NG HD (NG3)\correlation_arb512` |
|
||
|
|
| **Test/Dev** | `C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict` |
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Related Documentation
|
||
|
|
|
||
|
|
- **ZST_Compressed_Matrix_DOLPHIN_format_spec.md** - Detailed format specification for `.arb512.pkl.zst` files
|
||
|
|
- **run_joint_encoder_pipeline.py** - Pipeline using this data
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
*Last updated: 2026-01-10*
|