7-Stage CRISP-DM Research Pipeline
CRISP-DM: Cross-Industry Standard Process for Data Mining
The MS-GARCH regime detection system progressed through 7 distinct research stages,
each building on the previous with rigorous statistical validation.
Click any stage in the diagram to navigate to its research article.
graph LR
A["Stage 1: Foundational Research
HMM Regime Detection
Sharpe: 0.32-0.81"] -->|CRISP-DM| B["Stage 2: Data Understanding
Statistical Validation
7,841 obs per asset"]
B --> C["Stage 3: Model Development
BIC Selection
2-regime GJR-GARCH"]
C --> D["Stage 4: Backtesting
Walk-Forward Validation
Sharpe 1.69, +32% alpha"]
D --> E["Stage 5: Weekly Optimization
Frequency Analysis
8.38x longer regimes"]
E --> F["Stage 6: Synthesis Report
6 Lessons + Code
+$191K validation"]
F --> G["Stage 7: Production Deploy
Hamilton Filter
4-state Kelly mapping"]
click A "/research/hmm-regime-detection"
click B "/research/ms-garch-data-exploration"
click C "/research/ms-garch-model-development"
click D "/research/ms-garch-backtesting"
click E "/research/ms-garch-weekly-optimization"
click F "/research/ms-garch-enhancement-report"
click G "/research/ms-garch-enhancement-report"
style A fill:#3B82F6,stroke:#1e40af,stroke-width:2px,color:#fff
style B fill:#3B82F6,stroke:#1e40af,stroke-width:2px,color:#fff
style C fill:#3B82F6,stroke:#1e40af,stroke-width:2px,color:#fff
style D fill:#10B981,stroke:#065f46,stroke-width:2px,color:#fff
style E fill:#10B981,stroke:#065f46,stroke-width:2px,color:#fff
style F fill:#10B981,stroke:#065f46,stroke-width:2px,color:#fff
style G fill:#F59E0B,stroke:#92400e,stroke-width:2px,color:#000
Research Outcome: The 7-stage pipeline transformed an academic hypothesis
(4-state HMM with Sharpe 0.32-0.81) into a production-deployed system achieving Sharpe 1.69,
70% fewer transactions, and $11,800/year in cost savings per $100K capital.
Total research duration: 4 months (Oct 2025 - Jan 2026).
Stage Details
Each stage produced peer-reviewed quality research with complete reproducibility.
Click any stage to expand its details, key metrics, and research artifacts.
- Established 4-state HMM framework: Bear, Neutral, Bull, Crisis
- Standalone Sharpe ratio: 0.32 (BTC) to 0.81 (SOL) -- insufficient for standalone trading
- Proved regime detection enhances position sizing rather than direct signal generation
- Literature review: Hamilton (1989) filtering, Ang & Bekaert (2002) regime switching
Impact: Defined the research direction -- regime detection as a position sizing enhancer,
not a standalone strategy. This insight saved months of dead-end research.
- 7,841 4H observations per asset (Jan 2022 - Jul 2025)
- ARCH effects validated: LM statistic 367-1,802 (p < 0.001 all assets)
- Extreme fat tails: SOL kurtosis 12.97, BTC 7.29, ETH 8.74
- Non-normality confirmed: Jarque-Bera test rejected for all assets
Impact: Justified GARCH modeling over normal distribution assumptions.
The extreme kurtosis (12.97 for SOL) proved that Gaussian models would catastrophically underestimate tail risk.
- BIC model selection: 2-regime GJR-GARCH optimal (+1,410.81 BIC improvement)
- Low-volatility regime: 74.3% frequency (dominant market state)
- High-volatility regime: 25.7% frequency (crisis/opportunity periods)
- GJR leverage parameter captures asymmetric volatility response
Impact: Data-driven model selection eliminated subjective bias.
BIC penalizes model complexity, ensuring the 2-regime model generalizes better than 3 or 4-regime alternatives.
- Walk-forward validation with 200-bar purge gap (institutional standard)
- Sharpe ratio: 1.69 (regime-adaptive) vs 0.82 (buy-and-hold)
- Alpha generation: +32% over benchmark
- Maximum drawdown: -29.9% (vs -52.1% buy-and-hold)
Impact: Proved regime-adaptive Kelly sizing doubles risk-adjusted returns
while cutting maximum drawdown by 43%. Walk-forward validation prevents overfitting --
results hold out-of-sample.
- 8.38x longer regime durations: 27.32 days (weekly) vs 3.26 days (daily)
- 70% fewer regime switches: 13/year (weekly) vs 112/year (daily)
- Transaction cost savings: $11,800/year per $100K capital
- Signal stability improved: regime persistence matches economic intuition
Impact: Weekly frequency was the breakthrough insight. Daily regime detection
produced noisy, expensive regime switches. Weekly classification aligns with actual
market regime persistence (weeks to months, not hours).
- 6 key lessons synthesized from 5 preceding research stages
- Backtest validation: +$191,150 profit vs inverted multipliers (-$191,150 loss)
- Regime multiplier calibration: Bull 67%, Neutral 50%, Bear 25%, Crisis 17%
- 35 Python research files consolidated into production-ready specifications
Impact: The $191K validation (correct vs inverted multipliers) provided
definitive proof that regime detection adds real economic value --
not just statistical artifacts.
- Hamilton Filter implementation: <15 microsecond inference latency
- 4-state Kelly mapping: Bear 25%, Neutral 50%, Bull 67%, Crisis 17%
- 1,192 lines of production code (RegimeDetectionActor)
- Integration with NautilusTrader event-driven architecture
Impact: Research-grade models translated into sub-15-microsecond
production inference. The Hamilton Filter provides real-time regime classification
without the computational overhead of full MS-GARCH re-estimation.
Research vs Production Comparison
The journey from research prototype to production system required significant engineering
decisions. This table highlights the key transformations at each dimension.
| Aspect |
Research (Oct 2025) |
Production (Jan 2026) |
Transformation |
| Model Type |
2-regime GJR-GARCH (BIC optimal) |
4-state derived (2x2 volatility x direction) |
✓ Richer state space for Kelly mapping |
| Regime Duration |
Low-Vol: 5.1 weeks, High-Vol: 1.6 weeks |
Similar persistence observed in live |
✓ Research predictions validated |
| Update Frequency |
Daily (112 switches/year) |
Weekly (13 switches/year) |
✓ 70% fewer transactions, $11.8K savings |
| Sharpe Improvement |
+0.87 (0.82 to 1.69 in backtest) |
+4.6% (Week 49 production validation) |
✓ Live performance confirms research |
| Transaction Costs |
$11,800/year savings potential |
Realized in production trading |
✓ Theory to practice confirmed |
| Codebase |
35 Python research files (Jupyter + scripts) |
1,192 LOC production actor |
✓ 97% code reduction, 100% type-hinted |
| Filter Algorithm |
Research-grade (full MLE, slow) |
Hamilton Filter (<15 microseconds) |
✓ 1000x+ latency improvement |
| Validation |
Walk-forward backtest, statistical tests |
Live P&L, Prometheus metrics, JADE Index |
✓ From hypothesis to measured reality |
Key Insight: The largest transformation was update frequency.
Research initially used daily regime updates (112 switches/year), but Stage 5 analysis revealed
weekly updates (13 switches/year) produce 8.38x longer regime durations that better match
actual crypto market dynamics -- saving $11,800/year in transaction costs per $100K capital
while maintaining equivalent risk-adjusted returns.