DNS Research

Step-by-Step Guide to Backtesting with Historical Data

advertisement

Step 1: Define a Precise Hypothesis Before Touching Data
Every credible backtest begins with a falsifiable hypothesis, not a vague idea like “moving averages work.” Specify the instrument universe (e.g., S&P 500 constituents, EUR/USD spot, BTC/USDT perpetuals), the signal logic (e.g., 50-period EMA crosses above 200-period EMA), the entry and exit rules, position sizing, and the exact time horizon. Write the hypothesis in a version-controlled document with a timestamp. This pre-registration prevents hindsight bias and data snooping, the two most common causes of backtest overfitting. A good hypothesis includes expected edge magnitude, expected win rate range, and the market regime in which it should perform (trending, mean-reverting, high-volatility). If you cannot articulate why the edge should exist economically or behaviorally—risk premium, liquidity provision, underreaction—the backtest is a fishing expedition, and its results are noise.

Step 2: Select and Audit Historical Data Sources
Data quality determines backtest validity more than any model choice. Prioritize survivorship-bias-free databases (e.g., CRSP for equities, Refinitiv, Bloomberg, Polygon, Databento) over free sources that only list current index members. For crypto, use exchange-native tick data from Binance, Coinbase, or Kraken with verified trade and quote files. Check for: adjusted vs. unadjusted prices (dividends and splits must be handled consistently), timestamp alignment across time zones, duplicate or missing bars, and bid-ask spread availability. Run a data audit script that flags gaps larger than 1% of the expected bar count, negative prices, zero-volume bars, and outliers beyond 10 standard deviations. Document every cleaning decision. For survivorship bias, reconstruct point-in-time index membership; for look-ahead bias, ensure fundamentals are lagged to their actual publication dates (e.g., 10-K filings appear 60–90 days after fiscal year end).

Step 3: Build a Point-in-Time Data Pipeline
Store raw data immutably and create a versioned, point-in-time (PIT) layer. Each record must carry an “as-of” timestamp representing when the information became publicly available. Use a columnar store (Parquet, DuckDB) partitioned by date and instrument for fast replay. Implement a replay engine that, for any simulation timestamp t, returns only data with as-of ≤ t. This eliminates look-ahead bias in corporate actions, earnings, and index rebalances. Validate the pipeline with unit tests: assert that querying 2015-03-15 returns no data published after that date, and that adjusted prices reconcile to raw prices times cumulative adjustment factors. Version your datasets with semantic versioning (e.g., equities_us_v2.3.1) and hash checksums so every backtest result is reproducible.

Step 4: Choose the Right Backtesting Engine
Select an engine matched to strategy complexity. For vectorized, daily-frequency strategies, use pandas, vectorbt, or zipline-reloaded. For event-driven simulation with realistic order types, use Backtrader, NautilusTrader, or QuantConnect LEAN. For derivatives and portfolio-level risk, consider Zipline or custom C++/Rust engines for speed. Evaluate engines on: order type support (market, limit, stop, IOC, FOK), partial fills, latency modeling, commission and slippage hooks, and multi-asset portfolio accounting. Avoid engines that silently fill orders at the close using the same bar’s high/low—this creates unrealistic fills. Confirm the engine’s bar-close semantics: signals computed on bar t should execute at bar t+1 open, not bar t close, unless you explicitly model intrabar execution.

Step 5: Model Transaction Costs and Slippage Realistically
Gross returns are fiction. Model commissions per share or per contract (e.g., $0.005/share plus $0.01/contract), exchange fees, and regulatory fees. Model slippage as a function of order size relative to average daily volume (ADV): a common approach is half-spread plus a square-root impact term, slippage = k * sqrt(order_size / ADV) * volatility. For illiquid assets, add a participation-rate cap (e.g., no more than 5% of bar volume). For futures, include roll costs and margin financing. For short sales, include borrow fees (hard-to-borrow names can cost 10–50% annualized) and recall risk. Calibrate parameters from your own execution data or published venue statistics, then stress-test at 2x and 3x assumed costs. If the strategy’s edge disappears at 2x costs, it is not robust.

Step 6: Implement Walk-Forward Optimization
Replace single in-sample/out-of-sample splits with walk-forward analysis (WFA). Define an optimization window (e.g., 3 years), a test window (e.g., 6 months), and a step size (e.g., 6 months). For each fold: optimize parameters on the in-sample window, freeze them, then trade the out-of-sample window. Concatenate out-of-sample equity curves to form the walk-forward equity curve, which is the only performance series you should trust. Use anchored (expanding) or rolling windows based on regime assumptions. Report the parameter stability across folds—large swings indicate overfitting. Complement WFA with combinatorial purged cross-validation (CPCV) as described by López de Prado, which purges overlapping labels and embargoes data near fold boundaries to prevent leakage.

Step 7: Run Robustness and Sensitivity Tests
A single backtest result is a point estimate. Stress it along multiple axes. Parameter sensitivity: perturb each parameter by ±10–20% and confirm performance degrades gracefully rather than collapsing. Monte Carlo: resample trade sequences (bootstrapping with replacement) 10,000 times to build a distribution of terminal wealth, max drawdown, and Sharpe ratio; report the 5th percentile. Synthetic data: shuffle returns to preserve volatility clustering (block bootstrap) and confirm the strategy does not rely on specific historical order. Regime analysis: slice performance by volatility quartile, trend vs. range (ADX or Hurst exponent), and macro regimes (rate hiking/cutting). Cost sensitivity: rerun at 1.5x, 2x, 3x costs. If results hold across all these, the edge is more likely real.

Step 8: Evaluate Performance With Risk-Adjusted Metrics
Go beyond total return and Sharpe. Compute annualized return, volatility, Sharpe, Sortino, Calmar, and maximum drawdown with recovery time. Report hit rate, payoff ratio, profit factor, and average trade duration. Analyze the equity curve’s autocorrelation and the distribution of drawdowns. Use the Deflated Sharpe Ratio (DSR) to adjust for the number of trials—if you tested 500 parameter combinations, the best Sharpe is inflated; DSR accounts for selection bias. Compute the Probabilistic Sharpe Ratio (PSR) and Minimum Track Record Length (MinTRL) to determine how much live data is needed before you can statistically distinguish the strategy from zero skill at a chosen confidence level. Report turnover, capacity (AUM at which slippage erodes edge), and beta/alpha decomposition against relevant benchmarks.

Step 9: Diagnose and Eliminate Common Biases
Systematically audit for: survivorship bias (delisted and bankrupt names included), look-ahead bias (PIT data verified), data snooping (pre-registered hypothesis, DSR applied), transaction cost underestimation (calibrated slippage), and optimization bias (WFA + CPCV). Check for the “backtest overfitting” signature: smooth equity curve in-sample, erratic out-of-sample, and parameter settings on a knife-edge. Verify timezone consistency—a daily bar stamped 00:00 UTC for US equities misaligns signals. Confirm that signals use only information available at decision time; a common leak is using the current bar’s close to generate a signal that executes at that same close. Run a “random signal” control: replace your alpha signal with random entries but identical exits; if performance is similar, your signal adds nothing.

Step 10: Paper Trade and Forward Test Before Capital
Backtests are hypotheses; live markets are the experiment. Before deploying capital, run the exact strategy in paper trading or a shadow portfolio for at least 3–6 months, or until you accumulate the MinTRL number of trades. Compare paper results to backtest expectations on a rolling basis: track slippage vs. model, fill rates, signal-to-order latency, and drawdown behavior. If paper performance deviates beyond a pre-specified tolerance (e.g., 2 standard deviations of expected Sharpe), halt and diagnose. This forward test catches implementation bugs, data feed issues, and regime shifts that no historical simulation can reveal. Only after consistent alignment between paper and backtest should you allocate capital, starting at a fraction (e.g., 10–25%) of target size and scaling as live results confirm the model.

advertisement

latest posts

Something went wrong. Please refresh the page and/or try again.