1. Distinguish In-Sample From Out-of-Sample Performance
Every credible backtest divides data into an in-sample (IS) period used to design and tune the strategy and an out-of-sample (OOS) period the strategy never “saw” during development. Professionals read the OOS column first, because IS results are contaminated by optimization. A strategy producing a 2.1 Sharpe in-sample and a 0.4 Sharpe out-of-sample is not a 2.1 Sharpe strategy—it is evidence of curve-fitting. Demand a minimum OOS window of 20–30% of total data, ideally spanning a different market regime (bull, bear, sideways, high and low volatility). If the developer cannot clearly state where IS ends and OOS begins, treat the entire result as promotional material.
2. Scrutinize the Equity Curve, Not Just the Headline Return
Total return and CAGR hide the path taken to get there. Open the equity curve and ask three questions: Is the growth smooth or stair-stepped with violent drawdowns? Does performance cluster in a few explosive periods while flatlining otherwise? Does the curve slope upward consistently across years, or does one year carry the entire result? A professional prefers a 12% annualized curve with a maximum drawdown of 8% and positive returns in 9 of 11 years over a 25% annualized curve that lost 40% once and made everything back in a single quarter. Plot the curve on a log scale to judge compounding fairly, and overlay the benchmark to see whether returns came from the strategy or simply from beta exposure.
3. Decompose the Drawdown Profile
Maximum drawdown alone is insufficient. Read the full drawdown table: depth, duration, recovery time, and frequency. A 20% max drawdown that recovered in three weeks is a fundamentally different risk than a 20% drawdown that took two years to recover. Calculate the average drawdown, the average time underwater, and the Calmar ratio (CAGR divided by max drawdown). Professionals also check drawdown behavior during known stress windows—2008, the 2011 euro crisis, March 2020, the 2022 rate shock—because a strategy that survives only benign conditions is untested. If the backtest does not include at least one crisis period, its risk metrics are optimistic by construction.
4. Examine Trade-Level Statistics
Aggregate metrics conceal trade-level pathologies. Pull the full trade list and compute: number of trades (fewer than 30–50 trades produces statistically meaningless results), win rate, average win, average loss, payoff ratio, profit factor, and the largest single winner as a percentage of total profit. If one trade accounts for more than 20–25% of net profit, the strategy is a lottery ticket, not an edge. Check the distribution of returns per trade—fat tails on the loss side mean rare catastrophic outcomes are underrepresented in the average. Also verify expectancy: (win rate × average win) − (loss rate × average loss) must be positive after costs, and the confidence interval around expectancy should exclude zero.
5. Recalculate Performance Net of Realistic Costs
Amateur backtests assume fills at the close or mid-price with zero friction. Professionals rebuild the P&L with commissions, bid-ask spread, slippage, market impact, and financing costs. For liquid futures or large-cap equities, model 1–3 basis points per side; for small caps, illiquid ETFs, or wide-spread instruments, model 10–50 basis points or more. Then stress it: double the assumed slippage and confirm the edge survives. A high-frequency strategy showing a 15% annual return before costs and 2% after costs is not viable. Also verify how the backtest handled dividends, splits, and survivorship bias in the universe—using today’s index constituents to test 2010 performance silently deletes every company that failed.
6. Test Robustness Through Parameter Sensitivity
A professional assumes every optimized parameter is a potential overfit until proven otherwise. Take each key parameter (lookback window, stop distance, threshold) and vary it ±20–50%. A robust strategy shows a broad plateau of similar performance across neighboring values; a fragile one shows a sharp spike where the chosen value sits isolated, surrounded by losses. Plot a heat map of two parameters against each other and look for the plateau. Additionally, run the strategy with one fewer and one more position, with 10% higher transaction costs, and with a one-bar execution delay. If any of these trivial perturbations collapses performance, the original result was an artifact.
7. Verify Statistical Significance and Sample Size
Returns must be distinguished from noise. Compute the t-statistic of mean trade returns and require roughly t > 2 (or a Sharpe ratio confidence interval that excludes zero). With fewer than 100 trades, even a 60% win rate can be indistinguishable from randomness—use bootstrap resampling to generate confidence intervals for CAGR, Sharpe, and drawdown rather than trusting point estimates. Check the Deflated Sharpe Ratio or comparable multiple-testing adjustments if the developer tried many variants; testing 500 parameter combinations and reporting the best one inflates the apparent Sharpe by a large factor. Professionals ask: how many versions were tested before this one was presented?
8. Match the Benchmark and Risk-Adjusted Metrics
Compare against the correct opportunity cost: an equity long-short strategy benchmarks against the S&P 500 plus a hurdle, a market-neutral strategy against T-bills or SOFR, a futures strategy against a managed-futures index. Report alpha, beta, Sharpe, Sortino, and information ratio side by side, and interrogate beta—if a “market-neutral” strategy shows 0.6 beta to equities, its returns are disguised directional exposure. Confirm whether Sharpe was computed on daily, weekly, or monthly returns, since annualization assumptions differ, and verify that the risk-free rate used matches the test period rather than today’s rate.
9. Inspect Assumptions About Liquidity and Capacity
A backtest that fills 50,000 shares of a stock averaging 20,000 daily volume is fiction. Professionals check position sizes against historical average daily volume (ADV), typically capping participation at 1–5% of ADV, and model capacity: at what asset level does slippage erase the edge? Strategies that look stellar at $1 million often fail at $50 million. Review how the engine handled gaps, limit-order fills, partial fills, and halted sessions. Any assumption that every signal executes at the exact price the model wanted—especially on gap opens or during volatile sessions—deserves skepticism.
10. Read the Assumption Log and Data Provenance
Before trusting any number, audit the inputs: data vendor, adjustment methodology, timestamp conventions, universe definition, delisting handling, and whether prices are consolidated or single-venue. Survivorship bias, look-ahead bias (using a signal computed from data not yet available at execution time), and point-in-time violations in fundamentals are the three silent killers of backtests. Professionals insist on point-in-time datasets and confirm that indicators use only completed bars. A result cannot be better than its data; a strategy backtested on restated earnings or a curated list of current index members is measuring hindsight, not skill.
11. Compare Against a Null Model and Live Tracking
Finally, benchmark the strategy against trivial alternatives: buy-and-hold, random entry with identical risk controls, and a coin-flip signal run through the same cost model. If the sophisticated strategy barely beats random entry with the same stop and position sizing, the “edge” is the risk management, not the signal. Professionals also compare backtest expectations to paper-trading or small live results, watching for divergence in fill quality, signal frequency, and slippage. A backtest is a hypothesis; live tracking is the experiment, and any material gap between the two is a finding worth more than the original equity curve.
