Bitcoin Strategy Research: What a Backtest Must Survive
Prepared for this website. This editorial analysis has not been verified as a statement of Leonard Osborne's personal views.
Key takeaways
- A historical simulation is evidence about a model and a dataset, not proof of future profit.
- The number of ideas tested matters when interpreting the best result.
- Validation needs a record of decisions, not just a final performance chart.
Contents
- Define the hypothesis before the result
- Treat repeated experimentation as part of the evidence
- Review the time boundary
- Ask what survives changes in assumptions
- Limits
- Source
Define the hypothesis before the result
A strategy research project should state the behavior it intends to study before selecting parameters. What is the proposed mechanism? Which observations would count against it? Which instruments, dates, and transaction assumptions belong to the test? Without these definitions, a researcher can unknowingly turn a question into a search for an attractive historical picture.
A useful research record preserves failed variants as well as successful ones. If only the final version remains, readers cannot judge how much experimentation produced it. A polished chart can conceal a long sequence of decisions that were influenced by the same data.
Treat repeated experimentation as part of the evidence
Bailey and co-authors examine the probability of backtest overfitting and propose a framework for evaluating it [1]. Their work motivates a broader discipline: the strongest historical result should not be interpreted without considering the selection process that produced it. A conventional holdout period is not a universal remedy for repeated research choices.
This does not mean every historical pattern is meaningless. It means that the experiment history belongs in the interpretation. A research note should describe how variants were selected, which information was available when, and how much of the evaluation remained untouched by later revisions.
Review the time boundary
A Bitcoin test should make the ordering of information explicit. Features must be based on information that would have been available at the decision time. Execution assumptions must follow that decision rather than quietly granting a favorable price from the past. Data corrections and exchange-specific timestamps also deserve documentation.
When observations overlap or decisions depend on shared windows, a validation design needs to address that dependence rather than assuming every row is an independent experiment. The appropriate method depends on the strategy and dataset; there is no single checklist that guarantees validity.
Ask what survives changes in assumptions
A result becomes more interesting when its interpretation survives reasonable variations in costs, parameters, sample periods, and data handling. A dramatic collapse after a small change is information that should remain visible. Results should also be compared with clearly described, relevant alternatives rather than with an uninformative zero benchmark alone.
The next stage is further evaluation, not a claim of deployable certainty. Simulation, paper observation, and live operation are different evidence categories and should be labeled separately.
Limits
This note proposes a review framework; it does not present a tested strategy, a validated algorithm, or Leonard Osborne's investment performance. No backtest, including one that passes several checks, guarantees profitable future trading.
Source
- David H. Bailey, Jonathan M. Borwein, Marcos López de Prado, and Qiji Jim Zhu, The Probability of Backtest Overfitting. The paper supports the discussion of selection risk, not a claim that a strategy in this article has been validated: https://www.davidhbailey.com/dhbpapers/backtest-prob.pdf
For informational and educational purposes only. No live trading performance is presented. Hypothetical examples, historical simulations, and forward-looking scenarios are not guarantees of future results. Digital assets involve significant risk, including loss of principal. This material is not personalized investment advice.