I was going to say this about your posts. The paper clearly states this, and lays out some ground rules for when it is and is not applicable. Obviously, you use thew two terms interchangeably which is not correct. No eBook available Amazon. Who is David Aronson? The book is intended for users who wish to master TSSB without support or with at most very minimal support.
TSSB is a free software platform from Hood River Research designed for rapid research and development of a statistically sound predictive model based trading systems via machine learning.
The two articles I posted did not mention this problem but it is given extensive treatment in our book and software. Monte Carlo tests cannot deal with data-mining bias because they cannot reject perfectly fitted systems. Neither you nor Masters are aware of that but we do because we actually trade. The Monte Carlo analysis is good only when it is applied for a sound trading system and not to an over-fitted one. Also I am surprised you mix data-mining and data-snooping bias. These are two different things.
Your program works by actually enforcing data-snooping bias and there is not sound known statistical tests other than the SPA test to deal with that. All that you can show a superior rule exists in the universe of rules you deal with but it is impossible to know which one it is. If you know of a test that can determine whether a specific trading rule will work in the future and it is not just a result of data-mining please let us know so we can contact Stockholm and nominate you for the Nobel Prize.
From your response it appears as if you are thinking of another application of Monte Carlo that we do not employ. I am happy to respond further but let first see if we are on the same page. Hello again Bob sorry about Joe can you tell me what you understand to be the distinction between data mining bias and data snooping bias. We are familiar with Dr. Masters work and aware that he makes various significant and unfounded assumptions like for example that all random pairing of system returns with daily raw returns are equally likely, something that amounts to esoteric statistics in this particular case.
Then his methods apply only to systems that are invested all the time otherwise considering daily raw returns in a MC or bootstrap does not make sense. As someone commented in Amazon about your book, these are statistical acrobatics because there is no sound theoretical framework that underpins it. This is reversal of the burden. You claim to be able to deal with these so please you help us clarify the distinction. Obviously, you use thew two terms interchangeably which is not correct. To conclude, if your statistical methods worked it would only make sense to have kept them secret.
Otherwise, revealing them to the public and giving the software away for free is not something that someone with an edge would do. Actually most people in the field of machine learning and data mining do use the terms interchangeably. Robert Hansen does in his most recent article on SPA I recently chaired the machine-learning track of a conference for traders in Chicago sponsored by Terrapin and there as well speakers used them interchangeably.
While I did make a distinction between data mining and data snooping in my book, the later relating to obtaining hypotheses from others research papers, I came to find others did not make that distinction. You seem to think that we are permuting the order of trades produced by an over fitted trading system.
Moreover your assumptions about Dr. Masters current work are also wrong. This latest version has not be put in the public domain. They do not work for machine-learning, which involves a guided search through the hypothesis space. These tests only work if the search is either exhaustive or random.
It is precisely for this reason that the MCP test incorporated into TSSB was designed to be valid under the condition of a guided searche machine learning. Your prior posts suggest your opinions are entrenched and any further attempt to correct your errors will prove fruitless. I will leave you the last word as you seem to need that.
I have no doubt it too will be filled with inaccuracies and insults. He is correct in asserting that curve fitting, learning noise instead of authentic patterns, et cetera, result in an optimistic bias in performance figures. But what he does not seem to understand is that the main driving force behind the design of TSSB is to defeat this exact problem!
The version of the MCPT documented in the several papers written by me and available online is an early design, excellent in some narrowly defined applications but worthless in a broader environment. The paper clearly states this, and lays out some ground rules for when it is and is not applicable. This algorithm has subsequently been refined to be far more robust, and it is this refined version that is incorporated into TSSB.
In a general sense, here is how a properly designed MCPT test operates. Stipulate that the process of developing a model-based trading system training models and committees, setting thresholds, selecting the best performers, et cetera produces optimistically biased performance figures.
If the models are excessively powerful and overfit the data, this bias can be extreme. So to assess and account for this we deliberately destroy any predictive power in the data. This may be done by randomly permuting market changes, permuting targets relative to indicators, permuting predicted market moves relative to optimized thresholds, or in any other number of ways, each of which has its benefits and limitations.
If this is done a large number of times hundreds or thousands we can assess the degree to which the system development process inflates performance. If, for example, we see that the performance results obtained from the actual data greatly exceeds that produced from the permuted data, we can be confident that our developed system has true power above and beyond that produced by the bias inherent in the development cycle.
These issues are discussed in greater detail in the the book Statistically Sound Machine Learning. Pages describe a particular MCPT useful for quick-and-dirty screening of indicators for predictive power. Page describes another MCPT that can be used to test the true predictive power that above and beyond any data mining bias of an individual model. Pages describe an extremely powerful MCPT that completely removes the effects of data mining bias throughout the entire chain of decision making models, committees, oracles in order to assess the true capabilities of the system.
In fact, Pages describe how it can even account for the interaction between long-term market bias and trading systems that are unbalanced in their long-short positions.
I do make the point several times in the book that many of these algorithms are recent developments and lack rigorous theoretical justification. On the other hand, I do have a PhD in mathematical statistics. Finally, let me say that if anyone has a specific technical criticism of any of these algorithms, I would be happy to hear this criticism, and if I think it is valid I will answer in some way. Critics may, in fact, help me to improve the program. I am always eager for constructive criticism.
Again, you are both trying to accuse me of plain criticism but the truth is I have tried to explain to you a few times that a perfectly curve fitted system will always appear good based on your tests.
So you are basically achieving nothing. This is a fundamental problem with MC simulation and no matter how one modifies it it will be there.
I was going to say this about your posts. MC simulation can rule out random and bad system but will not rule out well fitted systems, the types that are produced by your software or other machine learning software, and it appears that you are relying on a method or modification thereof that does not work. If you know of a method that will guarantee that a system your software produces will work in unseen data in forward mode please let us know. You do not because nobody does.
You reduce the data-mining bias inherent in bad system and you retain the best fitted systems because the MC, no matter how it is modified, cannot reject those. I have been clear so please do not accuse me of being vague or just criticizing your work. You basically have nothing and you know that. Interesting discussion—especially since it sounds like Bob is trying to expose another fraud, of which the financial landscape is littered.
Unfortunately, while I am no dummy, I do not have a Ph. This leaves the specific points of debate somewhat over my head. I think datasnooping is one of the biggest problem with backtesting and should be discussed more.
An argument in favor of David is that Jaffray Woodriff one of the most successful hedge managers seem to do similar things: Here Woodriff explains that he looks at the difference in outcomes between the model on the real data and on the data with random disturbances. Although I do not hold a PhD in statistics, I have studied this topic fairly deeply.
I read the fxhackers blog post per the link that Bob posted in the comment. That blog post Bob suggested talks generically about Monte Carlo methods. Unfortunately the meaning of the term Monte Carlo is overloaded.
There are many methods of resampling including bootstrap, jacknife, and permutation. The method discussed at fxhackers is actually a simple bootstrap using a set of trade results from a back-test. The term Monte Carlo in this case simply means random re-ordering of individual trade results.
This method is the most naive use of resampling applied to trading system evaluation there is. If you call that a marketing scheme, then so be it. We incorporated the best that is out there. But other features OPstring models, Split Linear models, the Purify Transform, Oracles intelligent ensembles , intelligent dimensional reduction, just to name a few are new and not found elsewhere at any price, let alone free.
What is also new is the ability to test the statistical soundness of a predictive-model based trading system, a model ensemble or even a portfolios of trading systems developed via machine learning. The test, Monte Carlo Permutation, was proposed by my co-author Dr.
If you are aware of any other software free or otherwise that permits the computation of a p-value that is robust to the effects of data mining bias please post a new comment. Had you read the book you realize how ridiculous such a comment is. Trading systems developed with TSSB are explicitly based on predictive modeling rather than humanly proposed rules. Of course it does offer a full range of descriptive statistics.
But even in this area TSSB offers entirely new things such as a histograms based on thresholding logic and marginal density plots. The book is intended for users who wish to master TSSB without support or with at most very minimal support.
We developed TSSB over a seven year period at a considerable cost for our own consulting work because we were not able to find a product at any price that does this. The book is not free. But I must warn you that TSSB is a complicated comprehensive platform intended only for the most sophisticated trading system developers.
However, you will be able to develop a stand alone trading system and a trading system signal filter by the end of the 2nd chapter.
Why do I have to complete a CAPTCHA?
Co-designer of TSSB (Trading System Synthesis and Boosting) a software platform for the automated development of statistically sound predictive model based trading systems. Author & editor of Statistically Sound Machine Learning for the Algorithmic Trading of Financial Instruments: Developing Predictive-Model-Based Trading Systems Using . tssbutil. This is an automation framework for Trading System Synthesis and Boosting (TSSB). TSSB is nice package available here from Hood River Research for the development of predictive model-based trading systems, but right now it is GUI only and the output is in verbose log files. The tssbutil framework uses pywinauto to enable a . “Second, this book shows how the free program TSSB (Trading System Synthesis & Boosting) can be used to develop and test trading systems. The machine learning and statistical algorithms available in TSSB go far beyond those available in other off-the-shelf development software.