Post New

Mechanical Investing using machine learning initial observations.

Well, I told everyone last year that I was stepping back from continually trying to outperform the market
to just sit back and enjoy life. But I can’t help myself.
I had SIP historical data in a database and P123 started allowing downloads of historical data for a price.
Free open source very capable software like Scikit-learn has the tools to make predictions.
I’m still in the exploration stage experimenting with different features, different learning algorithms and
am not ready to put real money into the pot yet.

The practitioners of financial ML predictions have come up with methods to build ML models that
supposedly eliminate some of the unreliable post discovery problems MI screens have had in the past.
When we build a MI screen in 2024 we make a model based on what we already know about what
already happened.

ML evaluates a model differently. They first define an algorithm to take any data and predict the future
(like a neural network or many tree search). Then the break the historical data into shorter subsets. For
each subset you train the predictor model on typically 60 to 70% of the subset and test the results.
The model is judged on the results of all the train-test sequences where no train-test sequence has any
future data. A model that does well did well looking only at the data it had at that point in time.

My ML results are different from the results we are used to from screens. With screens we typically
trying to come up with a method using a very limited set of features that pick the top 5, 10 or 15 stocks
out of a universe of one or two thousand. ML regressions are designed to accurately predict both
winners and losers accurately. For ML I’ve been putting in 60 to 100 financial and momentum features
and letting the algorithms decide what works. Others are using 300 or more. My initial results trying for
the top 1% (15 out of 1500) give good but very inconsistent results. However top 5% is very good which
unfortunately, is a very large portfolio.

My initial impressions:
My ML is not yet up to the performance level other individuals (some obviously professional) are reporting.
My ML doesn’t do as well in the S&P500, probably because they are well evaluated by the big boys.
My ML does much better on mid-caps and even better on small caps.
My ML seems to especially outperform during the 2008 and covid dips. The down is slightly less and the
up is exceptionally good.

I’m still in the exploration stage experimenting with different features, different learning algorithms and
am not ready to put real money into the pot yet.
I haven’t got to the point where I model actual trades with turnover and friction.

ML evaluates a model differently. They first define an algorithm to take any data and predict the future
(like a neural network or many tree search). Then the break the historical data into shorter subsets. For
each subset you train the predictor model on typically 60 to 70% of the subset and test the results.
The model is judged on the results of all the train-test sequences where no train-test sequence has any
future data. A model that does well did well looking only at the data it had at that point in time.

This is how all data mining is done, when it's done correctly. Same with "classic" MI screens.
(terminology clarification: data mining is a good thing, overmining is a problem)

The problem is this: the process you have described, including seeing which models worked on the withheld out-of-sample validation subset and killing those that didn't, is itself another layer of data mining, another step in a single larger and more complex model building process. That "greater" process has no out-of-sample validation. Once you have culled your set of models by looking at the effectiveness within the validation data set you held back, that validation data set is contaminated and is now in sample, not out of sample.

This might be seen to be "OK" if you only did it once (depending on your strictness), but nobody does this just once. There just isn't enough financial history. In effect all history gets used, and it's all in sample. The only out of sample is the stuff that actually happened after you stopped modelling, and (to be strict) after you stopped culling your set of models.

Combined with machine learning with a lot of parameters, the ability of your final model to have memorized the data set AND the validation set is nigh unbounded. That's not to say it's impossible to find a new and useful insight this way, but it's a very thorny patch in which to be hunting.

Jim

Jim:
That's not to say it's impossible to find a new and useful insight this way, but it's a very thorny patch in which to be hunting.

I understand what you are saying and agree it is a complex problem to design models that don’t memorize the past but generalize on the methods that work post discovery. But ML has been working on techniques to minimize this for a long time. That’s not to say it still can’t be a problem but much less likely than just a few years ago.

In my models I trained on different 5 year periods of data and tested on following 2 years after a gap to prevent data leakage from data they had not seen. over 100’s of times they have in over 85% of the time performed well on the unseen data and never worse than average. And no I have never had to retrain them.

Marcos Lopez De Prado’s “Advances in Financial Machine Learning” has the best overall coverage of the problems making ML work post learning.

But from where the rubber hits the road I’ll have to acknowledge that ML doesn’t have a stellar history with capital management firms.

I’m still in the experimental stage, before I throw real money at this game I’ll have some better evidence it’s working post prediction.

Rob

Hi RAMc..... First of all kudos to trying out something new. Except as Jim mentions a LOT of it is basically like old wine in a new bottle.

Traditional MI ( esp the GTR1 kind) is basically an intersection of rulesets - which in a modeling sense is equivalent to partitioning the data into rectangular portions (Hyperplanes in multidimensional spaces) - the closest ML type algorithm for this is the Decision Tree :CART/C4.5 etc

I have been using them for a while - but not in the traditional MI Screen way. So my 0.02

(1) Algorithm : Dont get too hung up with this. Infact if you understand the underlying mechanism which Jim also alluded to, the basic issue is going to be dealing with Noise and MOST IMPORTANTLY Time Invariant ( ie Non-Stationary Data). If you have read some good books - I especially liked the fact that in the "Deep Learning" book - it clearly documents how Deep Learners are not exactly suited to these kind of problems - ie you have to go thru some hoops to make them work. While on the other hand - LTSMs beat traditional Time Series (ARIMA class) models - but what most people dont get is that Time Series models are not exactly great forecasters - they are like the hammer you try on every problem/nail.

Try Statistical MLs and some simple Deep Learners , infact no harm also to benchmark NNs. Choose the ones which have highest %age of Stable features across Dev, Validation, Test, Out-of-Time ( Not necessarily performance)

(2) Features : There's going to be Non-Stationarity in your features , its a given. Additionally be very very careful of choosing unscaled ones - a lot of times with MLs the absolute value of the underlying is basically the embedded information which is being used to discriminate. Eg for whatever reason - VIX in the late 90s rarely ever breached 30 and 20 used to be local maxima. This largely held thru to 2007 and even early summer 2008 and then BAM - it went close to 90. Nowadays the VIX is some of the lowest it has even been. Direct usage of this - will potentially impact your models ability to perform "Out-of-Sample"

(3) Validation/Out-of-Sample Data: You need to ensure you have rolling-cross/validation samples. Have a holdout which is contiguous time period with decent representation of both Bullish, Bearish periods. And have an OOT or Out-of-Time - This is like post-discovery - it should be the LAST STEP ie DO NOT LET INFO LEAK into the model from this. You judge your performance after all due diligence and ensure that Validation and OOT performance do not diverge too much ( or maybe diverge a bit- but consistently across methods. This would indicate unhandled non-stationarity, sometimes you just have to accept it)

Most importantly - PLEASE DO NOT SACRIFICE YOUR EFFORTS on the "ALTER OF BEST FIT" - what fits best in the lookback typically wont survive out of time ie post-discovery. Try to choose stable but decent performers.

All the best!

QUOTE : In my models I trained on different 5 year periods of data and tested on following 2 years after a gap to prevent data leakage from data they had not seen. over 100’s of times they have in over 85% of the time performed well on the unseen data and never worse than average. And no I have never had to retrain them.
ENDQUOTE

I of course do not know your objective ie success parameters of your model - so this step is absolutely sensible but not enough. You need to ensure that you separately evaluate performance at points of inflection. Some are obvious - but in the post 2009+ Mega Bull ..... your holdout/validation years need to have 2011,2016,2018,2020 and 2022 separated out. If something works well treating these as unseen ie using data before these years and validated on these post - and of course large scale would be 1990s validated on 2000-2002 and pre-2008 validated 2008-09 - then you will have confidence in your models

Best

anchak;

Thanks for the suggestions.
I too have been using ML for a while. Way back in 1986 the company I was working for became aware that IBM was in competition with us using ML. A group of us were sent off to learn the capabilities of ML. We ended up developing our own array processors but, in the end, IBM outperformed us. So, way back in the late 80’s and early 90’s a significant percentage of the current ML algorithms were known but limited in capability by the processing capabilities and supporting software.
I have been using NN’s for my investing since 1999 so I am well aware of data mining over fit.
However I haven’t been working in the field for over 20 years so have only kept up by playing at home.

I am using time sample cross validation with a gap and holdout. But I have found that rolling-cross/validation results are not a good as simple time series CV, it appears that the longer past history gives the latter a slight advantage.
I previously thought that as the market is changing over time more recent data would be more valuable. But so far, the longer the training historical data the better results. Training from 2000 for 5 years and testing for 2 years after a gap isn’t as good as training for 17 years and testing for 2 after a gap. Almost always better with a longer training set.

Additionally I’m trying to get an Ensemble of models to build a portfolio.
Aurelien Geron “Hands on Machine Learning with Scikit-Learn, Keras & Tensor Flow” points out in chapter 7 voting classifiers that Ensemble methods work best when the predictors are as independent as possible.
Voting doesn’t seem appropriate to regrssors.

I’ll have to rethink treating where I train/test bull and bear markets, up to now I have been letting them fall into the arbitrary fixed time folds. It was surprising to me to see bear ML test worked on models trained in bull periods without any special considerations.

Thanks again for the advice, you defiantly are more up to date than I on the subject. As I said I’m going to reevaluate some of the methods I’m using. I’m still in the learning mode.

Rob

I’m still in the learning mode.

We saw what you did there ; )

Jim

Ram, thank you SO MUCH for posting this. I've been looking for some lever to learn more about ML, but more specifically a tool that may help with some basic predictive modeling for these long-used timing signals - statistical analysis beyond brute-force single-factor correlation formulas. Without building simplistic python scripts or spreadsheet regressions or R models to come up with difficult-to-interpret and disappointing r2s.

So I'm going to try out scikit-learn.

FC

FC ....

"So I'm going to try out scikit-learn." - Its python's ML Toolkit ..... ie you have to be decent with Python

Best

FlyingCircus:
First although I have only used R for a few very basic manipulations I believe that it has a full set of ML capabilities. I know they have versions of NN’s, Random Forests and most of the other same toolsets that Python with SciKit-Learn has.

However Kaggle on the Pros and Cons of R vs Python Sci-Kit learn; “In summary, machine learning in R tends to be a more tiresome experience than machine learning in Python once you have moved beyond the basics”.

Before you jump into Scikit-learn. As you said you have written python scripts check out pandas if you haven’t already. Pandas is good for mungeing the data and interface to Scikit-learn. Pandas is just Excel with Visual Basic on steroids.
A little off topic but Wes McKinney was a MIT math grad initially worked in quantitative finance at AQR capital management. In 2008 he wasn’t satisfied with the tools they were using so he started writing pandas and somehow convinced management to allow him to release it as open source. He conceived of pandas and got the ball rolling but at this point over 2000 volunteer contributors have made additions and many corporations have donated either money or their employees time to enhance it.

Wes McKinney has now made the 3rd edition of his Python for Data Analysis available for free at
Python for Data Analysis, 3E (wesmckinney.com)
I bought the paper copy just for convince.

Good Luck

FC & Bob ..... I think the Kaggle evaluation is a bit misguided. R and Python are an increasingly convergent place - infact I went to the website of Wes Mckinney - he now works at Posit .... ie erstwhile R-Studio labs. Posit is clearly aiming at bringing massive convergence and fungibility between these 2 platforms.

Most tools are common - but there's a key differentiator typically for Python - its a development tool, while R is primarily research oriented ( Shiny being exceptional attempt). Because both are in-memory -computing architecture is critical and this is where R has overtaken Python in the last few years. A lot of this may not be very relevant at all for the ML exercise undertaken for MI type attributes ( Data is atmost Daily and most firm related attributes change only on Quarterly basis) - but if the universe is all stocks ( like CRSP) - computing efficiency will start to matter.

If you are into writing your own code - pick whichever you are comfortable with.

But if you are just after something which involves minimal code or you want GUI look-and-feel my suggestions would be

H2O.ai : That team is primarily responsible for most of the Big data breakthrus in R and that platform is also cross - infact their Python one is a bit more feature rich.
But they use Distributed computing - so you can pretty much throw a lot at it.

Infact once you setup an ML pipeline in H2O it would practically work across algorithms/methods. And they also now have Deep Learners

MLR3 : R specific ML pipelines

Rattle : This is practically a defunct R package - but it does basic ML modeling in a GUI driven interface - however most of its dependencies are not supported on CRAN anymore. And SIZE of DATA will be an issue. I dont think it can efficiently process something like CRSP.

Best of luck!

Bob : Ensembling is the way to go .... that's what how I use mine anyway!

You can always do mean or median on Regressors as a proxy to Ensembling Classifiers or any other central tendency hashing looking at the distribution of your regression estimates. Classifiers typically work the same way - especially if you do the logit(sigmoid) transform to output the class probabilities instead of accepting the inbuilt thresholding . At that point in time you are ensembling continuous probabilities.

FC : If you are comfortable with R - the corresponding and actually WAAY more feature rich package to scikit-learn is CARET. In fact that's the most popular modeling package in R.

Best

For EVERYONE who's interested and reading this : MLs are not like some panacea .....

In any modeling you do undertake - please ensure you benchmark it against a standard GLM using elastic-net - like in H2O.

All the other methods introduce massive amounts of non-linearity and complexity - you need to PROVE/CONVINCE that its warranted. Else go with the simpler model!

Best

For EVERYONE who's interested
Absolutely! This for me is by far the most interesting thread since ages, as end of 80's I tried MI first with statistical means (failed), from 89-94 then with NN ("Brainmaker" for the NN, writing C code for the rest), failed again.

I had no idea that some people here are so deep into ML. Any links/references for a now total newbie how to get to grips with the current state?

"better on smallcaps"

Please post criteria for examination even if interim

Anchak; One super big thank you! You are a gold mine of information. I have run a GAM (Generalized Additive Model) which is supposedly an extension of GLM on a couple of different universes which came up with good results. However your Elastic Net Regularization suggestion was new to me. Looking through some docs and “is a logical pairing with GLMs — it removes unimportant and highly correlated features, which can hurt both accuracy and inference”. Thanks again this is exactly what I was looking for.

Said; “I tried MI first with statistical means (failed), from 89-94 then with NN ("Brainmaker" for the NN, writing C code for the rest), failed again”.
Seems we were running in the same universe. I was also a Brainmaker user . I was never able to use it with individual stocks nor with the market timing I tried. What kept me going was a very successful proprietary application that worked for a while, crashed but could quickly relearn and run again.

Mullelmant; Criteria for examination? Not sure what you mean but for small caps I tried both the SP600 and even smaller listed companies > $500K aveddvol. Model returns go us as the market cap goes down.
But I did not consider the friction or turnover , friction and volatility is obviously much higher in the small caps. I believe the sweet spot for most of us small timers is anything not in the SP500 with average daily dollar vol much greater than you plan to invest.

My primary concern with all of this was brought out by Loren Cobb back in 2000.

To me, all of this thunder and lightning about datamining seems to miss the point. Datamining works as a concept only within the impoverished world of statistical models. Outside of that domain there are gorgeously complex mathematical models whose validity derives from well-understood mechanisms of cause and effect. When causation is poorly understood, as for example in the stock market, then our models are necessarily simpler and more ad hoc, and the cry of Datamining! has at least some semblance of applicability. But is it helpful? I think not. We need better models of cause and effect, not purer statistics.

http://www.datahelper.com/mi/search.phtml?nofool=y...

The question then is: How do you insure using approaches such as this that there is truly a reasonable cause/effect relationship going on?

It reminds me of this:

https://www.investing.com/analysis/the-bangladesh-...

Zee; “How do you insure . . . .that there is truly a reasonable cause/effect relationship going on”

Agreed and a very good point. ML will identify patterns and correlations in data without actually establishing causal relationships. But hopefully if the same ML patterns are determined in multiple time periods with different universes of samples there will be a greater probability that the observed relationships will be causal.

I'm not proclaiming to have the answer, but as I said I still experimenting.

Zee.... There are always pros and cons .... but one always has to attempt to frame the issue with an unbiased mind

(1) World (Data Science) has advanced exponentially from 2000s. Majority of the "at-play" stuff today are breakthrus in 2010s-2015 ish periods
(2) I know who Lorren Cobb is/was - IIRC he used to be a prof in a Colorado University. Background in physical sciences. The reason why I mention this is that physicists tend to have this bias - "everything is a system" , ie some closed form representation exists.

Of course , we ( ie Statisticians/ML Scientists) also tend to have a bias - we have a good tool ie Hammer - so everything starts to look like a nail. ( Infact this is almost a direct quote from one of the best Deep Learning books)

(3) Frankly, this sort of a broad brush is disrespectful to highly erudite and honorable folks. My Linear Models prof ( He's like Top 5 in LM/Exp Designs) told me once "learn to figure out how a model goes awry first. That's the key to building a good one"

And in the years and years in practice we developed a saying "All models are wrong in the long run, some happen to be useful".

I think going in with a presumption that there must exist a STABLE Cause-Effect relationship is flawed - especially with Financial Markets. Weirdly, if there's one and you feed enough relevant info to a Deep Learner today - its practically guaranteed it will LEARN it from the data and give it back to you. As Bob (RAMc) says - if you do rolling cross-validations (enough of them thru cycles) and certain relationships come out time and again - then you have proved a Causal driver relationship from the data itself.

Again purely from a practitioners standpoint - these are the KEY steps IMHO:

(a) GIGO ( Garbage In/Out) ie Data Quality : Supremely important and in financial series - most critical is to ensure there's no "look-forward" leakage. Eg for whatever reason one chooses to use Q2 Company (SEC filings/VL type) data to forecast July returns. Issue is , even if the data is marked as Q2 - companies report thru July-August so the data really is not available till fully possibly Aug mid.

The biggest hole where I see this issue is actually macro data. Most econometric data is released at various lags - but if you look at historical repositories (eg FRED) - they mark it contemporaneously

If your performance ie Outcome periods are 1 yr out etc - this wont matter that much - but can still dilute your run-time ie actual/RL(real-life) post performance

(b)Benchmark yourself against a simple linear model. If you see not much difference between methods - stick to the simplest model always. This is the Occam's Razor problem

(c) Ensure a proper validation process. Read some books/articles - sequencing matters in financial time series - your Validation samples should have NO LEAKAGE from development time periods

(d) Understand what the model is saying - this in MLs is very time consuming and computationally expensive - so sometimes ignored. DONT. Field is called Model Explanations - you should be able to explain your model. IN ESSENCE - this is the crux of what started the topic - your end result shouldnt be some magic or the Butter indicator.

It may very much end up being Jim's ROE/Cash model - and there's no harm or shame in that - you just learnt/re-learnt that if a Firm has a healthy Return on Equity and Generates adequate Cash-Flow it IS A GOOD FIRM - and long term produces healthy returns on its stock.

Best

Love to hear about people's experiences with ML, some going back further than me :)

My first NN projects were in the 90s. I remember Brainmaker, but I think the best package was SNNS:
https://en.wikipedia.org/wiki/SNNS

Development on SNNS stopped by the late 90s, but it was the best thing available until 2010 or so.

From 2001-2010 I used NNs I developed for all my retirement account investments, similar to what others here were doing with MI. I also had similar results - about 24-29% CAGR. Until 2008, when things went south, including for my NN models. I gave up on those in 2011.

I've been working in Python for about 8 years. I've developed some XGBoost models on time series data that are better than a coin toss, which doesn't necessarily make them actionable. I tried RNNs on the same data, thinking they would do better with the time series. But the so far the XGBoost models have performed better. That seems to be consistent with this persons experience:
https://towardsdatascience.com/why-you-currently-d....

I'm not sure a backtest on data more than a couple years back has much value. If it hasn't worked in the last two years I'd assume the gig is up on that model. If I found something that worked, I'd expect it would be arbitraged away in a year or two, and consider myself lucky if it continued to work after that.

John

And, (extremely simplistically), the causal relationships have changed over time and will continue to change over time.

20-30 years ago, stock price effects had different momentum, time factor and fundamental causes as well as much less machine-based buying.

Seemingly even slightly predictive/probabilistic factors we may "find" now will probably not last for long. That said, good MLMs can help adapt, IF designed and maintained well.

How do you model the reliability of and change in seasonality indicators (decay / improvement) over time, just for one example? In the trending of the reliability of a trend indicator?

All interesting stuff, and ethereal, except for the long standing price x volume.

FC.... "And, (extremely simplistically), the causal relationships have changed over time and will continue to change over time."

Correct and technically - deadly statement = Non-Stationarity
and puts a spanner in the works for most common methods.

In most cases - one has to work with at least temporal stationarity assumption. The problem with that is "then what?"

There are 2 options - at least what I have tried ....

(1) Staged model - try to built a time series model ( you can use your method of choice , although LSTMs are known to beat ARIMAs but its typically not by much , I have stuck to ARIMAs) to handle the time variance and then assume stationarity for the rest.

This one is very flexible but the issue is forward forecast - you dont really know what really is changing on the LT horizons. Really only useful for simulations etc to understand HOW MUCH of a spanner damage it can cause. Additionally if you assume migrating errors - almost all bets are off ( Some people will say GARCH - you can try that - but its all more and more complexity and tons of assumptions - which may or may not hold)

(2) Or incorporate Time variance regressors in the the main model itself. This assumes the time variance itself is a bit stable ( This works for very Short Term types of situations)

NET MSG: Non-Stationarity is a bane - anyone saying its not , well ......

John and others ..... Here's a link to the original M5 ( Makradakis) competition submissions and outcome

"I've been working in Python for about 8 years. I've developed some XGBoost models on time series data that are better than a coin toss, which doesn't necessarily make them actionable. I tried RNNs on the same data, thinking they would do better with the time series. But the so far the XGBoost models have performed better. That seems to be consistent with this persons experience:"

https://www.sciencedirect.com/science/article/pii/...

The winning methods were all practically based on LGBMs .... this is a version of a booster developed at Revolution R ( now part of Microsoft) .... it has some distinct advantages - especially computationally ....but its not a 1-to-1 XGB beater. Infact the latest versions of XGB has tried to incorporate some of the advances from LGBM into it. I think the lintree ( Linear tree) especially is well suited for Time Series problems. I practically almost always use LGBM if using boosting ( with XGB as a benchmarker).


Hope this helps!

Well I unintentionally stirred up a hornets nest. The two most esteemed old time members of the board mungofitch and zeelotes both seem to dismiss my ML attempt at stock prediction as a complete data mining folly.

But trying to respond;
FlyingCircut; “And, (extremely simplistically), the causal relationships have changed over time and will continue to change over time.”

Another good point! It’s obvious to anyone doing MI for many years that factors that worked 15 years ago don’t work today. As they say mispricing is discovered and arbitraged away. That’s why I thought the ML training would work better using say only the last 5 years of history. However, the few tests that I’ve done show using longer training periods are giving better results. On my list to try is using all recent data along with subsampled longer-term data to emphasize the recent without leaving out all the past.

Looking at how the ML models are approaching the prediction problem I see advantages and disadvantages. The advantages are an automated tool that finds general relationships covering the whole data set at a speed no individual person could hope to match.

There are still very distinct advantages to the very skilled individual building a classic MI screen.
If you have followed Yuval Taylor over the last several years as of 11/23, he states he has had an eight-year CAGR of 44%. That’s almost unbelievable for someone with my much less stellular results. But if you observe how Yuval a person with no finance background but an outstanding mind is approaching the problem after reading his posts for several years, I believe him. Perhaps P123 where he was running his screens and could monitor his results also did, they hired him. He has systematically gone through the detailed analysis of financial factors, digging deeper and modifying to his preferences value ratios, earnings growth, share turnover and many more. For each on he tries to understand how they work in different sectors/industries.
The point is I believe an intelligent person with the right tools with a full time multi year effort can always perform better than the ML can. Yuval has been working at P123 alongside the team developing their ML system and he has stated he at least for the present doesn’t see the value in it for him.

Hello Boss..... I come from a school of thought - where if you are at Zero - anything positive is an improvement.... even if you are at 45 and failing the course. You have every hope of passing AND with a learner - you might go all the way past 80. Yeah - you wont look so cool as your genius friends who come and seem to conjure up 90+s all the time- but who cares!

QUOTE :
Well I unintentionally stirred up a hornets nest. The two most esteemed old time members of the board mungofitch and zeelotes both seem to dismiss my ML attempt at stock prediction as a complete data mining folly.
: ENDQUOTE

(1) I definitely think you misinterpreted what Jim said. Please re-read. His concern and that's the universal concern with MLs is its ability to remember the whole sequence and just reproduce it forward. In our days in school "Commit to memory , vomit to paper" - till you come up against someone like BKS with questions always from the left most field and over the horizon - ala the MARKET!

NET : Please please ensure you have an OOT ( Out-of-Time) Sample - this is pristine.(at min this should be FY20-23) You will ONLY touch it - after you have your final model. And then perform an OOT validation as a completely separate exercise - if it bombs - well SOL. I have oodles of folders with stuff like that.

(2) Zee : He's from a different school of thought. Man spends hours and hours on developing expert systems with massive amounts of data and fine tuning by hand.
Think of him as your Alpha-GO challenge :)

"The advantages are an automated tool that finds general relationships covering the whole data set at a speed no individual person could hope to match." BINGO!
As per my thought process most modeling involves Search and Optimization - automating it helps you get to a better place - for the larger audience

"The point is I believe an intelligent person with the right tools with a full time multi year effort can always perform better than the ML can. Yuval has been working at P123 alongside the team developing their ML system and he has stated he at least for the present doesn’t see the value in it for him."

True statement - Quote from Deep Learning - Most people cant overcome a computer today at chess. But a human expert with a computer learner is Unbeatable!

The point is - there's unlikely to be a LOT of secret sauce in what you are doing, because I am sure Wall St professionals have possibly already gone thru a few versions of it. But what it hopefully will/should provide the community , just like GTR1 is a consistency of approach and refreshing paradigm on which some of the stuff hereabouts can be measured.

Just a few days before this "ML for MI" thread was initiated, I had drafted some notes to myself about using AI in combination with MI, with the thought of possibly sharing them on this board. FWIW, here are the notes:

I'm new to this Shrewd forum, so my apologies if this has already been beaten to death. I followed Mechanical Investing assiduously for many years in the now distant past, but haven't kept up for a long time.

My question is whether AI can effectively configure trading strategies optimized for specifiable short-term and long-term financial objectives such as reducing risk, increasing gain, reliably affording income.

It might be responsive to all of our obsessions, from present financial status to age, health, life-expectancy, anticipations ... whatever.

It would access a large library of backtests, and be capable of specifying more, as well as situationally updating an ever-elaborating library of relevant history.

That may sound like a large order, but if it's understandable it's likely achievable, given adequate developmental resources.

Successfully undertaking something like that would require considerable investments of time and other resources.

Some or all of that investment might be recovered by affording an interface for general use that, rather than giving advice, suggests available explorations.

The advice issue is a big deal. You can’t suggest buying a particular stock, like BRK for example, but you can supply a variety of progressions positioning BRK within relevant contexts.

Your business preferences — integrity, solvency, price, sales, earnings, book value, dividends; as well as their momenta and collectively evolving ratios — proffer clues as to whether or not to buy, hold, or sell at any given moment.

Having read what's been shared so far in this thread, I've grown acutely aware of my near-total ignorance in the arena of ML. My thoughts have been related more to employing AI to customize MI. Some of Jim's strategies come to mind as objects of incorporation.

Unq

PS: I honestly can't wrap my head around the notion of developing an investment strategy based upon just two or three years of sample data.

To unquarked's It would access a large library of backtests, and be capable of specifying more, as well as situationally updating an ever-elaborating library of relevant history.

TL;DR version: discouragingly (sorry), expectations should be very low due to the scope. Plebes like us could learn the ML tools to varying degrees, test out a few handfuls of promising correlation ideas by trial & error on the limited raw data we have access to. It's all "when this scenario of a, b, c, d and/or e" happens, the forward returns on index / stock have been good / bad". If ML tools can identify for us, faster, some promising *causes* of positive forward returns, that's where it will add value.

*****
Unfortunately, the data needed to do this is only available to us publicans in GTR1 (or to those who pay for Pinnacle), and - although GTR1 is by far the cleanest and most expansive set of stock and index price and fundamental data publicly available for all history - it's limited to the data fields from its sources and the formula functions that one phenomenal developer (Robbie) has coded it to handle.

Now, sites like Barchart, Stockcharts, et al have vast arrays of technical indicator formulas to use to generate current stock picks from screening criteria (below low ATR, Chaikin Flow is a specific value, RSI in a specific range, etc etc etc ad crazium) - but they're not easy or cheap to backtest if they even can. And, they would not / do not allow scraping-type access. I'm not sure what their APIs do.

So, the institutions / hedge funds / database vendors (like FactSet, especially) have built all this data for use by themselves and for very expensive sale to others; and are paying thousands of highly trained data developers to make & backtest ideas to get trading edges (or just cheating and frontrunning trades).

To really build some predictive models, both kinds of data are needed, and a nearly infinite set of variable combinations and formula definitions are possible to test.

Wrapping up - discouragingly (sorry), the only thing plebes like us might be able to get done is learning the ML tools somewhat and testing out a handful of promising correlation ideas by trial & error on the limited raw data we have access to.

More than that is expecting to boil the ocean.

P.S. Long ago now, Bill2m's legendary weekly trade results tracker of every screen was the only "What's working lately" forward tracking database out there - and unfortunately nobody could take over that process when he stopped (which was partly because it was showing The Problem.)

How did Zee come up with his (reasonably reliable) indicators? With (1) paid access to historical data sources, (2) a narrow concept area to focus on: handling extremes in data to identify high-probability opportunities, fairly contrarian and (3) A LOT - YEARS - of time and effort in testing.

XGBoost, pandas and numpy etc may have accelerated those discoveries faster. (Who knows, maybe he used them.)

FC... First the GOOD news --- there are some intrinsic strengths in MLs

(1) Transformations - Not required. MLs figure out the topology themselves - ie literally you can throw just Price and Volume and the algo will learn it for you. EXCEPT of course if your idea is to FIND which ones - then you have to feed the transform as a feature itself.

That being said - Price and Volume in RAW formats itself and especially in combinations produce a degree of uniqueness which if fed in raw - the learner will remember by heart and not be able to generalize forward. So its important to provide it in some standardized formats ( Stochastics, MA diff, anything you have found useful...)

(2) Feature Importance ie subset selection : Not required also. Algo figures out for you - so more you throw at it the merrier. Although in practice I have personally found that if you follow a final step of variable redundancy between Development and Holdout - it tends to generalize better in Out-of-time ie Post-discovery simulation/testing

(3) However GOOD DATA is the key. Infact you are basically tethered/constrained by your data. But based on GTR1 capabilities - if someone knows how to manipulate it properly it should be possible to generate very good features for ML usage.

One of the simplest ideas could be an ML-SOS.

But again ML is not panacea ..... and infact if you consider and approach it as a "plebe" ( no offense) - your chances of success are fairly limited. I know there's a LOT out there about AI democratization - but they didnt really put fresh graduates on the Alpha-GO advisory team. Financial data is notoriously tough problem - you can easily land on misleadingly rosy results in backtest which will essentially lose you money in the forward.

My personal view would be for this group

(a) Try to band together a group and work on it both independently and together to cross-check ideas etc.
(b) Use MLs as a primary search tool - ie cull down the universe - you can simply eliminate a whole lot of stuff that doesn't work and then use the smaller set to build your own ideas

Best

FlyingCircus wrote: How did Zee come up with his (reasonably reliable) indicators? With (1) paid access to historical data sources, (2) a narrow concept area to focus on: handling extremes in data to identify high-probability opportunities, fairly contrarian and (3) A LOT - YEARS - of time and effort in testing.

XGBoost, pandas and numpy etc may have accelerated those discoveries faster. (Who knows, maybe he used them.)

No, I did not, and will not.

My primary source of inspiration has been a special shower head. :)

RAMc wrote: The two most esteemed old time members of the board mungofitch and zeelotes both seem to dismiss my ML attempt at stock prediction as a complete data mining folly.

Most esteemed! Say what! I think not. Mungofitch has that badge of honor but I most definitely do not.

Folly? I never said that, and wouldn't say that. I celebrate your efforts and anyone else who will willing put in the time and effort to truly understand how the stock market works and the various (many!) influences that guide it on a daily basis. My comment was simply a warning. If there is no reasonable cause/effect relationship be wary of what flows out of it. It has to be understandable. Behavioral science and macroeconomics are the two major influencers. MI investing has pretty much ignored both of these and limited their study to financial statements. It is just not that simple! And this is why I personally am convinced that MI has consistently failed over the years.

Well I unintentionally stirred up a hornets nest. The two most esteemed old time members of the board mungofitch and zeelotes both seem to dismiss my ML attempt at stock prediction as a complete data mining folly.

I don't dismiss it.

Looking at the whole subject from a great distance, there are two things that using smarter tools might accomplish.
* Exploring unexamined "corners" of the parameter space and discovering previously undiscovered pockets that tend to lead to good performance.
* Adding more degrees of freedom in the models.

The first one is a very good thing, provided only that the out-of-sample validation is done enough, and well enough, prior to trusting real money to it. Same with any investment method.

But the second one is generally a Very Bad Thing. There are good reasons that we have always liked screens with very few tuning parameters: less to tune means less chance of overtuning to the training set. It's astounding what a model will memorize if you don't watch it like a hawk.

I think there's a very good chance that you might accomplish good things on the first bullet, and I don't dismiss that at all. My prior post was merely intended to caution about the second...compounded by the hidden problems of doing out-of-sample validation cleanly.

An anecdote: back in the 90's when my team was paid to do predictive modelling on very large data sets, we mostly used a whole lot of linear factors, and it worked. We souped it up by feeding that result into a neural net (not many layers back then), which polished the model to get a few extra percentage points of accuracy, which (as we understood the data set quite well) was quite a surprise to us, even though the benefit was modest. These things can work. Fortunately we had a lot of validation data; we generally held back around 20% for that.

Data mining is defined as the discovery of new, useful, non-obvious and predictive rules in old data. Go for it.

Jim

Most esteemed! Say what! I think not. Mungofitch has that badge of honor but I most definitely do not.

I admit that I write more stuff.
But your stuff is more often useful. That gets my esteem!

Jim

But the second one is generally a Very Bad Thing. There are good reasons that we have always liked screens with very few tuning parameters: less to tune means less chance of overtuning to the training set. It's astounding what a model will memorize if you don't watch it like a hawk.

A bit OT -

The whole subject makes me recall work I did many years ago in designing aspheric optical lenses for display systems (believe it or not, cathode ray tubes lol).
We needed to define the exposure lens surface in a very precise way in order to produce a physical lens that exposed the phosphor stripes exactly where they needed to be.
Essentially, the optics needed to match the electromagnetic deflection coil as well as the geometry of the CRT, with all of its asymmetries.
It also needed to include deflection of the electron beam due to magnetic fields, some stray, some not.
There were also thermal and mechanical effects, process effects, vacuum effects, and all kinds of things that made "lens design" a huge industry problem.
Very fancy software (for the day) and procedures were developed and honed by huge corporations - Philips/Sylvania (where I worked), GE, RCA, Sony, Zenith, and others.

We desired to fit these surfaces with simple polynomials, but that rarely worked well enough.
You had to add higher orders. If you included up to sixth or eighth order terms, you could make a very nicely working lens.
We liked to stick to even orders, but sometimes you had to throw in odd order terms to make things work - without understanding why they were needed.
And the problem was, to manufacture the lens, you needed to hold it in various fixtures. And those fixtures held the lens outside the used region.
That was called the "extrapolated region", where you had to extend the polynomial (or other) fit in a way to produce a continuous, smooth, real surface.
That region was what was gripped and held during lens manufacturing, and during the exposure process as well.

Junior lens designers quickly learned that a sixth order polynomial made a wonderful lens in the used (in-sample) region, able to match most any dataset.
But in the extrapolated (out of sample) region, it was exceedingly difficult to extend those to a surface that could be held by lens grinding equipment.
The fit could "blow up" within millimeters of leaving the used region.
We would spend a lot of time trying to smooth and extend those complicated mathematical fits.
And senior lens designers would always say, if you need a sixth or eighth order term, you are trying to fit erroneous data, or noise.

I've always had this mental model of MI screen/strategy development being like my experience with lens design.
The better your fit in the used or in sample region, the less trustworthy is in the out of sample region.

But hey, that was 35 years ago. Our FORTRAN programs got replaced by C and then more sophisticated software, and then CAD came along.
You fit surfaces with meshes, and like magic these fit and extrapolation problems disappeared. (It was still challenging to make a well working exposure lens).
Then, CRT's went away, and the whole problem went away, to be replaced by new problems in plasma displays, LCD's, OLEDs, quantum dots, etc.

I think ML has a place at the table.


Mark

also OT define the exposure lens surface in a very precise way in order to produce a physical lens that exposed the phosphor stripes exactly where they needed to be.... followed by a lot of incredibly technical me and ee terms...

WOW that stuff was HARD!

In programming software I never felt like the stuff we were doing was really hard, it was just kind of tedious to put together the right sequence of commands using the rules and constraints of the human-developed tools we were using. We could get it to do stuff, and as more developers released cooler tools to make items easier to get done it was always evolving and mostly cool.

That stuff you guys were working on? Hard. Math. Mech and Electrical engineering, properties of physical materials at the molecular level. 6th order polynomials?? Jesus... I barely survived freshman calculus with three attempts at passing quizzes on first & second derivatives.

FC

RAMc: Seems we were running in the same universe. I was also a Brainmaker user

Interesting for me is that this seems to be the case for many (most?) here. When FlyingCircus says "In programming software..." it reminds of my long coding history. Jim has such too, Julian (Manlobbi) also. Apparently MI attracts computer nerds --- not surprising, a natural fit.

More surprising: That many of us (most?) are also Berkshire shareholders, which is kind of the opposite. Personal theory: Holding boring but reliable Berkshire is our "insurance", is balancing the scale.

Right or wrong, a lot of people with similar histories and mentalities seem to be here (So all of you former computer nerds also are now, after the end of your active career, into Yoga, Meditation etc.? Just (half) kidding :)

Interesting for me is that this seems to be the case for many (most?) here. When FlyingCircus says "In programming software..." it reminds of my long coding history. Jim has such too, Julian (Manlobbi) also. Apparently MI attracts computer nerds --- not surprising, a natural fit.

More surprising: That many of us (most?) are also Berkshire shareholders, which is kind of the opposite. Personal theory: Holding boring but reliable Berkshire is our "insurance", is balancing the scale.

I resemble these remarks.

Elan :-)

(So all of you former computer nerds also are now, after the end of your active career, into Yoga, Meditation etc.? Just (half) kidding :)

Hmmm. Suffice to say I spent some time in India

Jim

Sure... I turned my programming interest into a (somewhat forced) corporate career "progression" into an IT program manager. Should have just stayed technical.

Got an MBA and basic training in stock investing which led me to Voldemort just before TMF started up on AOL & the original MI. Combination of the above and the web sucked me completely in to data mining, edge-finding...

The 401K was my "insurance"; the leftover money from selling my deceased-early father's house was my growth plan. Which worked really well in the late 90s when the bull market made all those VL RS/IBD screens look brilliant and made us all look like geniuses for using them.

Suffice to say I spent some time in India

Ahhh, all those coincidences - which actually are not just coincidences. Ten years ago in a yoga retreat on Crete I met one of the main developers of the world's 1st Multiprotocol Router (at a company named Wellfleet). Later, in 2000, her boss told her: "We have to save money. You must fire some of your team". She: "They are my friends! Before I fire them I go". And off she was, to India, were she spent years in Ashrams, as she says "Learning Yoga, meditating, sweeping the floor and making tea" (Nowadays while still doing yoga she also is into IT again :)

What did Buddha say when he sat down at the lunch counter?
"Make me one with everything"

Post New

US Policy FAQ

---

Contact Shrewd'm
Contact the developer of these message boards.