Unbalanced Classes in Machine Learning and the Stock Market

Many assets exhibit bull or bear trends which persist for long periods of time. This presents an interesting problem  for anyone trying to predict the future return of an asset: a lack of diversity in your training set. This problem is known as unbalanced classes in the machine learning field.

The basic issue is that many classification methods work best when your training data is roughly uniform between each class. However, bull and bear markets produce historical training sets in which one class dominates. For example, in a bull market our historical dataset will contain a majority of time periods in which a stock moves up. Likewise, bear markets produce a majority of down periods.

Why do unbalanced classes mess up machine learning models? Because you can’t know the darkness without knowing the light. Consider the extreme case where your training set is 100% one class, e.g. all up days/weeks/months. Now a trivial strategy of buying all the time is our best bet. Any machine learning algo would be hard pressed to beat that level of accuracy.

How do we combat this problem? Admittedly, I am grappling with the problem myself on a daily basis. Like many posts on this blog, this one was initially motivated by my own search for a satisfactory solution. It would seem a popular solution is up-sampling rare classes and down sampling ubiquitous classes until a 50/50 balance is achieved, but that seems like it requires some acrobatics when you try to transfer your model back to the real world, as you’ve significantly altered the training set from what your autonomous trading minion will encounter in real life. Restricting yourself to learning one-class is another method you see in literature and I’ve resorted to in practice. One-class classification can also be useful in quantifying normal vs anomalous behavior.

Another interesting method from this source is to apply a higher confidence threshold to the majority class. So if your training set is a majority of up days, only predict further up movement when your classifier is 75% sure its an up day versus a 50% threshold. The authors also bring up a good point that this can be especially useful when the costs of mis-classification are different for each class. In trading I would venture that this is probably the case: misclassifying an up move could be worse than a down move. This might be due to the stylized fact that down moves in the stock market are quicker and more severe than price increases. Misclassifying a single down day means you miss out on a small increase in price. Misclassifying a single up day could wipe out weeks/months of profit. Thus, applying a higher “hurdle” to your up predictions could make sense in a bull market, especially at the latter stages.

Here are some other papers I’m studying at the moment. Realizing now that we have it easy in finance compared to some unbalanced datasets…

A Survey of Recent Trends in One Class Classification [uwaterloo.ca]

Feature selection for unbalanced class distribution and naive bayes [ResearchGate]

Editorial: Special Issue on Learning from Imbalanced Data Sets [nd.edu]

kNN Approach to Unbalanced Data Distributions: A Case Study involving Information Extraction [AWS]

Man vs. Computer: Benchmarking Machine Learning Algorithms for Traffic Sign Recognition [ini.ruhr-uni-bochum.de]

Categories: Uncategorized

4 replies »

    • its an interesting idea! it certainly will bring your data classes back into balance, but now your estimating another parameter (the mean) which will need a sensible online updating mechanism when you go live. gets me thinking though, if you just fixed some minimum threshold return (instead of just using the mean) you could build a series of classifiers, then look at as an ensemble, altho i’m totally just spitballing here… certainly something worth investigating!

      Like

  1. There is also one interesting alternative to over/undersampling which usually also skews results.

    Train your classifiers based on relative rank – compare instruments with index for example. Instead of trying to predict direction you try to predict rank. If you want raw direction you can hedge with index and you will get nice almost zero beta algo

    normalizing data with mean is one option but I wouldn’t use it as to get useful results you then need also to predict mean..

    Liked by 1 person

  2. I would highly recommend taking a look at quantile regression. If you view your problem as classification you can usually perform quantile regression for different quantiles of interest and then bin the data appropriately for evaluation with classification metrics. You have much more control over your inputs than class balance, sample weighting, oversampling, etc. Keep in mind that none of these algorithms are Einstein. You need to spend as much time looking at residuals and confidence metrics as you do the output results. Once you know how to make your model fail and why, you are getting somewhere.

    http://data.library.virginia.edu/getting-started-with-quantile-regression/

    Liked by 1 person

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s