The Gamble of Large Numbers

Shakuntala Devi, Indian mathematician and calculating prodigy

The way of doing business has continuously evolved over the last few decades, and one of the most prominent factors affecting it is data. Data is being generated in enormous quantities, while at the same time it’s becoming much cheaper to store and process. Many organizations are unearthing their past data to identify patterns and generate insights that help them to estimate their future positions.

While the organizations are discovering their hidden potential through the language of data, can we be certain that the data is speaking the absolute truth? What if the data is telling only the pseudo-truth? What if our estimates are based on some false equivalences which might never hold true again? How can we be sure of the patterns/insights generated by our data? Well, this is where good old mathematics can come to our rescue!

The Law of Large Numbers

The Law of Large Numbers is a key concept whenever you want to make predictions based on patterns you observe in data. This mathematical “law” basically means that as we perform some experiment “n” number of times, the probability of the expected outcome gets closer to its theoretical probability as n approaches infinity.

Let’s take an example of tossing a coin (the mathematician’s favorite). Tossing an unbiased coin can have two possible outcomes: heads or tails, each having a probability of 0.5. Here, the law states that if we toss a coin a large number of times – say 100 or 1,000 – the probability of getting heads will be close to 0.5, which is its theoretically expected probability.

Easy, right? Almost… with a catch!

The Law of Large Numbers does not guarantee that a given sample will always reflect the true characteristics of a population. If a given sample deviates from the expected mean, you can’t expect that subsequent observations will bring the observed average nearer to the expected mean.

Let’s continue with tossing a coin.

If you toss the coin three times and get heads all three times, the probability of getting heads in those tosses now becomes 1. However, we expect that in the next three tosses, the coin must come up tails three times in a row to reflect the true probability of 0.5. But can we guarantee that tails will show up three consecutive times? No!

The Gambler’s Fallacy

The Gambler’s Fallacy is a phenomenon in which an individual erroneously believes that a certain random and independent event is less or more likely to occur based on the outcome of a previous event. This is not true. The outcome of previous events does not affect the probability of future random and independent events.

Continuing with tossing the coin, if we get three heads in a row after tossing the coin three times, the individual might predict that next time the coin will definitely land on tails. However, the actual probability of getting “tails” still remains at 0.5. This is a how an individual commits a Gambler’s Fallacy.

A classic example of the Gambler’s Fallacy occurred at the Monte Carlo Casino in 1913. The roulette wheel’s ball fell on black several times in a row. This led people to believe that the ball would soon fall on red, and they started placing their bets with this assumption. However, the ball only fell on red after 27 spins of the wheel. Millions of francs were lost by then!

Impact on Business

We humans have a tendency to find patterns in all events. We are uncomfortable with ambiguity and randomness. This is what makes us find patterns that simply do not exist in the actual world. With businesses today still being run by humans (pre-Terminator, at least), decisions that don’t take into account the Law of Large Numbers and the Gambler’s Fallacy can prove disastrous. These mathematical concepts are applicable to every domain of business. Let’s look at some:

Insurance

Insurance companies need some mechanism to estimate the value and frequency of all their future claims. With a good foresight of the future claims, the companies can fix their premium rates at levels that let them stay profitable while making advantageous offers to customers Here is where the Law of Large Numbers can help. The law states that the actual total of compensation claims will eventually equal the expected total in the long run as the number of policyholders keeps increasing. Thus, insurance companies have basic confidence that their calculations are reliable models for future outcomes.

However, when a pandemic like COVID looms, all these models crumble because the probability itself changes significantly; the number and amount of compensation claims can mount much higher than the expected average. This can lead to large potential losses.

Startups

The technology industry is famous for startup companies that become incredibly valuable. India alone has 80 unicorns (startups worth more than $1 billion). An early investment in one of these companies would have paid off handsomely. But what about a similar investment now? It’s important to know whether the startup that interests you will continue to grow just as fast in the long run.

This is where the Law of Large Numbers kicks in. It says that eventually, the growth rate of these unicorns will level off from a steep growth curve to a more predictable, slower-rising line, approaching the average growth of other big organizations. Thus, the law can prevent the unicorns from estimating their growth rate based on some false equivalences.

Stock prices

Speaking of investments, how often have you heard a comment like this? “Equity funds are hitting the upper limit from the last three trading sessions. Let’s not invest – it will surely fall now.” But what often gets unnoticed is that many external factors can affect a stock price. Probability does play a role in overall stock price movement, but gravity doesn’t. It isn’t true that if a stock keeps moving up, then it has to come down eventually. Thus, knowing about the Law of Large Numbers and the Gambler’s Fallacy can also save you from bad investments.

Using mathematical fundamentals to guide our work

At Digitate, we have built our flagship product, ignio™, as the “perfect” data scientist. It has been trained to distinguish events with a definite pattern from those that are merely random. Moreover, it avoids letting its “judgment” get biased, using statistical de-biasing techniques that highlight independent events and their inability to affect each other. The patterns and correlations that ignio generates are structured and well-defined – ignio’s “Spidey sense”! Well, not really; these results are not based on tingling intuition but guided by mathematical concepts.

Parting Thoughts

The Law of Large Numbers continues to present mysteries to unravel. Can we use historical macro-trends and patterns to infer micro-behavior? How should the recent past influence the prediction of future micro-events? To continue the coin toss analogy, if for 100 tosses, all we get is heads, should the prediction for the next toss be skewed towards heads since we have had 100 out of 100 so far?

Or should the future be more skewed towards tails, since we need to catch up with the law of averages? Well… ignio knows. Do you?