“The normal distribution is the most common distribution in nature.”

I heard this countless times in school…

The bell curve itself makes intuitive sense—there are few very short people, few very tall people and lots in the middle, right? But something deeper bothered me. My teachers were making an extraordinary claim: among infinite possible bell-shaped curves, nature consistently chooses one specific mathematical form.

How could this be?

Subscribe now

The answer lies in a remarkable property. When you add up many small components, their sum tends toward a normal distribution. If human height is the sum of many body segments, then height will be approximately normally distributed.

But there’s a catch—it won’t always work. Several conditions must be met:

• Components are independent

• Each component must have finite variance (the distribution tail decreases sufficiently fast)

• No single component can dominate the others in terms of variance (Lindeberg’s condition)

• You need enough components

Are these conditions realistic? Do they actually occur in nature?

The Human Height Example

Adult height serves as an empirical example of a normally distributed biological trait in textbooks on statistics¹.

Let’s examine human height more closely. At a basic level, we might divide the body into major segments, say: foot, ankle, calf, knee, thigh, hips, belly, chest, neck, and head height. For these to sum to a normal distribution:

• they cannot influence each other

• their individual variances must be fairly similar

• and with so few components, each would need to be nearly normal itself

The number of components is the key.

Instead of a few major segments, imagine dividing the ankle into all the bones and other tissues that comprise it. Go further—consider parts of these bones, or even smaller, unspecified fragments beyond human nomenclature. Suddenly, we have a large number of micro-components. Now the requirements become much less strict:

• still independent

• but the component heights can have varied distributions, far from Normal

• and more different variances

Yet, their sum will still converge to a normal distribution.

Central Limit Theorem

This remarkable property of the normal distribution has a name: the Central Limit Theorem (CLT).

The requirements described here apply to the Lindeberg-Feller version, but other formulations exist with different conditions. You can often trade one requirement for another—or several others. The more you remove, the fancier the new ones become.

Either way, one of the versions is apparently often satisfied in nature, and that’s why we can observe the Normal distribution so frequently.

P.S.

This post was inspired by reading The Art of Statistics: Learning from Data by David Spiegelhalter. As I mentioned in my previous post, I’m going back to basics. I’m on page 100 of 400 and I’m fascinated. My notebook is full of ideas for future posts. We’ll see how much my upcoming newborn will let me actually write them ;)

¹ Slavskii, S.A., et al. (2021). “The limits of normal approximation for adult height.” European Journal of Human Genetics, 29, 1082–1091. https://www.nature.com/articles/s41431-021-00836-7

During my studies, I took courses that included statistical tests. So I started to refresh my knowledge. I realized I could perform the test, calculate the p-value, and reject the null hypothesis, but I felt overwhelmed by the feeling that I had no idea what I was doing. The number of “why” questions was devastating.

• Why this test and not another?

• Why am I modeling data with probability theory at all when reality is more complex? At what point am I “paying” for simplifying real data into probability theory?

• Why is the p-value usually taken as 0.05? Why sometimes 0.005, etc.?

• What does the p-value tell me at all? And what does it not tell me? Why are interpretations on the internet contradictory?

• Why does the p-value take into account events more extreme than those observed, if I am not interested in them?

  and more popping up along the way.

How was I supposed to make any business decision with such a limited understanding of the decision-making tool?

The number of these questions made me suspect that these statistical tests are worthless. When I read various materials available on the internet on this subject, I had the impression that everyone was parroting what I had learned in college, rather than asking WHY questions.

I had an ambitious dream of finding answers to all my questions and describing them in a comprehensive blog article.

I was tempted by the bold thesis that tests are worthless. I started reading and searching. I learned more about different types of tests, assumptions, CLT, and the history of statistical tests. Unfortunately, I was constantly struck by the impression that the sources I was reading were often contradictory. Some repeated interpretative errors that others criticized. I still couldn't find the depth and certainty I was looking for. Something didn't add up; it didn't fit together coherently, eluding me. Nevertheless, I gathered a lot of fragments related to the components of statistical tests, from which I tried to assemble a coherent whole that was easy to understand, despite its extensive nature.

The main thesis was that at no point do we “pay” for simplifying real data to probability theory.

When I use probability in machine learning, I can check how well such modeling works on a test set. In the case of statistical tests, I have to accept the result uncritically.

Additionally, I identified several issues that further undermine credibility. The potential for arbitrary selection of the level of statistical significance. An overwhelming amount of overinterpretation of p-values and vast confusion on the internet regarding this issue. I found “The ASA’s Statement on p-Values” by the American Statistical Association, from which I concluded that it is practically impossible to move from the formal definition of p-values to any useful conclusions about the phenomenon under study.

Privately, I was waiting for my second child's birthday while caring for my first baby. I read about statistical tests on the sidewalk. I had my phone in one hand and pushed the stroller with the other. Time was at a premium, so progress was slow. After the birth, I went into survival mode, when only work and family mattered. The space for reflection disappeared. The article dragged on for several months.

The children became a little more independent, and I got back into the game. In search of answers to my questions, I went back to basics. While reading about CLT, I turned to more serious sources and realized I didn't even know what a random variable was. During my studies and when using ML after graduation, it was enough for me to equate a random variable with a probability distribution. It turns out that this is not the case… My confidence in my own knowledge collapsed for a moment. ChatGPT cheered me up, saying that it's not that I don't know anything, but that I'm just going deeper.

In the meantime, I asked numerous questions on Cross Validated. Surrounded by educated, serious people, I felt very incompetent. But I believed that reading on my own was not enough. I have to bounce questions and thoughts off people who have vastly more experience.

I found it challenging to understand some of the answers because they employed concepts I had never encountered before. When I read that to understand the definition of a random variable, I needed to know the concepts of sample space, measurable space, and σ-algebra, I was taken aback. I don't remember ever seeing these concepts on any slides during my master's studies. They focused more on covering descriptive statistics for EDA, then quickly jumped into classifiers, regression, and neural networks. The concept of a random variable emerged in the context of Bayesian Neural Networks (BNNs) or Variational Autoencoders (VARs), but treating it as a probability distribution was sufficient for my purposes.

Concerned, I asked ChatGPT what I need to read to fill this gap. I selected three books whose tables of contents encouraged me.

1. The Art of Statistics (Spiegelhalter)

2. Mathematical Statistics and Data Analysis (Rice)

3. Statistical Inference (Casella & Berger)

I will certainly not get through all of them (especially since my third child is on the way and another period of “time out” is coming), but I want to fill in the fundamental gaps.

And well, we'll see what happens next :)

Wikipedia is not the best resource for Data Science and Machine Learning beginners. In contrast to academia, online courses, and books, it doesn't order the topics in a way that one builds upon another. Instead, we get unordered articles with a very high entry level.

However, at the current stage of my Data Science journey, almost 4 years after graduation, I found many Wikipedia articles in-depth and rich. At the same time, I usually struggle to break its strict way of explaining things, being frequently ashamed and dissatisfied.

Today, I will tell you my short story with the Likelihood function article on Wikipedia, from confusion to the joy of understanding and new insights.

When reading through a Wikipedia article about the Likelihood function, I struggled to understand the given definition and example.

Something was missing there for me. I remember my Data Science classes, where we covered the Likelihood function when training neural networks for classification with negative log-likelihood as a loss function or inside a Naive Bayes model. Always in the context of some datasets like ancient diabetes or iris. I used to deal with input variables (like sepal width or length) to classify data. However, I was unable to match that experience with the formulas I saw in the Wikipedia article.

So I immediately went through the coin flip example (they say an example is worth a thousand words…), but it still seemed very unfamiliar to me.

After reading it several times and taking a moment to reflect, the fog finally cleared, and I understood what was going on.

In both the example and the provided definition, there are no input variables that I used to work with. And there’s absolutely nothing wrong with that! It’s a perfectly fine situation, but it was very awkward for me to get at first glance.

The thing is that the given model is a probability distribution. Or I should say, a random variable described by a probability distribution. It indeed has no inputs but can produce outputs.

From the Machine Learning perspective, the model in the example could take any form. Linear function, deep neural network, decision tree, probabilistic graphical model, basically anything that produces outputs! Not only a random variable.

The random variable part of the definition wasn’t clear to me because I incorrectly assumed it was somehow connected to the idea of the Likelihood itself, while it was actually not.

To better explain what I mean, I need to go step-by-step through the definition.

Let’s unravel two provided formulas. It starts with the “probability density or mass function” according to the article.

We can read this right arrow sign as “function f maps x values to outputs“ and its main goal is to introduce an assumption that θ is a fixed(constant) parameter while x remains adjustable.

Since the article says we have a random variable X, then x are all possible values of this random variable, and f is its probability density function.
The most trivial probability density function (PDF) that comes to my mind is the Normal Distribution. It takes exactly two parameters: mean (μ) and standard deviation (σ). Thus, the parameters set θ simply consists of μ and σ.

The chart shows an example of the f function - the Normal Distribution probability density function.

We use this function to say what the probability of sampling a value from a particular range from the distribution is.

In the second formula, we get to the point and see the definition of the Likelihood function:

which reads “function f maps parameters set θ to outputs” and introduces an absolutely crucial assumption that from now on, x is a fixed (constant) parameter and θ is adjustable.

At this stage, the article says, “x is a realization of the random variable X“, which basically means x is a value sampled from our distribution. Let’s say we really sampled a value using an online tool or a Python script, and we got 1.52. We treat x as a fixed, ground-truth value here and can’t change it. θ is what we can freely change in this formula.

Let’s select some specific θ, then. For our Normal Distribution, I pick mean μ = 1 and standard deviation σ = 0.5, whatever. For these parameter values, the probability density of sampling 1.52 is equal to 0.46. (Please notice I use the term probability density because the probability itself is an area under the curve for continuous distributions.)

And that’s it. This is how the likelihood works. Given an observed value, we select a parameters set, and the likelihood says what is the probability of sampling this value.

The most useful thing I’m aware of we can use this for is to find what distribution parameters θ are most likely to produce the sampled value. By trying many different mean and standard deviation values, we will find the ones that are most likely to generate our sampled value 1.52. This task is called maximum likelihood estimation (MLE).

Noooooow, the best thing is, our function f from the definition can be a Probability Density Function or Probability Mass Function (for discrete distributions) describing not only a random variable. From the machine learning perspective, it can describe any model we can think of. So let’s make things more sexy now and pick a Large Language Model (LLM) as an example.

X now becomes our LLM. A small x from the formula becomes a data sample the LLM works with. And since LLMs are trained to predict the next token in a sequence, our x becomes a tuple (“Data Science is”, ”fun“), where the first sequence is input to the model, and the second word is an expected output. The full sequence “Data Science is fun“ is our ground-truth observation scraped from the internet or maybe found in a book.

LLMs are neural networks with parameters set θ consisting of weights and biases. When feeding the model with our input “Data Science is” we will get probabilities of all the possible completion tokens in the model’s dictionary. Let’s say we work with some state-of-the-art model found on the web, and it assigns a probability of 0.8 to the “fun“ token. Good.

We can change the weights and the biases in some way, pass the input sequence “Data Science is” through the model again, and see if the probability of sampling “fun“ has increased. At the highest level of abstraction, this is how language models learn. Yeah.

From now on, I’m able to split the model being a random variable from the concept of the likelihood itself to understand both the Wikipedia’s definition and the example.

And I could've stopped here here, but my mind ran further. I swiftly categorized the potential models into two groups. Deterministic and non-deterministic ones. Deterministic models usually have some kind of input variables because when they don’t, they become constant functions, which is not very useful…

Non-deterministic models introduce some randomness. They can take no inputs, like the coin flip in Wikipedia’s example. However, we can mix them with anything from basic math operations on input variables to large operation graph modules. For example, Variational Auto Encoders consist of a deterministic encoder, decoder, and some normal distribution sampling in the middle.

That’s it. I will have more respect for the coin flips now and be aware that even the ordinary coin is a serious machine-learning model…

We’ll see if this new point of view will help me to get through more articles quickly. I hope it will for you. Cheers!

Did you know that this most popular and confusing softmax formula can be broken down into two simple operations?

Instead of this

I prefer this

In the first step (1), we convert all input values into positive ones using the exponential function property.

In the second step (2), we normalize our positive values by their sum to fit in the range [0, 1] and make them add to 1.

Clearer now? It was much more clear for me this way!

Many of the most skilled people publish both cutting-edge and basics every day, and you have free access to their knowledge and results.

Wouldn’t it boost your learning to work with the best Google, DeepMind, PyTorch, or OpenAI researchers, engineers, PhDs, CTOs, and CEOs? Get closer this way.

You don’t even need to look for them — just follow a few of your favorite libraries and Twitter recommendations will lead you further.

LinkedIn will do the job too.

Below is the first page of the 62 accounts I follow:

Social media addiction is your friend in this case.

2. Subscribe to your favorite libraries’ release notes

This kind of message in my mailbox is a golden nugget. I can’t find words to express how cool it is.

The amount of hard work compressed in this message is enormous every time. It’s like your own army of researchers, developers, and a copywriter who research their ideas, publish papers, implement them, and document precisely.

All you need to do is open a finely crafted tl;dr message with a summary of SOTA methods selection and ready-to-use implementations.

Here’s how to subscribe to GitHub release notes:

And here’s a link to the transformers releases: https://github.com/huggingface/transformers/releases

3. Medium.com subscription

This one is the best at the beginning of your adventure. It helps you gain missing basic knowledge very quickly.

Medium’s algorithm selects the best articles and messages for you every day. After a few months of reading one article per day, you will be a different person. Make it a routine.

This is what it looks like:

That’s it. Share your ways in the comments so we can all become better at what we are passionate about.

Subscribe now

Why this, why now

Hi, my name is Maciek. I dedicate this space mostly to my Data Science hobby/job. It started in 2019 when I decided to enter a Data Science master's degree at the Wroclaw University of Science and Technology in Poland. I was a little bit afraid to drop the web app development and switch to a more scientific area, but it was worth it.

I successfully graduated, having a lot of fun from crazy projects, new friends, and genuine satisfaction from completing the most demanding specialization I could choose.

My first serious Data Science position was at Surfer. A small startup that grew rapidly then. I faced real business problems and had a chance to challenge my graduate skills. Combined with great company culture, people, and expertise, we keep successfully shipping broadly used ML solutions to this day.

In the meantime, I keep becoming more aware of stuff I don’t know, or I covered carelessly during my studies. I want to fill all the gaps and go beyond. Writing short articles seems to be a very helpful tool.

I have some dreams but limited time in life. As a happy husband and father living outside any greater city, I can’t afford a Ph.D. degree for now. However, I can sit tight in my room in the evening, read some text about a topic I don’t profoundly understand yet, and share some precious insights with the public.

What kind of community am I looking to build here

I believe there are Data Scientists who will find something for themselves in the dump of my experiences and aha moments. I include a pinch of emotional load in all my texts to hopefully make someone feel “Oh, I was thinking the same. I understand this guy. I had the same doubts about it. I wasn’t alone then!“

At the same time, I’d love to welcome constructive criticism from experts. It’s necessary for me (us?) to make sure I keep moving toward the truth.

Maybe you are one of those people?