In my previous article, I explored what covariance and correlation are but something felt incomplete.
I had accepted the formulas …… without truly earning them.
So, In this piece I will do what i should have done first: derive everything from scratch, question each step, and extract insights I haven’t seen discussed elsewhere. Understanding the derivation is what allows you to manipulate data for advanced techniques like whitening or dimensionality reduction.
Consider two variables, X and Y, which represent two different datasets you wish to analyze. To analyse how these variables relate to one another, we must move from their individual characteristics to their joint behavior through a mathematical derivation.
1. Covariance — Derivation from First Principles
Covariance, at its core, is just a way of asking — do these two variables move together?
Before diving into the proofs, let’s take a look at the starting components.
X and Y: Two random variables representing your datasets.
x̄ and ȳ: The arithmetic means (centers) of those datasets.
n: The total number of data points.
Step 1: The Individual Deviation:
To see how variables move together, we first measure how each point deviates from its own average.
Step 2: Calculate the Product of Deviations:
We multiply these two deviations together for every data point. This product (xi - x̄)(yi - ȳ) will be positive if both variables move in the same direction and negative if they move in opposite directions.
Step 3: Aggregate the Movement:
We sum these products across all n data points to find the total shared movement:
Step 4: Finalize the Average:
To find the average joint variability, we divide the sum by the number of observations n.
Image illustrating different types of covariance
The Direction: Covariance Trends
Covariance acts as a directional compass for data. It tells us whether two variables are moving in harmony or in opposition.
Positive Covariance: This upward slope indicates a direct relationship where variables increase or decrease together.
Negative Covariance: This downward slope represents an inverse relationship; as one variable climbs, the other falls.
Zero Covariance: The random cloud indicates that the variables are independent in a linear sense, meaning one does not provide information about the movement of the other.
2. Correlation (r): The Standardized Strength
I now normalise covariance to remove units.
Pearson’s Correlation (r) is a standardized version of covariance that measures both the strength and direction of a linear relationship within a fixed range of -1 to +1.
Step 1: Identify the Scaling Problem:
Covariance is scale-dependent; if you change the units of X or Y, the covariance value changes, making it difficult to interpret the strength of the relationship.
Step 2: Apply Normalization:
To create a unitless measure, we divide the covariance by the product of the individual standard deviations, σX and σY.
Step 3: Expand the Components:
By substituting the full mathematical definitions for covariance and standard deviation, we get a comprehensive ratio.
Step 4: Final Simplification:
The 1/n th terms in the numerator and denominator cancel out, leaving the finalized Pearson Correlation Coefficient.
The Strength: Correlation Gradient:
While covariance identifies the path, the correlation coefficient (r) measures the tightness or reliability of that path.
The Extremes (r = 1 and r = -1): These represent perfect linear certainty. In these scenarios, the data points align perfectly, showing that the relationship is completely predictable without any noise.
The Moderate States (r = 0.5 and r = -0.5): These are more common in complex environments like financial markets. They show a clear trend but include "noise," indicating that while a relationship exists, other factors are also at play.
The Neutral State (r = 0): This confirms the zero covariance finding. There is no linear trend visible, which is a critical signal to stop looking for a direct linear link between those specific variables.
By observing these transitions, we move beyond simple numbers to see the geometric reality of data relationships—a perspective that is vital for building robust predictive models.