Modules
05/10
Probability

Contents

Covariance Matrices

The shape of data in high dimensions.

Multidimensional Variance

In 1D, variance tells you how spread out data is along a single axis. But real data lives in many dimensions. Light bulb quality might be characterized by filament thickness, gas pressure, and glass clarity simultaneously.

In 2D (or n-D), data can be spread out in different directions. The Covariance Matrix captures this shape. It tells us not just how much each variable varies, but how they vary together.

The Covariance Matrix (Σ\Sigma)

For nn variables, it's an n×nn \times n symmetric matrix.

Σ=[Var(X1)Cov(X1,X2)⋯Cov(X1,Xn)Cov(X2,X1)Var(X2)⋯Cov(X2,Xn)⋮⋮⋱⋮Cov(Xn,X1)Cov(Xn,X2)⋯Var(Xn)]\Sigma = \begin{bmatrix} Var(X_1) & Cov(X_1,X_2) & \cdots & Cov(X_1,X_n) \\ Cov(X_2,X_1) & Var(X_2) & \cdots & Cov(X_2,X_n) \\ \vdots & \vdots & \ddots & \vdots \\ Cov(X_n,X_1) & Cov(X_n,X_2) & \cdots & Var(X_n) \end{bmatrix}

Diagonal Elements

Variances of each variable. How spread out each variable is on its own.

Off-Diagonal Elements

Covariances between pairs of variables. How they move together.

Interactive Simulator

Manipulate the covariance matrix for 3 variables (X, Y, Z). See how changing the covariances changes the 2D projections of the data cloud.

Covariance Playground

Covariance Matrix Σ

X
Y
Z
X
1.0
0.80
0.50
Y
0.80
1.0
0.20
Z
0.50
0.20
1.0
How to use:Adjust sliders in the upper triangle. The matrix is symmetric, so changes reflect in the lower triangle automatically. Trying to set strong pairwise correlations (e.g. X-Y=0.9, Y-Z=0.9, X-Z=-0.9) will break geometry and show an error.
X vs Yr=0.80
X vs Zr=0.50
Y vs Zr=0.20

Key Properties

1. Symmetry

Cov(X,Y)=Cov(Y,X)Cov(X,Y) = Cov(Y,X). The matrix is symmetric, so Σ=ΣT\Sigma = \Sigma^T.

2. Positive Semi-Definite (PSD)

xTΣx≥0x^T \Sigma x \geq 0 for any vector xx. Why? Because variance cannot be negative. The quadratic form represents a variance.

3. Linear Transformation

If Y=AXY = AX, then ΣY=AΣXAT\Sigma_Y = A \Sigma_X A^T. This is how covariance propagates through linear operations.

4. Correlation Matrix

Normalize by dividing each entry by σiσj\sigma_i \sigma_j. Correlation matrix has 1s on diagonal, values between −1-1 and 11 off-diagonal.

Geometric Intuition

Think of the Covariance Matrix as defining an ellipsoid (a stretched sphere) in n-dimensional space. The equation (x−μ)TΣ−1(x−μ)=c(x - \mu)^T \Sigma^{-1} (x - \mu) = c defines an elliptical contour of constant probability.

Spherical (Identity Covariance)

Variables are uncorrelated with equal variance. Data cloud is a perfect sphere.

Ellipsoidal (General Covariance)

Variables have different variances and correlations. Data cloud is a tilted ellipsoid.

Eigen-Decomposition (Spectral Theorem)

This is the bridge between Linear Algebra and Statistics. Since Σ\Sigma is symmetric PSD, it has a special decomposition:

Σ=VΛVT\Sigma = V \Lambda V^T
Eigenvectors (VV):

The columns of VV point in the directions of the principal axes of the ellipsoid. These are the "natural" coordinate axes of the data, where variables are uncorrelated.

Eigenvalues (Λ\Lambda):

The diagonal matrix where λi\lambda_i is the variance along the i-th principal axis. Larger eigenvalue = more spread in that direction.

Why This Matters

By rotating data into the eigenvector basis, you decorrelate the variables. The covariance matrix in the new basis is diagonal. This is exactly what PCA does.

Case Study: Bulb Quality Control

The Scenario

You measure 3 properties of each bulb: Filament Resistance (Ω), Luminosity (Lumens), and Heat Output (°C). You have data from 1,000 bulbs. How do you detect anomalies?

The Covariance Matrix

Σ = [[2.5, 1.8, 0.9], [1.8, 3.2, 1.4], [0.9, 1.4, 1.8]]

The off-diagonal values (1.8, 0.9, 1.4) are positive, meaning all three properties are positively correlated. A brighter bulb is also hotter.

Mahalanobis Distance (Anomaly Detection)

DM=(x−μ)TΣ−1(x−μ)D_M = \sqrt{(x - \mu)^T \Sigma^{-1} (x - \mu)}

Unlike Euclidean distance (which ignores correlations), Mahalanobis distance accounts for the covariance structure. A bulb that's hot but not bright is an outlier, even if each value alone seems normal.

PCA (Principal Component Analysis)

PCA is literally just finding the eigenvectors of the Covariance Matrix and projecting data onto the top ones.

The Algorithm

  1. Center the data (subtract mean from each column).
  2. Compute Covariance Matrix Σ=1n−1XTX\Sigma = \frac{1}{n-1} X^T X.
  3. Find Eigenvectors and Eigenvalues of Σ\Sigma.
  4. Sort by Eigenvalues (Largest = Most Variance = PC1).
  5. Project data: Z=XVZ = XV, where VV is the matrix of top kk eigenvectors.

Variance Explained

The proportion of variance explained by PC ii is λi/∑λj\lambda_i / \sum \lambda_j. If the first 2 PCs explain 95% of variance, you can visualize 100D data in 2D with minimal information loss.

Whitening / Sphering

After PCA, scale each PC by 1/λi1/\sqrt{\lambda_i}. This transforms the ellipsoid back into a sphere (uncorrelated, unit variance). Common preprocessing for neural networks.