Central Limit Theorem: MGFs, Proof, and Random Walks

Complex Systems · 1/8 · Series index · Notation

The Central Limit Theorem: Why Sums of Many Random Contributions Are Approximately Normal

A falling-ball apparatus illustrates a random walk: at each level, a ball moves randomly to the left or right, and its final position is the sum of many random displacements. The central limit theorem explains why suitably standardized sums often approach a normal distribution.

Distinguishing Random Variables, Distributions, and Moments

A random variable $X$ assigns a numerical value to the outcome of a random experiment; $x$ denotes one of its possible values. A continuous random variable is described by a probability density function (PDF), $p_X(x)$ :

\begin{aligned} p_X(x)&\geq 0,\\ \int_{-\infty}^{\infty}p_X(x)\,dx&=1,\\ P(a\leq X\leq b)&=\int_a^b p_X(x)\,dx. \end{aligned}

The height of a density is not the probability of that exact value; a continuous distribution has $P(X=x)=0$ . A discrete random variable instead uses a probability mass function (PMF), $p_X(x)=P(X=x)$ , and integration is replaced by summation over all possible values. For example, $\sum_xp_X(x)=1$ .

Mean, Variance, and Raw Moments

The mean describes location, the variance measures squared fluctuations around the mean, and the standard deviation $\sigma$ has the same units as $X$ :

\begin{aligned} \mu&=E[X]=\int_{-\infty}^{\infty}x p_X(x)\,dx,\\ \sigma^2&=\operatorname{Var}(X)=E[(X-\mu)^2]\\ &=E[X^2]-(E[X])^2. \end{aligned}

The $k$ th raw moment is $m_k=E[X^k]$ . Therefore,

m_0=1,\qquad \mu=m_1,\qquad \sigma^2=m_2-m_1^2.

The second raw moment $m_2$ equals the variance only when $\mu=0$ . Do not confuse raw moments with central moments, $E[(X-\mu)^k]$ .

General and Standard Normal Distributions

A general normal, or Gaussian, distribution is written as $X\sim\mathcal N(\mu,\sigma^2)$ . Its second parameter is the variance:

p_X(x)=\frac{1}{\sigma\sqrt{2\pi}} \exp\!\left[-\frac{(x-\mu)^2}{2\sigma^2}\right],\qquad \sigma>0.

A standard normal variable $Z\sim\mathcal N(0,1)$ has density

\phi(z)=\frac{1}{\sqrt{2\pi}}e^{-z^2/2},\qquad E[Z]=0,\quad \operatorname{Var}(Z)=1.

If $X\sim\mathcal N(\mu,\sigma^2)$ , then $(X-\mu)/\sigma\sim\mathcal N(0,1)$ . Define the standard normal cumulative distribution function by

\Phi(z)=\int_{-\infty}^z\phi(u)\,du.

An interval probability is $P(a\leq Z\leq b)=\Phi(b)-\Phi(a)$ , and symmetry gives $\Phi(-z)=1-\Phi(z)$ .

The Precise CLT Statement and Its Assumptions

Let $X_1,\ldots,X_n$ be independent and identically distributed (i.i.d.), with

E[X_i]=\mu\ \text{finite},\qquad 0<\operatorname{Var}(X_i)=\sigma^2<\infty.

“Identically distributed” means that all variables follow the same probability law; it does not mean that every observed value is identical. Independence means that the joint probability law factorizes: learning outcomes for any subset does not change the joint distribution of the remaining variables. Define the sum and sample mean as

S_n=\sum_{i=1}^nX_i,\qquad \overline X_n=\frac{S_n}{n}.

The central limit theorem (CLT) states that the standardized sum converges in distribution:

\boxed{ \begin{aligned} Z_n&=\frac{S_n-n\mu}{\sigma\sqrt n} =\frac{\sqrt n(\overline X_n-\mu)}{\sigma},\\ Z_n&\xrightarrow{d}\mathcal N(0,1),\qquad n\to\infty. \end{aligned} }

Here, convergence in distribution means that $P(Z_n\leq z)\to\Phi(z)$ for every fixed real number $z$ . The original variables $X_i$ may be discrete or non-normal. The theorem does not assert exact normality at finite $n$ , nor does it guarantee pointwise convergence of arbitrary probability densities.

Why Standardization Uses the Square Root of $n$

The scaling of means and variances explains the appearance of $\sqrt n$ :

\begin{aligned} E[S_n]&=n\mu,\\ \operatorname{Var}(S_n)&=n\sigma^2,\\ E[\overline X_n]&=\mu,\\ \operatorname{Var}(\overline X_n)&=\frac{\sigma^2}{n}. \end{aligned}

Independence makes the cross-covariances zero. The sum has standard deviation $\sigma\sqrt n$ , whereas the mean has standard deviation $\sigma/\sqrt n$ . For large $n$ , we therefore often write the approximations

\begin{aligned} S_n&\approx\mathcal N(n\mu,n\sigma^2),\\ \overline X_n&\approx\mathcal N\!\left(\mu,\frac{\sigma^2}{n}\right). \end{aligned}

When $\mu=0,\sigma^2=1$ , $Z_n=\sqrt n\,\overline X_n=S_n/\sqrt n$ ; the square root covers only $n$ . The fluctuations of the unstandardized mean shrink to zero as it approaches the constant $\mu$ ; we must not write $\overline X_n\to\mathcal N(0,1)$ . A density approximation must also include the correct normalization factor:

p_{\overline X_n}(x)\approx\frac{\sqrt n}{\sigma\sqrt{2\pi}} \exp\!\left[-\frac{n(x-\mu)^2}{2\sigma^2}\right].

Why the CLT Is Useful, and Why Its Assumptions Matter

Wealth, stock prices, earthquakes, and supermarket sales are influenced by many factors, which can motivate a normal model as an initial hypothesis. Before applying this reasoning, check whether the contributions add, whether they are independent, and whether the relevant distributional and finite-variance assumptions hold. The examples themselves do not prove that these quantities are normally distributed. Correlated interactions, multiplicative growth, or very heavy tails can invalidate this simple argument.

The version used here assumes i.i.d. variables with finite variance. Having equal means and variances alone does not imply identical distributions. More general CLTs can relax assumptions, but require additional theorems. There is also no universal threshold such as “ $n\geq30$ ” that guarantees a reliable approximation for every distribution; skewness and heavy tails affect the rate of convergence.

Moment-Generating Functions: Encoding Moments in One Function

The moment-generating function (MGF) is

M_X(t)=E[e^{tX}] =\int_{-\infty}^{\infty}e^{tx}p_X(x)\,dx.

For a discrete variable, use $M_X(t)=\sum_x e^{tx}p_X(x)$ instead.

Generating Moments by Differentiating at Zero

Expand $e^{tX}$ around $t=0$ . When summation and expectation can be interchanged,

\begin{aligned} e^{tX}&=1+tX+\frac{t^2X^2}{2!}+\frac{t^3X^3}{3!}+\cdots,\\ M_X(t)&=1+m_1t+\frac{m_2t^2}{2!}+\frac{m_3t^3}{3!}+\cdots. \end{aligned}

Consequently,

\boxed{M_X^{(k)}(0)=m_k=E[X^k].}

The notation $M_X^{(k)}$ denotes the $k$ th derivative, not the $k$ th power of $M_X$ . Differentiate with respect to $t$ first, then set $t=0$ . Substituting $t=0$ directly into the MGF gives only $M_X(0)=1$ .

A sufficient condition for these standard MGF operations is that $M_X(t)$ be finite on an open neighborhood of $0$ . Not every distribution has such an MGF; even the existence of all raw moments does not automatically guarantee it. The usual finite-variance CLT does not require an MGF to exist; the MGF proof given here imposes an additional existence assumption.

Translation, Scaling, and Independent Sums

The proof below uses two rules:

\begin{aligned} M_{aX+b}(t)&=e^{bt}M_X(at),\\ M_{X+Y}(t)&=M_X(t)M_Y(t)\qquad (X,Y\ \text{independent}). \end{aligned}

The first follows from $e^{t(aX+b)}=e^{bt}e^{atX}$ . The second uses independence to obtain $E[e^{tX}e^{tY}]=E[e^{tX}]E[e^{tY}]$ . Being merely uncorrelated is generally insufficient for the second rule.

The Standard Normal MGF: Complete the Square

Let $Z\sim\mathcal N(0,1)$ . Then

\begin{aligned} M_Z(t) &=\int_{-\infty}^{\infty}\frac{1}{\sqrt{2\pi}}e^{-z^2/2+tz}\,dz\\ &=e^{t^2/2}\int_{-\infty}^{\infty}\frac{1}{\sqrt{2\pi}}e^{-(z-t)^2/2}\,dz\\ &=\boxed{e^{t^2/2}}. \end{aligned}

The last integral is the integral of a normal density with mean $t$ and variance $1$ , so it equals $1$ . Writing $X=\mu+\sigma Z$ then gives the MGF of a general normal distribution:

M_X(t)=\exp\!\left(\mu t+\frac{\sigma^2t^2}{2}\right).

The first four derivatives of the standard normal MGF are

\begin{aligned} M'_Z(t)&=t e^{t^2/2},\\ M''_Z(t)&=(1+t^2)e^{t^2/2},\\ M'''_Z(t)&=(3t+t^3)e^{t^2/2},\\ M^{(4)}_Z(t)&=(3+6t^2+t^4)e^{t^2/2}. \end{aligned}

Thus $m_0=1,m_1=0,m_2=1,m_3=0,m_4=3$ .

Distinguishing Raw Moments, Skewness, and Kurtosis

For a general distribution, skewness and kurtosis are defined as

\begin{aligned} \gamma_1&=\frac{E[(X-\mu)^3]}{\sigma^3},\\ \beta_2&=\frac{E[(X-\mu)^4]}{\sigma^4},\\ \gamma_2&=\beta_2-3\qquad\text{(excess kurtosis)}. \end{aligned}

The identities $m_3=\gamma_1$ and $m_4=\beta_2$ hold only for standardized variables. A normal distribution has skewness $0$ , kurtosis $3$ , and excess kurtosis $0$ ; keep these three quantities distinct.

The MGF Proof of the CLT: Where Each Assumption Is Used

Consider the simplified case $E[X_i]=0$ , $E[X_i^2]=1$ , and assumes that the MGF of an individual $X_i$ is finite near $0$ . The goal is to show that the MGF of $Z_n=S_n/\sqrt n$ converges to $e^{t^2/2}$ .

Step 1: Turn the Exponential of a Sum into a Product

\begin{aligned} M_{Z_n}(t) &=E\!\left[\exp\!\left(\frac{t}{\sqrt n}\sum_{i=1}^nX_i\right)\right]\\ &=E\!\left[\prod_{i=1}^n e^{tX_i/\sqrt n}\right]. \end{aligned}

Step 2: Factor the Expectation by Independence, Then Use Identical Distributions

\begin{aligned} M_{Z_n}(t)&=\prod_{i=1}^n E[e^{tX_i/\sqrt n}]\\ &=\left[M_X\!\left(\frac{t}{\sqrt n}\right)\right]^n. \end{aligned}

This is precisely where independence and identical distributions enter the proof.

Step 3: Use a Second-Order Taylor Expansion at Zero

For fixed $t$ , let $u=t/\sqrt n\to0$ . Since $M_X(0)=1$ , $M'_X(0)=0$ , and $M''_X(0)=1$ ,

\begin{aligned} M_X(u)&=M_X(0)+M'_X(0)u+\frac{M''_X(0)}{2}u^2+o(u^2)\\ &=1+\frac{u^2}{2}+o(u^2). \end{aligned}

Therefore,

M_{Z_n}(t)=\left[1+\frac{t^2}{2n}+o\!\left(\frac1n\right)\right]^n.

The derivatives must be evaluated at the expansion point $0$ . At finite $n$ , we cannot discard the remainder and retain an exact equality.

Step 4: Take the Limit

Using $\log(1+v)=v+o(v)$ ,

\begin{aligned} \log M_{Z_n}(t) &=n\log\!\left[1+\frac{t^2}{2n}+o\!\left(\frac1n\right)\right]\\ &\longrightarrow\frac{t^2}{2}. \end{aligned}

Hence $M_{Z_n}(t)\to e^{t^2/2}$ . The continuity theorem for MGFs gives $Z_n\xrightarrow{d}\mathcal N(0,1)$ .

Step 5: Restore a General Mean and Variance

Repeat the argument for $Y_i=(X_i-\mu)/\sigma$ . Since $E[Y_i]=0$ and $\operatorname{Var}(Y_i)=1$ ,

\frac1{\sqrt n}\sum_{i=1}^nY_i =\frac{S_n-n\mu}{\sigma\sqrt n}\xrightarrow{d}\mathcal N(0,1).

The complete chain is: standardize → turn the exponential of a sum into a product → factor by independence → use identical distributions → expand to second order at zero → apply $(1+a/n)^n\to e^a$ . The finite mean determines the centering term, and the finite variance determines the scale.

Why the Finite-Variance CLT Is More General Than This Proof

We can instead use the characteristic function $\varphi_X(t)=E[e^{itX}]$ , which always exists. For a variable with mean $0$ and variance $1$ ,

\varphi_X(u)=1-\frac{u^2}{2}+o(u^2),

and the same argument gives

\left[\varphi_X\!\left(\frac{t}{\sqrt n}\right)\right]^n\to e^{-t^2/2}.

This explains how to address the gap caused by possible nonexistence of an MGF; the main derivation above still follows the MGF route.

Random Walks: From One Step to the Endpoint Distribution

Consider a walk in which each step is $+1$ to the right or $-1$ to the left, with probability $1/2$ each. Steps are independent, and the position after $n$ steps is $S_n=X_1+\cdots+X_n$ . Repeated left or right deflections in the falling-ball apparatus provide an intuitive analogy.

The Distribution and Moments of One Step

The step distribution is discrete, so it is described by a PMF:

P(X=1)=P(X=-1)=\frac12,\qquad M_X(t)=\frac{e^t+e^{-t}}{2}=\cosh t.

Since $\cosh' t=\sinh t$ and $\sinh' t=\cosh t$ ,

m_1=0,\quad m_2=1,\quad m_3=0,\quad m_4=1.

We can also see this directly: an odd power of $X$ equals $X$ , while an even power is always $1$ . A single step therefore has mean $0$ and variance $1$ . Its distribution is not normal: its fourth moment is $1$ , whereas the fourth moment of a standard normal variable is $3$ .

The Exact Distribution After $n$ Steps

Let $K$ be the number of steps to the right. Then $K\sim\operatorname{Binomial}(n,1/2)$ and $S_n=K-(n-K)=2K-n$ , so

P(S_n=s)=2^{-n}\binom{n}{(n+s)/2},\qquad s=-n,-n+2,\ldots,n.

All other values of $s$ have probability $0$ . The endpoint has the same parity as $n$ . The MGF of $S_n$ is $(\cosh t)^n$ , and

\begin{aligned} E[S_n]&=0,\\ E[S_n^2]&=n,\\ E[S_n^3]&=0,\\ E[S_n^4]&=3n^2-2n. \end{aligned}

The fourth moment can also be derived by counting terms. In the expansion of $(\sum_iX_i)^4$ , the only terms with nonzero expectation are those containing one index four times, or two distinct indices twice each. Hence

\begin{aligned} E[S_n^4]&=nE[X^4]+6\binom{n}{2}(E[X^2])^2\\ &=n+3n(n-1)\\ &=3n^2-2n. \end{aligned}

The standardized endpoint $Z_n=S_n/\sqrt n$ has mean $0$ , variance $1$ , skewness $0$ , and kurtosis

E[Z_n^4]=3-\frac2n\longrightarrow3.

Its MGF is

M_{Z_n}(t)=\left[\cosh\!\left(\frac{t}{\sqrt n}\right)\right]^n \longrightarrow e^{t^2/2}.

Thus each step can have only two possible values while the standardized endpoint after many steps approaches a normal distribution.

Simulating the Random Walk and Interpreting the Results

Use $n=5,25,125$ . For each $n$ , perform $1000$ independent repetitions. In each repetition, generate $n$ equally likely $\pm1$ steps and sum them, giving $1000$ endpoints. Plot a histogram of the endpoints and compare it with $\mathcal N(0,n)$ ; alternatively, plot $S_n/\sqrt n$ and compare it with $\mathcal N(0,1)$ .

Number of steps $n$	$\operatorname{SD}(S_n)$	Exact kurtosis of $Z_n$
5	$\sqrt5\approx2.236$	$2.600$
25	$5$	$2.920$
125	$\sqrt{125}\approx11.180$	$2.984$

Keep the following distinctions in mind when interpreting the results:

$n$ is the number of steps in each walk, which determines the theoretical approximation. The $1000$ repetitions are independent trials, which determine the sampling noise in the estimated histogram. Increasing the number of repetitions does not turn the true distribution of a fixed $5$ -step walk into a continuous normal distribution.
All three values of $n$ are odd, and possible endpoints are separated by $2$ . Histogram bins of width $2$ , centered on the allowed endpoints, are appropriate. Normalize the histogram as a density when overlaying a PDF. If the bars instead show PMF values, compare them with the normal distribution’s probability areas over the corresponding intervals, rather than directly with density heights.
The theoretical parameters are $\mu=0,\sigma=\sqrt n$ . When fitting the simulated sample, report the sample mean and fitted standard deviation; these will generally differ slightly because of sampling fluctuations.
As $n$ increases, the overall profile of the standardized endpoints generally becomes closer to a bell shape. At finite $n$ , the true distribution remains discrete, and a finite set of $1000$ simulations retains sampling noise.

Practice Questions and a General Solution Method

The following exercises combine probability, standardization, and moment-generating functions.

Question 1: Normal Approximations for a Sum and a Mean

Independent, identically distributed variables $X_i$ have mean $10$ and standard deviation $2$ . Let $n=100$ . Find the approximate distributions of $S_n$ and $\overline X_n$ , and estimate $P(9.6\leq\overline X_n\leq10.4)$ .

Solution: First note that $\sigma^2=4$ , so

\begin{aligned} S_{100}&\approx\mathcal N(1000,400),\\ \overline X_{100}&\approx\mathcal N(10,0.04). \end{aligned}

The standard deviation of the mean is $0.2$ , so the required probability is approximately $\Phi(2)-\Phi(-2)\approx0.9545$ . Common errors are treating $0.04$ as the standard deviation or incorrectly writing the variance of the sum as $100^2\times4$ .

Question 2: Reading Moments from an MGF

Given $M_X(t)=\exp(3t+2t^2)$ , find the mean, variance, and second raw moment.

Solution: Comparing with the normal MGF $\exp(\mu t+\sigma^2t^2/2)$ gives $\mu=3$ and $\sigma^2=4$ . Therefore, $m_2=\sigma^2+\mu^2=13$ . Alternatively, differentiate to obtain $M'_X(0)=3$ and $M''_X(0)=13$ ; the latter is not the variance.

Question 3: An Exact Random-Walk Probability

What is the probability of reaching $s=1$ after $5$ steps?

Solution: This requires $K=(5+1)/2=3$ rightward steps. Therefore,

P(S_5=1)=\binom{5}{3}2^{-5}=\frac{10}{32}=\frac5{16}.

Also, $P(S_5=0)=0$ , because an odd number of steps cannot lead to an even endpoint. A normal approximation cannot replace this restriction on the discrete support.

Question 4: Why Having Many Terms Is Not Enough

Suppose $X_1=\cdots=X_n=Y$ , where $E[Y]=0$ and $\operatorname{Var}(Y)=1$ . Can the ordinary i.i.d. CLT establish $S_n/\sqrt n\to\mathcal N(0,1)$ ?

Solution: No. These variables are perfectly correlated, and $S_n=nY$ . Thus $\operatorname{Var}(S_n)=n^2$ and $\operatorname{Var}(S_n/\sqrt n)=n$ ; the independence assumption fails. Identical distributions do not imply independence, and the covariance terms cannot be dropped.

Question 5: Two Claims About the Proof

Are the following statements correct: “Finite variance guarantees that an MGF exists near $0$ ” and “The MGF of the mean is $[M_X(t/\sqrt n)]^n$ ”?

Solution: Both are incorrect. The finite-variance CLT may hold even when the MGF method cannot be applied directly. The MGF of $\overline X_n$ is $[M_X(t/n)]^n$ . When $\mu=0,\sigma=1$ , the expression $[M_X(t/\sqrt n)]^n$ instead corresponds to $\sqrt n\,\overline X_n$ .

A General Sequence for Calculations

Identify the distribution of an individual variable and establish independence.
Calculate $\mu,\sigma^2$ .
Determine whether the target is $S_n$ or $\overline X_n$ .
Calculate the mean and variance of that target.
Standardize to $Z$ .
Finish using $\Phi$ , an exact discrete probability, or an MGF.

In an explanatory answer, state the theorem, its assumptions, and what a finite-sample approximation means.

← Introduction · Series index · Scale-free distributions →