1. The latent-variable model

Observed data , latent variables , parameters . The model specifies a joint , and the observed-data likelihood marginalizes the latents:

Coin instance. , , , so with , , .

The difficulty is structural: contains , a log-of-a-sum, which is not separable in and admits no closed-form maximizer.

2. Jensen’s inequality and a corollary

Theorem (Jensen). If is concave and a random variable with finite mean, then , with equality iff is almost surely constant or is affine on the support of . For a finite convex combination, with , .

Corollary (Gibbs’ inequality; non-negativity of KL). For distributions over the same finite set,

Proof. Apply Jensen to the concave :

Equality in Jensen requires constant on the support, i.e. .

Note that this is the same inequality that makes the ELBO a lower bound — seen once as and once as .

3. The ELBO and the exact decomposition

For any distribution with the same support as the posterior, define the evidence lower bound

Theorem (decomposition). For every and ,

Proof (one experiment; sum over at the end). Since and using ,

Insert inside the log and split the resulting product:

Because , we recover for all , with equality iff for all .

4. The E-step: optimal q

Proposition. For fixed , , and at this optimum .

Proof. In the decomposition , the term does not depend on . Hence maximizing over is equivalent to minimizing the (non-negative) KL sum, which attains its floor of exactly at .

Coin instance (responsibilities). By Bayes’ rule, with the binomial coefficient cancelling,

5. The M-step: maximizing Q

With fixed (entropy term constant in ), maximizing over is maximizing

Coin instance. Using ,

Setting this to zero — and noting is a constant here (frozen by the E-step) — gives a linear equation with the closed-form root

The two updates are decoupled because ’s terms in carry coefficient and vanish under .

6. Why no closed form exists for ℓ directly

It is instructive to differentiate the intractable and see precisely where it jams. With ,

The chain rule on the outer drops a whose denominator contains both and . Equivalently, the prefactor is itself a nonlinear function of . Setting therefore yields

an implicit, transcendental, coupled system: appears on the left and (through ) nonlinearly on the right, and the twin equation for mirrors it. No finite algebraic formula solves it.

EM’s resolution is exactly the comparison of Sections 5 and 6: freezing at converts the same stationarity equation into a linear one with a closed-form root. EM is thus a fixed-point iteration for the coupled score equations, solved by alternation; a fixed point satisfies the true (Section 8).

7. The monotonicity theorem

Theorem. Let (E-step) and where (M-step). Then

Proof. Write the decomposition with and abbreviate (the -free entropy), so that . Then

Since is non-decreasing and (here) bounded above, the sequence converges.

8. Convergence to a stationary point

Define the gap as a function of with fixed: . By construction , so is a global minimizer of , whence (assuming differentiability) . Differentiating the decomposition at gives the tangency identity

Consequently, if is a fixed point of EM, then maximizes , so , and by tangency : every EM fixed point is a stationary point of the marginal log-likelihood. (EM may converge to a local maximum or saddle, not necessarily the global maximum; initialization matters.)

The tangency also explains the rate: near each bound peaks almost directly above its contact point, so steps shrink and the tail is linear (slow).

The figure (a 1-D slice fixing ) shows each amber ELBO touching at a green point where equals the posterior (KL ), the M-step reaching its peak (orange diamond), and the reopened KL gap between each diamond and the next contact point above it.

9. Numerical appendix

EM on , , , seed . Responsibilities are computed at ; the row shows the resulting and the new .

	(per experiment)
0		0.7073	0.5765
1		0.7435	0.5691
2		0.7671	0.5500
3	—	0.7826	0.5350
4	—	0.7908	0.5265
5	—	0.7944	0.5225
	converged

Worked first update (verification):

The log-likelihood column is monotone non-decreasing (Section 7); is non-monotone (it overshoots to then relaxes), consistent with the theorem guaranteeing ascent only in , not in individual coordinates.

← Part 2 — Math with Intuition | Back to Part 1 — The Intuition

Notation

Per experiment : = heads observed (data), = which coin was used (latent). Parameters ; mixing priors ; flips. One coin gives a binomial:

For brevity write (and similarly ).

The thing we actually want to maximize

We only see the , so the quantity to maximize is the marginal log-likelihood, summing the latent coin out:

Reading it from the inside out:

• The joint — “pick a coin, then flip it.” For the two coins: and .
• The inner sum is marginalization: you can’t see the coin, so you add up over every value it could take. The result is the per-experiment marginal, a mixture of two binomials:

• The puts it on log scale; the outer adds across the (independent) experiments.

The obstacle: log of a sum

Look at the shape of the right-hand side: a wrapped around a sum.

You can’t distribute it — — so the two unknowns , stay tangled inside that log, with no clean derivative and no closed-form maximizer. This single obstacle is the entire reason EM exists.

Contrast the easy world: if the coin labels were known, the log would sit on a single binomial, and — a tidy, separable expression you can maximize in closed form. The sum is what ruins this.

Slipping in a helper distribution q

Introduce a free distribution over the latent coin — just two numbers and summing to 1. “Free” means we choose it, and the algebra below holds for any choice.

Multiply and divide inside the sum by (which equals multiplying by 1):

That last equality is just the definition of expectation, . So the marginal is now “log of an expectation,” .

Jensen’s inequality

For a concave function (like ), the function of an average is at least the average of the function:

Geometrically: a concave curve bulges above its chords, so evaluating the curve at the average (a point on the curve) sits at or above averaging two curve-heights (a point on the chord).

The green dot is “function of the average” (on the curve); the orange dot is “average of the function” (on the chord); the vertical distance is the Jensen gap, which is zero only when the two inputs coincide.

Applying this to our — concave , so push it inside at the cost of an inequality:

That right-hand side is the ELBO, written . The now sits outside the sum, on single product terms — exactly the splittable, differentiable form we wanted.

The exact gap is a KL divergence

The inequality has an exact bookkeeping form. For any :

The gap between the true log-likelihood and the ELBO is exactly the KL divergence between your chosen and the posterior . The KL has two properties that drive everything:

• always — so the ELBO is always a lower bound;
• iff — so the bound is tight exactly when is the posterior.

On the coins, with and posterior , the KL is just

a concrete number measuring how far your working belief is from the row’s actual posterior — zero when they match.

The two steps, and the geometry

Rearranging the boxed identity, . Hold fixed: is a constant, so maximizing the ELBO over = minimizing the KL over — a seesaw. Both bottom out at $q = $ posterior.

E-step (touch). Set , the posterior at your current spot. This zeroes the KL at , so the ELBO rises up to touch the true curve right where you stand.

M-step (climb). Freeze that and maximize the ELBO over — . Since the bound is easy and decoupled, this is a closed-form update.

Why climbing the bound raises the truth — chain three facts:

End to end: . The true log-likelihood never decreases.

Because the dome reopens a gap once you step off , you rebuild it each round — a staircase of bounds walking up the true curve:

The steps shrink near the top: as you approach the optimum, each bound’s peak sits almost directly above its touch point (the bound is tangent to there), so there’s less left to climb — which is both why EM converges and why its tail is slow.

The E-step and M-step on the coins

E-step (responsibilities). Setting to the posterior gives, by Bayes’ rule,

M-step (weighted MLE). Maximizing the ELBO over reduces to maximizing the expected complete-data log-likelihood,

whose maximizers are soft heads-over-flips counts:

Each row contributes to each coin in proportion to its responsibility — the “soft counting” from Part 1, now derived.

Watching the bounds touch and climb, for real

Running EM on the five experiments from a seed of , produces this trajectory:

And here is the actual ELBO geometry (1-D slice fixing , varying ). The blue curve is the true log-likelihood; each amber dome is a real ELBO built at one iterate:

Read it left to right:

• Each amber dome touches the blue curve at a green dot (where was set to the posterior, so KL = 0).
• The M-step climbs the dome to its peak — the orange diamond.
• Look directly above each diamond: the blue curve is higher. That vertical gap is the KL reopening, and it’s why the next green dot (the next E-step) sits up on blue, not down at the diamond. You gained more true likelihood than the bound alone promised.
• The green dots march monotonically up blue toward .

The loop, compactly

Repeat until stops moving.

← Part 1 — The Intuition | Continue to Part 3 — The Rigorous Treatment →

The problem: a chicken-and-egg puzzle

The Expectation–Maximization (EM) algorithm exists to solve one frustrating situation: you want to estimate some parameters, but to estimate them you’d need to know a hidden label, and to know the hidden label you’d need the parameters. Each one would unlock the other, and you have neither.

EM breaks the loop by guessing, then refining.

The running example: two coins

Imagine two biased coins, A and B, with unknown head-probabilities and . Someone runs five experiments. In each one they secretly grab one of the two coins, flip it 10 times, and write down only the number of heads — not which coin they used:

Your goal is to recover and . The annoying part is the missing label — which coin produced each row. That hidden label is the latent variable.

The fix: guess, soften, repeat

Start with a guess, say , . Now instead of forcing a hard “this row is A” decision, EM does something gentler, alternating two steps:

E-step (Expectation). For each row, use the current guesses to compute the probability it came from each coin. The 9-heads row comes out to roughly 80% A / 20% B; the 4-heads row to roughly 35% A / 65% B. These soft probabilities are called responsibilities — coin A is “responsible” for 80% of the 9-heads row.

M-step (Maximization). Re-estimate the coins, but let every row contribute to both coins in proportion to those responsibilities. The 9-heads row dumps 80% of its 9 heads into A’s tally and 20% into B’s. It’s a weighted version of the trivial counting you’d do if labels were known — using soft counts instead of hard ones.

Then take the new ’s back to the E-step and go around again. Each loop the responsibilities sharpen and the estimates improve. The hidden labels and the parameters bootstrap each other into existence.

Why it keeps getting better: the surrogate-dome picture

Here’s the mental model that makes EM click. Picture the quality of a parameter guess as a height on a hill — the true likelihood of the data. You want the summit, but the hill is shrouded in fog: you can’t survey it or optimize it directly.

So at your feet you lay down a smooth dome — a simpler surrogate — engineered with two properties:

1. it touches the hill exactly where you’re standing, and
2. it never pokes above the hill anywhere.

You can easily find the top of the dome (that’s the M-step). Walk there. Because the dome never exceeded the hill, the hill at your new spot is at least as high as the dome’s peak — which was at least as high as where you started. So you’ve climbed the real hill without ever seeing it. Then you lay a fresh dome at the new spot and repeat.

That dome is the ELBO (Evidence Lower BOund), and the “touch then climb” rhythm is exactly the E-step and M-step. The fact that the dome stays below the hill is what makes climbing it safe — you can never be tricked into walking somewhere that looks good on the surrogate but is bad in reality.

The deeper “why” — why the dome touches where it does, why it stays below, and the inequality that guarantees the climb — is the heart of Part 2.

Watching it actually converge

Here’s EM run for real on the five experiments above, deliberately seeded off-target at , :

Two things worth noticing, both of which the theory predicts:

• The log-likelihood (bottom panel) never decreases. That’s EM’s core guarantee — every round either improves the fit of the data or leaves it unchanged.
• overshoots. It jumps up to on the first step, then drifts back down to . That’s allowed: EM only promises the overall likelihood climbs monotonically, not that each individual parameter marches straight to its target. From a bad seed the first responsibilities are misjudged, so the first update over-corrects, and later rounds walk it back.

The run settles at , — the biases that best explain the five observed heads-counts.

The whole algorithm in one breath

Fill in, then re-fit. The E-step fills in the missing labels with their expected (soft) values given where you currently stand; the M-step re-fits the parameters as if those soft labels were the data. Because the fill-in depended on the old parameters, you fill in again at the new spot — and around you go, until nothing moves.

Continue to Part 2 — Math with Intuition →