The Math Behind Image Compression: How JPEG Works

How does a 20-megapixel photo, which is about 60 million numbers, shrink down to a 2MB file you can text to a friend in two seconds?

You might guess it's some clever way of listing pixels more efficiently. It's not. JPEG barely thinks about pixels at all. Instead, it asks a stranger question: what if we described this image the way a musician describes a chord, as a small set of pure notes added together, instead of a long list of individual sounds?

That single idea, plus one fact about how your eyes work, is the whole trick. Let's build it from scratch.

A row of pixels is just a wave problem

Picture one row of 8 pixels from a photo, each one a brightness value from 0 (black) to 255 (white). Something like a soft edge, dark on the left, bright on the right.

You could store all 8 numbers directly. That works, but it's not clever. Here's the interesting question instead: could you describe this row as a sum of a few smooth waves, added together, instead of 8 separate numbers?

Turns out, yes. Any row of 8 values can be written as a sum of 8 cosine waves of increasing frequency, the flattest one first (basically the average brightness), then a wave that swings once across the row, then twice, and so on up to the busiest one. Add enough of them and you reconstruct the row exactly.

But here's the payoff: you usually don't need all 8. The first few waves already capture the overall shape. The later ones just add fine detail most eyes won't miss.

Eight brightness values, rebuilt from only the first K waves. Watch how few you need before the curve locks onto the dots.

K = 2 waves

Waves used2 / 8

RMS error17.25

Notice how the curve locks onto the dots with just 3 or 4 waves. That last handful of high-frequency waves is doing very little work, which is exactly the opening JPEG needs.

Writing that down: the discrete cosine transform

Now for the notation. We write the row of pixel values as $x_0, x_1, \dots, x_7$ . The wave decomposition is called the discrete cosine transform, or DCT, and it turns those 8 pixel values into 8 "wave strengths" $X_0, X_1, \dots, X_7$ :

X_k = \alpha(k) \sum_{n=0}^{7} x_n \cos\left[\frac{\pi}{8}\left(n + \tfrac{1}{2}\right)k\right]

Don't worry about $\alpha(k)$ , it's just a scaling constant that keeps the energy consistent. The important part is the shape: each $X_k$ measures how much of wave frequency $k$ is present in the row. $X_0$ is the average brightness. $X_7$ is the fastest-wiggling detail.

Compression is now a simple decision: keep the low-frequency $X_k$ values, and throw away the high-frequency ones the eye barely notices.

From one row to a real image block

An actual photo isn't one row, it's a 2D grid. JPEG splits the image into 8×8 pixel blocks and runs the same wave-decomposition trick in both directions at once, horizontally and vertically. Instead of 8 possible waves, an 8×8 block has 64: some purely horizontal, some purely vertical, some diagonal checkerboards, ranging from "flat" all the way to "busiest pattern possible in 8 pixels."

D_{v,u} = \alpha(u)\,\alpha(v) \sum_{y=0}^{7}\sum_{x=0}^{7} p_{x,y} \cos\left[\frac{\pi}{8}\left(x+\tfrac12\right)u\right]\cos\left[\frac{\pi}{8}\left(y+\tfrac12\right)v\right]

Same idea as before, just doubled up. $D_{v,u}$ tells you how much of the pattern at horizontal frequency $u$ and vertical frequency $v$ is hiding in that block. And just like the single row, most real photo blocks put almost all their energy into the low-frequency corner.

Slide K and watch how many of the 64 frequency patterns the block actually needs before it looks right.

K = 10 / 64

Coefficients kept10 / 64

Discarded84%

RMS error6.13

Slide K up from 1 and watch the reconstructed block sharpen. By the time you're past 15 or 20 coefficients out of 64, most blocks are already visually close to the original. That's roughly 70% of the data gone, and you can barely tell.

The color trick nobody notices

There's a second compression lever, and it has nothing to do with frequency. It's about how your eyes are built.

Your retina has far more receptors tuned to brightness than to color. You're excellent at spotting a sharp edge in light and dark. You're mediocre at spotting a sharp edge in color alone. JPEG exploits this directly.

First, it converts the image from red-green-blue into a format that separates brightness from color: one channel for brightness ( $Y$ ), and two for color ( $C_b$ , $C_r$ ).

Y = 0.299R + 0.587G + 0.114B

C_b = 128 - 0.169R - 0.331G + 0.5B, \qquad C_r = 128 + 0.5R - 0.419G - 0.081B

Then it keeps the brightness channel at full resolution, but shrinks the two color channels, often to a quarter of their original size, by averaging blocks of pixels together. Same trick as before: throw away detail the eye won't miss, just applied to color instead of frequency.

Same block size, two different channels cut. Toggle which one gets blurred and see why JPEG always picks color.

block = 3×3

Data saved59%

Avg color error29.7

Toggle the comparison. Cutting color resolution at a given block size barely registers. Cutting brightness resolution by that same amount looks broken almost immediately. That asymmetry is the entire reason JPEG (and video, and your phone's camera pipeline) always subsamples color, never brightness.

Turning near-zeros into a tiny file

We've thrown away high-frequency wave patterns and extra color resolution. Now for the last step: actually shrinking the file.

After the DCT, JPEG divides each of the 64 coefficients by a number from a quantization table, then rounds to the nearest integer. Low-frequency coefficients get divided by small numbers, so they survive mostly intact. High-frequency coefficients get divided by large numbers, so most of them round straight down to zero.

Q_{v,u} = \text{round}\left(\frac{D_{v,u}}{\text{step} \cdot (u+v+1)}\right)

Bigger "step" means more aggressive rounding, means more zeros, means a smaller file and lower quality. This is the exact slider behind every "JPEG quality: 1-100" export dialog you've ever used.

The 64 quantized values then get read off in a zig-zag pattern, low frequency first, which bunches all those zeros together at the end. A long run of trailing zeros compresses beautifully, you just write "and then 40 more zeros" instead of listing 40 individual zeros. That's the last piece, entropy coding, and it's why a heavily quantized block collapses to almost nothing.

This is the quality slider from every "Save as JPEG" dialog. Push it up and watch the scan fill with zeros.

quant step = 3

Nonzero coefficients14 / 64

Est. size (vs 512 bits raw)171 bits

RMS error2.33

Push the quantization step up and watch the scan light up mostly gray. Every gray cell is a coefficient that costs the file nothing.

The short version

JPEG doesn't compress pixels directly. It rewrites each 8×8 block as a sum of frequency patterns using the discrete cosine transform, keeps the low frequencies and discards the high ones your eyes barely register, shrinks the color channels because your eyes are far less sensitive to color detail than brightness detail, and finally rounds the leftover numbers aggressively enough that most of them become zero, so the file can just say "then a long run of nothing" instead of storing every value.

Three separate appeals to the same fact, human vision doesn't notice everything, stacked on top of one elegant piece of math, the discrete cosine transform. That's how a 60-million-number photo becomes a 2MB file, and why it still looks like a photo.

All visualizations run entirely in your browser: real 1D and 2D discrete cosine transforms, real YCbCr color conversion, and a real zig-zag quantization scan, computed live in plain JavaScript. No libraries beyond React.