How Regression Analysis Works: From Scatter Plot to Equation

You're staring at a scatter of dots. Temperature on one axis, ice cream sales on the other. It's obviously going up and to the right, warmer days sell more ice cream. But if someone asked you to draw "the" line through those dots, could you do it?

Not just any line. The best one.

Turns out "best" isn't a vibe. It's a number. And once you can measure how good a line is, you can search for the line that scores lowest, no guessing required.

That's regression analysis. Here's how it actually works, from scatter plot to equation.

What does "best line" even mean?

Look at the scatter below. You could draw a line through it a hundred different ways, and most of them would look fine. Reasonable, even.

But "reasonable" doesn't cut it in math. We need a number that tells us exactly how wrong a line is, so we can compare two lines and say which one is better.

Drag either endpoint. See those dashed segments connecting each dot to the line? Those are the errors, sometimes called residuals. The gap between what the line predicts and what actually happened.

Add up all those gaps and you get a score. Lower score, better line. That's the whole game.

But here's a subtlety you probably just ran into: some gaps point up, some point down. Positive errors and negative errors. Add them directly and they cancel out, a line that's way too high in some places and way too low in others could score a perfect zero, even though it fits terribly.

We need a way to punish being wrong that doesn't care about direction.

Why square the errors?

The obvious fix: use absolute value instead of the raw gap, $|e|$ instead of $e$. That would work, actually. Statisticians do use it sometimes.

But there's a more natural fix, and it comes from physics.

Imagine every dot connected to the line by a spring. Stretch a spring and it stores energy, not proportional to how far you stretched it, proportional to the stretch squared. Pull it twice as far, and it stores four times the energy. That's just how springs work, it's Hooke's law.

Now imagine the line is free to slide and tilt, pulled by all those springs at once. Where does it settle? At the position where the total stored energy is smallest. Physics does the minimizing for you.

That's exactly what squared error is: energy. Tilt the line above and watch the squares grow, each one is the "energy" of that point's spring. The total area of every square is the score we're trying to minimize.

And hey, notice something else: squaring has a practical bonus too. Squared functions are smooth and curved, easy for calculus to work with. Absolute value has a sharp corner right at zero, which makes the math annoying. Squaring gives us the best of both, a real physical meaning, and clean math.

We write the total error as:

Here $m$ is the slope, $b$ is the intercept, and $(x_i, y_i)$ is each dot. $SSE$ stands for "sum of squared errors." Minimize that, and you've found the best line.

How do we find the minimum automatically?

So we have a score. Now we need to search for the line that gets the lowest one. Try every possible slope and intercept? There are infinitely many.

But here's the thing: SSE isn't a random, bumpy landscape. Fix the intercept, vary only the slope, and plot the score against that slope... you get a parabola. A smooth bowl with exactly one bottom.

Press play. Watch the ball roll downhill, always moving in the direction that decreases the score fastest, slowing as the bowl flattens near the bottom. That's gradient descent, and it's the same idea a lot of machine learning uses to find its answers too.

But for a straight line, we don't even need to roll downhill step by step. Because the bowl is a perfect parabola, calculus can jump straight to the bottom in one move. Set the bowl's slope, the derivative, to zero, solve, and you land exactly on the minimum.

Doing that for both $m$ and $b$ at once gives two clean formulas, called the normal equations:

where $\bar x$ and $\bar y$ are just the averages of your $x$ and $y$ values. No dragging, no rolling, no guessing. Plug in your data, and out comes the best possible line, instantly.

Is the fit actually good?

Here's a trap. You can always compute a "best" line, even for data with no real trend at all. The formula doesn't know or care whether the pattern is meaningful.

So we need a second number: not "what's the best line for this data," but "how much does this data even behave like a line?"

Slide the spread up and watch the dots scatter away from the line. Notice $R^2$ drop. Slide it back down, the dots tighten around the line, and $R^2$ climbs back toward 1.

...and that's the idea behind $R^2$. It compares two errors: how wrong the line is (small, if the fit is tight), against how wrong you'd be if you just guessed the average every time (a flat, lazy line). Writing the line's prediction as $\hat y_i$, we get:

An $R^2$ of 1 means the line explains every bit of the pattern. An $R^2$ near 0 means the line barely beats just guessing the average. It's the "how much can I trust this line" number.

The short version

Fitting a line to data means finding the slope and intercept that minimize the sum of squared errors, the total "spring energy" between every point and the line. Because that error is a smooth bowl shape, calculus can jump straight to the minimum with two formulas instead of guessing. Once you have that line, $R^2$ tells you whether it's actually a meaningful fit or just the best of a bad situation.

Every trend line you've ever seen on a stock chart, a weather report, or a scientific paper came from exactly this process: define a score, minimize it, then check whether the answer deserves your trust.

All visualizations are interactive React components running entirely in your browser, computing least squares and gradient descent live from the data shown. No libraries beyond React.