What Is a Neural Net? A Beginner Walkthrough

What is a neural net? A neural net, or neural network, is a program built from many small units that each take in some numbers, multiply each by a weight, add the results, and pass the total on. Stack those units in layers, let a training process adjust the weights until the outputs match examples, and you have the machine behind image recognition, speech-to-text and every chatbot you've used. Stanford's Institute for Human-Centered AI defines it as a computational model inspired by the structure of the human brain, consisting of interconnected layers of artificial "neurons" that process and transmit information.

The word neuron makes it sound biological. It isn't. Each unit is a few lines of arithmetic, and by the end of this page you'll have run one by hand with a pencil, which is the fastest way I know to make the term stop feeling like magic.

One Neuron, on Paper

Here's a neuron that decides whether to take an umbrella. The numbers are invented for teaching, and I picked them so the arithmetic stays easy.

It reads three inputs, each on a scale from 0 to 1. Cloud cover. The forecast chance of rain. How long a walk you're facing. Each input has a weight, which says how much that input matters, and there's one extra number called a bias that sets how hard the neuron is to trigger. The rule is simpler than the diagrams make it look, which I think is the main reason the diagrams put people off. Multiply each input by its weight, add everything up including the bias, and if the total is above zero, fire.

Say today's cloud cover is 0.8, the forecast says 0.6, and the walk is a middling 0.5. The weights are 0.5, 1.0 and 0.3, and the bias is minus 0.7.

The total is 0.45, which is above zero, so the neuron fires. Take the umbrella.

Now a different day. Cloud cover 0.2, forecast 0.1, but a long walk at 0.9. Same weights, same bias. Multiply and add. 0.10 plus 0.10 plus 0.27, minus 0.70, gives minus 0.23. Below zero. No umbrella.

That's a neuron. The whole thing. It's the "all-or-none" unit that McCulloch and Pitts described in 1943, a threshold that excitation has to exceed before anything happens, and it's exactly the formula the IBM page in these search results writes as z equals the sum of the weights times the inputs, plus the bias. The formula and the table are the same object. One of them you can do in your head, and I'd read the table before the formula every time.

The Parts, Named

Every neural network explainer throws the same six words at you, and my complaint with most of them is that the words never get pinned to a number. Here they are, pinned to the example so they stop floating.

The last row is the one to remember when you read a headline. GPT-3 was described in its own paper as having 175 billion parameters. That means 175 billion of those adjustable numbers, arranged in far more complicated layers than the umbrella neuron, but the same kind of number doing the same kind of job.

One Training Step, on Paper

Nobody set the weights above by hand in a real system. They get learned, and learning is nothing more mysterious than nudging the numbers when the neuron gets something wrong.

Go back to the second day, the long walk with little cloud. The neuron said no umbrella. Suppose it rained. The neuron was wrong by one whole step, so call the error 1. The simplest learning rule, the perceptron rule that traces back to Frank Rosenblatt's 1958 paper, says to move each weight by a small learning rate, times the error, times that weight's input. Use a learning rate of 0.1.

Run the second day again with the new numbers. 0.104 plus 0.101 plus 0.351, minus 0.60, comes to minus 0.044. Still no umbrella, but the total moved from minus 0.23 to almost zero. One more rainy long-walk day and it tips over.

Look at which weight moved most. The walk weight, because the walk input was the biggest number on the day the neuron got it wrong. The rule pushes hardest on whatever was present at the moment of the mistake. That's the entire intuition behind training, and it holds at every scale I've read about.

Training a real network is this nudge repeated across thousands or millions of examples, over and over, until the mistakes get rare. With many layers, the harder question is how much of the blame each earlier weight deserves for a mistake at the output. The 1986 paper by Rumelhart, Hinton and Williams gave the standard answer, back-propagation. The 2015 review by LeCun, Bengio and Hinton describes it as the method that indicates "how a machine should change its internal parameters that are used to compute the representation in each layer from the representation in the previous layer." Same nudge, routed backwards through the stack.

If you want the surrounding lifecycle, the training run, the frozen model, and what happens when you press send, that's in how AI models work. This article stays inside the neuron.

Why Stack Them Into Layers

One neuron can only draw a straight line. It splits every possible day into two regions, umbrella and no umbrella, and the boundary between them is flat. Try to express "take it if the walk is long and it's cloudy, or if rain is forecast regardless" and a single weighted sum can't bend that way. I'd encourage you to attempt it with the weights above. You'll find no set of three numbers and a bias that gets all the cases right, and the failure is more instructive than the explanation.

Layers fix it. The first layer holds several neurons that each learn a simple feature of the input. The next layer reads their outputs, so it's combining features rather than raw inputs, and can express "this and that, or the other." Add more layers and the features get more abstract at each step. The 2015 review's opening sentence says it in one breath. Deep learning "allows computational models that are composed of multiple processing layers to learn representations of data with multiple levels of abstraction."

That's all "deep" means. Many layers. The sibling piece on machine learning versus deep learning draws the boundary properly, and for pictures, where early layers find edges and later ones find shapes, the convolutional network explainer is the next step.

I'd resist the urge to picture something brain-shaped. My own picture is a spreadsheet with columns of numbers feeding columns of numbers. The reason nets look like diagrams of dots and lines is that the diagram is easier to draw than the spreadsheet, and the diagram is where most of the mystique comes from.

Is ChatGPT a Neural Network?

Yes. The model underneath ChatGPT is a large language model, and Jurafsky and Martin's textbook states the general point plainly. Large language models are based on neural networks. The specific kind is called a transformer.

The concrete numbers are what make it less abstract, at least for me. The GPT-3 paper lists eight sizes of the model, from 125 million parameters up to the 175 billion version that got the name, and its architecture table gives the largest one 96 layers. Ninety-six stacks of the arithmetic you did above, each far wider than three inputs, with a transformer's attention machinery deciding which earlier tokens each layer should weigh. The umbrella neuron and GPT-3 are the same kind of object at wildly different scale, and the sibling piece on what LLM stands for unpacks the "large" and "language" parts of that name.

What the neural net doesn't tell you is what the model does with all those layers, which is predict the next token. The LLM explainer covers that, and it's why a network this size can be fluent and wrong in the same sentence. Nothing in the arithmetic checks facts. It only fits patterns.

Do Humans Have a Neural Network?

Yes, the original one. The 1943 paper that started this field wasn't about computers. McCulloch and Pitts open their introduction with "The nervous system is a net of neurons," and their threshold unit was an attempt to describe real nerve cells with logic. The artificial version borrowed the sketch and kept the name, which is why neuroscientists and machine learning people both say "neural network" and mean different things.

The comparison gets stretched in two directions, so here are the numbers and why they don't line up. A 2009 study by Suzana Herculano-Houzel counted an average of 86 billion neurons in the adult male human brain. GPT-3 has 175 billion parameters. Those aren't the same unit. A parameter is a weight, which is closer to the strength of a connection between two cells than to a cell itself, and a biological neuron does things the umbrella unit doesn't, in timing and chemistry, that I'm not qualified to summarise. Wikipedia's phrasing is the right level of caution, a "simplification of neurons in a brain." Inspired by, not a copy of.

So when a headline says a model has more parameters than the brain has neurons, my reading is that it's comparing wires to houses. Interesting, and not what it sounds like.

Are Neural Nets Real?

Real in the plainest sense. They're software running arithmetic on chips, and the "neurons" are rows of multiplication and addition like the one you did above. There's no organism and no electricity flowing through anything neuron-shaped. A computer runs the same operations you ran by hand, billions of times per second, on numbers it learned from examples.

If the word "real" is asking whether they work, I'd point at the 2015 review's list rather than at any vendor page. It credits deep networks with improving speech recognition, visual object recognition and object detection, and names drug discovery and genomics among the other domains, with convolutional nets for images and audio and recurrent nets for text and speech. That was a decade ago. The transformer behind ChatGPT arrived after that review, and it's built from the same parts.

A Short, Sourced History

The pages ranking for this search disagree on dates, with one giving 1944 and 1957 and another 1943 and 1958. The papers settle it, and I'd rather cite them than pick a side.

Nothing in the last row would surprise the authors of the first. It's the same threshold unit, learning by the same kind of nudge, at a scale they had no hardware to imagine.

Try One Change

Take the umbrella neuron and set the bias to minus 1.0 instead of minus 0.7. Rerun both days. The first day drops from 0.45 to 0.15, still firing, and the second day drops to minus 0.53, further from firing. You've made the neuron pickier without touching a single weight, and you've felt what a bias does in a way I don't think any definition can.

Then try a third day of your own and decide, before you calculate, whether you'd take the umbrella. If you and the neuron disagree, work out which weight you'd nudge, and in which direction. That's the whole of machine learning, done once, on paper. Everything else is doing it 175 billion times.