What Is a Convolutional Network? Images Made Simple

What is a convolutional network? It's a neural network where at least one layer works by sliding a small grid of numbers, called a filter, across the input and multiplying as it goes. Google's machine learning glossary defines a convolutional neural network as "a neural network in which at least one layer is a convolutional layer," usually mixed with pooling layers and ordinary dense layers, and notes that these networks "have had great success in certain kinds of problems, such as image recognition." That's the honest one-line answer. The filter is the whole trick, and I'd rather show it than describe it, because the reason it works on pictures is that a picture's interesting parts, edges, corners, eyes, can show up anywhere in the frame.

The neural net walkthrough on this site ran one neuron by hand. This page does the same for one filter, on a picture small enough to check with a pencil, and then shows you why the 2012 network that changed computer vision is that same filter stacked deep.

One Filter, on Paper

Here's a 6 by 6 picture. Each number is a pixel's brightness, 0 for black and 9 for white. I've made it as boring as possible on purpose. The left half is dark, the right half is bright, so there's one vertical edge running down the middle.

And here's a filter, a 3 by 3 grid of weights. This one is designed to notice vertical edges, and you'll see why in a moment.

The operation, in the glossary's exact words, is two steps. "Element-wise multiplication of the convolutional filter and a slice of an input matrix," then "summation of all the values in the resulting product matrix." Lay the filter over the top-left 3 by 3 patch of the picture, multiply each pixel by the weight sitting on it, add the nine products, and write down the total. Then slide the filter one column to the right and do it again.

Take the top-left patch first. Its three columns are 0, 0, 0. Every product is zero, so the total is 0. Now slide one step right. The patch's columns are 0, 0, 9. Each row gives minus 1 times 0, plus 0 times 0, plus 1 times 9, which is 9, and three rows make 27. One more step, columns 0, 9, 9. Each row gives 0 plus 0 plus 9, so 27 again. Last step, columns 9, 9, 9. Each row gives minus 9 plus 0 plus 9, which is 0.

Slide down a row and repeat. The picture is the same in every row, so every row of the output is the same. The result is a 4 by 4 grid, which the 2015 review by LeCun, Bengio and Hinton calls a feature map.

Read it as a heat map. The filter stayed quiet on the flat dark region, stayed quiet on the flat bright region, and lit up exactly where dark meets bright. Nine numbers just found the edge.

The Wrong Filter Finds Nothing

Now the part that made this click for me. Run the same picture through a filter built for horizontal edges, with the minus ones along the top row and the ones along the bottom.

Every patch has identical top and bottom rows, because the picture doesn't change from row to row. So the top row's products cancel the bottom row's exactly, and every one of the sixteen totals is 0. A feature map of pure zeros. This filter is deaf to vertical edges, and if you flip the picture on its side, so the edge runs across, the two filters swap roles and it's the horizontal one that reads 27 down the middle rows.

That's what a filter is. It's a tiny question asked at every position. Is there a vertical edge here? Is there a horizontal one? A real convolutional layer asks dozens of these questions at once, one filter each. The review calls the weights in one filter a filter bank, and its description of the layer matches the tables above. Units are "organized in feature maps," each "connected to local patches" of the input, and "all units in a feature map share the same filter bank."

Nobody wrote the two filters above by hand in a real network, by the way. I did, for teaching. In a trained network the nine weights start random and get nudged until they turn into whatever question helps most. The AlexNet paper from 2012 reports that its first layer ended up with "a variety of frequency- and orientation-selective kernels, as well as various colored blobs." Edge detectors, learned rather than designed, which is what Wikipedia's definition means by a network "that learns features via filter (or kernel) optimization."

Why It's Called Convolutional

The slide-multiply-add routine has a name in mathematics. The review says it plainly. "Mathematically, the filtering operation performed by a feature map is a discrete convolution, hence the name." Convolution is an old idea from signal processing, and a convolutional network is one that runs that operation as a layer, with the numbers inside the filter learned from data.

The reason the same nine weights get reused at every position is the sentence I'd remember from the whole review. "If a motif can appear in one part of the image, it could appear anywhere, hence the idea of units at different locations sharing the same weights and detecting the same pattern in different parts of the array." A cat's ear is a cat's ear whether it's top-left or bottom-right. So the network learns one ear detector and slides it everywhere, instead of learning a separate one for every spot.

Weight sharing is also why these networks are affordable. My 3 by 3 filter is 9 numbers, reused at all 16 positions of the 6 by 6 picture. A layer that instead wired all 36 pixels to 16 outputs would need 36 times 16, or 576 weights, for a job the filter does with 9.

Pooling, the Shrink Step

After a convolutional layer usually comes a pooling layer, and the glossary defines it as "reducing a matrix (or matrixes) created by an earlier convolutional layer to a smaller matrix," usually by "taking either the maximum or average value across the pooled area."

Do it to the feature map above. Chop the 4 by 4 into four 2 by 2 windows and keep the biggest number in each.

The 4 by 4 map became a 2 by 2 map that says, in effect, there's a strong vertical edge in every quarter of this picture. The exact column it was on has been thrown away. That sounds like a loss. I thought so too, and it's the point. The review says the pooling layer's job is "to merge semantically similar features into one," and that shrinking the map this way creates "an invariance to small shifts and distortions."

Here's the shift, in numbers. Move the edge one column to the left, so the picture's bright half starts at column three. The feature map's rows become 27, 27, 0, 0 instead of 0, 27, 27, 0. Different map, same edge. But the top-left pooling window still reports 27, because the strongest response landed somewhere inside it. Pooling makes the network care that the edge is there more than it cares exactly where, which is what you want if you're trying to recognise a face that might be a few pixels off centre.

The Four Ideas, Pinned to the Example

The review says there are "four key ideas behind ConvNets that take advantage of the properties of natural signals." Here's each one attached to a number from the tables above, because that's how I finally stopped confusing them.

The last row is where a single filter turns into something that can tell a tabby from a terrier. The review describes the stack in one line. "In images, local combinations of edges form motifs, motifs assemble into parts, and parts form objects." The first layer's filters find edges, exactly like ours. The second layer's filters slide across the first layer's feature maps, so they're finding arrangements of edges, corners and curves. The third finds arrangements of those. In the review's description, "two or three stages of convolution, non-linearity and pooling are stacked, followed by more convolutional and fully-connected layers," and those last layers turn the final feature maps into a decision.

Nobody hand-writes any of it. The sibling piece on machine learning versus deep learning has the scoreboard showing what happened when learned filters replaced hand-designed features on a million photos.

The 2012 Network, in Numbers

The network that produced that scoreboard is described in its own paper with figures a beginner can actually hold. It has "60 million parameters and 650,000 neurons," and it "consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers." Five rounds of the filter trick, some followed by the shrink step, then three ordinary layers to decide among 1,000 categories. In the 2012 competition its top-5 error was 15.3%, "compared to 26.2% achieved by the second-best entry."

The first layer is worth a small calculation, and it's the one I'd do if you only do one. The paper says it "filters the 224 x 224 x 3 input image with 96 kernels of size 11 x 11 x 3." Three is the colour channels, red, green and blue, so each filter is an 11 by 11 grid three deep. Stanford's CS230 cheatsheet, which ranks for this search, gives the parameter count for a convolutional layer as filter height times width times channels, plus one bias, times the number of filters. So 11 times 11 times 3 is 363, plus 1 is 364, times 96 filters is 34,944 numbers. That's the entire first layer of the network that halved the error rate, and it reads an input of 224 times 224 times 3, or 150,528 pixel values. Under 35,000 learned numbers, applied everywhere, looking at 150,000. That ratio is weight sharing doing its job.

What Convolutional Networks Are Mainly Used For

Anything that arrives as a grid. My rough test is whether the data has neighbours, and the review is precise about the shapes. ConvNets "are designed to process data that come in the form of multiple arrays," listed as "1D for signals and sequences, including language; 2D for images or audio spectrograms; and 3D for video or volumetric images." Pictures are the famous case. A sound recording turned into a spectrogram is a 2D grid too, and the same edge-finding filters work on it.

The oldest deployed example I know of is money. The review reports that a convolutional network reading handwritten digits was, "by the late 1990s," reading "over 10% of all the cheques in the United States." Medical scans, the photo search on your phone, the lane-keeping camera in a car all followed. In 2015 the review named Mobileye and NVIDIA as using ConvNet-based methods "in their upcoming vision systems for cars," and called ConvNets "the dominant approach for almost all recognition and detection tasks."

Are they AI? Yes, in the ordinary sense, and I wouldn't overthink it. A convolutional network is a kind of neural network, which is a kind of deep learning, which sits inside machine learning, which sits inside AI. The lifecycle around all of that is in how AI models work, and none of it changes when you add the word convolutional.

Is YOLO Just a Convolutional Network?

Mostly yes, and the part I find interesting is what it does differently with one. YOLO, from the 2015 paper by Redmon, Divvala, Girshick and Farhadi, is an object detector, which means it doesn't just say "there's a dog," it draws the box around the dog. Earlier detectors, the paper says, "repurpose classifiers to perform detection." YOLO treats detection as one prediction instead. "A single neural network predicts bounding boxes and class probabilities directly from full images in one evaluation." The paper's body states that the model is implemented "as a convolutional neural network," 24 convolutional layers followed by two fully connected ones.

So the filters are the same idea as our 3 by 3 grid, stacked 24 deep. What changed is the question asked at the end, box coordinates and a class for each region rather than a single label for the whole image. The paper reports the base model running at 45 frames per second, a faster version at 155, and admits the trade, "more localization errors" but "far less likely to predict false detections where nothing exists."

Is ChatGPT a Convolutional Network?

No, though it is a neural network. The model behind ChatGPT is a large language model, and GPT-3 is described in its paper as "an autoregressive language model with 175 billion parameters." Its architecture is the transformer, and the paper that introduced transformers in 2017, Attention Is All You Need, says in its abstract that the design is "based solely on attention mechanisms, dispensing with recurrence and convolutions entirely."

So the two most famous kinds of deep network use different tricks. Convolutional networks slide a shared filter across a grid. Transformers let every token in a sequence weigh every other token. Both are stacks of learned weights trained by the same nudge, and the LLM explainer covers what the transformer stack does with its layers. My dinner-table version is that ChatGPT reads and convolutional networks look.

A Short History, With the Papers

IBM's page, which ranks second for this search, gives 1980 and 1989 as its dates and no journals, and the other ranking pages disagree on the origin. I went to the journals' own registry records instead.

The review draws the line from the first row to the rest itself. The convolutional and pooling layers are "directly inspired by the classic notions of simple cells and complex cells in visual neuroscience," and ConvNets "have their roots in the neocognitron," which "did not have an end-to-end supervised-learning algorithm such as backpropagation." The 1989 paper added that. Everything since is the same design with more layers and more data.

Try One Change

Take the vertical filter and flip its signs, so the left column is 1 and the right column is minus 1. Run it over the original picture. The feature map comes out as 0, minus 27, minus 27, 0 on every row. Same edge, opposite sign, because now the filter is asking "bright on the left, dark on the right" and our picture is the other way round. A real first layer keeps both versions, and the sign tells the next layer which way the edge faces.

Then draw your own 6 by 6 picture with a single bright square in the middle and run both filters over it. You'll get a vertical response down the square's left and right sides, opposite in sign, and a horizontal one along its top and bottom, and where those responses meet in the second layer is what a corner detector looks like. You've just built two layers of a convolutional network with a pencil. The 2012 one has five, and I hope they now look less like magic and more like arithmetic with patience.