ml.lab
Python sleeps until you run code
09 From LSTMs to today's models

Lesson 5 of 5

Vision and what comes next

Convolution shares one small filter across a whole image, the way a recurrent network shares its weights across time. From there, the R-CNN family finds objects by sharing computation, segmentation labels every pixel, vision transformers read patches as tokens, and CLIP puts images and text in one space.

About 65 minutes
By the end you can
  • Compute a 2D convolution by hand, with stride, and predict the output size of a layer.

  • Count the parameters of a convolutional layer and the receptive field of a stack of layers.

  • Implement convolution, pooling, a multi-channel layer and an edge detector with loops.

  • Choose between classification, detection, semantic and instance segmentation methods for a problem, and explain how the R-CNN family shares computation.

  • Explain how a vision transformer turns an image into tokens and how CLIP classifies images it was never trained to label.

A dense layer from module 6 connects every input number to every output. For a small color image of 32 by 32 pixels, that is 32×32×3=3,07232 \times 32 \times 3 = 3{,}072 inputs, and producing 16 numbers for every pixel position (32×32×16=16,38432 \times 32 \times 16 = 16{,}384 outputs) takes 3,072×16,384=50,331,6483{,}072 \times 16{,}384 = 50{,}331{,}648 weights. Worse than the size is the waste: a cat in the top-left corner and the same cat in the bottom-right corner fall on entirely different weights, so the network has to learn what a cat looks like separately at every position.

Recurrent networks met the same problem in time, and solved it by using the same weights at every step, as you saw in module 7. Convolution uses the same weights at every position in space. With 16 filters of size 3 by 3 over the 3 color channels, the layer above needs 3×3×3×16+16=4483 \times 3 \times 3 \times 16 + 16 = 448 parameters instead of fifty million.

This lesson builds convolution by hand, follows it into object detection and segmentation, then shows how transformers and text entered vision. It ends with a map of what the rest of ML Lab covers.

Convolution: one small filter, slid everywhere

A kernel (or filter) is a small grid of weights, say k×kk \times k. Place it on the top-left corner of the image, multiply each kernel weight by the pixel under it, and add everything up: that is one output number. Slide the kernel one pixel to the right and repeat, across and then down. The grid of outputs is a feature map. In symbols, with rows and columns counted from 0,

Yij=∑a=0k−1∑b=0k−1Kab Xi+a, j+b.Y_{ij} = \sum_{a=0}^{k-1}\sum_{b=0}^{k-1} K_{ab}\,X_{i+a,\,j+b} .

Each output is a dot product, in the sense of module 1, between the kernel and the patch under it. It is large when the patch looks like the kernel. A kernel is a pattern detector, and the feature map shows where the pattern occurs.

Here is an image that is dark on the left and bright on the right, and a 2 by 2 kernel that subtracts the left column from the right column:

X=[0011001100110011],K=[−11−11].\mathbf{X} = \begin{bmatrix} 0 & 0 & 1 & 1 \\ 0 & 0 & 1 & 1 \\ 0 & 0 & 1 & 1 \\ 0 & 0 & 1 & 1 \end{bmatrix}, \qquad \mathbf{K} = \begin{bmatrix} -1 & 1 \\ -1 & 1 \end{bmatrix}.

At the top-left position the patch is all zeros, so Y00=0Y_{00} = 0. One step right, the patch is [0101]\begin{bmatrix} 0 & 1 \\ 0 & 1 \end{bmatrix}, and Y01=(−1)(0)+(1)(1)+(−1)(0)+(1)(1)=2Y_{01} = (-1)(0) + (1)(1) + (-1)(0) + (1)(1) = 2. At the third position the patch is all ones and the −1-1s cancel the +1+1s: Y02=0Y_{02} = 0. Every row is the same, so

Y=[020020020]:\mathbf{Y} = \begin{bmatrix} 0 & 2 & 0 \\ 0 & 2 & 0 \\ 0 & 2 & 0 \end{bmatrix}:

the edge lights up and the flat regions stay dark. A 4 by 4 image and a 2 by 2 kernel give a 3 by 3 output, because the kernel fits in 3 positions along each axis.

The widget does the same on a larger image: a bright 6 by 6 square on a dark 12 by 12 background, with a 3 by 3 kernel. The white square on the image is the window, the small colored squares inside it are the kernel's weights, and the output map fills in cell by cell, coral for positive, blue for negative and dark for 0. Under the figure, the arithmetic for the current window is written out. The starting kernel, edge-x, is a Sobel filter: its left column is (−1,−2,−1)(-1,\allowbreak -2,\allowbreak -1), its middle column 0 and its right column (1,2,1)(1,\allowbreak 2,\allowbreak 1), so it responds where brightness changes from left to right.

Two more knobs control the size of the output. The stride ss is how far the kernel moves each step. Padding pp adds pp rings of zeros around the border, so that the kernel can be centered on edge pixels too. Along one axis of length nn, the kernel of size kk fits at

⌊n+2p−ks⌋+1\left\lfloor \frac{n + 2p - k}{s} \right\rfloor + 1

positions, where ⌊⋅⌋\lfloor\cdot\rfloor rounds down. For example, a 224×224224 \times 224 image with a 7×77 \times 7 kernel, stride 2 and padding 3 gives ⌊223/2⌋+1=112\lfloor 223/2 \rfloor + 1 = 112 positions per side. With k=3k = 3, s=1s = 1 and p=1p = 1, the output keeps the input's size.

Why sharing works. The same kernel at every position gives two properties. The parameter count does not depend on the image size. And the layer is translation equivariant: shift the image and the feature map shifts the same way, so a pattern detector learned in one place works everywhere.

To recap: a convolution slides one small kernel over the image, and each output is the dot product of the kernel with the patch under it. The output has ⌊(n+2p−k)/s⌋+1\lfloor (n + 2p - k)/s \rfloor + 1 positions per side. Now do it by hand, with a stride and with padding.

On paperConvolve by hand

Take the image and kernel

X=[1201312001322011],K=[10−110−110−1].\mathbf{X} = \begin{bmatrix} 1 & 2 & 0 & 1 \\ 3 & 1 & 2 & 0 \\ 0 & 1 & 3 & 2 \\ 2 & 0 & 1 & 1 \end{bmatrix},\allowbreak \qquad \mathbf{K} = \begin{bmatrix} 1 & 0 & -1 \\ 1 & 0 & -1 \\ 1 & 0 & -1 \end{bmatrix}.

  1. Compute the feature map (stride 1, no padding, kernel not flipped).
  2. Now use the 2×22 \times 2 kernel of all ones with stride 2. Compute the output.
  3. How large would the output of part 1 be with padding 1?

Work it on real paper: writing each step is the point. Then check your final answer here and compare your working with the walk-through.

The feature map from part 1

One entry per box, row by row (Tab moves along a row). 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

The next one has a kernel with a zero in it, which makes each output a sum of only three terms. Watch the orientation: no flip.

Work it outA feature map

Convolve X=[2013110203111220]\mathbf{X} = \begin{bmatrix} 2 & 0 & 1 & 3 \\ 1 & 1 & 0 & 2 \\ 0 & 3 & 1 & 1 \\ 1 & 2 & 2 & 0 \end{bmatrix} with K=[1−102]\mathbf{K} = \begin{bmatrix} 1 & -1 \\ 0 & 2 \end{bmatrix}, stride 1, no padding, without flipping the kernel. Enter the output Y\mathbf{Y} row by row, with Y00Y_{00} at the top left.

Y

One entry per box, row by row (Tab moves along a row). 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

A few more, with new numbers each time:

Practice setConvolution reps
3 correct in a row completes the set. A miss starts the count again.

A convolution slides a 1×11 \times 1 kernel over a 7×77 \times 7 input with padding 1 (1 row and column of zeros added on every side) and stride 1. How wide is the output?

width

Type a number: 0.25, -2, 3/4 and sqrt(2) all work. Enter checks.

Channels, pooling and receptive fields

Real images have three color channels, and the feature maps inside a network have many more. A convolutional layer takes CinC_{\text{in}} input channels and produces CoutC_{\text{out}} output channels. Each output channel has its own kernel of shape k×k×Cink \times k \times C_{\text{in}}: it slides over all input channels at once, multiplying and adding across the whole block, and then adds one bias. So the layer has

k2 Cin Cout+Coutk^2\,C_{\text{in}}\,C_{\text{out}} + C_{\text{out}}

parameters. For 3×33 \times 3 kernels from 3 channels to 16, that is 9×3×16+16=4489 \times 3 \times 16 + 16 = 448, the number in the opening.

Pooling shrinks a feature map. Max pooling with a 2×22 \times 2 window and stride 2 keeps the largest value in each block, halving the height and width. It keeps the strongest response in each neighborhood and makes the output a little less sensitive to small shifts.

Receptive fields. One output of a 3×33 \times 3 layer depends on a 3×33 \times 3 patch of its input. An output of a second 3×33 \times 3 layer depends on a 3×33 \times 3 patch of the first layer's outputs. Along one row, those are 3 neighboring outputs, each seeing 3 input pixels and shifted by one pixel from the last, so together they span 3+1+1=53 + 1 + 1 = 5 pixels. The same holds down a column, so the field is 5×55 \times 5 input pixels. Each additional layer with a k×kk \times k kernel and stride 1 widens the field by k−1k - 1:

receptive field after L layers=1+L(k−1).\text{receptive field after } L \text{ layers} = 1 + L(k - 1).

Three 3×33 \times 3 layers see 7×77 \times 7 pixels. Strides and pooling make the field grow much faster, because each later step skips over more input. This is how a CNN goes from local to global: early layers respond to edges and small textures, and deeper layers, whose fields cover much of the image, respond to larger parts and whole objects. Visualizations of trained networks broadly support this picture, although individual units are often harder to name.

Before the exercises, here is the widget's convolution in code, written with plain loops. It runs the widget's square through edge-x (sobel_x) and edge-y (its transpose), then combines the two maps into an edge strength gx2+gy2\sqrt{g_x^2 + g_y^2}. The second heatmap is the map you filled in the widget: two columns of 4s along the square's left side and two of −4-4s along its right.

⌘+Enter runs · edit freelyPython sleeps until you run code

The parameter formula above is worth checking at a realistic size, where the comparison with a dense layer is dramatic.

Work it outParameters in a convolutional layer

A convolutional layer maps 64 input channels to 128 output channels with 3×33 \times 3 kernels and one bias per output channel. How many parameters does it have?

parameters

Type a number: 0.25, -2, 3/4 and sqrt(2) all work. Enter checks.

Now write the pieces yourself: convolution with a stride, max pooling, a layer with channels, and an edge detector.

Code itConvolution, pooling and an edge detector

Write the pieces of a convolutional network with plain loops, then use them.

  1. conv2d(image, kernel, stride=1): the feature map, with no padding and no kernel flip.
  2. max_pool(fmap, size=2): the largest value in each non-overlapping block, dropping leftover rows and columns.
  3. conv_layer(images, kernels, bias): a layer with channels. Output channel o sums conv2d(images[c], kernels[o, c]) over the input channels c and adds bias[o].
  4. edge_strength(image): gx2+gy2\sqrt{g_x^2 + g_y^2}, where gxg_x uses the Sobel kernel SOBEL_X and gyg_y its transpose.
⌘+Enter runsPython sleeps until you run code
Write your code where the starter says raise NotImplementedError, then press Run tests. Each check says what it expects.

From labels to boxes to pixels

A classifier answers one question about a whole image: what is in it. Many problems need more.

  • Object detection: a box and a label for every object.
  • Semantic segmentation: a class for every pixel (road, car, person, sky), without telling two cars apart.
  • Instance segmentation: a separate mask for each object, so two people standing together are two masks.

The widget compares how the classic approaches spend their computation, on one 640 by 480 scene with four shapes: a large circle, a small circle, a square and a triangle. The Method switch picks the approach, Play or Result runs it, a status line says what each stage is doing, and the chart counts the expensive network runs on a log scale.

Sliding windows run a classifier on every window at every position, size and shape. As the big picture computed, a single window size on one 640 by 480 image already gives thousands of windows, each needing a full CNN run.

R-CNN (Girshick and colleagues, 2014) used a classical algorithm (selective search) to propose about 2,000 regions likely to contain an object, warped each region to a fixed size, and ran the CNN on each one, then classified the features and refined the box. Better than blind sliding, but still about 2,000 CNN runs per image, much of it recomputing the same pixels in overlapping regions.

Fast R-CNN (Girshick, 2015) ran the CNN once, on the whole image, to get a feature map. For each proposed region it cut out the matching part of the feature map and pooled it to a fixed grid, such as 7×77 \times 7, by max pooling within each cell (RoI pooling, for "region of interest"). A small network then classified each region and refined its box. The expensive work became shared.

Faster R-CNN (Ren, He, Girshick and Sun, 2015) replaced the slow hand-designed proposals with a region proposal network running on the same feature map. At every position it considers a fixed set of reference boxes called anchors, 9 of them in the paper (3 sizes times 3 shapes), and predicts for each whether it contains an object and how to adjust it. Proposals became learned and nearly free.

Mask R-CNN (He, Gkioxari, Dollár and Girshick, 2017) added a small branch that predicts a pixel mask for each detected object, which is instance segmentation. It also replaced RoI pooling's rounding to whole feature-map cells with RoIAlign, which samples the feature map at exact positions, because masks need pixel accuracy.

Fully convolutional networks (Long, Shelhamer and Darrell, 2015) replaced a classifier's final dense layers with convolutions, so the network outputs a map of class scores rather than one label, and upsampled that map to full resolution: semantic segmentation in one pass. U-Net (Ronneberger, Fischer and Brox, 2015), designed for biomedical images, pairs a contracting path that shrinks the image with an expanding path that grows it back, and copies high-resolution features across with skip connections so that fine boundaries survive.

A different branch skipped proposals altogether: one-stage detectors such as YOLO and SSD (both 2016) predict boxes and classes directly from the feature map in a single pass, trading some accuracy for speed at the time. The thread through all of it is the one from the big picture: stop recomputing what can be shared.

To recap: detection finds a box and a label per object, semantic segmentation labels every pixel, and instance segmentation gives each object its own mask. The methods differ mostly in how much expensive computation they share. The next question asks you to match a method to a real problem.

Quick checkWhich method for which problem

A car's camera system must label every pixel of each frame as road, sidewalk, vehicle, person or sky. Telling two touching vehicles apart is not required. Which approach fits best?

Choose one answer, then check.

Transformers for images, and images with words

The vision transformer (ViT), introduced by Dosovitskiy and colleagues (2020), applies the transformer of Inside a transformer to images with almost no changes. Cut a 224×224224 \times 224 color image into 16×1616 \times 16 patches: 224/16=14224/16 = 14 patches per side, so 14×14=19614 \times 14 = 196 patches. Flatten each patch into 16×16×3=76816 \times 16 \times 3 = 768 numbers and multiply by a learned matrix to get one token vector. Add a learned position vector to each, add one extra learned token whose final vector is used for classification, and run a standard transformer encoder, with no causal mask: every patch attends to every patch, 1972=38,809197^2 = 38{,}809 scores per head per layer. A CNN has locality and weight sharing built in. A ViT has neither, and has to learn from data which patches relate. In the original experiments it outperformed strong CNNs only when pretrained on very large datasets, and it has since become a standard backbone.

CLIP (Radford and colleagues, 2021) connects images and text. It has two encoders: an image encoder (a CNN or a ViT) and a text encoder (a transformer), each ending in a vector in the same embedding space, normalized to length 1. It was trained on 400 million image and caption pairs collected from the internet. In a batch of NN pairs, it computes all N×NN \times N cosine similarities between images and captions (the similarity of module 1), scaled by a learned temperature, and applies cross-entropy twice: each image must pick its own caption out of the NN, and each caption its own image. Matching pairs are pulled together and every mismatched pair in the batch is pushed apart.

The payoff is zero-shot classification. To sort photos into classes the model was never trained on, write a caption for each class, such as "a photo of a dog" and "a photo of a cat", embed the captions once, and label each image with the class whose caption has the highest cosine similarity with it. The labels are just text, so changing the classes means changing a list of strings.

⌘+Enter runs · edit freelyPython sleeps until you run code

The diagonal of the heatmap holds the matching pairs, and training pushes it up while pushing everything else down. The loss is close to 0 here (about 0.0016) because each image's own caption already wins by a wide margin once the similarities are divided by the temperature 0.1. Make the change the last comment suggests and the cat image becomes more similar to the dog caption (0.96) than to its own (0.53): the loss jumps to about 0.84, and the zero-shot label for that image flips to dog.

The last exercise asks you to explain CLIP's zero-shot trick to someone who will rely on it.

In your own wordsHow CLIP labels what it never trained on

Explain to a colleague how CLIP can sort photos into classes it was never trained on, such as a company's own product categories. Connect each step to something you built in Part 1, and say where you would expect it to fail.

A few sentences first: 0 of 60 characters.

Saved in this browser as you type.

What comes next

Part 1 rebuilt the machinery: vectors, matrices, eigenvalues and Jacobians, backpropagation, recurrent networks and LSTMs, and in this module the ideas that replaced and extended them. The rest of ML Lab, on the roadmap, builds on each piece.

  • Part 2, attention and transformers. A small GPT built from scratch and trained: tokenization, embeddings and positions, the transformer block of Inside a transformer, training and sampling, and the KV cache. Your attention and block code are its core.
  • Part 3, pretraining at scale. Data pipelines, scaling laws, optimizers and learning-rate schedules, mixed precision and parallelism. The 6ND6ND estimate of Pretraining and fine-tuning and the Adam you wrote in module 6 are the starting points.
  • Part 4, post-training. SFT, reward models, RLHF and DPO, reinforcement learning with verifiable rewards, LoRA and distillation, at a scale where they produce real assistants. The objectives are the ones in Pretraining and fine-tuning and the previous lesson.
  • Part 5, vision. CNNs in depth, detection and segmentation, vision transformers and CLIP, built and trained.
  • Part 6, models in products. Serving, quantization, retrieval (the cosine search of module 1 at scale), evaluation and agents.

The thread that ran through Part 1 keeps running: a model is matrices and nonlinearities; training is gradients flowing backward through Jacobians; and whether those gradients survive a long path, through time in an RNN or through depth in a transformer, decides what can be learned.

Next: the module checkpoint