Checkpoint

Derivatives, gradients and Jacobians

Eleven questions on the whole module: derivatives and their rules, gradients, Jacobians and the chain rule, and gradient descent. The numbers are new, and several options are the classic mistakes. Work on paper where it helps; the code item is short. You need 80 percent to pass.

1
What is the derivative of $h(x) = \sigma(3x)$ at $x = 0$ , where $\sigma$ is the sigmoid?
h'(0)
2
You check a derivative $f'(x)$ of a smooth function with a central difference. With $h = 0.1$ the estimate is off by $0.012$ . Ignoring rounding error, about how far off will it be with $h = 0.05$ ?
error with h = 0.05
3
A pre-activation $z = 0$ passes through either a sigmoid unit or a tanh unit, and the upstream gradient is $1$ . How much gradient does each unit pass back to $z$ ?
4
The loss on one example is $L(\mathbf{w}) = (\mathbf{w}\cdot\mathbf{x} - y)^2$ with $\mathbf{x} = (1, 2)$ and $y = 1$ . Find $\nabla_{\mathbf{w}}L$ at $\mathbf{w} = (1, 1)$ .
∇L at w = (1, 1)
5
At a point, $\nabla f = (2, -1)$ . You take a small step in the unit direction $\mathbf{u} = (1, 2)/\sqrt{5}$ . To first order in the step (the linear approximation), what happens to $f$ ?
6
A feed-forward block maps a 768-entry vector up to 3,072 entries with its first layer (Jacobian $\mathbf{J}_1$ ), then back down to 768 with its second (Jacobian $\mathbf{J}_2$ ). What is the Jacobian of the whole block?
7
Let $f(\mathbf{x}) = \mathrm{ReLU}(\mathbf{W}\mathbf{x})$ , with ReLU applied to each entry and $\mathbf{W} = \begin{pmatrix} 1 & 2 \\ 3 & -1 \end{pmatrix}$ . Find the Jacobian of $f$ at $\mathbf{x} = (-1, 1)$ .
Jacobian at (-1, 1)
8
Three logits are equal: $\mathbf{z} = (0, 0, 0)$ . For $\mathbf{p} = \mathrm{softmax}(\mathbf{z})$ , what is $\frac{\partial p_1}{\partial z_2}$ ? Give a fraction or 3 decimal places.
∂p₁/∂z₂
9
Gradient descent on $L(w) = 2(w - 1)^2$ starts at $w_0 = 3$ with learning rate $\eta = 0.3$ . What is $w_2$ ?
w₂
10
Near a minimum, the Hessian has eigenvalues $0.5$ and $50$ . You run gradient descent with $\eta = 0.05$ . What happens?
11
Code it
Write input_grad(W1, W2, x): the gradient with respect to x of $L = \sum_i \bigl[\tanh(\mathbf{W}_2\tanh(\mathbf{W}_1\mathbf{x}))\bigr]_i$ . Use vector-Jacobian products only: never build a Jacobian.
⌘↵ runsPython sleeps until you run code

11 still unanswered. Unanswered questions count as wrong.