Build Micrograd from Scratch

You build a small automatic differentiation engine in the style of micrograd, then use it to train a neural network. The plan has 25 NumPy and PyTorch problems in seven sections, starting from numerical derivatives and ending with a full training loop on your own engine.

Python, and the idea of a derivative from school calculus. The chain rule and backpropagation lessons cover the math used here.

1. Derivatives and gradient checks: estimate derivatives numerically, check gradients and take a gradient-descent step.
2. Value objects and expression graphs: build addition and multiplication nodes, expression graphs and graph traversal.
3. Neurons and activation gradients: differentiate tanh, then evaluate, backpropagate through and gradient-check a neuron.
4. Automatic reverse-mode autodiff: local vector-Jacobian rules, topological sorting and a full backward pass.
5. Accumulation and scalar operators: accumulate gradients on shared paths and differentiate exponentials and powers.
6. Neural-network modules: neurons, dense layers, a multi-layer perceptron and recursive parameter collection.
7. Losses and training: squared-error gradients, an SGD step with zeroed gradients, and training a tiny MLP.

Apply the same ideas at scale in Language Modeling from Scratch, or see autograd in a real framework with the PyTorch study plan.