Mathematics for Machine Learning 2: Calculus and Optimization for Training Models

What you will achieve

By the end of this course you will be able to work with derivatives of functions that take and return more than one number, and to trust your own results.

Concretely, you will be able to:

• Compute partial derivatives by hand and assemble them into a gradient vector.
• Build the Jacobian of a vector-valued function and read it as a local linear map.
• Apply the chain rule to composed functions, including multi-layer compositions where the Jacobians multiply.
• Write down a Hessian and use it to classify critical points as minima, maxima, or saddle points.
• Expand a function with a Taylor series to second order and use the result to reason about curvature.
• Test a function for convexity and explain why convexity makes local minima global.
• Set up and solve constrained problems with Lagrange multipliers.
• Implement gradient descent, choose a learning rate, and diagnose divergence, slow progress, and oscillation from the loss curve.
• Check every hand-derived gradient or Jacobian against a finite-difference estimate in Python.

That last ability is the spine of the course. A derivation you cannot verify is a guess. Here you will derive first, then confirm numerically, every time.

Who this course is for

This is an intermediate course. It assumes you are already comfortable with the pieces that calculus for multiple variables is built on, so it does not re-teach them.

You should come in with:

• Vectors, matrices and matrix multiplication. You can multiply a matrix by a vector and know what the shapes must be.
• Comfort reading function notation. Expressions like $f: \mathbb{R}^n \to \mathbb{R}$ and $g(f(x))$ should not slow you down.
• Basic Python. You can write a small function, call NumPy, and print an array.

If those three are solid, you are ready for lesson 1. If matrix multiplication still feels shaky, review it first: the Jacobian and the chain rule are matrix products in disguise, and the course moves at a steady pace rather than a remedial one.

This course suits learners who want the fundamentals done properly: students preparing for machine learning or optimization coursework, engineers who use gradient-based methods and want to know why they work, and anyone who has called an autodiff library and wants to see the arithmetic underneath it.

Course roadmap

Module 1: First-order derivatives: gradients, Jacobians and the chain rule

• 1.1 From partial derivatives to the gradient vector. You differentiate one variable at a time, then stack the results into the gradient and interpret it as the direction of steepest increase.
• 1.2 Jacobians of vector-valued functions. You extend the gradient to functions with several outputs, building the matrix of all first partial derivatives.
• 1.3 Chain rule for composed functions. You differentiate nested functions by multiplying Jacobians, and track the shapes so the product is legal.
• 1.4 Checkpoint: first-order derivatives. A short quiz that confirms the first-order toolkit before curvature enters the picture.

Module 2: Second-order behavior and convexity

• 2.1 Hessians and second-order behavior. You collect second partial derivatives into the Hessian and read curvature, including the mixed-partial symmetry you can rely on.
• 2.2 Taylor expansions, convexity and local minima. You approximate functions to second order, define convexity through the Hessian, and connect convexity to the guarantee that a local minimum is the global one.
• 2.3 Checkpoint: curvature and convexity. A quiz on classification of critical points and convexity tests.

Module 3: Constrained optimization and gradient descent

• 3.1 Lagrange multipliers for constrained problems. You optimize subject to an equality constraint by introducing multipliers and solving the resulting stationarity conditions.
• 3.2 Gradient descent and learning rate behavior. You turn the gradient into an algorithm, then study how the learning rate controls convergence, oscillation, and divergence.
• 3.3 Checkpoint: constrained optimization and descent. A final quiz tying constraints, curvature, and step size together.

How the course works

The course runs about 295 minutes across 10 lessons, so roughly five hours of focused study. Most learners spread it over a week or two.

Each lesson is short theory followed by exercises. Worked examples live inside the theory lessons: you will see a function, watch the derivative derived by hand, and then see the same derivative confirmed with a finite-difference check in Python. There are no separate labs, so you can do everything in one sitting with a notebook and a Python session open beside it.

You will meet three kinds of lesson:

• Theory lessons with worked examples, where the derivation and the numeric check appear side by side.
• Checkpoint quizzes at the end of each module, which test recall and short computation rather than long proofs.
• Exercise sets woven into the theory, sized to be finished in a few minutes each.

How to use it. The course is built to be taken linearly. Module 2 assumes the Jacobian from Module 1, and Module 3 assumes the Hessian from Module 2, so skipping ahead tends to cost more time than it saves. If you already know first-order derivatives cold, you can skim 1.1 and 1.2 and take the 1.4 checkpoint as your entry test; a clean pass means you can start at 2.1. Otherwise, start at the beginning and keep a Python window open. Do the numeric check on every example you derive by hand, even when you are confident. That habit is the point of the course, and it is what will keep your gradients honest long after the lessons end.