← Back to teaching
Fall 2026Graduate Course

Mathematical Foundations in AI

Modern AI rests on a small set of mathematical ideas. This course develops four of them — linear algebra, probability, stochastic processes, and optimization — and shows how each underpins today's AI systems. We study how linear algebra describes the computations inside neural networks, how probability and stochastic processes model data, uncertainty, and sequential decision-making, and how optimization turns these models into algorithms that learn. Throughout, theory is paired with applications in deep learning, reinforcement learning, and large language models, giving students the mathematical tools to understand, analyze, and improve modern AI methods.

Peking University · MELON Research Group

9am – 12pm Thursday, C7-731511 lectures posted
About this course

Course information

Instructor
Teaching assistants
Classroom
9am – 12pm Thursday, C7-7315
Office hour
2pm – 3pm Thursday, C7-7315
Further reading

References

  1. Optimization for Machine LearningMartin Jaggi and Nicolas Flammarion · EPFL Class CS-439
  2. Advanced Machine Learning SystemsChris De Sa · Cornell CS6787
  3. Optimization MethodsZaiwen Wen · PKU 2024 Fall
  4. Introduction to LLMKun Yuan · PKU 2025 Spring
Full course notes

Lecture notes

Slides & notes

Materials

LECTURES 01–11
02

Linear Algebra I

  1. Part IGradient and HessianNotes ↓
  2. Part IILinear Transform; Eigenvalue and EigenvectorNotes ↓
03

Linear Algebra II

  1. Part IJacobian Matrix; Hessian Matrix; Chain RuleNotes ↓
  2. Part IIInner Product; Positive-definite Matrix; Matrix Functions; Forward-backward PropagationNotes ↓
  3. Part IIINorms; Singular-value DecompositionNotes ↓
04

Probability and Statistics

  1. Part IProbability; Random Variables; DistributionNotes ↓
  2. Part IIExpectation; Variance; Marginal/Conditional DistributionNotes ↓
05

Gradient Descent

  1. Part IConvex Set; Convex Functions; Convex Problems; Gradient DescentSlides ↓Notes ↓
  2. Part IIMomentum Gradient Descent; Nesterov Acceleration; Anderson AccelerationSlides ↓Notes ↓
07

Basics in Transformer and LLM I

  1. Part IWord Embedding; Seq2Seq; Attention; TransformerSlides ↓
08

Basics in Transformer and LLM II

  1. Part IParameters, Computations, and Memory Costs in TransformerSlides01 ↓Slides02 ↓
  2. Part IIMemory Access Cost; Kernel Fusion; Flash AttentionSlides ↓
09

Advanced Optimizer for LLM

  1. Part IShampoo; SOAP; Muon; Lite; MuonMSlides ↓
10

Subspace Optimizer for LLM

  1. Part IAdam-mini; Memory-efficient OptimizersSlides ↓
  2. Part IIBlock Coordinate Descent; Coordinate Friendly StructureSlides ↓
  3. Part IIIBlock-wise Training in LLMsSlides ↓
11

Zeroth-Order Optimizer

  1. Part IFinite Difference; Linear Interpolation; Sphere SmoothingSlides ↓Notes ↓
  2. Part IIMemory-efficient Zeroth-Order Optimization; Low-rank Zeroth-Order OptimizationSlides ↓
Back to top ↑