Overview
This course will provide a detailed and deep treatment for commonly used methods in continuous optimization, with applications in machine learning, statistics, data science, operations research, among others. The main focus of this course is on continuous optimization algorithms and as well as some core optimization theory as a foundation for the development of these algorithms. The discussions of algorithms will be accompanied with representative applications.
Course Outline
- Introduction:
- Machine learning paradigm and empirical risk minimization
- Mathematical optimization
- Representative models and applications in machine learning, data science, operations research, signal/image processing, and others
- Basic Theory
- Convex analysis and optimization theory
- Optimality conditions
- Smooth convex functions
- First-Order Methods
- Gradient descent
- Accelerated gradient methods and variants
- Stochastic Methods (Basic)
- Stochastic gradient method and variants
- Stochastic Methods (Advanced)
- Variance reduction methods
- Adaptive stochastic gradient methods
- First-Order Methods (Advanced)
- Coordinate descent
- Methods for constrained optimization: projected and conditional gradient
- Methods for nonsmooth optimization: subgradient method and proximal gradient
- Duality:
- Lagrangian duality theory and optimality conditions
- Augmented Lagrangian method
- ADMM
- Second-Order Methods
- Newton method
- Quasi-Newton methods
- Randomized variants
- Algorithms for parallel and distributed computing
- Communication vs computation tradeoffs
- Synchronous and asynchronous methods
- Federated learning
- Applications (some of these applications may be merged into other topics where appropriate)
- Least squares, lasso, and logistic regression
- Support vector machines
- Low-rank matrix completion/sensing
- Neural networks and deep learning
## Coding
We will use Python for coding exercises in this course.
Syllabus
You can find the syllabus here. Course material will be updated throughout the semester on Canvas.