ANU COMP 4670 / 8600 - Statistical Machine Learning
A broad but thorough intermediate level course of statistical machine learning, emphasising the mathematical, statistical, and computational aspects
Overview
Statistical Machine Learning plays a key role in science and technology. Some of the basic questions raised are:
What is a good model for the available data?
How can we fit the parameters of the model to the available data?
How will a model perform on data which has yet to be observed?
This course provides a broad but thorough intermediate-level study of the methods and practices of statistical machine learning, emphasising the mathematical, statistical, and computational aspects. Students will learn how to implement efficient machine-learning algorithms on a computer based on principled mathematical foundations. Topics covered will include Bayesian inference and maximum likelihood modelling; regression, classification, density estimation, clustering, principal and independent component analysis; parametric, semi-parametric, and non-parametric models; basis functions, neural networks, kernel methods, and graphical models; deterministic and stochastic optimisation; overfitting, regularisation, and validation.
The course will use Python 3 and Jupyter notebook for all tutorials and assignment/exam questions involving programming.
Course schedules
Timetable
Week 1A (Feb 21)
Course Introduction
Week 1B (Feb 22)
Machine Learning 101,
Probabilities, Model Selection
Self-Assessment
Assignment 0 - self-assessment
Self-Assessment
Jupyter Notebook, Matrices
Week 2A (Feb 28)
Linear Regression
Week 2B (Feb 29)
Bayesian Linear Regression
Assignment
Assignment 1 released
Video Assignment released
Week 2 Reading
Pre-read: Gaussian Mixture Model, Mathematics for Machine Learning, Chapter 11
Week 2 Tutorial
Linear Algebra, Optimisation, Probabilities
Week 3A (March 7)
Linear Classification
Week 3B (March 8)
Expectation Maximisation,
Mixture Models
Week 3 Reading
Pre-read: Kmeans,
Pattern Recognition and Machine Learning, Chapter 9.1
Week 3 Tutorial
Regression
Week 4A (March 14)
Generalisation
Week 4B (March 15)
Neural Networks
Quiz 1
Week 4 Reading
Pre-read: Patterns, Predictions and Actions: A Story about Machine Learning, Chapter 5
Week 4 Tutorial
Classification
Week 5A (March 21)
Automatic Differentiation
and its application to Neural Networks
Week 5B (March 22)
Linear and Non-linear Component Analysis, including Auto Encoder, Recommender System 101
Week 5 Reading
Pre-read: Principal Component Analysis, Mathematics for Machine Learning, Chapter 10
Week 5 Tutorial
Expectation Maximisation
Week 6A (March 28)
Kernel Methods
Week 6B (March 29)
Kernel Machines
Assignment
Assignment 1 due [Mon 12:00 noon]
Week 6 Tutorial
Neural Networks
Semester Break (Apr 3 - Apr 16)
Week 7A (Apr 18)
Gaussian Processes Regression
Week 7B (Apr 19)
Gaussian Process Classification
Assignment
Assignment 2 released
Week 7 Tutorial
Dimension Reduction
(ANZAC day, no lecture)
Week 8B (Apr 26)
Guest Lecture (Thang Bui) - Sparse Gaussian Process
Quiz 2
Week 8 Tutorial
Kernels
Week 9A (May 2)
Sampling
Week 9B (May 3)
Markov Chain Monte Carlo Theory
Week 9 Tutorial
Gaussian Process
Week 10A (May 9)
Markov Chain Monte Carlo Implementation
Week 10B (May 10)
Graphical Model - Bayesian Networks
Week 10 Tutorial
Sampling
Week 11A (May 16)
Graphical Models - Markov Random Fields
Week 11B (May 18)
Graphical Models - Factor Graphs
Assignment
Assignment 2 due
[Monday 12:00 noon]
Week 11 Tutorial
Graphical Models
(No Lecture)
Week 12B (May 24)
Machine Learning Perspective
+ Course Review
Assignment
Video Assignment due
[Monday 12:00 noon]
Labs and tutorials: refer to mytimetable
Exam Timetable
- Final exam: June 2, 2023, 9:50am-12:50pm
Course staff
Lecturers
School of Computing
School of Computing /
Research School of Astronomy and Astrophysics
Tutors
Chamin Hewa KONEPUTUGODAGE
(Head Tutor)
Alexander SOEN
(Head Tutor)
Anupama ARUKGODA
Dillon CHEN
Ziyu CHEN
Lydia LUCCHESI
Samuel JOLLEY
Buddhi KOTHALAWALA
Shidi LI
James Yuanchu LIANG
Evan MARKOU
Lachlan McGINNES
Josh NGUYEN
Vimukthini PINTO
Taylor Zishan QIN
Belona SONNA
Chunyi SUN
Tianyu WANG
Zhifeng WANG
Hansheng XUE
Allen Qinyu ZHAO
Haiqing ZHU
Textbooks
Christopher M. Bishop:
Pattern Recognition and Machine Learning
Springer, 2006 (selected parts)
We also recommend
Deisenroth, Faisal, and Ong, Mathematics for Machine Learning. Cambridge University Press.
Moritz Hardt and Benjamin Recht, Patterns, Predictions and Actions: A Story about Machine Learning
MacKay, Information Theory, Inference, and Learning Algorithms, Cambridge University Press
Murphy, Probabilistic Machine Learning: An Introduction, MIT Press, 2021
Course sites
All read-only content will be on the course web page -- this page!
The lectures and most tutorials will be in person. Microsoft teams (ANU edition) will be used to stream lectures via video (in addition to echo360) and host online tutorials/labs. Lectures will be recorded, but online tutorials will not be recorded.
EdSTEM is a new platform replacing piazza (with a few more functionalities)
Wattle will be used for exams, quizzes and surveys. SML 2023 wattle site can be found here.
A brief class FAQ can be found here.
Assessments
Quizzes (2% x 2)
Assignments (18% x 2)
Video assignment (20%)
Final exam (40%)
Online quiz expectations
The quiz will be conducted on Wattle. It will be automatically graded with answers released after the closing date.
Students can attempt the quiz once, with no time limit.
Open book -- students are expected to complete the quiz by themselves and are free to consult the textbook, notes, or relevant internet resources.
The quiz will be redeemable with the final exam, i.e. score for each quiz is calculated as Qx' = max(Qx, Final), where Qx is the raw quiz score for Quiz 1 (Q1) or Quiz 2 (Q2), out of 100. Final is the score for the final exam out of 100.
There will be no late period for either quiz. Special consideration requests will also not be accepted due to the rapid feedback cycle and redeemable nature of the quizzes.
Assignment
Assignments
Assignments 1 and 2 are individual assignments with conceptual, mathematical, and programming components. Submission instructions will be made available closer to time.
Video assignment
The video assignment is an individual assignment.
Each student is expected to upload a video discussing one topic from the assignments or labs and the thinking behind it.
The length of the video should be between 4 to 8 minutes, with an under- and over-length penalty being 1 point per 10 seconds (or part thereof).
The grading scheme for the video assignment will be made available in advance of the due date.
Late policy
This policy applies to Assignment 1, Assignment 2, and the video assignment.
Assignment submissions that are late from 1 min to 24 hours attract a 5% penalty (of possible marks available).
Submissions late by more than 24 hours will not be accepted.
Enrollment
To enrol in this course, you must have completed the prerequisites as per the COMP4670 or COMP8600 course description.
The topics covered in this course overlap with several courses in the major of Statistical Data Analytics. Please look at the first few tutorial sheets for an indication of the kinds of mathematics and statistics we will build upon.
Other enrollment info, including obtaining permission codes, is covered in the FAQ.