This course takes place in the traditional, ancestral and unceded territory of the xʷməθkʷəy̓əm (Musqueam). It is important to reflect upon this fact as we study, work, and grow on this land. (https://indigenous.ubc.ca.)
Description. Bayesian optimization (BO) is a well-established probabilistic approach to blackbox optimization: the process of optimizing functions that (1) lack an explicit mathematical form and (2) are often expensive to evaluate. While its origins trace back to 1962, BO gained popularity in 2012 when a seminal paper noted its efficacy at hyperparameter tuning for machine learning models. Recently, an explosion of developments have made BO a promising approach for challenging design problems, ranging from nuclear fusion control to drug discovery.
This graduate-level topics course aims to provide (1) a rigorous technical introduction of BO, (2) a survey of recent research developments in the field, and (3) the skills needed to utilize BO methods in practice. The first half of the course will be focused on technical foundations, basic algorithms, implementation, and theory (see below for topics). The second half of the course will explore special topics from recent literature, such as high-dimensional optimization, latent-space optimization, grey-box optimization, and/or mixed-feature spaces (see below for potential topics).
Learning objectives. If we're successful, you will:
Prerequisites. If you are worried that you won't satisfy the following prerequisites, please speak to me after the first class:
Values. This course will be safe and inclusive learning environment for all. I expect everyone to abide by the code of conduct. If you are treated unfairly or disrespectfully—whether by another student or myself—please (1) contact me directly (if you feel comfortable), (2) submit an anonymous feedback form, or (3) work with the UBC ombudsperson office. I also aim to accommodate neurodiversity, mental health, and access needs within the course. Please contact me for requests with special accommodation requests, or take advantage of UBC's student resources.
About me. I am a new assistant professor in the stats department. My research generally encompasses Gaussian processes/spatiotemporal modeling, Bayesian optimization, and "reliable" deep learning.
(This is my first course, so please be nice!)
Office hours will be held in my office (ESB 3142):
I'll also stick around to chat for ~10 minutes after class on Tuesdays.
How to contact me. Send me a message through Canvas, the UBC Stats Slack (preferred, if you have access), or email (only if you absolutely need to). Please note that I will not respond on weekends or after 6PM on weekdays.
You will learn through the following activities:
Class is where we will cover most of the technical content in this course. We will also have lively discussions on cool applications, current limitations, etc.
The textbook chapters will help to solidify the technical foundations that I will cover in lecture.
The research papers will give you a flavor of the current limitations/possibilities of Bayesian optimization.
There will be a single problem set on Gaussian distributions that will be due at the end of the first week of class. It will ensure that you are comfortable with the properties of Gaussian distributions, which will be at the center of all the technical content we will discuss throughout this course. If you have experience with Bayesian statistics, this problem set will (hopefully) be a nice review. If you struggle with this problem set, you may have difficulty with the rest of the material in this course.
(I realize that it's tough having an assignment due within the first week. I will release the problem set before the first of class to give you a head start. I will also provide extensions with no late penalty, provided that you contact me first.)
You will have a short coding exercise to help get you familiar with AX and/or BoTorch, the predominant software libraries for Bayesian optimization application and research. The exercise will ensure that you can apply what you learn in class to real-world problems.
This option is best for students who (1) want to apply Bayesian optimization to a research project/application area, or (2) are interested in Bayesian optimization methodology/theory research. Expect about 4 weeks to start and complete a project. You may work alone or in pairs, and you should contact me before selecting a topic.
Double dipping is encouraged! (If Bayesian optimization fits into your existing research or another class project, then please use that research/course project as your final for this course!) The deliverables for this option are:
| Deliverable | Date |
|---|---|
| Project proposal. | 9 Nov 2023. |
| 15-minute final presentation. | During exams period (after 7 Dec 2023). Exact date TBD. |
| Code repository and/or written proofs. | During exams period (after 7 Dec 2023). Exact date TBD. |
Note that there is no final report due. However, if your project is theoretical in nature, you should expect to turn in a document with your primary theorems/proofs. More information to follow!
For students who don't want to complete a research project, or who simply want to learn as much about BO as possible, you instead have the option of a final assessment geared more towards learning about current research topics.
In the second half of the course, each class will focus on 1-2 current BO papers. 1-2 students (who choose this final assessment option) will lead the class, giving a 30-60 minute presentation on the selected papers. All other students (except those doing final projects) will be responsible for writing a review/reflection on one of the papers before the start of each class.
| Deliverable | Date |
|---|---|
| One 30-60 minute presentation on 1-2 current BO papers. | Between Nov 14 to Dec 7. |
| 6-7 reflections/reviews on current BO papers. | Every class from Nov 14 to Dec 7. |
| An additional coding exercise. | TBD. |
More information to follow!
Dates. 20 Oct 2023 — 12 or 14 Nov 2023.
Class format. I will give lectures on foundational material. There will be assigned readings from the textbook, which should ideally be completed before each class. I trust that you will complete the readings on whatever timeline supports your schedule and learning style. Please note that if you fall behind in the readings you may have trouble engaging with the research papers covered in the second half of the course.
Assignments. Beyond readings, there will also be (1) the diagnostic problem set and (2) a short coding exercise. Additionally, those of you choosing the research project option for final assessment will submit your project proposal.
Topics will include:
Dates. 14 or 19 Nov 2023 — 7 Dec 2023.
Class format. For the second half of the course, the classroom will be flipped. We will be discussing 1-2 current research papers each class. Each student (who isn't choosing the research project option for their final assessment) will be responsible for leading at least one of the topics classes. The class will consist of 30-minute presentations on one of the papers, followed by 15 minute discussions. (See above for more details.)
(Details and scheduling for this portion of the class will be ironed out after everyone chooses their final assessment preference.)
Assignments. For the students who aren't leading a particular class, you will be responsible for reading the papers before the class starts. Furthermore, you will also have to write a conference/journal-style review of at least one of the papers presented each class. (Students who choose the research project option will be exempt from writing reviews.)
Topics will include:
Specific paper recommendations for each topic will be posted soon!
| Category | Contribution | Notes |
|---|---|---|
| Class participation | 5% | |
| Diagnostic problem set | 15% | See description above. |
| Short coding exercise | 20% | See description above. |
| Final assessment | 60% | See description above. Specific grading breakdown TBD. |
Rubrics will be released before each assignment is due.
Please contact me as soon as possible if you will have trouble completing an assignment on time and/or participating in class. I understand the effects of life events, health circumstances (mental or physical), and scheduling, and I will work with you to best structure your learning/assessment. Please also refer to UBC's policies on academic concession..
We will be following the brand-new textbook Bayesian Optimization written by Roman Garnett. It is available as a free PDF online, or (for those of you who are old-school) available as a hardcover book from Amazon.
We will use Canvas for 1) announcements, 2) discussions, and 3) uploading assignments. Please post all non-personal questions as a discussion—your fellow classmates might be able to answer them for you!
I will occasionally present small toy demonstrations during class. They will be made available on this Github repository.
UBC provides resources to support student learning and to maintain healthy lifestyles but recognizes that sometimes crises arise and so there are additional resources to access including those for survivors of sexual violence. UBC values respect for the person and ideas of all members of the academic community. Harassment and discrimination are not tolerated nor is suppression of academic freedom. UBC provides appropriate accommodation for students with disabilities and for religious, spiritual and cultural observances. UBC values academic honesty and students are expected to acknowledge the ideas generated by others and to uphold the highest academic standards in all of their actions. Details of the policies and how to access support are available here.