EE787: Fundamentals of Machine Learning

Course Info.

Course descriptions

• Fundamental concepts and theories in machine learning, supervised and unsupervised learning, regression and classification, loss function selection and its effect on learning, regularization and robustness to outliers, numerical experiments on data from a wide variety of engineering and other discplines.

Lectures

• Tue/Thr 13:30-14:45 (Rm.211-1)

Office hours

• Tue/Thr 15:00-16:00 (Rm.516), or by appointments if you cannot meet them.

Prerequisites

• Previous exposure to linear algebra, probability, and programming.

• Working knowledge on optimization will be a plus.

Reference textbooks

• There are no required textbooks.

• Introduction to applied linear algebra - vectors, matrices, and least squares by Boyd and Vandenberghe will be a useful reference.

Grading policy

• No exams.

• Students will be evaluated by their homework assignments.

Lecture notes

The course material is reproduced from the EE104: Introduction to machine learning by Sanjay Lall and Stephen Boyd at Stanford university, under their kind permission.

1. Course overview

2. Supervised learning via empirical risk minimization

3. Least squares linear regression

4. Validation

5. Features

6. Regularization

7. House prices example

8. Non-quadratic losses

9. Non-quadratic regularizers

10. Optimization

11. Prox-gradient method

12. Boolean classification

13. Multi-class classification

14. Neural networks

15. Unsupervised learning

16. Principal component analysis

Assignments

Several sets of occasional homeworks will be assigned. You are encouraged to work in groups, however everyone should turn in his/her own work.

1. Homework 1 (due 10/4): nearest_neighbor_data.json, fitting_outliers.json, all_pairs_data.json

2. Homework 2 (due 10/18): inductor_data.json, rational_fit_data.json, prostate_cancer_data.json, to_one_hot.jl

3. Homework 3 (due 11/15): power_demand_data.json

4. Homework 4 (due 11/27): tomodata_fullysampled.json, tomodata_undersampled.json, line_pixel_length.jl, TV_inpainting.ipynb

5. Homework 5 (optional): homework_scores.csv

Julia

Julia language

• We will be using Julia, which excels in high performance technical computing, for homework assignments.

• You are not expected to have a strong background in programming (with Julia or otherwise), because the program you will write will use only a tiny subset of Julia's (many and powerful) features.

Reference webpages

• The official Julia webpage

• JuliaBox: an easy-to-use online platform of Julia

• A variety of Julia product packages including JuliaPro

• A short Julia tutorial from EE103: Introduction to Matrix Methods at Stanford university.

• Julia language companion by Boyd and Vandenberghe.

Files

These are some data and the Julia codes in .ipynb notebook files that we are using for lectures or homework assignments. To run .ipynb files, first download the code by right-clicking “code” and clicking “Save link as…”. Then, upload the file to JuliaBox with the “upload” button under the “Jupyter” tab.

1. Julia example: KHU logo was a guitar pick. (khulogo.jpg, khu_logo_compression.ipynb)

2. Julia example: Huber-based dynamical system estimation. (huber-based_dynamical_system_estimation.ipynb)

3. JHK is safe from diabetes. (diabetes.data, diabetes.ipynb)

4. Polynomial fit. (polynomial_fit.ipynb)

5. Monomial-exponential fit. (monomial-exponential_fit.ipynb)

6. .json file read example. (readclassjson.jl)

7. Different norms. (Different_norms.ipynb)

8. Sound signal recovery. (Sound_signal_recovery.ipynb)

9. TV_inpainting. (neo_bw.png, neo_bw_corrupted.png, corrupted_image.json, TV_inpainting.ipynb)

10. Network topology identification. (Network_topology.ipynb)

11. Least squares classifier, logistic regression, and support vector machine. (ls_lr_svm.ipynb)

12. Iris classification. (iris.csv, iris.ipynb)

13. Handwriting image classification. (mnist_train.csv, mnist_test.csv, handwriting.ipynb)