Deep Learning is a subfield of machine learning mostly concerned with concepts and techniques built on top of the artificial neural network which in turn was inspired by the structure and functionality of human brains. This branch of machine learning is increasingly gaining popularity as deep learning systems are taking over all artificial intelligent tasks, ranging from image classification, language modeling, machine translation, playing games, autonomous vehicle driving, speech recognition, cancer detection and numerous other applications and is behind many recent advances in Artificial Intelligence (AI). In this course you will gain both theoretical and practical knowledge of deep learning concepts and techniques. So, welcome aboard!

Course objectives

By the end of the course you are expected to gain the following skills:

1. Understand fundamentals of artificial neural network, and deep neural networks.
2. Develop an understanding on how to train a neural network.
3. Determine how a deep neural network can be designed, and implemented to solve real world problems.
4. Demonstrate an in-depth understanding of one/more concepts introduced in the deep learning course through a final project.
5. Understand and apply the principles of responsible AI system design: fairness, inclusiveness, transparency, reliability and safety, privacy and security, accountability, limits of capabilities.

Recommended background

1. Good understanding of Calculus,
2. Good understanding of Linear Algebra,
3. Good understanding of Statistics and Probability theories,
4. Good Programming skills (in either Python, or Julia or C++),
5. Machine learning

Required Textbooks

None

Recommended Textbooks

1. Bishop, Chris and Bishop, Hugh (2024). Deep Learning – foundations and concepts. [official-web + book reader]. Springer.
2. Goodfellow, Ian and Bengio, Yoshua and Courville, Aaron (2016). Deep learning. [official-web + book reader] MIT press Cambridge.
3. Zhang, Aston and Lipton, Zachary C. and Li, Mu and Smola, Alexander J. (2020). Dive into Deep Learning. [official-web + book reader]

Other online resources

Besides the textbooks listed above the instructor may refer you to review online resources as the course proceeds.

Schedule

Module 1 : Machine Learning Preliminaries and setting up compute instances

1. Introduction to Machine learning en route to Deep Learning [ PDF ] [ Video]
2. Getting you ready to upcoming compute challenges throughout the course. Get familiar with compute resources you can use [Video]
3. Nuts and Bolts of Machine Learning (part 1): The types of learning [ PDF ][ Video]
4. Nuts and Bolts of Machine Learning (part 2): The task and the datasets [ PDF ] [ Video]
5. Nuts and Bolts of Machine Learning (part 3): How to tell the machine learning model is good? [ PDF ] [ Video]
6. Evaluating Regression models [ PDF, slide 50 ] [ Video]
7. Another look at evaluating linear regression models [Slide PDF] [ Video]
8. Bias-variance tradeoffs, Overfitting, Underfitting [Slide PDF] [ Video]
9. Dataset splits: training, testing, validation, cross-validation [Slide PDF] [ Video]
10. Label encoding, One-hot encoding [Slide PDF] [Video]
11. Nuts and Bolts of Machine Learning (part 4): Exploratory Data analysis – a case study: House price prediction [ zip ] [ Video]

Module 2 : Introduction to Deep Learning with Artificial Neural Networks

1. Now, let’s begin: Introduction to Artificial neural networks (ANN) and deep learning [ PDF ] [Video]
2. Artificial Neural Networks – the basics [ PDF ], [ Jupyter notebook ]
   • The preliminaries [notebook][ Video]
   • The math, the backpropgation algorithm and the implementation [notebook][Video]
3. A spelled out introduction to artificial neural networks and backpropagation – building a micrograd. A lecture by Andrej Karpathy [Video]
   • Git repo [https://github.com/karpathy/micrograd.git]
   • Jupyter notebook [https://github.com/karpathy/nn-zero-to-hero/tree/master/lectures/micrograd]

Module 3: Another introduction to artificial neural network with compute frameworks – Tensorflow

1. Compute framework for Artificial Neural Network + Deep Learning [ PDF ] [Video Recording]
2. Tensorflow 2.0 [ PDF ] [Video]
3. Introduction to Tensors [Jupyter Notebook] [Video]
4. Backpropagation with Tensorflow 2.0 [Jupyter Notebook] [Video Recording]
5. Tensorflow 2.0 with Tensorboard [ Getting started guide ] [ Deep Dive Into Tensorboard ] [ Talk ]

Module 4: Another introduction to artificial neural network with compute frameworks – PyTorch, etc.

1. Introduction to PyTorch [This is a 2 part video lecture:]
   • A short lecture by Andrej Karpathy, former student of Prof. Fei Fei Li who works in the AI / Auto-pilot team at Tesla. Here he explains how PyTorch was used to develop the model. [Lecture video]
   • Introduction to working with PyTorch – another deep learning compute framework and is not Tensorflow. [Jupyter Notebook] [Lecture recording]
   • Neural nets with PyTorch [Lecture notebook] [Video Lecture]
2. Special topic on ANN - 1: Batch normalization [Slides] [Video Recording]
3. Special topic on ANN - 2: Dropouts [Slides] [Video Recording]
4. Regularization [Slides+Codes] [Video Recording]
5. Digging deep into Deep Learning lecture slides used on 9/5/2023 during classroom https://docs.google.com/presentation/d/1kTXA88A39kPkqwhOiJGaxGmvn5aTAkg5/edit?usp=share_link&ouid=111045799100246528781&rtpof=true&sd=true

Module 5: Introduction to Convolution Neural Networks with compute frameworks

1. Introduction to Convolution Neural Network [ Notebooks ][Video Recording]
2. Convolution Operation, 1x1 conv, implementation aspects [Notebooks] [Video Recording]
3. The pooling layer [Notebook] [Video Recording]
4. Flattened layer and fully connected layer [Notebooks] [Video Recording]
5. Few examples [Notebooks] [Video Recording]
6. Comments on weight sharing property [Notebooks] [[Video Recording](https://youtu.be/rjwEVbtPjeQ]

Module 6: Advanced Convolution Neural Network Models / architectures

1. CNN architectures – [ Lecture slide ] [ Video Lecture]
2. ImageNet – what is it, and where is it going? [Lecture slides] [Video Lecture by Dr. Fei Fei Li]
3. What are ResNets? This is a 2 -part lecture: First a talk by Kaiming He, the inventor of ResNet , Second, my lecture to recap ResNet
   • A talk by Kaiming He at CVPR 2016 conference on “Deep Residual Learning for Image Recognition” [Lecture slides] [Video Lecture]
   • My talk to recap ResNet [Lecture slides] [Video Lecture]
4. CNNs with PyTorch [Lecture notebook] [Video Lecture]
5. Visualizing CNN components [Lecture notebook] [Video Lecture]
6. Let’s summarize CNN (once and for all!) [Lecture slides] [Video Lecture]

Module 7 – Recurrent Neural Networks and language modeling basics

1. Introduction to Recurrent Neural Networks [Notebooks] [Video Recording]
2. Basic language modeling with Recurrent Neural Networks [Notebooks] [Video Recording]
3. Vanilla architecture of RNN [Notebooks] [Video Recording]
4. Long short term memory (LSTM) architecture of RNN [Notebooks] [[Video Recording](https://youtu.be/lJRLhCeh0Xk]

Module 8 – Reserved for midterm

1. Midterm (online)

Module 9 : Transformer is all you need and another introduction to language modeling

1. Transformers [ link to required study materials]
2. Natural Language Processing (Part 1)
   • Bag of words model [Slides]
   • Word Embedding [Slides]
3. Unsupervised Deep Learning [Slides/supporting materials] [Video lecture]
4. Autoencoders [Slides/supporting materials] [Video lecture]

Module 10 : More on transformers and large language models

1. Transfer learning revisited [Lecture slides] [Lecture notebook] [Video Lecture]
   • A thorough example of PyTorch transfer learning PyTorch Transfer learning at Zero to Mastery Learn PyTorch for Deep Learning book by Daniel Bourke
2. LLMs – a lecture by Andrej Karpathy: Slides pdf format, key format, Video recording,
3. Customizing existing LLMs Fine-tuning a pre-trained model

Module 11 : Image Segmentation and Generative Adversarial networks

1. Image Segmentation [Slides/supporting materials] [Video lecture]
2. Generative Adversarial Nets (GANs) [Slides, supporting materials] [Video lecture p1] [Video lecture p2]

Module 12 : Reinforcement Learning and deep reinforcement learning

1. Deep Reinforcement Learning
   • Lecture 1 [Slide1, Code: QL-01.py] [Video1]
   • Lecture 2 (by David Silver) [Slide2] [Video2]

Module 13 : More about reinforcement learning and how useful it is in other learning paradigms

1. Module 12 repeated… or study more on the topics.

Module 14 : Wrapping up the course

1. I am wondering you want to become a data scientists [ PDF ] [Video-Recording]
2. Challenges with Machine learning system design [ PDF ] [Video-Recording]
3. Stories of Machine Learning [ PDF ][ Video-Recording]
4. What was covered and where to go from there

Module 15: Reserved for final exam

1. Final exam (online)