# (Fall’23): CSCI-4931 Deep Learning

Undergraduate course, *SCIENCE-2001 (Tue only)*, 2023

Deep Learning is a subfield of machine learning mostly concerned with concepts and techniques built on top of the artificial neural network which in turn was inspired by the structure and functionality of human brains. This branch of machine learning is increasingly gaining popularity as deep learning systems are taking over all artificial intelligent tasks, ranging from image classification, language modeling, machine translation, playing games, autonomous vehicle driving, speech recognition, cancer detection and numerous other applications and is behind many recent advances in Artificial Intelligence (AI). In this course you will gain both theoretical and practical knowledge of deep learning concepts and techniques. So, welcome aboard!

# Course objectives

By the end of the course you are expected to gain the following skills:

- Understand fundamentals of artificial neural network, and deep neural networks.
- Develop an understanding on how to train a neural network.
- Determine how a deep neural network can be designed, and implemented to solve real world problems.
- Demonstrate an in-depth understanding of one/more concepts introduced in the deep learning course through a final project.
- Understand and apply the principles of
`responsible AI`

system design:`fairness`

,`inclusiveness`

,`transparency`

,`reliability and safety`

,`privacy and security`

,`accountability`

,`limits of capabilities`

.

# Prerequisites

- MATH-3195 (Linear algebra and differential equations) or equivalent,
- CSCI-3412 (Algorithms) or equivalent.

# Recommended Textbooks

- Goodfellow, Ian, Yoshua Bengio, Aaron Courville, and Yoshua Bengio (2016). Deep learning. [official-web] MIT press Cambridge.
- Zhang, Aston, Zachary C. Lipton, Mu Li, and Alexander J. Smola (2020). Dive into Deep Learning. [official-web]

# Schedule

### Week 1 [8/21/2023 – 8/25/2023]

- Introduction to Machine learning en route to Deep Learning [ PDF ] [ Video-Recording, 19:46]
- I am wondering you want to become a data scientists [ PDF ] [Video-Recording, 17:00]
- Challenges with Machine learning system design [ PDF ] [Video-Recording, 26:09]
- Stories of Machine Learning [ PDF ][ Video-Recording, 58:06]
- Nuts and Bolts of Machine Learning (part 1): The types of learning [ PDF ][ Video-Recording, 34:31]
- Nuts and Bolts of Machine Learning (part 2): The task and the datasets [ PDF ] [ Video-Recording, 32:37 ]
- Nuts and Bolts of Machine Learning (part 3): How to tell the machine learning model is good? [ PDF ] [ Video-Recording, 52:21 ]

### Week 2 [8/28/2023 – 9/1/2023]

- Evaluating Regression models [ PDF, slide 50 ] [ Video-Recording, 13:05 ]
- Another look at evaluating linear regression models [Slide PDF] [ Video-Recording, 28:41]
- Bias-variance tradeoffs, Overfitting, Underfitting [Slide PDF] [ Video-Recording, 26:50 ]
- Dataset splits: training, testing, validation, cross-validation [Slide PDF] [ Video-Recording, 16:23 ]
- Label encoding, One-hot encoding [Slide PDF] [Video_Recording, 10:57]
- Nuts and Bolts of Machine Learning (part 4): Exploratory Data analysis – a case study: House price prediction [ zip ] [ Video-Recording, 23:02 ]

### Week 3 and 4 [9/4/2023 – 9/15/2023]

- Now, let’s begin: Introduction to Artificial neural networks (ANN) and deep learning [ PDF ] [Video Recording, 24:19]
- Artificial Neural Networks – the basics [ PDF ], [ Jupyter notebook ]
**The preliminaries**[ Video-Recording, 44:15 ]**The math, the backpropgation algorithm and the implementation**[Video-recording, 33:55]

- Compute framework for Artificial Neural Network + Deep Learning [ PDF ] [Video Recording, 01:00:03]
- Tensorflow 2.0 [ PDF ] [Video Recording, 32:56]
- Introduction to Tensors [Jupyter Notebook] [Video Recording, 01:02:38]
- Backpropagation with Tensorflow 2.0 [Jupyter Notebook] [Video Recording, 01:08:55]
- Batch normalization [Slides] [Video Recording, 16:05]
- Dropouts [Slides] [Video Recording, 17:15 ]
- Regularization [Slides+Codes] [Video Recording, 14:54]
- Digging deep into Deep Learning lecture slides used on 9/5/2023 during classroom https://docs.google.com/presentation/d/1kTXA88A39kPkqwhOiJGaxGmvn5aTAkg5/edit?usp=share_link&ouid=111045799100246528781&rtpof=true&sd=true

### Week 5 [09/18/2023 – 09/22/2023]

- Introduction to Convolution Neural Network [ Notebooks ][Video Recording, 30:52]
- Convolution Operation, 1x1 conv, implementation aspects [Notebooks] [Video Recording, 01:27:10]
- The pooling layer [Notebook] [Video Recording, 11:11]
- Flattened layer and fully connected layer [Notebooks] [Video Recording, 14:08]
- Few examples [Notebooks] [Video Recording, 14:16]
- Epilog + comment on weight sharing property [Notebooks] [Video Recording, 09:02]

### Week 6 and 7 [09/25/2023 – 10/6/2023]

- Introduction to Recurrent Neural Networks [Notebooks] [Video Recording, 38:20]
- Basic language modeling with Recurrent Neural Networks [Notebooks] [Video Recording, 29:15]
- Vanilla architecture of RNN [Notebooks] [Video Recording, 58:49]
- Long short term memory (LSTM) architecture of RNN [Notebooks] [Video Recording, 44:17]

### Week 8 [10/9/2023 – 10/13/2023]

- Reserved for midterm

### Week 9 [10/16/2023 – 10/20/2023]

- Tensorflow 2.0 with Tensorboard [ Getting started guide ] [ Deep Dive Into Tensorboard ] [ Talk ]
- CNN architectures – [ Lecture slide ] [ Video Lecture]
- ImageNet – what is it, and where is it going? [Lecture slides] [Video Lecture by Dr. Fei Fei Li]
- What are ResNets?
**This is a 2 -part lecture: First a talk by Kaiming He, the inventor of ResNet , Second, my lecture to recap ResNet**- A talk by Kaiming He at CVPR 2016 conference on “Deep Residual Learning for Image Recognition” [Lecture slides] [Video Lecture]
- My talk to recap ResNet [Lecture slides] [Video Lecture]

- Introduction to PyTorch [
**This is a 2 part video lecture:**]- A short lecture by Andrej Karpathy, former student of Prof. Fei Fei Li who works in the AI / Auto-pilot team at Tesla. Here he explains how PyTorch was used to develop the model. [Lecture video]
- Introduction to working with PyTorch – another deep learning compute framework and is not Tensorflow. [Jupyter Notebook] [Lecture recording]
- Neural nets with PyTorch [Lecture notebook] [Video Lecture]
- CNNs with PyTorch [Lecture notebook] [Video Lecture]
- Visualizing CNN components [Lecture notebook] [Video Lecture]

- Let’s summarize CNN (once and for all!) [Lecture slides] [Video Lecture]

### Week 10 [10/23/2023 – 10/27/2023]

- Transfer learning revisited [Lecture slides] [Lecture notebook] [Video Lecture]
- A thorough example of PyTorch transfer learning PyTorch Transfer learning at Zero to Mastery Learn PyTorch for Deep Learning book by Daniel Bourke

- Generative Adversarial Nets (GANs) [Slides, supporting materials] [Video lecture p1] [Video lecture p2]

### Week 11 [10/30/2023 – 11/3/2023]

- Transformers [ link to required study materials]
- Natural Language Processing (Part 1)
- Unsupervised Deep Learning [Slides/supporting materials] [Video lecture]
- Autoencoders [Slides/supporting materials] [Video lecture]
- Image Segmentation [Slides/supporting materials] [Video lecture]
- Generative Adversarial Nets (GANs) [Slides, supporting materials] [Video lecture p1] [Video lecture p2]