Deep Learning
Deep Learning Using PyTorch
Index:
Course Overview:
In this course, you will learn the foundations of Deep Learning, understand how to build
neural networks, and learn how to lead successful machine learning projects. You will learn
about Convolutional networks, RNNs, LSTM, Adam, Dropout, BatchNorm, and more.
Main TextBooks:
Slides and Papers:
Recommended Slides & Papers:
-
Introduction (1 Session)
- Video of lecture by Ian Goodfellow and discussion of Chapter 1 at a reading group in San Francisco organized by Alena Kruchkova
- Paper: On the Origin of Deep Learning by Haohan Wang and Bhiksha Raj
Applied Mathematics and Machine Learning Basics:
-
-
- Blog: Learning PyTorch with Exampls by Justin Johnson.
This tutorial introduces the fundamental concepts of PyTorch through self-contained examples.
Building Dynamic Models Using the Subclassing API:
-
Deep Feedforward Networks (6 Sessions)
-
-
Regularization for Deep Learning (5 Sessions)
-
-
Optimization for Training Deep Models (5 Sessions)
- Lecture Note: Matrix Norms and Condition Numbers by Ralucca Gera
- Blog: Initializing Neural Networks by Katanforoosh & Kunin, deeplearning.ai, 2019
- Blog: How to Initialize Deep Neural Networks? Xavier and Kaiming Initialization by Pierre Ouannes
- Blog: What Is Covariate Shift? by Saeed Izadi
- Book: Dataset Shift in Machine Learning by by Joaquin Quinonero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D. Lawrence
- Blog: Stay Hungry, Stay Foolish: This interesting blog contains the computation of back propagation of different layers of deep learning prepared by Aditya Agrawal
- Paper: FitNets: Hints for Thin Deep Nets by Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio
- Paper: Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization by Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar
-
Convolutional Networks (3 Sessions)
-
-
Sequence Modeling: Recurrent and Recursive Networks (4 Sessions)
-
- Deep Learning via Pytorch by Ayoosh Kathuria
-
Practical Methodology (2 Sessions)
-
Ray provides a simple, universal API for building distributed applications:
Optuna is an automatic hyperparameter optimization software framework, particularly designed for machine learning:
-
Applications (1 Session)
-
Autoencoders (2 Sessions)
-
Generative Adversarial Networks (1 Session)
Slide: Generative Adversarial Networks (GANs) by Binglin, Shashank, and Bhargav
Paper: NIPS 2016 Tutorial: Generative Adversarial Networks by Ian Goodfellow
-
Graph Neural Networks (1 Session)
-
Additional Resources:
Class Time and Location:
Saturday and Monday 10:30-12:00 AM (Fall 2020)
Recitation and Assignments:
Tuesday 16:00-18:00 PM (Fall 2020),
Refer to the following link to check the assignments.
Projects:
Projects are programming assignments that cover the topic of this course. Any project is written by Jupyter Notebook. Projects will require the use of Python 3.7, as well as additional Python libraries.
Google Colab:
Google Colab is a free cloud service and it supports free GPU!
Fascinating Guides For Machine Learning:
Latex:
The students can include mathematical notation within markdown cells using LaTeX in their Jupyter Notebooks.
- A Brief Introduction to LaTeX PDF
- Math in LaTeX PDF
- Sample Document PDF
- TikZ: A collection Latex files of PGF/TikZ figures (including various neural networks) by Petar Veličković.
Grading:
- Projects and Midterm – 50%
- Endterm – 50%
ُThree Exams:
- First Midterm Examination: Saturday 1399/09/01, 10:30-12:00
- Second Midterm Examination: Saturday 1399/10/06, 10:30-12:00
- Final Examination: Wednesday 1399/10/24, 14:00-16:00
Prerequisites:
General mathematical sophistication; and a solid understanding of Algorithms, Linear Algebra, and
Probability Theory, at the advanced undergraduate or beginning graduate level, or equivalent.
Linear Algebra:
Probability and Statistics:
Topics:
Have a look at some reports of Kaggle or Stanford students (CS224N, CS224D) to get some general inspiration.
Account:
It is necessary to have a GitHub account to share your projects. It offers
plans for both private repositories and free accounts. Github is like the hammer in your toolbox,
therefore, you need to have it!
Academic Honor Code:
Honesty and integrity are vital elements of the academic works. All your submitted assignments must be entirely your own (or your own group’s).
We will follow the standard of Department of Mathematical Sciences approach:
- You can get help, but you MUST acknowledge the help on the work you hand in
- Failure to acknowledge your sources is a violation of the Honor Code
- You can talk to others about the algorithm(s) to be used to solve a homework problem; as long as you then mention their name(s) on the work you submit
- You should not use code of others or be looking at code of others when you write your own: You can talk to people but have to write your own solution/code
Questions?
I will be having office hours for this course on Saturday (09:00 AM–10:00 AM). If this is not convenient, email me at hhaji@sbu.ac.ir or talk to me after class.