DL Fundamentals & Optimization Algorithms

October 8, 2026 deep-learning

Handwritten notes on deep learning fundamentals: neural network architectures, activation functions, backpropagation, and optimization algorithms (SGD, Adam, RMSProp).

These notes cover the foundational building blocks of deep learning and the optimizers that train them:

Neural Network Architecture: Perceptrons, multi-layer networks, forward pass, and universal approximation.
Activation Functions: Sigmoid, tanh, ReLU, Leaky ReLU, GELU — properties, gradients, and when to use each.
Backpropagation: Computational graphs, chain rule, gradient flow, and vanishing/exploding gradients.
Optimization Algorithms: SGD, momentum, RMSProp, Adam, learning rate scheduling, and convergence properties.