Theory Guide - Neural Networks Suite

Neural Network Fundamentals

The Perceptron

The foundation of neural networks begins with the perceptron, a simple linear classifier that forms the basis of modern deep learning.

y = σ(Wx + b) where: - W: weight matrix - x: input vector - b: bias term - σ: activation function

Activation Functions

Non-linear activation functions enable neural networks to learn complex patterns:

ReLU: f(x) = max(0, x)
Sigmoid: f(x) = 1 / (1 + e^(-x))
Tanh: f(x) = (e^x - e^(-x)) / (e^x + e^(-x))
GELU: f(x) = x * Φ(x)
Swish: f(x) = x * sigmoid(x)

Backpropagation

The core algorithm for training neural networks through gradient descent:

∂L/∂W = ∂L/∂y · ∂y/∂W Chain Rule Application: - Forward pass: compute outputs - Backward pass: compute gradients - Update weights: W = W - η∇L

Neural Network Architectures

Convolutional Neural Networks (CNNs)

Specialized for processing grid-like data such as images:

class ConvBlock(nn.Module):
    def __init__(self, in_channels, out_channels):
        super().__init__()
        self.conv = nn.Conv2d(in_channels, out_channels, 3, padding=1)
        self.bn = nn.BatchNorm2d(out_channels)
        self.relu = nn.ReLU(inplace=True)
        
    def forward(self, x):
        return self.relu(self.bn(self.conv(x)))

Vision Transformers (ViT)

Applying transformer architecture to computer vision:

Patch Embedding: Divide image into patches
Position Encoding: Add spatial information
Multi-Head Self-Attention: Global receptive field
MLP Head: Classification layer

Attention(Q, K, V) = softmax(QK^T / √d_k)V

Hybrid Architectures

Combining CNNs and Transformers for optimal performance:

CNN backbone for local feature extraction
Transformer layers for global context
Best of both worlds: efficiency and accuracy

Attention Mechanisms

Channel Attention (SE Blocks)

Squeeze-and-Excitation blocks recalibrate channel-wise feature responses:

1. Squeeze: z_c = F_sq(u_c) = (1/HW) Σ Σ u_c(i,j) 2. Excitation: s = σ(W_2 · ReLU(W_1 · z)) 3. Scale: x̃_c = s_c · u_c

Spatial Attention

Focus on where to attend in the spatial dimensions:

class SpatialAttention(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = nn.Conv2d(2, 1, 7, padding=3)
        self.sigmoid = nn.Sigmoid()
        
    def forward(self, x):
        avg_out = torch.mean(x, dim=1, keepdim=True)
        max_out, _ = torch.max(x, dim=1, keepdim=True)
        x = torch.cat([avg_out, max_out], dim=1)
        x = self.conv(x)
        return self.sigmoid(x)

CBAM (Convolutional Block Attention Module)

Combines both channel and spatial attention mechanisms:

Sequential application of channel and spatial attention
Adaptive feature refinement
Improved feature representation
Minimal computational overhead

Training Theory

Loss Functions

Objective functions for different tasks:

Cross-Entropy: Classification tasks
MSE: Regression tasks
Focal Loss: Class imbalance
Contrastive Loss: Similarity learning

Regularization Techniques

Preventing overfitting and improving generalization:

Regularization Methods

Dropout: Randomly zero activations
Weight Decay: L2 penalty on weights
Data Augmentation: Increase data diversity
Early Stopping: Prevent overtraining
Batch Normalization: Normalize activations

Learning Rate Scheduling

Adaptive learning rate strategies:

Cosine Annealing: η_t = η_min + 0.5(η_max - η_min)(1 + cos(πt/T)) Step Decay: η_t = η_0 × γ^(floor(t/step_size)) One Cycle: Triangular schedule with momentum inverse

Optimization Algorithms

Gradient Descent Variants

# SGD with Momentum
v_t = β × v_{t-1} + η × ∇L
W_t = W_{t-1} - v_t

# Adam Optimizer
m_t = β₁ × m_{t-1} + (1 - β₁) × ∇L
v_t = β₂ × v_{t-1} + (1 - β₂) × (∇L)²
m̂_t = m_t / (1 - β₁^t)
v̂_t = v_t / (1 - β₂^t)
W_t = W_{t-1} - η × m̂_t / (√v̂_t + ε)

Advanced Optimization

AdamW: Decoupled weight decay
RAdam: Rectified Adam
Lookahead: Stabilized optimization
LAMB: Layer-wise adaptive learning

Mixed Precision Training

Accelerate training with automatic mixed precision:

from torch.cuda.amp import autocast, GradScaler

scaler = GradScaler()

with autocast():
    output = model(input)
    loss = criterion(output, target)

scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()