Overview
This project traces the evolution of convolutional neural network architectures from the original LeNet-5 (1998) to the deep residual networks that transformed the field in 2015. Each architecture was built from scratch in PyTorch without relying on pretrained weights, trained and evaluated on FashionMNIST.
Part 1 — LeNet and Modernized LeNet
Original LeNet-5
Implemented the original LeCun (1998) architecture with Sigmoid activations and average pooling, trained on both MNIST and FashionMNIST across 5, 10, and 20 epoch configurations.
Modernized LeNet
Upgraded the original architecture by replacing Sigmoid with ReLU activations and swapping average pooling for max pooling — both standard practices in modern CNNs that improve gradient flow and feature selection.
Key changes: Sigmoid → ReLU · AvgPool → MaxPool · faster convergence · better accuracy on FashionMNIST
Part 2 — Batch Normalization from Scratch
Implemented batch normalization manually in PyTorch — computing per-batch mean and variance during training, switching to running statistics during inference, with learnable scale (γ) and shift (β) parameters.
Three normalization variants implemented and compared:
| Variant | Formula | Effect |
|---|---|---|
full | Normalize by mean and variance | Standard batch norm |
mean_only | Subtract mean, skip variance | Faster, less stable |
variance_only | Divide by std, skip mean | Partial normalization |
Ablation Study
Tested selective batch normalization — enabling/disabling BN at individual layers using a bn_layers=[True, True, True, True] flag — to isolate which layers benefit most from normalization.
| Configuration | BN Layers Active | Observation |
|---|---|---|
| Full BN | All conv + FC layers | Best convergence speed |
| Conv only | Conv layers only | Good stability, slower FC |
| No BN (baseline) | None | Slower, less stable training |
Part 3 — ResNet-18, ResNet-34, ResNet-50
Implemented the full ResNet family with two residual block types — BasicResidual for ResNet-18/34 and ResidualBottleneck for ResNet-50 — matching the original He et al. (2015) paper specifications.
BasicResidual Block (ResNet-18 / 34)
1×1 projection shortcut when channels change or stride > 1
Bottleneck Block (ResNet-50)
mid_channels = out_channels // 4 — reduces computation while preserving capacity
Model Configurations
| Model | Block Type | Layer Config | Parameters |
|---|---|---|---|
ResNet-18 | BasicResidual | [2, 2, 2, 2] | ~11M |
ResNet-34 | BasicResidual | [3, 4, 6, 2] | ~21M |
ResNet-50 | Bottleneck | [3, 4, 6, 3] | ~25M |
Training Results
ResNet-18 — Epoch-by-Epoch
| Epoch | Train Loss | Test Accuracy |
|---|---|---|
| 1 | 0.6332 | 83.75% |
| 2 | 0.3605 | 87.45% |
| 3 | 0.3079 | 88.28% |
| 5 | 0.2683 | 89.72% |
| 7 | 0.2364 | 90.37% |
| 10 | 0.2081 | 90.08% |
ResNet-34 — Epoch-by-Epoch
| Epoch | Train Loss | Test Accuracy |
|---|---|---|
| 1 | 0.9302 | 82.05% |
| 3 | 0.3490 | 87.27% |
| 6 | 0.2709 | 89.10% |
| 9 | 0.2333 | 89.70% |
| 10 | 0.2327 | 89.66% |
ResNet-50 — Epoch-by-Epoch
| Epoch | Train Loss | Test Accuracy |
|---|---|---|
| 1 | 0.7961 | 79.40% |
| 4 | 0.4032 | 81.76% |
| 6 | 0.3434 | 85.70% |
| 8 | 0.3145 | 88.27% |
| 10 | 0.2756 | 87.45% |
Key Observations
ResNet-18 achieved the highest peak accuracy (90.37%) despite having fewer parameters than ResNet-34 and ResNet-50 — suggesting the shallower network generalizes better on FashionMNIST's relatively simple patterns. ResNet-50's bottleneck design shows slower initial convergence but stabilizes well by epoch 8. All three models show consistent loss decrease without overfitting, validating the residual connections.
