Neural Style Transfer from Scratch
A minimal PyTorch implementation of Gatys et al. optimisation-based neural style transfer: feature extraction with frozen VGG-19, Gram-matrix style loss, and direct pixel gradient descent.
Timeline
Jan 2025
Role
Solo — implementation & experimentation
Status
CompletedTechnology Stack
Key Challenges
- Optimizing the image pixel tensor directly rather than neural network weights
- Balancing content and style loss magnitudes to preserve structure while applying texture
- Selecting optimal feature tap points across deep and shallow convolutional layers
Key Learnings
- Gram matrices capture spatial texture correlations while discarding absolute positioning
- Freezing pretrained network backbones provides robust feature extractors for zero-shot tasks
- Image tensor denormalization is required before disk export
Overview
Neural Style Transfer (Gatys et al., 2015) demonstrates that convolutional neural networks represent content structure and artistic style in separate feature spaces.
Instead of training network parameters, the network remains frozen and gradient descent optimizes the image pixels directly to minimize a composite loss function.
How It Works
1. Multi-Scale Feature Extraction
A pretrained VGG-19 model extracts activations across 5 strategic stages (conv1_1, conv2_1, conv3_1, conv4_1, conv5_1). Shallow layers capture edges and localized color palettes, while deeper layers preserve structural semantics.
2. Dual Loss Formulation
- Content Loss: Mean Squared Error (MSE) between feature activations of the content image and target image in deep layers (
conv4_2/conv5_1). - Style Loss: MSE between the Gram matrices of the style reference and target image across all tapped layers. The Gram matrix () computes inner products of flattened feature channels, capturing texture correlations independent of spatial positioning.
3. Optimization Loop
The target image is initialized (either as random noise or a content clone) and iteratively refined using the Adam or L-BFGS optimizer over 1,000–2,000 steps.
Tech Stack Summary
- Framework: PyTorch, Torchvision
- Model: Pretrained VGG-19
- Core Concepts: Gram Matrices, Feature Reconstruction, Pixel-Level Gradient Descent