ResNet: The architecture that changed ML forever | Legendary paper with 260k+ citations скачать в хорошем качестве

ResNet: The architecture that changed ML forever | Legendary paper with 260k+ citations

Residual Networks (ResNet): How Microsoft Revolutionized Deep Learning The field of Machine Learning has evolved rapidly over the years, but few innovations have been as pivotal as Residual Networks (ResNet). Introduced by a Microsoft Research team in 2015, ResNet not only refined existing deep learning methods but also introduced a new perspective on how to design neural networks. Today, it remains foundational in state-of-the-art computer vision systems and beyond. The Vanishing Gradient Problem Deep neural networks become more powerful as they add layers, but this increased depth often leads to an issue called the vanishing gradient: What happens? -When networks grow deeper, the gradients (which guide the weight updates) become extremely small for earlier layers. Why is it bad? -These tiny gradients barely adjust the initial layers’ weights, causing poor performance or even non-convergence. -For many years, this barrier limited the effective depth of neural networks, preventing researchers from successfully training very deep models. ResNet’s Core Innovation ResNet addressed the vanishing gradient problem with residual (skip) connections: Key Insight: Instead of learning the entire output mapping, each layer learns only the “residual”—the difference needed to transform the input into the desired output. Mathematical Description: If a layer receives an input X, it learns a function F(X) and produces X + F(X). This simple formula ensures that earlier layers maintain a direct connection to the final output. As a result, gradients flow more smoothly backward through the network, greatly reducing the chance they will vanish. Record-Breaking Performances After its debut, ResNet quickly dominated major competitions in 2015: ImageNet Large Scale Visual Recognition Challenge (ILSVRC) -Achieved the best top-5 error rates in image classification. -Took first place in detection, localization, and segmentation. Microsoft COCO -Led key detection tasks, further establishing ResNet’s impact. These wins confirmed that very deep networks—with 50, 101, or even 152 layers—could be trained successfully using residual connections. Why Residual Learning Is a Game-Changer Simplicity Much of an input (e.g., a blurry image) is already correct. You only need to learn how to improve it—rather than relearn everything from scratch. Stability Gradients remain strong because the network only focuses on the “correction” component. This stability allows networks to scale in depth more easily. How Does a “Same-Dimension” Block Lead to Classification? ResNet blocks often preserve the same input and output dimensions (e.g., 64 channels in, 64 channels out). The question is: How does the network ever reach class scores? Stage-by-Stage Downsampling ResNet uses strided convolutions between groups of blocks to reduce the spatial dimensions (e.g., 56×56 to 28×28) while increasing channels (e.g., from 64 to 128). Global Pooling and Dense Layers By the final stage, the feature maps are smaller (like 7×7), but have more channels (like 512 or 2048). A global average pooling layer then converts this 7×7 feature map into a single vector, which a fully connected layer transforms into class scores. Seeing ResNet as an Euler Method Interestingly, each residual block resembles a single “step” in the numerical solution of differential equations (Euler method). The network iteratively refines its representation by adding a small correction at each layer. Popular ResNet Variants ResNet-18, ResNet-34 U se simpler “basic” blocks with two 3×3 convolutions per block. ResNet-50, ResNet-101, ResNet-152 Employ “bottleneck” blocks containing three convolutions (1×1, 3×3, 1×1). This design uses parameters more efficiently and helps scale to greater depths. The numbers (18, 34, 50, 101, 152) correspond to the total number of layers dedicated to feature transformation. Here is the code you can implement: https://open.substack.com/pub/aivizua...