WK12 - MIT How to AI Almost Anything - Discussion 6: Large multimodal models скачать в хорошем качестве

WK12 - MIT How to AI Almost Anything - Discussion 6: Large multimodal models

The research highlights a significant shift toward parameter-efficient fine-tuning (PEFT) and linear-time architectures to overcome the high computational costs and quadratic complexity of traditional Transformer-based models. LLaMA-Adapter demonstrates that a large frozen model can be successfully adapted for instruction following and multi-modal tasks by adding only 1.2M learnable parameters, using a zero-initialized attention mechanism with a gating factor to ensure training stability. Moving beyond the Transformer bottleneck, the Cobra model utilizes Mamba (a State Space Model) to achieve linear scalability and inference speeds that are 3x to 4x faster than existing efficient Transformer baselines like LLaVA-Phi. These advancements reveal that high-performance multi-modal reasoning—including improved spatial relationship judgment and reduced object hallucination—can be achieved through the strategic fusion of vision encoders (like DINOv2 and SigLIP) with lightweight, efficient backbones, often with a fraction of the parameters and training time required by full fine-tuning Course website: https://mit-mi.github.io/how2ai-cours... 🔔 Don't get left behind in the AI era. Hit subscribe to get more deep dives into the AI breakthroughs. #machinelearning #llamaadapter #mambassm #multimodalai #efficientinference #how2ai #multimodalai #peft