Day 10: Persistent Storage in Kubernetes: Managing Model Weights and Datasets with MinIO скачать в хорошем качестве

Day 10: Persistent Storage in Kubernetes: Managing Model Weights and Datasets with MinIO

In this lesson, we tackle one of the biggest challenges in machine learning infrastructure — managing persistent storage inside Kubernetes. Since Kubernetes Pods are ephemeral, storing critical ML assets like datasets, checkpoints, and trained model weights requires a scalable storage solution. This tutorial demonstrates how to deploy MinIO, an S3-compatible object storage system, directly inside your Kubernetes cluster to manage ML artifacts efficiently. You'll learn how Persistent Volumes (PV) and Persistent Volume Claims (PVC) work, why object storage is better for large-scale MLOps pipelines, and how MinIO allows multiple training and inference Pods to access shared datasets and model artifacts concurrently. In this hands-on demo we will: • Deploy MinIO inside Kubernetes • Create an object storage bucket for ML artifacts • Use the MinIO Client (mc) to manage data • Run a Python script in a Kubernetes Pod that uploads and downloads files using S3 APIs (boto3) • Verify dataset and model storage operations through Pod logs This architecture forms the foundation of scalable MLOps systems used by companies running high-performance AI workloads.