The Hash Tree Trick That Saves Terabytes скачать в хорошем качестве

The Hash Tree Trick That Saves Terabytes

Remember back on Day 19 (Gossip Protocol)? We got our servers talking to each other to stay updated. And on Day 18 (Heartbeats), we figured out how to know when a server dies. But today is about the aftermath. What happens when that offline server wakes back up after being offline for an entire hour? It missed a lot of Gossip. How does it catch up? Welcome to Day 22 of "100 Days of System Design." Pull up a chair, because today I am revealing a massive performance hack used in distributed systems (straight out of Designing Data-Intensive Applications) to keep massive databases perfectly in sync. In this fast-paced insider class, we cover: The Nightmare Problem (Anti-Entropy) 🛑 Imagine the absolute nightmare of comparing two 10-Terabyte databases just to find one missing tweet. Reality check: If you try to send all 10TB over the network just to compare them, you are sending a massive, slow-moving garbage truck to jam up a glowing neon superhighway. You will crash the entire system. The Solution: Merkle Trees (Hash Trees) 🌳 Here is the kicker: We don't compare the data. We compare the hashes of the data. Visualizing the Neon Pyramid 🔺 Imagine our vibrant minimalist vector art: a glowing neon pyramid. At the bottom, there are millions of tiny green data cubes representing individual blocks of data (the leaves). We hash those cubes. Then, we group those hashes and hash them together, forming glowing blue triangles. We repeat this, merging upward, until we have one single, shining golden orb at the very top—the Root Hash. The Anomaly (Plot Twist!) 🚨 Plot twist: A user deletes a photo. Down at the bottom of the pyramid, one tiny green cube turns red. Instantly, the red glow shoots straight up the branch, turning the golden orb at the top red. The system immediately spots the anomaly. The Lightning-Fast Search Magic Trick ⚡ Here is how the sync actually happens. Server A and Server B just compare their golden Root Hashes. If they match? Perfect, do nothing. If they are different? We step down one level and compare the left and right branches. You keep following the mismatched hashes down the glowing tree until you pinpoint the exact corrupted or missing block. You only download that one single block over the network. Now our databases can heal themselves instantly. But what if a user searches for a username that doesn't even exist? How do we stop the database from wasting time looking for it? Tomorrow, Day 23, we build Bloom Filters. Hit subscribe! 👇 References: Designing Data-Intensive Applications by Martin Kleppmann #SystemDesign #MerkleTrees #DistributedSystems #DataSynchronization #BackendEngineering #SoftwareArchitecture #DDIA #TechInterview #100DaysOfSystemDesign #Cryptography #Hashing