Google BigQuery 101 & how to ingest data to BigQuery скачать в хорошем качестве

Google BigQuery 101 & how to ingest data to BigQuery

BigQuery Sandbox Blog: https://estuary.dev/bigquery-sandbox/ In this video, we'll be talking about Google BigQuery – what it is, some background, and how to build a data pipeline with it using an ETL tool Estuary Flow. If you're looking to get started with big data, then this video is for you! We'll be discussing Google BigQuery, its features, and how you can use it to help streamline your data management process. After watching this video, you'll be well on your way to big data success! 🏄🏻‍♂️ If you're interested in Surfshark, sign up via this link for an exclusive discount and free months: https://get.surfshark.net/aff_c?offer... 🍺 If you've learned something new from this video, buy me a coffee to support this channel =) https://bmc.link/jennyman 0:00 Intro 0:47 What is BigQuery 0:57 Petabytes of data 1:11 BigQuery Characteristics 1:23 BigQuery History 1:35 BigQuery Benefits 2:09 BigQuery Architecture 2:50 Under the hood 3:24 Ingesting data 4:10 Surfshark 5:27 BigQuery Sandbox 6:00 What is a materialization 6:37 Tutorial starts References: https://en.wikipedia.org/wiki/BigQuery   / bigquery-explained-overview   https://estuary.dev/cloud-sql-to-bigq... https://cloud.google.com/blog/product... What is BigQuery? BigQuery is Google's fully managed, serverless data warehouse that enables scalable analysis over petabytes of data. How much is petabytes? One Petabyte is the equivalent of 20 million tall filing cabinets, or 500 billion pages of standard printed text. It’s A LOT of data! Also, BigQuery is a Platform as a Service (PaaS) that supports querying using dialects of SQL. It also has built-in machine learning capabilities, which everyone is going after these days. When did BigQuery come about? BigQuery was announced in May 2010 and made generally available in November 2011.[1] So it’s been around for over a decade! What are some benefits of using BigQuery? There are many, but here are just a few: With BigQuery, you no longer have to provision and forecast compute and storage resources beforehand. BigQuery allocates all the resources based on usage, dynamically. Also, BigQuery provides super fast analytics on a petabyte scale through its unique capabilities and architecture, which we’ll talk about. And, since BigQuery uses a columnar data store, you can enjoy the highest data compression with minimized data scanning in the usual data warehouse deployments. What is the BigQuery Architecture like? BigQuery’s serverless architecture decouples storage and compute and allows them to scale independently on demand. This structure offers both flexibility and cost controls for users because they don’t need to keep their expensive compute resources up and running all the time. This is very different from traditional node-based cloud data warehouse solutions or on-premise MPP systems. This approach also allows users of any size to bring their data into the data warehouse and start analyzing their data using Standard SQL, without worrying about database operations and system engineering. Under the hood, BigQuery employs a vast set of multi-tenant services driven by low-level Google infrastructure technologies like Dremel, Colossus, Jupiter and Borg. Compute is Dremel, a large multi-tenant cluster that executes SQL queries. Storage is Colossus, Google’s global storage system. Compute and storage talk to each other through the petabit Jupiter network. BigQuery is orchestrated via Borg, Google’s precursor to Kubernetes. Ingesting Data Now, at some point, you’ll probably need to ingest data into or from BigQuery because every organization today relies on multiple data sources, databases, data warehouses, so it’s likely that BigQuery alone cannot contain all of your data needs and pipelines. The good news is: BigQuery supports several ways to ingest data into its managed storage. The specific ingestion method depends on the origin of the data. If your data sources are in GCP, then some of them support direct exports to BigQuery. However, if your data sources are outside of GCP. Or, you don’t want to manually handle exports, there are several third-party ETL solutions in the market that can help you ingest data to and from BigQuery fairly easily. #bigquery #dataengineering #dataengineeringessentials #datapipeline