𝐀𝐖𝐒 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫 𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠 Day5: KMS issue fix| Create IAM user | First Glue spark job скачать в хорошем качестве

𝐀𝐖𝐒 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫 𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠 Day5: KMS issue fix| Create IAM user | First Glue spark job

AWS Data Engineer Day 5 ====================== 1) How to create a IAM user and use it to login into the account 2) How to solve the KMS problem will solve the IAM user problem first then will solve the IAM role problem ( this is what you will do 95% in real time) 3) Delete the KMS key , talk about the use case when about the "schedule key deletion" 4) How to create glue spark job and what all things we need to take care of to save cost while doing hands on 5) How Crawler helps in updating glue catalog in case of partitions 6) How to update glue catalog using alter command in athena 7) How to create a custom IAM policy -- alter command to update glue catalog ALTER TABLE order_target ADD PARTITION (order_date='2013-07-27') LOCATION 's3://source-sb-2023/order_target/order_date=2013-07-27/'; -- spark job import sys from awsglue.transforms import * from pyspark.context import SparkContext from awsglue.context import GlueContext from pyspark.sql.functions import to_timestamp, to_date sc = SparkContext() glueContext = GlueContext(sc) spark = glueContext.spark_session Read CSV file from S3 df = spark.read.option("header", "true").csv("s3://source-sb-2023/source_files/") df.printSchema() Convert order_date to timestamp and then extract the date part df = df.withColumn("order_date", to_date(to_timestamp("order_date", "yyyy-MM-dd HH:mm:ss.S"))) Save the file back to S3, partitioning by order_date df.write \ .mode("overwrite") \ .partitionBy("order_date") \ .option("header", "true") \ .csv("s3://source-sb-2023/order_target/") print("Glue Job completed successfully!")