Wrangle PDFs with Custom User Defined Functions (UDF) in Daft скачать в хорошем качестве

Wrangle PDFs with Custom User Defined Functions (UDF) in Daft

Wrangle PDFs from start to finish with custom User Defined Functions (UDFs) in Daft. Software Engineer Malcolm Greaves   / malcolm-greaves   walks you through every step of a PDF processing pipeline. By the end of the video, you will have a fully functional pipeline that: • Starts with downloading PDFs from an S3 bucket • Extracts text boxes using OCR or by reading the file format • Performs spatial layout analysis to group text boxes into lines or paragraphs • Computes embeddings using a lightweight LLM, running locally • Saves everything to Parquet Build a singular PDF processing pipeline and have complete control over all of it, no more stitching together fragmented tools for these types of workloads. Notebook to follow along: https://docs.daft.ai/en/stable/resour... Try it yourself and get started today: pip install daft 🩷 Get to know Daft ‣ Learn more about Daft: https://www.daft.ai ‣ Join our Distributed Data Slack Community: https://www.daft.ai/slack ‣ Star Daft Github: https://github.com/Eventual-Inc/Daft ‣ Subscribe to Daft Engineering Blog: https://www.daft.ai/blog 📲 Follow us ‣ LinkedIn:   / daftengine   ‣ X/Twitter:   / daftengine   #daft #distributed #multimodal #data #dataengineering 00:00 Introduction 00:35 Download Daft & Dependencies 00:58 Pull S3 urls of PDFs 01:53 Download PDFs from S3 02:38 Use Pydantic classes 04:39 Generating Daft Datatypes from Pydantic 05:10 Load & Parse PDFs Using UDFs 07:53 Perform OCR and Extract Text on First PDF 08:49 Document Processing 11:10 Text Embedding with SentenceTransformer 12:00 Entire End-to-End Pipeline 12:52 Step 1: Enumerate S3 Keys 13:06 Step 2: Download PDFs 13:12 Step 3: Load PDFs, Maybe Apply OCR 13:46 Explaining Daft UDF Application 14:31 Step 4: Text Box Processing 16:27 Explaining Structure Access Expressions 18:09 Step 5: Text Embeddings 19:07 Execute and Write to Parquet