Inspecting websites to find JSON data APIs | Marcos Huerta | Data Science Lab скачать в хорошем качестве

Inspecting websites to find JSON data APIs | Marcos Huerta | Data Science Lab

The Data Science Lab is a live weekly call. Register at pos.it/dslab! Discord invites go out each week on lives calls. We'd love to have you! The Lab is an open, messy space for learning and asking questions. Think of it like pair coding with a friend or two. Learn something new, and share what you know to help others grow. On this call, Libby Heeren is joined by Marcos Huerta, a Data Science Manager at Carmax, as he walks us through the guts of websites looking for data we can play with. He shows us how to find hidden REST/JSON APIs by using the web inspector in Safari/Firefox and then how to get what's necessary to pull the same data programmatically in python or R. Hosting crew from Posit: Libby Heeren, Isabella Velasquez, Daniel Chen Marcos's urls: Website: https://marcoshuerta.com GitHub: https://github.com/astrowonk/ Resources from the hosts and from participants in the Discord chat: 🔗 Postman: https://www.postman.com/ 🔗 Insomnia (open source alternative to Postman): https://insomnia.rest/ 🔗 Baseball Savant website Marcos is using: https://baseballsavant.mlb.com/gamefe... 🔗 Isabella Velasquez's blog on using {polite} R package to help scrape Wikipedia: https://ivelasq.rbind.io/blog/politel... 🔗 Festivas Mac app Marcos used to add the lights to his desktop: https://festivitas.app/ 🔗 Ted Laderas blog post on parsing JSON in R: https://laderast.github.io/intro_apis... 🔗 New rvest read_html_live() function: https://rvest.tidyverse.org/reference... 🔗 yyjsonr R package: https://github.com/coolbutuseless/yyj... 🔗 tuber R package: https://github.com/gojiplus/tuber 🔗 WikipediaR R package: https://www.quantargo.com/help/r/late... 🔗 rookiepy python package: https://pypi.org/project/rookiepy/ ► Subscribe to Our Channel Here: https://bit.ly/2TzgcOu Follow Us Here: Website: https://www.posit.co The Lab: https://pos.it/dslab Hangout: https://pos.it/dsh LinkedIn:   / posit-software   Bluesky: https://bsky.app/profile/posit.co Thanks for learning with us! 💛 Timestamps 00:00 Introduction 03:05 Web scraping vs. API calls 04:12 Server-side rendering vs. client-side JSON 06:12 Warning: Rate limits and business ethics (ahem) 08:39 Demo: Baseball Savant website 08:57 Using browser Developer Tools and the Network tab 12:15 "What is curl?" 13:30 Importing curl into Postman 16:03 Generating Python code from Postman 16:50 "Are there open source alternatives to Postman?" 17:50 Using the generated code in Python/Jupyter 22:28 R packages for JSON (jsonlite, yyjsonr) 25:09 Demo: Massachusetts Lottery website 28:17 Example: scripts Marcos automated with Cron jobs 30:17 Handling logins and cookies with RookiePie 32:19 Demo: CNN Election Data 34:26 Inspecting ESPN's website 36:58 "Can you scrape YouTube?" 38:19 Finding hidden JSON in CardsMania history 45:00 Benefits of API inspection over Beautiful Soup 46:59 New rvest function: read_html_live 50:40 Inspecting LinkedIn and finding GraphQL 53:58 Encouragement on handling API pagination