This site is powered by
course builder. Create your online course today.
Start now
Create your course
with
Autoplay
Autocomplete
Previous Lesson
Complete and Continue
Web Scraping for Data Science-Python: BS4, Selenium & Scrapy
Introduction
What is the best web scraping library? BS4 vs Selenium vs Scrapy (3:30)
Read This Before You Start
Installing Python and PyCharm (4:57)
Python Basics for Web Scraping (30:31)
Web Scraping Cheat Sheet for The Course (Don't skip it)
HTML for Web Scraping: Tags and Elements (7:49)
HTML for Web Scraping: Tree Structure (3:52)
Introduction to Beautiful Soup
Section Overview
Installing Beautiful Soup (3:13)
Beautiful Soup Basics (4:01)
How To Get The HTML from a Website (4:41)
How to Scrape a Single Page (10:34)
Exporting Data to a TXT File (3:20)
Scraping Multiple Pages with Beautiful Soup
Section Overview
Scraping Multiple Links within The Same Page (16:27)
Pagination (18:27)
XPath
Important Note (Don't skip it!)
XPath Syntax, Functions and Operators (3:41)
Test Your XPath (5:29)
Special Characters and Syntax (4:37)
Introduction to Selenium
Section Overview (+ Important Note)
How to Identify Java Script Driven Websites (3:25)
Installing Selenium and ChromeDriver (2:19)
Importing Libraries and Creating The Driver (3:37)
How to Find Elements with Selenium (3:41)
Clicking on a Button (6:36)
Extracting Data from a Table (12:26)
Exporting Data to a CSV File with Pandas (5:14)
Selecting Elements within a Dropdown (7:41)
Project #1: Dealing with Multiple Pages using Selenium (Amazon Audible Bot)
Important Note (Update)
Building the Bot (20:42)
Headless Mode (5:04)
Handling Pagination (17:05)
Implicit vs Explicit Waits (9:03)
Project #2: Infinite Scrolling & Login (Twitter Bot)
Important Note
How to Login to Websites (11:51)
Building a Function (22:17)
Infinite Scrolling (8:33)
Scraping the Website with Infinite Scrolling (12:28)
Introduction to Scrapy
Section Overview
Scrapy Installation (2:23)
Setting up Scrapy with Pycharm (2:39)
Note: How to Show Hidden Folders on Mac (1:07)
Scrapy's Commands (4:13)
Creating our First Project and Spider (6:00)
Scrapy Templates and How to Find Elements with Scrapy (3:21)
Scrapy's Shell Command (4:08)
Building a Spider (2:59)
Getting links listed in a website (6:19)
Relative vs Absolute Links (5:10)
Scraping Data from Multiple Links (10:53)
Dealing with Multiple Pages using Scrapy
Building the Spider (First Page) (14:24)
Pagination (6:22)
Changing User-Agent (7:03)
Project #3: Building Crawlers with Scrapy
Crawler - Rules (11:25)
Crawler - Pagination (5:10)
Crawler - Changing User Agent (4:30)
Exporting Data to a Database: MongoDB and SQLite
Pipelines (6:00)
MongoDB (9:29)
SQLite3 (9:37)
Scraping APIs & Login to website with Scrapy
Scraping APIs - Building the Spider (11:34)
Scraping APIs - Dealing with Pagination (5:31)
Login to website - Building the Spider (10:04)
Splash Basics
Section Overview
Setting up Splash with Docker (4:30)
Introduction to Splash (6:02)
How to Find Elements with Splash (5:10)
Project #4: Scraping JavaScript websites using Scrapy/Splash
Building the Script in Splash (8:15)
Using Splash with Scrapy (6:45)
Parsing (5:45)
Changing User-Agent (Splash) (4:28)
How to Make Money Using Web Scraping
Section Overview
How I Monetized My Web Scraping Skills Over The Past 2 Years (14:45)
Getting Started - Part 1 (13:20)
Getting Started - Part 2 (20:17)
ChatGPT for Web Scraping
Section Overview
Web Scraping with ChatGPT - Part 1 (11:48)
Web Scraping with ChatGPT - Part 2 (14:57)
Congratulations!
Building a Spider
Lesson content locked
If you're already enrolled,
you'll need to login
.
Enroll in Course to Unlock