Add examples for new quotes.toscrape.com endpoints demonstrating modern web scraping techniques

[rjanain]

Overview

The quotes.toscrape.com website has expanded significantly with new endpoints designed to teach modern web scraping challenges. The quotesbot project currently only covers the basic endpoints (/ with CSS and XPath), but the sandbox now includes 6+ additional endpoints that simulate real-world scenarios students need to learn.

New Endpoints Available on quotes.toscrape.com

The quotes.toscrape.com sandbox now includes these endpoints specifically designed for teaching:

Endpoint	Purpose	Modern Technique Demonstrated
/js/	JavaScript-rendered content	Extracting data from JS variables/JSON in script tags
/scroll/	Infinite scroll page	API consumption, JSON parsing
/api/quotes?page=N	REST API endpoint	Working with APIs instead of HTML scraping
/login	Login-protected pages	Form authentication, CSRF tokens, session handling
/tableful/	Table layout	Parsing structured table data
/search.aspx	ViewState forms	Handling stateful ASP.NET applications
/random	Dynamic content	Scraping endpoints with changing content

Problem

quotesbot currently only has 2 spiders (toscrape-css and toscrape-xpath) that both scrape the same basic endpoint. This leaves students without examples for all the modern techniques that quotes.toscrape.com is designed to teach.

When students encounter:

• JavaScript-heavy websites
• Infinite scroll interfaces
• Login-protected content
• API-driven applications
• Complex form submissions

Proposed Solution

Add spiders for each of the new quotes.toscrape.com endpoints to provide complete coverage of the learning sandbox.

New Spiders to Add

1. toscrape-js.py → /js/ endpoint

   • Extract data embedded in JavaScript <script> tags
   • Parse JSON from var data = [...] declarations
   • Example of reverse-engineering JS-rendered pages

2. toscrape-scroll.py → /api/quotes?page=N endpoint

   • Consume REST API directly instead of scraping HTML
   • Parse JSON responses
   • Handle pagination via API parameters (has_next, page fields)

3. toscrape-login.py → /login endpoint

   • Demonstrate FormRequest.from_response() for CSRF handling
   • Show proper authentication flow
   • Scrape content only accessible after login

4. toscrape-table.py → /tableful/ endpoint

   • Parse quotes organized in HTML table structure
   • Iterate through rows and extract cells
   • Handle table-based layouts

5. toscrape-viewstate.py → /search.aspx endpoint

   • Extract and submit ASP.NET ViewState hidden fields
   • Handle stateful form submissions
   • Demonstrate techniques for legacy enterprise applications

6. toscrape-random.py → /random endpoint

   • Scrape endpoints that return different content each time
   • Show how to handle dynamic/random content