Skip to content

fix: modernize scraping for current Amazon DE/UK pages - #38

Open
apoapostolov wants to merge 4 commits into
asherAgs:masterfrom
apoapostolov:fix/modern-amazon-scraping-de-uk
Open

fix: modernize scraping for current Amazon DE/UK pages#38
apoapostolov wants to merge 4 commits into
asherAgs:masterfrom
apoapostolov:fix/modern-amazon-scraping-de-uk

Conversation

@apoapostolov

Copy link
Copy Markdown

Summary

This PR modernizes the core HTML parsing in amzSear so it works again on current Amazon .de (default) and .uk search pages. The original selectors were outdated and returned 0 results on live pages.

Key Changes

  • AmzRating.py (_get_from_html):

    • New priority list of modern selectors: span.a-icon-alt, i.a-icon-star span, aria-labels for "Sternen", "stars", "out of 5", German text support.
    • Falls back gracefully to old selectors if needed.
  • AmzProduct.py (_get_from_html):

    • Reliable title extraction using h2 span, h2 a span, span.a-text-normal.
    • Better product URL logic (prefers /dp/ links).
    • img.s-image for product images.
    • Prices now prioritize span.a-price .a-offscreen (the clean price text Amazon currently uses).
  • AmzProduct.get_prices():

    • Fixed European number parsing (comma as decimal separator).
    • "80,66 €" → 80.66 correctly.
    • Handles "1.234,56" style as well.
  • AmzSear.py:

    • Updated result container selectors to modern Amazon markup:
      [data-component-type="s-search-result"], div[data-asin], .s-result-item.
  • Minor: Updated get_asin regex to match current ASIN format.

Testing

Tested with real queries on DE (primary) and UK (fallback):

  • "Logitech MX Master 3S"
  • "Anker USB C Kabel"
  • "wireless headphones"

Results now return titles, prices (with correct decimals), ratings like "4,6 von 5 Sternen", ASINs, and direct links.

A standalone wrapper (for reference) is also available in a related skill at https://github.com/the user may link.

Why DE/UK focus

Per the use case: DE is the default, UK is rare fallback. No changes or support added for .com or other locales.

Related

This revives functionality that had silently broken due to Amazon's frequent page structure changes.

Please review the diffs for the three core files. Happy to iterate.

- AmzRating: Replace brittle old selectors with robust fallbacks
  (span.a-icon-alt, i.a-icon-star, aria-labels for "Sternen"/"stars",
   German "Bewertungen", etc.) + old fallbacks.

- AmzProduct: Rewrite _get_from_html for modern search result HTML
  - Titles via h2 span / span.a-text-normal
  - Reliable /dp/ product URLs
  - img.s-image for images
  - Prioritize span.a-price .a-offscreen for prices

- AmzProduct.get_prices(): Fix EU decimal parsing
  (e.g. "80,66 €" correctly becomes 80.66 instead of 8066).
  Handles 1.234,56 and simple comma/dot cases.

- AmzSear: Update result container selectors to current
  [data-component-type="s-search-result"], div[data-asin], .s-result-item

- Minor: Improve get_asin regex (B[0-9A-Z]{9})

Restores functionality for .de (default) and .uk Amazon searches.
Tested with real queries like "Logitech MX Master 3S".
@apoapostolov

Copy link
Copy Markdown
Author

Hey @asherAgs (and anyone watching),

I had a local use case that required reliable Amazon DE (primary) + UK (fallback) price lookups. The existing scraping was returning zero results because Amazon's search page markup had changed significantly.

I updated the three core parsing methods with modern selectors and EU number handling that actually work on live .de/.co.uk pages today.

Details in the PR description + diffs.

Tested with multiple real products. A side-effect was also a small standalone wrapper for easier usage outside the original classes.

Happy to adjust anything or add tests if you'd like to merge.

PR: #38

- AmzProduct: add is_sponsored flag + aggressive regional domain normalization for product URLs (fixes .com links on DE/UK searches)
- Add get_main_price() and get_currency_hint() convenience methods
- AmzRating: extract selectors to class constants for easier future maintenance
- AmzSear: add exclude_sponsored=True option; pass region to products; clean indentation
- .gitignore: expanded with standard Python ignores (egg-info, dist, etc.)
- README: added DE/UK focus section with examples and changelog of fixes
@apoapostolov

Copy link
Copy Markdown
Author

Additional polish pushed to the branch:

  • Regional product URL normalization (now correctly returns amazon.de / amazon.co.uk links instead of .com sponsored redirects)
  • is_sponsored attribute on products + exclude_sponsored=True on AmzSear
  • Convenience methods: prod.get_main_price() and prod.get_currency_hint()
  • Selectors extracted to class constants (easier to maintain as Amazon changes layouts)
  • Expanded .gitignore
  • README updated with DE/UK examples and notes

All changes are on top of the original modernization fix.

Branch: https://github.com/apoapostolov/amzSear/tree/fix/modern-amazon-scraping-de-uk

…s, docs, and CLI

Full overhaul of the DE/UK marketplace scraping pipeline:
- Updated AmzBase, AmzProduct, AmzRating, AmzSear to current Amazon DOM
- Fixed CLI entry points and consts for region-specific routing
- Regenerated docs and egg-info metadata to match new API surface
- Updated legacy/v1 stubs for backward compatibility
- Cleaned up .gitignore, MANIFEST, setup files
Amazon DE uses "4,3 von 5" — old scraper stripped punctuation and
turned 4,3 into 43. Parse EU decimal commas correctly.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant