Scraping problem

[az65sd]

Hello,

I'm trying to scrape using the following command:

oddsharvester historic --sport football --season 2025-2026 --league england-premier-league --market 1x2 --headless --concurrency 3

I've tried using a VPN and proxy from netherlands, but I'm still being detected by the anti-bot system. I just want to know if you're experiencing the same issue on your side, or if it's just me.

Sincerly

[vladshwarz-ui]

Hi — same symptom on my side with upcoming (not only historic).

oddsharvester upcoming -s football -d 20260811 -l england-premier-league -m 1x2 \
  --period full_time --no-include-started --headless \
  --timezone Europe/Berlin -c 3 --request-delay 1.0 \
  -f json -o /tmp/oh-smoke

Observed:
Process exits with code 1
No …% success rate line (fails before a successful summary)
Same result with a working HTTP proxy and without proxy (different VPN)
OddsPortal opens fine in a normal browser on the same machine / same network setup
Started after updating to current master (v0.10.0)

Could you confirm whether this looks like a known/widespread anti-bot issue on OddsPortal’s side, and whether a fix is feasible in the near term? Happy to wait on a resolution — and to share stderr or headed-browser observations if that would help.

[vladshwarz-ui]

Follow-up: confirmed DOM change on upcoming listing.
URL: https://www.oddsportal.com/matches/football/20260812/ → Events

document.querySelectorAll("div[class*='eventRow']").length  → 0
document.querySelectorAll("[data-testid='game-row']").length → 169

OddsHarvester upcoming still uses class*=eventRow / ^eventRow
(see scrape_upcoming scroll + extract_match_rows), while live already
uses data-testid=game-row. That matches our logs: page loads, odds
format OK, Found 0 event rows → exit 1.

[az65sd]

While waiting for a fix to be officially deployed, could you be more specific about the changes to be made so that the app works again?

Thanks

[royalcohen]

Hello guys!

I'm also getting zero matches right now. OddsHarvester was upgraded to the newest version, but it looks like OddsPortal changed the page layout as well. Do you think that could be the reason? I've asked Jordan the same thing in another thread. Let's see what he says about that.

[vladshwarz-ui]

Follow-up after the listing finding (eventRow → 0, game-row → N).

With listing fixed (or using game-row), upcoming can collect match links and finish with a non-zero success rate, but odds are still empty:

• #react-event-header is gone (null); match meta is available via DOM (game-host / game-guest / game-time-item / breadcrumbs-line, team name in participant-name).
• bookies-filter-nav is also null.
• Market tabs often time out on legacy ul…odds-tabs selectors (sometimes recover via URL hash / “tab already active”).
• OddsParser reports many false “bookmakers” (Football, team names, Add to coupon) and ends with Successfully parsed odds for 0 bookmakers → JSON has "1x2_market": [].

So there appear to be (at least) three DOM drifts: listing rows, match header JSON, and the bookmaker odds table.

[jordantete]

Hi, thanks for your messages.
I’m away from the project for the next 10 days, so I won’t be able to dig into this properly before then.

I’ll take a proper look once I’m back

[jordantete]

Bug confirmed. This is not anti-bot detection: OddsPortal shipped a full frontend redesign and every selector generation the scraper relies on is gone. I reproduced all three DOM drifts reported by @vladshwarz-ui in a real browser today, plus a few more.

@az65sd to answer your question: there is no small config change that brings it back. The listing fix alone (swapping to game-row) only gets you match links; odds extraction, market navigation, match metadata and pagination all need to be rewritten against the new DOM, and all the HAR test fixtures need re-capturing. It is a significant rework, not a one-line patch.

I am starting on it. No ETA yet, but the diagnosis is complete and the new DOM structure is mapped, so this issue will track the fix.

[jordantete]

The rework is done and merged to master.

All commands are working again on the redesigned site: historic, upcoming, live and community. Verified live today across football, tennis, basketball and baseball. Thanks @vladshwarz-ui for the early diagnosis, it saved me a lot of time.

Small heads-up: a few fields the new site no longer exposes (like match_info and the in-play match clock) now come back null.

I will cut a release shortly. Until then you can run the fix from master.

[royalcohen]

Hey Jordan!

Thank you for the fix!

What do you mean with match info? Does it include metadata like venue, kickoff time, etc?

[jordantete]

No, match_info never included venue or kickoff time. Those are separate fields and they still work.

match_info was the "notes" line the old site showed under the match header: labels like "neutral venue", "first leg", "behind closed doors", or a broadcast note. It came from a JSON blob (eventData.staticInfo) that the redesigned site no longer embeds, and there is no DOM equivalent, so the field is now always null. It was already null on most ordinary matches before the redesign anyway.

What you are asking about is still extracted:

• kickoff time: match_date (UTC)
• teams, league, scores: home_team, away_team, league_name, home_score, away_score
• venue: venue, venue_town, venue_country

One caveat on the venue trio: it comes from the page's structured data, which only describes the next upcoming meeting between the two teams. So it is reliable for upcoming matches, but on historic matches it can come back null.