Scrape Team Information

[royalcohen]

I have a Google Sheets file with multiple sheets. One sheet contains matches, and the other sheets contain teams, and leagues, etc. The matches sheet is pulling information from the latter ones. Once in a while I'm updating the information, because sometimes OddsPortal is changing the names of the teams, and leagues. Sometimes it's just a small variation, and sometimes teams are getting rebranded, or dissolved. Another example would be leagues switching sponsors.

Actually, I wanted to do it with a simple browser extension, because it didn't seem like a very complex task. For teams, the name is visible, and scrapable on the team page. But, in some cases the name appearing on the team page is not identical with the one in the match list.

For example, Barcelona from Brazil:

https://www.oddsportal.com/football/team/barcelona-fc/WGt8En5I/

Match list: Barcelona
Team page: Barcelona FC

Since the sheets are designed to work like a database, the team, and league names in the matches sheet need to match the ones from their dedicated ones to correctly pull data via lookup formulas.

Can you add a module which scrapes the team name how it appears in the match list by providing the URL of the team page? This page also contains other metadata which could be scraped like country, city, stadium, and coach. Do you think other people might have a use case for this as well?

[jordantete]

Thanks, this is a reasonable ask and most of it is cheap to support.

I checked a couple of team pages. The team page ships a JSON blob in its server-rendered HTML (the <team-page :data="..."> component) that already contains, without any JS rendering: participantName, participantFullName, venue / venueTown / venueCountry, countryName, tournamentName, sportName and the logo URL. So a team command that takes a team page URL and returns that record is a single page load, no market navigation. Cheap to add.

Two caveats on what you actually asked for.

1. There is no coach anywhere in the payload or in the rendered page, so that field is not available.

2. participantName is not always the name shown in the match list, which is the one you need for your lookups. For Arsenal it matches (participantName "Arsenal", full name "Arsenal Football Club"). For your Brazilian Barcelona (WGt8En5I) the team page says participantName "Barcelona FC" and participantFullName "Barcelona de Ilheus", while the match page for Barcelona vs Juazeirense carries "home":"Barcelona". So the team metadata alone will not give you a reliable join key in exactly the cases you care about. The list-form name does appear on the team page, but only inside the fixtures/results rows and the form tooltips, which are rendered client side and can be hidden when the bookmaker filter leaves no rows. Extracting it is doable but best effort: a team with no listed matches gives nothing.

[royalcohen]

Thank you!

1. I wonder why that is, and where that information is loaded from then. Generally, the bigger teams tend to have more metadata on their page. So, I thought maybe just smaller teams like Barcelona FC from the example above don't have a coach there, but even this team has one. Anyway, it wouldn't be a big problem for me since I primarily want to scrape the match list name. The idea was just, when we're already there, and it's not taking much longer with one page load, we can collect the other information as well in case someone else wants to use it.

2. Okay, sounds good! So, you would add the match list name as a third name to the scraped metadata, and if a team has no matches listed, it returns just the other two? I previously tested accessing OddsPortal from a few different locations, and I'm currently using Canada. I haven't tested all of them, but Canada has a better coverage than USA for example. Even for matches in the early, or mid 2000's there are usually some bookmakers available. So, for the most teams, it should be possible to parse the name from the match list. If anyone knows some other countries with good, or better coverage, please let me know. That itself could actually be a new feature idea! 😅 Maybe something like testing bookmaker coverage by country with TOR IP? But, I don't know how high the demand for that is.

[jordantete]

Good catch on the coach, I was wrong. It is there, I had only looked at the server-rendered payload. It comes from a separate request that fills the header after page load, so it is scrapable on the same page load with no extra navigation. Barcelona FC shows "Paulo Sales", Remo shows "Osorio Juan". Same source also carries last 6 games form, over 2.5 %, BTTS % and average goals, but I will leave those out of the first version unless someone asks.

2. Yes, that is the plan. The record will be name, full_name and list_name, plus country, town, venue, coach, tournament, sport and logo. If the page shows no matches, list_name comes back null and the rest of the record is still returned, not an error.

Your geo observation matches what I see. Team pages have no bookmaker filter of their own, unlike match pages, so the rows follow whatever default bookmaker selection your IP gets and there is no way to override it. From a French IP the Barcelona FC page says there are no odds from the selected bookmakers and shows zero rows, so list_name would be null. From Canada you get rows. I will document that the field depends on where you scrape from.

When you do hit a null, a match page is the fallback: it carries the list form name and a link to the team page together, so scraping a league and reading the team links gives you the name plus the team id in one pass.

On the coverage testing idea, I would keep it out of this repo. Tor exits are heavily flagged by the anti-bot layer here, and what you would measure is the default bookmaker set per country rather than real data availability. If you want the answer for yourself, running the same scrape through two VPN exits and diffing the bookmaker list already gives it. Happy to track it as a separate issue if more people want it.

[royalcohen]

Okay, thank you!

How would scraping the logo work? Would it scrape the image link, or the image itself? I don't need it right now, but I'm curious.

What bookmaker filters do you mean? I can't find any on the match pages either.

So, it would scrape the league table of the team in the backend to find the teams list form name? That would be interesting, because that's how I've done it manually sometimes.

Okay, I understand. This function wouldn't be a huge problem solver anyway. It can be easily checked manually when deciding on the geo location before for the scraping.

[jordantete]

Logo: The URL, not the image. Header logo is served directly, e.g. /serve/images/team-logo/Men/W0eABFPt-f5FmjcSH.png. The query string on it is a global asset version stamp that changes over time, so I strip it and store the bare URL.

Bookmaker filter: Two different things. On a match page there is a tab row above the odds table (All Bookies / Classic Bookies / Crypto Bookies), that is what --bookies-filter drives. Separately, the "selected bookmakers" set comes from your IP or account, it is not a control on the page, and it is what empties the team page rows. Team pages have no tab row of their own, only the second one applies there.

List name: No league table scraping, and you will not need the match page fallback in the normal case. The Last 6 Games block ships in the initial HTML with names in list form, even when the odds rows are empty. From a French IP, Barcelona FC gives "Jequie - Barcelona", "Barcelona - Juazeirense", "Bahia - Barcelona" and so on. The team's own name is the one present in every entry, so list_name resolves to "Barcelona" with no extra request. Only a team with no recent played matches returns null.

Also, correcting myself on the coach: it is in the initial document too, next to venue, town and country. Those keys are often null on smaller teams.

[royalcohen]

Thank you very much!

Great to hear, that it's possible to get the list form names, even when there are no odds are available for the recent matches.

Can we already use this with the newest OddsHarvester version?