Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 33 additions & 1 deletion extensions/geolibre-chrome/scanner.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,31 @@ export function scanDocumentForDatasets() {
}
};

const huggingFaceHost = (url) => /^(?:huggingface\.co|hf\.co)$/i.test(url.hostname);

// The Hub links one file from seven routes -- blob, raw, blame, edit, delete,
// commits and the ?download=true button -- and every one of them ends in the
// file's own extension, so a repository page yields near-duplicate hits where
// only `resolve` (which 302s to the CDN) serves bytes a map source can read.
const huggingFaceFileUrl = (url) => {
if (!huggingFaceHost(url)) return null;
const parts = url.pathname.split("/").filter(Boolean);
const isRoute = (part) => /^(?:blob|raw|blame|edit|delete|commits|resolve)$/.test(part ?? "");
// /<owner>/<repo>/<route>/<revision>/<path> for models, one segment deeper
// for /datasets and /spaces (which still carry namespaceless legacy repos).
// Read the route from its fixed position rather than scanning, so an owner
// or repository named after a route cannot stand in for one.
const route = /^(?:datasets|spaces)$/.test(parts[0]) && isRoute(parts[3]) ? 3 : 2;
Comment thread
giswqs marked this conversation as resolved.
Outdated
if (!isRoute(parts[route]) || parts.length < route + 3) return null;
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated
parts[route] = "resolve";
Comment on lines +43 to +53

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug (medium confidence): the "prefer the deeper index" heuristic can misfire when a legacy (namespaceless) repo has a revision name that happens to collide with a route keyword and the file path is nested — both fairly plausible on the Hub.

Example: https://huggingface.co/datasets/mnist/blob/raw/data/train.csv, where mnist is a namespaceless dataset repo, blob is the real route, and raw is (coincidentally) the branch/revision name — not the download route.

Walking the algorithm: parts = ["datasets","mnist","blob","raw","data","train.csv"] (length 6). The namespaced candidate is checked first: isRoute(parts[3])isRoute("raw")true, and parts.length >= 3+36 >= 6true, so index 3 wins even though this repo has no namespace. parts[3] ("raw") gets overwritten with "resolve", but parts[2] ("blob", the actual route) is left untouched, producing /datasets/mnist/blob/resolve/data/train.csv — still a blob (HTML) URL, not a working direct-file link. The correct canonical URL is /datasets/mnist/resolve/raw/data/train.csv.

The length check disambiguates most of the time (as covered by the existing tests), but it can't distinguish "namespaced repo, route at index 3" from "namespaceless repo with nested path, whose revision name happens to look like a route" — both satisfy length >= 6. This is a narrow edge case (needs a revision literally named blob/raw/blame/edit/delete/commits/resolve), but namespaceless repos (glue, squad, mnist, gpt2, …) and nested file paths are both common on the Hub, so it's not purely theoretical.

const canonical = new URL(url.href);
canonical.pathname = `/${parts.join("/")}`;
canonical.searchParams.delete("download");
return canonical;
};
Comment thread
giswqs marked this conversation as resolved.

const canonicalUrl = (url) => {
if (huggingFaceHost(url)) return huggingFaceFileUrl(url) ?? url;
if (url.hostname !== "source.coop") return url;
const parts = url.pathname.split("/").filter(Boolean);
if (parts.length < 3) return url;
Expand Down Expand Up @@ -93,10 +117,18 @@ export function scanDocumentForDatasets() {
return;
}

// Every Hub route other than a file route is a UI page -- tree, viewer, the
// "Auto-converted to Parquet" branch -- so a hint-based match there would
// offer HTML as data.
const onHub = huggingFaceHost(url);
if (onHub && !huggingFaceFileUrl(url)) return;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit (low confidence, minor perf): huggingFaceFileUrl re-parses and re-runs the route-matching regex against the same URL twice per link — once inside canonicalUrl (called via canonicalHttpUrl on line 113) to build url, and again here to check it's a file route. Since url is already the post-canonicalization value, the second call is redundant (it'll always find resolve at whatever index it was rewritten to, or stay null if the first call already returned null). Not a correctness issue, just duplicate work on every Hub link — could be avoided by having canonicalUrl/canonicalHttpUrl also report whether a Hub URL resolved to a file, or by checking huggingFaceHost(url) && parts.includes("resolve")-style logic once.


const kind = classify(url, hint);
if (!kind) return;
const existing = datasets.get(url.href);
const name = label.trim() || cleanName(url);
// Hub links carry UI chrome as their text ("Download", "History", "308 kB
// xet"), so the file name has to come from the path.
const name = (onHub ? "" : label.trim()) || cleanName(url);
const candidate = {
url: url.href,
name,
Expand Down
77 changes: 77 additions & 0 deletions tests/chrome-extension.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,83 @@ describe("GeoLibre Chrome extension scanner", () => {
assert.equal(found.styleUrl, "https://data.source.coop/giswqs/opengeos/roads.style.json");
});

it("collapses every Hugging Face file route onto the direct resolve URL", () => {
const repo = "https://huggingface.co/datasets/giswqs/PACE-Water-Quality";
const file = "main/cogs/PACE_OCI-20260103-chla.tif";
const found = scan(
`
<a href="${repo}/blob/${file}">PACE_OCI-20260103-chla.tif</a>
<a href="${repo}/raw/${file}">Raw pointer file</a>
<a href="${repo}/blame/${file}">Blame</a>
<a href="${repo}/edit/${file}">Contribute</a>
<a href="${repo}/delete/${file}">Delete</a>
<a href="${repo}/commits/${file}">History</a>
<a href="${repo}/resolve/${file}?download=true">Download</a>
`,
`${repo}/blob/${file}`,
);
assert.deepEqual(found, [
{
url: `${repo}/resolve/${file}`,
name: "PACE_OCI-20260103-chla.tif",
format: "GeoTIFF",
kind: "raster",
styleUrl: null,
},
]);
});

it("canonicalizes Hugging Face model and Space files but leaves other Hub links alone", () => {
const found = scan(
`
<a href="https://hf.co/giswqs/model/blob/main/grid.geojson">grid</a>
<a href="https://huggingface.co/spaces/giswqs/demo/blob/main/roads.pmtiles">roads</a>
<a href="https://huggingface.co/datasets/giswqs/PACE-Water-Quality/tree/main/cogs">cogs</a>
<a href="https://huggingface.co/datasets/giswqs/PACE-Water-Quality/tree/refs%2Fconvert%2Fparquet/default">Auto-converted to Parquet</a>
`,
"https://huggingface.co/giswqs",
);
assert.deepEqual(
found.map((dataset) => dataset.url),
[
"https://hf.co/giswqs/model/resolve/main/grid.geojson",
"https://huggingface.co/spaces/giswqs/demo/resolve/main/roads.pmtiles",
],
);
});

it("reads the Hugging Face route from its position, not the first matching segment", () => {
const found = scan(
`
<a href="https://huggingface.co/datasets/giswqs/blob/resolve/main/roads.geojson">repo named blob</a>
<a href="https://huggingface.co/raw/model/blob/main/dem.tif">owner named raw</a>
<a href="https://huggingface.co/datasets/glue/blob/main/grid.pmtiles">legacy repo</a>
`,
"https://huggingface.co/giswqs",
);
assert.deepEqual(
found.map((dataset) => dataset.url),
[
"https://huggingface.co/raw/model/resolve/main/dem.tif",
"https://huggingface.co/datasets/glue/resolve/main/grid.pmtiles",
"https://huggingface.co/datasets/giswqs/blob/resolve/main/roads.geojson",
],
);
});

it("pairs a Hugging Face style file with its dataset across routes", () => {
const repo = "https://huggingface.co/datasets/giswqs/opengeos";
const [found] = scan(
`
<a href="${repo}/resolve/main/roads.geojson?download=true">Download</a>
<a href="${repo}/blob/main/roads.style.json">roads.style.json</a>
`,
`${repo}/tree/main`,
);
assert.equal(found.url, `${repo}/resolve/main/roads.geojson`);
assert.equal(found.styleUrl, `${repo}/resolve/main/roads.style.json`);
});

it("preserves a discovered style when stronger metadata replaces a dataset", () => {
const target = new URL("https://web.geolibre.app/");
target.searchParams.append("data", "https://data.example.com/roads.json");
Expand Down
Loading