Skip to content

CLAUDE.md

Latest commit

 

History

History
251 lines (199 loc) · 14.2 KB

CLAUDE.md

File metadata and controls

251 lines (199 loc) · 14.2 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code when working with code in this repository.

Project Overview

Research companion repo for the paper "A Bibliometric and Semantic ATLAS of AI Bias Research: Trends, Disparities, and Knowledge Graph Perspectives" (under revision, target: DMKD). Contains analysis scripts for the survey corpus, a metadata extraction notebook, and an interactive Next.js dashboard deployed to https://biasatlas.cair-nepal.org (custom domain; repo renamed to CAIRNepal/biasatlas).

Current corpus: 692 papers (deduped from 703: 7 exact duplicates + 4 near-duplicate pairs removed, 1 corrupted metadata row fixed via OpenAlex re-fetch). All 692 have abstracts (688 via OpenAlex, 4 manually curated for papers with no DOI).

Repository Structure

algorithmic-bias-survey/
├── evaluation/analysis/          # Python scripts → figures + CSVs
│   ├── papers_new.csv            # Primary corpus (692 papers, actively maintained)
│   ├── semantic_similarity.py    # SBERT embeddings + UMAP → semantic_clusters.csv + figures
│   ├── embedding_comparison.py   # Compare 7 embedding models for domain separation
│   ├── umap_grid_search.py       # Grid search n_neighbors × min_dist → best UMAP params
│   └── figures_new/              # Output figures (PNG/PDF)
├── generate_atlas_umap.py        # 3 SBERT models × 2D+3D UMAP + HDBSCAN → atlas_data.csv
├── openalex_enriched.csv         # OpenAlex metadata (abstracts, citations, countries, OA)
├── extract_publication_metadata.ipynb  # Jupyter notebook: fetches OpenAlex metadata by DOI
├── dashboard/                    # Next.js 15 dashboard
│   ├── public/
│   │   ├── atlas_data.csv        # Generated by generate_atlas_umap.py
│   │   ├── semantic_clusters.csv # Generated by semantic_similarity.py
│   │   └── papers.csv            # Copy of papers_new.csv for dashboard
│   └── src/app/                  # React components
└── .github/workflows/
    ├── deploy.yml                # Auto-deploys dashboard on push to main
    └── update-atlas.yml          # Manually triggered: regenerates atlas_data.csv in CI

Data Files

Primary Corpus

  • evaluation/analysis/papers_new.csv — 692 papers (actively maintained, all fields complete)
    • Columns: SN, Paper Title, DOI, Authors, Author Regions, Affiliations, Year, Focus Region, Domain, Source, ORC ID
    • Five domains: Health & Clinical AI, General Fairness & Bias Mitigation, Graph-Based Fairness & Bias Mitigation, LLMs & NLP, Recommender Systems
    • Country names normalized: UK→United Kingdom, DZ→Algeria, IE→Ireland, Finlad→Finland, Dubai→UAE
    • SN 633 author fix: Shloka Desai (not Sudarshan Desai); SN 35 author fix: Pessach/Tassa/Shmueli (not Oneto/Chiappa)
    • Author Regions and Affiliations must have the same number of semicolon-separated entries as Authors — a mismatch silently breaks per-author attribution

OpenAlex Enrichment

  • openalex_enriched.csv — metadata fetched via OpenAlex API (692 rows, all fields filled)
    • Columns: SN, corpus_doi, openalex_id, doi, title, year, cited_by_count, abstract, referenced_works, is_oa, oa_status, oa_url, openalex_authors, openalex_affiliations, openalex_countries, keywords
    • DOI used as primary identifier for matching (rekeyed by normalized DOI, not by SN — SN alone was previously found unreliable across the two files)
    • All 692 papers have abstracts (688 via OpenAlex, 4 manually curated for the 6 papers with no DOI — 2 of those 6 also lack an abstract entirely)

Generated CSVs

  • dashboard/public/atlas_data.csv — generated by generate_atlas_umap.py

    • 692 rows, 38 columns (688 embedded — 4 papers with no abstract are excluded from clustering)
    • Per-model 2D coords: umap_x_mpnet, umap_y_mpnet, umap_x_minilm, umap_y_minilm, umap_x_specter, umap_y_specter
    • Per-model 3D coords: umap_x3_mpnet, umap_y3_mpnet, umap_z3_mpnet (+ minilm, specter variants)
    • Default coords (mpnet): umap_x, umap_y, umap_x3, umap_y3, umap_z3
    • Cluster: cluster (int, -1 = noise), cluster_label (auto-labeled from top keywords)
    • Current: 4 clusters (0:93, 1:127, 2:301, 3:163), 8 noise points (1.2%)

  • dashboard/public/semantic_clusters.csv — generated by semantic_similarity.py

    • 692 rows (regenerated, complete)
    • Columns: SN, Paper Title, DOI, Domain, Year, umap_x, umap_y, umap_x_minilm, umap_y_minilm, umap_x_mpnet, umap_y_mpnet, umap_x_specter, umap_y_specter, cited_by_count, is_oa, oa_status, oa_url

CSV quoting: Use quoting=csv.QUOTE_ALL in pandas to avoid breaking Papa Parse.

Embedding Models & UMAP Parameters

Three SBERT models used throughout:

Key Model ID Dims Notes
mpnet all-mpnet-base-v2 768 Default/best — highest domain separation gap (0.111)
minilm all-MiniLM-L6-v2 384 Fast, general purpose (gap 0.100)
specter allenai-specter 768 Scientific paper trained (gap 0.059)

UMAP params (optimal, main analysis, MPNet): n_neighbors=30, min_dist=0.0, metric='cosine', random_state=42

  • Grid search confirmed these optimal (embedding silhouette=0.2564; MiniLM optimal at n_neighbors=15/min_dist=0.0, silhouette=0.2137; SPECTER optimal at n_neighbors=10/min_dist=0.05, silhouette=0.2052)
  • Grid search results are saved every run to evaluation/analysis/figures_new/embedding_grid_search_summary.csv and embedding_best_params_summary.csv — check these rather than re-deriving from stdout

UMAP params (robustness check): n_neighbors=15, min_dist=0.1 (library defaults)

  • Used in evaluation/analysis/umap_default_validation.py
  • Result: 3 clusters, 17 noise (2.5%), clustering silhouette=0.5331, ARI=0.36, NMI=0.41 (post-hoc vs domain labels)

HDBSCAN params (main): min_cluster_size=20, min_samples=10, metric='euclidean' — confirmed via grid search over the full corpus (do not assume min_samples=5 is still optimal after any corpus change; re-run the grid search in semantic_similarity.py, saved to hdbscan_grid_search_summary.csv) HDBSCAN params (robustness): same as main (min_cluster_size=20, min_samples=10)

  • Result: 4 clusters, 8 noise (1.2%), clustering silhouette=0.6852, ARI=0.53, NMI=0.47
  • Run on mpnet 2D coords only — cluster labels fixed regardless of which model is displayed
  • Domain labels are NEVER used in clustering; ARI/NMI computed post-hoc for validation only
  • Domain × cluster percentages should use the all-domain-papers denominator (matches the heatmap figures), not clustered-points-only — the two give different percentages when noise points exist, and only the all-domain version matches what's plotted

Key Paper Figures

  • evaluation/analysis/images/domain_cluster_heatmap.png/.pdf — main Domain×Cluster heatmap (optimal params). Run: cd algorithmic-bias-survey && python evaluation/analysis/domain_cluster_heatmap.py
  • evaluation/analysis/figures_new/robustness_heatmap.png/.pdf — robustness check (default UMAP). Run: python evaluation/analysis/umap_default_validation.py
  • Tex figure: 3 subfigures — (a) semantic_landscape_mpnet.png, (b) domain_cluster_heatmap.png, (c) robustness_heatmap.png

Full Pipeline (adding new papers)

For the contributor-facing version of this (with a data-flow diagram and common pitfalls), see CONTRIBUTING.md at the repo root.

# 1. Add rows to evaluation/analysis/papers_new.csv

# 2. Fetch OpenAlex metadata for new papers
#    Open extract_publication_metadata.ipynb and run for new DOIs
#    → updates openalex_enriched.csv

# 3. Regenerate semantic clusters (SBERT + UMAP, 3 models)
python3 evaluation/analysis/semantic_similarity.py
# → dashboard/public/semantic_clusters.csv
# → evaluation/analysis/figures_new/semantic_landscape_{mpnet,minilm,specter}.png

# 4. Regenerate atlas (SBERT + UMAP 2D+3D + HDBSCAN, 3 models)
python3 generate_atlas_umap.py
# → dashboard/public/atlas_data.csv

# 5. Update dashboard/public/papers.csv
python3 -c "
import pandas as pd, csv
df = pd.read_csv('evaluation/analysis/papers_new.csv')
df.to_csv('dashboard/public/papers.csv', index=False, quoting=csv.QUOTE_ALL)
"

# 6. Commit and push → deploy.yml auto-deploys to GitHub Pages
git add dashboard/public/ evaluation/analysis/papers_new.csv openalex_enriched.csv
git commit -m "Add N new papers, regenerate atlas and semantic clusters"
# Note: dashboard/public/papers.csv already has 692 rows with normalized country names
git push

Note: Steps 3 and 4 each take ~3–5 minutes locally (SBERT encoding + UMAP).

Analysis Scripts (evaluation/analysis/)

Script What it produces Needs Affiliations?
semantic_similarity.py UMAP figures (3 models) + semantic_clusters.csv No
embedding_comparison.py Per-model UMAP plots + domain separation ranking No
umap_grid_search.py Grid search heatmap + best params No
all_first_author_count_country_paper.py Country bar charts No
country_country_collab_heatmap.py Country collaboration matrix No
country_domain_heatmap_participation_all_authors_paper.py Country × Domain heatmap No
country_domain_heatmap_participation_first_author_paper.py Country × Domain heatmap (first author) No
domain_evolution_over_time.py Domain trends over years No
overall_domain_distribution_all_country.py Domain pie chart No
top_n_university_domain_all_author_paper.py Top 20 institutions heatmap Yes
top_n_university_domain_first_author_paper.py Top 20 institutions (first author) Yes

Note: Affiliations column in papers_new.csv is now populated for all 692 papers — the institution scripts run normally.

Dashboard

Run locally

cd dashboard
npm install
npm run dev    # localhost:3000

If you see Cannot find module './NNN.js': stale build cache. Run rm -rf .next then restart.

Page routing

Route Component Data source
/ AbstractAtlas public/atlas_data.csv
/atlas AbstractAtlas (same as home) public/atlas_data.csv
/analytics BiasResearchDashboard public/papers.csv
/advanced AdvancedAnalytics public/papers.csv

Key components

  • AbstractAtlas.tsx — Full-screen semantic atlas. Features:

    • 2D canvas UMAP (pan/zoom/lasso selection)
    • 3D canvas UMAP (auto-rotate, arcball drag, zoom)
    • Embedding model toggle: MPNet / MiniLM / SPECTER (switches both 2D and 3D coords)
    • World choropleth map + collaboration panel
    • Left sidebar: Domain + Cluster filters
    • Right panel: Filters & Analytics or DetailPanel (abstract, related papers, citation percentile, geography, cluster context)
    • Toolbar: search, color-mode (domain/cluster), lasso, map-view toggle, CSV download

  • BiasResearchDashboard.tsx — Research analytics. Loads papers.csv.

  • AdvancedAnalytics.tsx — Extended analytics. Loads papers.csv.

  • SemanticClusterChart.tsx — Dead file, not imported anywhere. Can be deleted.

Atlas internals

atlas_data.csv (Papa Parse, dynamicTyping: true)
  → papers[] (AtlasPaper type)
  → embedModel state ('mpnet' | 'minilm' | 'specter')
  → embedModelRef (synced each render for canvas callbacks)
  → dataRange / dataRange3 (recompute on embedModel change)
  → toScreen2 / project3 (useCallback on dataRange)
  → render2Ref / render3Ref (updated each render, called via rAF)
  → filteredPapers (domain + cluster + lasso + country + author + year + OA)

Stale-closure fix: render2/render3 captured in refs updated every render. All RAF and auto-rotate loops call renderXRef.current().

Deployment

Automatic via .github/workflows/deploy.yml on push to main. Builds static export → GitHub Pages.

// Base path handling (updated for custom domain)
const base = process.env.NEXT_PUBLIC_BASE_PATH || '';
Papa.parse(`${base}/atlas_data.csv`, ...);

Custom domain biasatlas.cair-nepal.org requires:

  • dashboard/public/CNAME with content biasatlas.cair-nepal.org
  • Empty basePath: "" in next.config.js
  • Custom domain enabled in GitHub repo Settings → Pages

Design System

  • Card style: rounded-2xl border border-gray-100 shadow-sm
  • Page background: bg-slate-50
  • Domain colours: Health=#e63946, General=#457b9d, Graph=#2a9d8f, LLMs=#e9c46a, Recommender=#f4a261
  • Cluster colours: Tol-bright 14-colour palette (colour-vision-friendly)
  • Atlas full-screen: height: calc(100vh - 72px), body overflow: hidden

Branches

  • main — Production, triggers GitHub Pages deploy on push
  • revision — Active revision branch for paper resubmission work

Repo Info

  • Renamed to CAIRNepal/biasatlas — update remote if needed: git remote set-url origin https://github.com/CAIRNepal/biasatlas.git
  • Dataset uploaded to Hugging Face: cair-nepal/ai-bias-research-landscape (dataset card: DATASET_CARD.md)
  • Dataset archived on Zenodo: https://doi.org/10.5281/zenodo.21221283

Gotchas (found the hard way)

  • Anchor script output paths to the script's own location (Path(__file__).resolve().parent), never a bare relative string. A bare relative path resolves against whatever directory you happen to run the script from — this previously caused several scripts to silently write to the repo root instead of evaluation/analysis/figures_new/, leaving stale data being read from the wrong place for an entire session.
  • Don't assume a previously-tuned hyperparameter is still optimal after the corpus changes. HDBSCAN min_samples was hardcoded at 5 from an earlier corpus size; re-running the grid search after dedup showed min_samples=10 was actually optimal. Re-check hdbscan_grid_search_summary.csv and embedding_best_params_summary.csv after any corpus change rather than reusing old hardcoded values.
  • Domain × cluster overlap percentages have two possible denominators (clustered-points-only vs. all-domain-papers) that give different numbers when noise points exist. The heatmap figures use all-domain-papers — match that convention when reporting percentages elsewhere, or the numbers won't agree with the figure.