Skip to content

DATA-5365: Fix case/punctuation-sensitive committee matching for USA events - #207

Merged
sinchana00 merged 1 commit into
mainfrom
DATA-5365/USA-events-and-committee-match
Sep 15, 2026
Merged

sinchana00 merged 1 commit into
mainfrom
DATA-5365/USA-events-and-committee-match

Conversation

@sinchana00

Copy link
Copy Markdown
Contributor

Replaced the formatting-guessing heuristic with normalized comparison on both sides:

  1. Expand & → and
  2. Strip all punctuation
  3. Collapse whitespace
  4. Uppercase
    This normalization is applied identically to the incoming scraped name (in Python) and to Organization.name + every entry in Organization.other_names (via a Postgres regexp_replace/jsonb_array_elements SQL expression), so matches no longer depend on guessing exact formatting conventions.

Testing

Added test_related_committee_event_other_names_case_mismatch unit test.
All tests passed (351/351)
Manually verified against real scraped event data for TX and IL (different scrapers, different naming conventions), confirming committees resolve correctly despite case, punctuation, Oxford comma, and &/and differences - verified directly against a local Postgres DB via psql.

@jessemortenson

Copy link
Copy Markdown
Member

Interesting solution! I need to verify some local testing results with real data, but looks promising

@jessemortenson

Copy link
Copy Markdown
Member

LGTM - we'll need to go over openstates-core release procedure, but if you want to take a shot on Monday you can. It's a few steps. Basically:

  1. Follow steps in openstates-core/RELEASE.md
  2. Those steps will create a new release of the openstates package with a new version number. It is "officially" released when that new version is available on pypi. Some steps of the CI will fail (that have to do with trying to push a separate "metadata" package - that's been broken for a while :( ). But if the new version is on PyPi and you can upgrade a local copy of openstates-scrapers to that new version of openstates, then it's good enough.
  3. Modify openstates-scrapers to use that new version of openstates and make and merge a PR. Once that second CI process completes, then the next round of scraper runs will actually pull an image that uses the new openstates-core version's code.

@sinchana00
sinchana00 merged commit b500b24 into main Sep 15, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants