Skip to content

[OpenLineageDao] creating cache in code for sources to prevent upsert…#59

Merged
lmassaoy merged 2 commits into
mainfrom
prevent-upsert-default-source
Aug 13, 2025
Merged

[OpenLineageDao] creating cache in code for sources to prevent upsert…#59
lmassaoy merged 2 commits into
mainfrom
prevent-upsert-default-source

Conversation

@jrosend

@jrosend jrosend commented Aug 13, 2025

Copy link
Copy Markdown
Contributor

This pull request introduces a caching mechanism for SourceRow objects in the OpenLineageDao to reduce redundant database upserts and improve performance. The main change is the addition of an in-memory cache and refactoring of the dataset upsert logic to utilize this cache. Below are the most important changes grouped by theme:

Performance Optimization

  • Added a static in-memory cache SOURCE_ROW_CACHE to the OpenLineageDao interface to store and reuse SourceRow objects, minimizing repeated upsert operations for sources with the same name.

Code Refactoring

  • Refactored the dataset upsert logic in upsertLineageDataset to use a new private helper method getOrUpsertSource, which checks the cache before performing an upsert and stores new sources in the cache. This replaces the previous logic that always performed a database upsert.

Imports

  • Added HashMap import to support the new caching mechanism.

@jrosend jrosend marked this pull request as ready for review August 13, 2025 17:58
@lmassaoy lmassaoy merged commit 3314ccc into main Aug 13, 2025
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants