Skip to content

[PLUGIN-1844] Add label and use storage client for bucket cleanup.#1552

Merged
itsankit-google merged 1 commit into
developfrom
PLUGIN-1844
Jun 3, 2025
Merged

[PLUGIN-1844] Add label and use storage client for bucket cleanup.#1552
itsankit-google merged 1 commit into
developfrom
PLUGIN-1844

Conversation

@itsankit-google

@itsankit-google itsankit-google commented Jun 2, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

context : It is observed multiple times that temp GCS buckets are left undeleted after pipeline execution.

Tested in local sandbox

2025-06-02 20:36:34,463 - DEBUG [SparkRunner-phase-1:i.c.p.g.b.u.BigQueryUtil@737] - Deleted GCS bucket 'bq-source-bucket-c533479c-eae1-449d-9ebe-1b29645ff395'.
2025-06-02 20:36:34,463 - DEBUG [SparkRunner-phase-1:i.c.p.g.c.GCPUtils@270] - Setting read timeout to 120 seconds.
2025-06-02 20:36:34,634 - DEBUG [SparkRunner-phase-1:i.c.p.g.b.s.BigQuerySourceUtils@186] - Deleted temporary table '_E2E_SOURCE_54983dcc_6cbe_4bf1_9617_045b7b885495_99a24d64_6dc9_4e11_8112_eff4487c0492'
2025-06-02 20:36:35,411 - DEBUG [SparkRunner-phase-1:i.c.p.g.b.u.BigQueryUtil@737] - Deleted GCS bucket 'bq-sink-bucket-0c7abd5d-38a4-4ffa-b6b1-cbdc32898fd0'

image

@itsankit-google
itsankit-google force-pushed the PLUGIN-1844 branch 4 times, most recently from 8e446fa to 166006e Compare June 2, 2025 18:12
@itsankit-google itsankit-google added the build Trigger unit test build label Jun 2, 2025
@itsankit-google
itsankit-google force-pushed the PLUGIN-1844 branch 2 times, most recently from 9acfa88 to 9214ad3 Compare June 2, 2025 18:20
@itsankit-google
itsankit-google force-pushed the PLUGIN-1844 branch 11 times, most recently from 3d46eba to a0da216 Compare June 2, 2025 22:47
@itsankit-google itsankit-google changed the title [PLUGIN-1844] Add support for cleanup of temp gcs buckets after pipeline completion [PLUGIN-1844] Add label and use storage client for bucket cleanup. Jun 2, 2025
@itsankit-google
itsankit-google merged commit c13bece into develop Jun 3, 2025
15 of 16 checks passed
@itsankit-google
itsankit-google deleted the PLUGIN-1844 branch June 3, 2025 08:27
This was referenced Jun 3, 2025
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sahusanket sahusanket sahusanket approved these changes

Assignees

No one assigned

Labels

build Trigger unit test build

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@itsankit-google @sahusanket