Skip to content

[DO NOT MERGE] Validate beam 2.74.0 RC1#3833

Draft
Amar3tto wants to merge 4 commits into
GoogleCloudPlatform:mainfrom
akvelon:validate-beam-274-rc1
Draft

[DO NOT MERGE] Validate beam 2.74.0 RC1#3833
Amar3tto wants to merge 4 commits into
GoogleCloudPlatform:mainfrom
akvelon:validate-beam-274-rc1

Conversation

@Amar3tto

@Amar3tto Amar3tto commented May 19, 2026

Copy link
Copy Markdown
Collaborator

No description provided.

@codecov

codecov Bot commented May 19, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 54.95%. Comparing base (28916c8) to head (d311c38).
⚠️ Report is 7 commits behind head on main.

Files with missing lines Patch % Lines
.../java/org/apache/beam/it/gcp/TemplateTestBase.java 0.00% 2 Missing ⚠️

❌ Your patch check has failed because the patch coverage (0.00%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files 
@@             Coverage Diff              @@
##               main    #3833      +/-   ##
============================================
+ Coverage     53.99%   54.95%   +0.95%     
- Complexity     6895     6923      +28     
============================================
  Files          1096     1096              
  Lines         67352    67355       +3     
  Branches       7558     7558              
============================================
+ Hits          36368    37016     +648     
+ Misses        28515    27866     -649     
- Partials       2469     2473       +4
Components Coverage Δ
spanner-templates 74.80% <ø> (+1.98%) ⬆️
spanner-import-export 68.51% <ø> (-0.02%) ⬇️
spanner-live-forward-migration 87.39% <ø> (+6.44%) ⬆️
spanner-live-reverse-replication 78.80% <ø> (+1.71%) ⬆️
spanner-bulk-migration 90.97% <ø> (+<0.01%) ⬆️
gcs-spanner-dv 85.75% <ø> (ø)
Files with missing lines Coverage Δ
.../java/org/apache/beam/it/gcp/TemplateTestBase.java 0.00% <0.00%> (ø)

... and 25 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Abacn

Abacn commented May 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Comparing with surefire-integration-test-result there is some test for v1 template stuck

Raw log as a table, only v1 tests are missing:
v2/astradb-to-bigquery/target/surefire-reports/TEST-com.google.cloud.teleport.v2.astradb.templates.AstraDbToBigQueryIT.xml 1 ✅ 938s
v2/azure-eventhub-to-pubsub/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.AzureEventhubToPubsubIT.xml 1 ⚪ 126s
v2/bigquery-to-parquet/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.BigQueryToParquetIT.xml 2 ✅ 992s
v2/cdc-parent/cdc-change-applier/target/surefire-reports/TEST-com.google.cloud.dataflow.cdc.applier.CdcToBigQueryChangeApplierPipelineIT.xml 2 ✅ 707s
v2/file-format-conversion/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.FileFormatConversionIT.xml 4 ✅ 1101s
v2/firestore-to-firestore/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.FirestoreToFirestoreIT.xml 3 ✅ 1 ⚪ 1700s
v2/googlecloud-and-jms/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.JmsToPubsubIT.xml 2 ✅ 1122s
v2/googlecloud-and-jms/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.PubsubToJmsIT.xml 2 ✅ 1137s
v2/googlecloud-and-mongodb/target/surefire-reports/TEST-com.google.cloud.teleport.v2.mongodb.templates.MongoDbToBigQueryIT.xml 2 ✅ 560s
v2/googlecloud-and-mongodb/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.PubSubToMongoDBIT.xml 3 ✅ 1369s
v2/googlecloud-to-clickhouse/target/surefire-reports/TEST-com.google.cloud.teleport.v2.clickhouse.BigQueryToClickHouseIT.xml 2 ✅ 1306s
v2/googlecloud-to-clickhouse/target/surefire-reports/TEST-com.google.cloud.teleport.v2.clickhouse.PubSubToClickHouseIT.xml 6 ✅ 1313s
v2/googlecloud-to-elasticsearch/target/surefire-reports/TEST-com.google.cloud.teleport.v2.elasticsearch.templates.BigQueryToElasticsearchIT.xml 4 ✅ 1451s
v2/googlecloud-to-elasticsearch/target/surefire-reports/TEST-com.google.cloud.teleport.v2.elasticsearch.templates.GCSToElasticsearchIT.xml 5 ✅ 1466s
v2/googlecloud-to-elasticsearch/target/surefire-reports/TEST-com.google.cloud.teleport.v2.elasticsearch.templates.PubSubToElasticsearchIT.xml 3 ✅ 1566s
v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.bigtablechangestreamstobigquery.BigtableChangeStreamsToBigQueryIT.xml 3 ✅ 729s
v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.bigtablechangestreamstogcs.BigtableChangeStreamsToGcsIT.xml 7 ✅ 929s
v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.bigtablechangestreamstopubsub.BigtableChangeStreamsToPubSubIT.xml 7 ✅ 3 ⚪ 835s
v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.bigtablechangestreamstovectorsearch.BigtableChangeStreamsToVectorSearchIT.xml 1 ⚪ 0ms
v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.DLPTextToBigQueryStreamingIT.xml 3 ✅ 653s
v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.PubSubToAvroIT.xml 2 ✅ 727s
v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.PubSubToBigQueryAutoIT.xml 3 ✅ 1 ⚪ 1704s
v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.PubSubToBigQueryIT.xml 4 ✅ 1745s
v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.pubsubtotext.PubsubToTextIT.xml 2 ✅ 680s
v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.spannerchangestreamstobigquery.SpannerChangeStreamsToBigQueryIT.xml 4 ✅ 1 ⚪ 816s
v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.SpannerChangeStreamsToGcsIT.xml 2 ✅ 2 ⚪ 789s
v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.SpannerChangeStreamsToPubSubIT.xml 2 ✅ 754s
v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.SpannerToBigQueryIT.xml 2 ✅ 723s
v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.TextIOToBigQueryIT.xml 3 ✅ 1650s
v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.TextToBigQueryStreamingIT.xml 3 ✅ 1714s
v2/googlecloud-to-neo4j/target/surefire-reports/TEST-com.google.cloud.teleport.v2.neo4j.templates.ConstraintsIndicesIT$Neo4j44CommunityIT.xml 3 ✅ 430s
v2/googlecloud-to-neo4j/target/surefire-reports/TEST-com.google.cloud.teleport.v2.neo4j.templates.ConstraintsIndicesIT$Neo4j44EnterpriseIT.xml 3 ✅ 520s
v2/googlecloud-to-neo4j/target/surefire-reports/TEST-com.google.cloud.teleport.v2.neo4j.templates.ConstraintsIndicesIT$Neo4j5CommunityIT.xml 3 ✅ 429s
v2/googlecloud-to-neo4j/target/surefire-reports/TEST-com.google.cloud.teleport.v2.neo4j.templates.ConstraintsIndicesIT$Neo4j5EnterpriseIT.xml 1 ⚪ 0ms
v2/googlecloud-to-neo4j/target/surefire-reports/TEST-com.google.cloud.teleport.v2.neo4j.templates.CustomNeo4jWriteQueryIT.xml 6 ✅ 561s
v2/googlecloud-to-neo4j/target/surefire-reports/TEST-com.google.cloud.teleport.v2.neo4j.templates.DataConversionIT.xml 12 ✅ 775s
v2/googlecloud-to-neo4j/target/surefire-reports/TEST-com.google.cloud.teleport.v2.neo4j.templates.SecretManagerIT.xml 1 ✅ 545s
v2/jdbc-and-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.JdbcToBigQueryIT.xml 9 ✅ 568s
v2/jdbc-and-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.JdbcToPubSubAutoIT.xml 1 ✅ 822s
v2/jdbc-and-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.JdbcToPubsubIT.xml 1 ✅ 543s
v2/jdbc-and-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.PubsubToJdbcIT.xml 1 ✅ 622s
v2/managed-io-to-managed-io/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.ManagedIOToManagedIOIT.xml 1 ✅ 543s
v2/mqtt-to-pubsub/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.MqttToPubsubTestIT.xml 1 ✅ 550s
v2/mysql-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.MySQLToBigQueryIT.xml 1 ✅ 578s
v2/oracle-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.OracleToBigQueryIT.xml 1 ✅ 600s
v2/postgresql-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.PostgreSQLToBigQueryIT.xml 1 ✅ 538s
v2/pubsub-binary-to-bigquery/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.PubsubAvroToBigQueryIT.xml 1 ✅ 518s
v2/pubsub-binary-to-bigquery/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.PubsubProtoToBigQueryIT.xml 2 ✅ 1475s
v2/pubsub-cdc-to-bigquery/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.PubSubCdcToBigQueryIT.xml 2 ✅ 538s
v2/pubsub-to-redis/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.PubSubToRedisIT.xml 3 ✅ 1313s
v2/sqlserver-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.SQLServerToBigQueryIT.xml 1 ✅ 505s
v2/streaming-data-generator/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.StreamingDataGeneratorIT.xml 9 ✅ 1 ⚪ 682s

Update: v1 tests passed if only run alone: https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/runs/26170831646/job/76992749193

@Abacn

Abacn commented May 20, 2026

Copy link
Copy Markdown
Contributor

I suspect it's some teardown step gets stuck. The last Dataflow job already finished after 2h run the test kick off.

@Abacn

Abacn commented May 20, 2026

Copy link
Copy Markdown
Contributor

It is com.google.cloud.teleport.templates.DatastoreToTextIT stuck indefinitely. ssh'd into the VM running the test and did a thread dump, suspicious thread is

name="testFirestoreToText" classname="com.google.cloud.teleport.templates.DatastoreToTextIT" time="13778.677"

"com.google.cloud.teleport.templates.DatastoreToText.main()" #279 prio=5 os_prio=0 cpu=14113.11ms elapsed=12914.50s tid=0x000070c6d81f2c90 nid=0x102572 waiting on condition  [0x000070c860885000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@17.0.17/Native Method)
	- parking to wait for  <0x00000000bd111400> (a java.util.concurrent.CompletableFuture$Signaller)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@17.0.17/LockSupport.java:252)
	at java.util.concurrent.CompletableFuture$Signaller.block(java.base@17.0.17/CompletableFuture.java:1866)
	at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@17.0.17/ForkJoinPool.java:3465)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@17.0.17/ForkJoinPool.java:3436)
	at java.util.concurrent.CompletableFuture.timedGet(java.base@17.0.17/CompletableFuture.java:1939)
	at java.util.concurrent.CompletableFuture.get(java.base@17.0.17/CompletableFuture.java:2095)
	at org.apache.beam.sdk.util.MoreFutures.get(MoreFutures.java:69)
	at org.apache.beam.runners.dataflow.util.PackageUtil.stageClasspathElements(PackageUtil.java:354)
	at org.apache.beam.runners.dataflow.util.PackageUtil.stageClasspathElements(PackageUtil.java:239)
	at org.apache.beam.runners.dataflow.util.GcsStager.stageFiles(GcsStager.java:53)
	at org.apache.beam.runners.dataflow.DataflowRunner.stageArtifacts(DataflowRunner.java:1179)
	at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:1355)
	at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:212)
	at org.apache.beam.sdk.Pipeline.run(Pipeline.java:325)
	at org.apache.beam.sdk.Pipeline.run(Pipeline.java:310)
	at com.google.cloud.teleport.templates.DatastoreToText.main(DatastoreToText.java:128)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@17.0.17/Native Method)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@17.0.17/NativeMethodAccessorImpl.java:77)
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17.0.17/DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(java.base@17.0.17/Method.java:569)
	at org.codehaus.mojo.exec.AbstractExecJavaBase.executeMainMethod(AbstractExecJavaBase.java:402)
	at org.codehaus.mojo.exec.ExecJavaMojo.executeMainMethod(ExecJavaMojo.java:142)
	at org.codehaus.mojo.exec.AbstractExecJavaBase.doExecClassLoader(AbstractExecJavaBase.java:377)
	at org.codehaus.mojo.exec.AbstractExecJavaBase.lambda$execute$0(AbstractExecJavaBase.java:287)
	at org.codehaus.mojo.exec.AbstractExecJavaBase$$Lambda$1998/0x000070c868cb27d8.run(Unknown Source)
	at java.lang.Thread.run(java.base@17.0.17/Thread.java:840)

and once I run kill -9 <pid> for the process running test, it does show a single test failure (time 13779s): https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/runs/26124431477?pr=3833

Error staging template" type="org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.util.concurrent.UncheckedExecutionException"><![CDATA[org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalArgumentException: Error staging template
	at org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2086)
	at org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.cache.LocalCache.get(LocalCache.java:4012)
	at org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4922)
	at org.apache.beam.it.gcp.TemplateTestBase.getSpecPath(TemplateTestBase.java:340)
	at org.apache.beam.it.gcp.TemplateTestBase.setUpBase(TemplateTestBase.java:264)

stuck at staging templates for some reason

@Abacn

Abacn commented May 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

I think we've hit JDK-8349206: https://bugs.openjdk.org/browse/JDK-8349206 EDIT: it's not related to JDK bug

There are many blocking threads

"pool-15-thread-1" #289 prio=5 os_prio=0 cpu=311.84ms elapsed=12855.24s tid=0x000070c6e1b6b050 nid=0x102d4c waiting for monitor entry  [0x000070c8c43fd000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at java.util.logging.StreamHandler.publish(java.logging@17.0.17/StreamHandler.java:194)
	- waiting to lock <0x00000000a3f7a578> (a java.util.logging.ConsoleHandler)
	at java.util.logging.ConsoleHandler.publish(java.logging@17.0.17/ConsoleHandler.java:95)
	at java.util.logging.Logger.log(java.logging@17.0.17/Logger.java:980)
	at com.google.common.flogger.backend.system.AbstractBackend.log(AbstractBackend.java:101)
	at com.google.common.flogger.backend.system.SimpleLoggerBackend.log(SimpleLoggerBackend.java:32)
	at com.google.common.flogger.AbstractLogger.write(AbstractLogger.java:163)
	at com.google.common.flogger.LogContext.logImpl(LogContext.java:659)
	at com.google.common.flogger.LogContext.log(LogContext.java:787)
	at com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.<init>(GoogleCloudStorageImpl.java:337)
	at com.google.cloud.hadoop.gcsio.AutoBuilder_GoogleCloudStorageImpl_Builder.build(AutoBuilder_GoogleCloudStorageImpl_Builder.java:68)
	at org.apache.beam.sdk.extensions.gcp.util.GcsUtilV1.createGoogleCloudStorage(GcsUtilV1.java:739)

due to some (benign) error happens when the uploaded artifact already exists:

if (canIgnoreException) {
                  logger.atInfo().log(

https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/73e2c447c2e9993957a04b016410c9ca596c3189/gcsio/src/main/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorageClientImpl.java#L363

and stuck inside logger.atInfo()).log(

This log line is added in GoogleCloudDataproc/hadoop-connectors#1094, due to gcsio 3.x upgrade: apache/beam#38419

@Abacn Abacn mentioned this pull request May 20, 2026
Draft
@Abacn Abacn mentioned this pull request May 21, 2026
Closed
3 tasks
@Abacn

Abacn commented May 21, 2026

Copy link
Copy Markdown
Contributor

Actually now I think it's an pre-existing bug in TemplateTestBase.java‎

It uses Process exec = Runtime.getRuntime().exec(mavenCmd); to start the process and only consumes input and errorstream after it's done. When java subprocess buffer size is full write to stderr is stuck.

When staging templates, slf4j logs are not emitted:

[Thread-15349] INFO org.apache.beam.it.gcp.TemplateTestBase - SLF4J(W): No SLF4J providers were found.
[Thread-15349] INFO org.apache.beam.it.gcp.TemplateTestBase - SLF4J(W): Defaulting to no-operation (NOP) logger implementation
[Thread-15349] INFO org.apache.beam.it.gcp.TemplateTestBase - SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.

Then gcsio 3.0 added many Google Fluent log, which does not use slf4j, and defaults to JUL's ConsoleLog, suddenly there are many logs in the buffer

Testing

0ca80be

on https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/runs/26236531476 now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

ignore-for-release size/XXL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Amar3tto @Abacn