Prevent potential GPU OOM in R2C with split retry by thirtiseven · Pull Request #14073 · NVIDIA/cudf-spark

thirtiseven · 2025-12-31T02:19:17Z

Description

#13842 added retry for R2C convert to prevent Host OOM. Following on from that, this pr aimed to add split and retry for GPU OOM when copying the converted results to GPU.

This PR:

Introduced HostColumnarBatchWithRowRange, a wrapper for host columns that supports logical slicing without copying underlying host memory. This allows splitting a large host batch into smaller chunks for transfer.
Updated GpuRowToColumnarExec to use split and retry when a GPU OOM occurs during transfer.

Note that when we split a batch into two halves, we can't free them until both halves are processed. So, the first half just borrows the data, but we pass the ownership of the host columns to the second half. This ensures the host memory stays alive exactly as long as needed and is freed when the last split is closed.

Performance tests:

Benchmark measures end-to-end time for R2C conversion + GPU aggregation on 100M rows. Data is created via sc.parallelize + createDataFrame to force CPU rows through GpuRowToColumnarExec. Three test groups:

String columns — (Int, String, Int), forces non-codegen RowToColumnarIterator path (PR's code path)
Nested columns — (Int, Array[Int], Struct[Int, String]), same path
Fixed-width columns — (Int, Int, Int, Double), codegen path (control, unaffected by this PR)

Each group: 5 warmup + 10 timed runs, reporting median. spark-shell --master local[4] --driver-memory 8g.

Configuration	String cols	Nested cols	Fixed-width (control)
main (baseline)	8,416 ms	19,349 ms	4,025 ms
This PR	8,611 ms	20,040 ms	4,044 ms
Overhead	+2.3%	+3.6%	+0.5% (noise)

benchmark code:
r2c_split_retry_benchmark.scala.zip

Checklists

Documentation

Updated for new or modified user-facing features or behaviors
No user-facing change

Testing

Added or modified tests to cover new code paths
Covered by existing tests
(Please provide the names of the existing tests in the PR description.)
Not required

Performance

Tests ran and results are added in the PR description
Issue filed with a link in the PR description
Not required

thirtiseven · 2025-12-31T02:19:56Z

@greptile full review

Copilot

Pull request overview

This PR adds split retry functionality to Row-to-Columnar (R2C) conversion to prevent GPU out-of-memory (OOM) errors by implementing a mechanism to split batches in half when GPU OOM occurs during host-to-GPU data transfer.

Key Changes

Introduced a new HostColumnarBatchWithRowRange class that wraps host columns with row range tracking and supports splitting
Modified RowToColumnarIterator to use split retry logic for GPU OOM scenarios, allowing single input batches to produce multiple output batches
Added buildHostColumnsWithoutOwnership() method to transfer ownership of host columns to the retry framework

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
`sql-plugin/src/main/scala/com/nvidia/spark/rapids/HostColumnarBatchWithRowRange.scala`	New class that wraps host columns with row range support, implements splitting logic for GPU OOM retry, and handles slicing of various column types (LIST, STRUCT, STRING, fixed-width)
`sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuRowToColumnarExec.scala`	Updated `RowToColumnarIterator` to build host columns separately, use split retry for TargetSize goals, and maintain a pending batch iterator for split outputs
`sql-plugin/src/main/java/com/nvidia/spark/rapids/GpuColumnVector.java`	Added `buildHostColumnsWithoutOwnership()` method to transfer host column ownership to caller
`tests/src/test/scala/com/nvidia/spark/rapids/RowToColumnarIteratorRetrySuite.scala`	Added comprehensive test coverage for GPU OOM split retry scenarios including single batch requirement enforcement, multiple batch production, multiple consecutive splits, and single row edge case

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

greptile-apps · 2025-12-31T02:31:47Z

Greptile Summary

This PR adds GPU OOM split-and-retry for the host-to-GPU transfer step in RowToColumnarIterator, addressing cases where a fully-converted host batch is too large to fit on the GPU in one shot. It introduces HostColumnarBatchWithRowRange, a ref-counted wrapper that enables zero-copy logical slicing of host columns by row range, and wires it into buildBatch() via withRetry + a splitInHalf split policy.

HostColumnarBatchWithRowRange implements ref-counting over HostColumnVector arrays, copyToGpu() with an optimised fast path (no slice when the range covers the full column), and a recursive sliceHostColumn that handles LIST, STRUCT, STRING, and fixed-width types including deeply nested schemas.
GpuRowToColumnarExec changes replace the old withRetryNoSplit(tryBuild) call with a two-phase approach: build host columns first, then use withRetry(hostBatch, splitHostBatchInHalf) for the GPU transfer; the first split result is returned immediately while the remaining iterator is held in pendingBatchIter and drained lazily via subsequent next() calls with semaphore re-acquisition.
Tests cover empty schema, single/multi OOM splits, the unsplittable single-row case, nested complex types, and GPU semaphore re-acquisition between deferred split batches.

Confidence Score: 5/5

Safe to merge; the new split-retry path is well-isolated, resource lifecycle follows ARM conventions throughout, and the retry framework's task-completion listener covers deferred iterator cleanup.

The change introduces non-trivial host-side slicing logic but it is thoroughly tested across string, nested, and fixed-width types. Resource management in HostColumnarBatchWithRowRange is correct: ref counts are incremented on construction and decremented on close, splitInHalf closes the input inside withResource, and both copyAllToGpu/copyRangeToGpu use closeOnExcept correctly around copyToDevice()+GpuColumnVector.from(). The two observations (unclosed child views and INT32 overflow on giant string slices) are speculative rather than present defects on any realistic input.

HostColumnarBatchWithRowRange.scala is the highest-complexity new file and deserves a careful second read, particularly sliceHostColumn for LIST/STRUCT recursion and the validity-slice logic.

Important Files Changed

Filename	Overview
sql-plugin/src/main/scala/com/nvidia/spark/rapids/HostColumnarBatchWithRowRange.scala	New class implementing logical slicing of host column batches for GPU OOM split-retry; ARM patterns used correctly throughout, one subtle concern with unclosed child column views from getChildColumnView
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuRowToColumnarExec.scala	Refactors host-to-GPU transfer step to use withRetry+splitHostBatchInHalf, introduces pendingBatchIter for deferred split delivery, recordOutput helper correctly accounts for split metrics
tests/src/test/scala/com/nvidia/spark/rapids/RowToColumnarIteratorRetrySuite.scala	Adds five new tests covering empty schema, single-OOM split, multi-OOM split, single-row unsplittable case, and nested types; resources properly closed in finally blocks

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Caller
    participant RowToColIter
    participant withRetry
    participant HostBatch as HostColumnarBatchWithRowRange
    participant GPU

    Caller->>RowToColIter: next()
    RowToColIter->>RowToColIter: pendingBatchIter.hasNext? false
    RowToColIter->>RowToColIter: buildBatch()
    RowToColIter->>RowToColIter: convertRows(builders) rowCount
    RowToColIter->>RowToColIter: builders.buildHostColumns() hostColumns[]
    RowToColIter->>HostBatch: HostColumnarBatchWithRowRange(hostColumns, rowCount)
    RowToColIter->>withRetry: withRetry(hostBatch, splitHostBatchInHalf)
    withRetry->>HostBatch: copyToGpu()
    HostBatch--xwithRetry: GpuSplitAndRetryOOM
    withRetry->>HostBatch: splitInHalf(batch)
    HostBatch-->>withRetry: [firstHalf, secondHalf]
    withRetry->>withRetry: push secondHalf to attemptStack
    withRetry->>HostBatch: firstHalf.copyToGpu()
    HostBatch->>GPU: H2D transfer rows 0..n/2
    GPU-->>HostBatch: ColumnarBatch first half
    HostBatch-->>withRetry: first GPU batch
    withRetry-->>RowToColIter: it iterator with secondHalf pending
    RowToColIter->>RowToColIter: it.next() first batch
    RowToColIter->>RowToColIter: "pendingBatchIter = it"
    RowToColIter-->>Caller: first batch rows 0..n/2
    Caller->>RowToColIter: next()
    RowToColIter->>RowToColIter: pendingBatchIter.hasNext? true
    RowToColIter->>RowToColIter: GpuSemaphore.acquireIfNecessary
    RowToColIter->>withRetry: pendingBatchIter.next()
    withRetry->>HostBatch: secondHalf.copyToGpu()
    HostBatch->>GPU: H2D transfer rows n/2..n
    GPU-->>HostBatch: ColumnarBatch second half
    withRetry-->>RowToColIter: second batch
    RowToColIter-->>Caller: second batch rows n/2..n

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Caller
    participant RowToColIter
    participant withRetry
    participant HostBatch as HostColumnarBatchWithRowRange
    participant GPU

    Caller->>RowToColIter: next()
    RowToColIter->>RowToColIter: pendingBatchIter.hasNext? false
    RowToColIter->>RowToColIter: buildBatch()
    RowToColIter->>RowToColIter: convertRows(builders) rowCount
    RowToColIter->>RowToColIter: builders.buildHostColumns() hostColumns[]
    RowToColIter->>HostBatch: HostColumnarBatchWithRowRange(hostColumns, rowCount)
    RowToColIter->>withRetry: withRetry(hostBatch, splitHostBatchInHalf)
    withRetry->>HostBatch: copyToGpu()
    HostBatch--xwithRetry: GpuSplitAndRetryOOM
    withRetry->>HostBatch: splitInHalf(batch)
    HostBatch-->>withRetry: [firstHalf, secondHalf]
    withRetry->>withRetry: push secondHalf to attemptStack
    withRetry->>HostBatch: firstHalf.copyToGpu()
    HostBatch->>GPU: H2D transfer rows 0..n/2
    GPU-->>HostBatch: ColumnarBatch first half
    HostBatch-->>withRetry: first GPU batch
    withRetry-->>RowToColIter: it iterator with secondHalf pending
    RowToColIter->>RowToColIter: it.next() first batch
    RowToColIter->>RowToColIter: "pendingBatchIter = it"
    RowToColIter-->>Caller: first batch rows 0..n/2
    Caller->>RowToColIter: next()
    RowToColIter->>RowToColIter: pendingBatchIter.hasNext? true
    RowToColIter->>RowToColIter: GpuSemaphore.acquireIfNecessary
    RowToColIter->>withRetry: pendingBatchIter.next()
    withRetry->>HostBatch: secondHalf.copyToGpu()
    HostBatch->>GPU: H2D transfer rows n/2..n
    GPU-->>HostBatch: ColumnarBatch second half
    withRetry-->>RowToColIter: second batch
    RowToColIter-->>Caller: second batch rows n/2..n

_{Reviews (2): Last reviewed commit: "address new comments" | Re-trigger Greptile}

greptile-apps

Additional Comments (1)

sql-plugin/src/main/scala/com/nvidia/spark/rapids/HostColumnarBatchWithRowRange.scala, line 176-184 (link)

logic: Workaround condition may not cover all cases. If dataLen == 0 but nullCount > 0 (empty strings with some null values), the code goes to the else branch and creates a 0-length slice, which could trigger the same copyToDevice issue mentioned in the comment.

_{4 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

thirtiseven · 2025-12-31T07:45:05Z

build

thirtiseven · 2025-12-31T10:44:44Z

build

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

greptile-apps

Additional Comments (1)

sql-plugin/src/main/scala/com/nvidia/spark/rapids/HostColumnarBatchWithRowRange.scala, line 178-188 (link)

style: Check that this empty string workaround handles all edge cases correctly. The 1-byte allocation when dataLen == 0 && nullCount == 0 works around copyToDevice behavior, but verify it handles: (1) all strings in slice are empty (not null), (2) mixed empty strings at different positions. This edge case could cause issues with certain data distributions.

_{4 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps

Additional Comments (1)

sql-plugin/src/main/scala/com/nvidia/spark/rapids/HostColumnarBatchWithRowRange.scala, line 180 (link)

syntax: typo: "Existsing" should be "Existing"

_{4 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps

Additional Comments (1)

sql-plugin/src/main/scala/com/nvidia/spark/rapids/HostColumnarBatchWithRowRange.scala, line 178-186 (link)

style: Consider if allocating a 1-byte dummy buffer for empty strings when dataLen == 0 but numRows > nullCount is necessary, or if cudf would accept null here to save allocations.

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

_{4 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

firestarman · 2026-01-06T03:08:04Z

@greptileai can you review it again ?

greptile-apps

Additional Comments (1)

sql-plugin/src/main/scala/com/nvidia/spark/rapids/HostColumnarBatchWithRowRange.scala, line 180-188 (link)

style: Edge case handling for empty strings is correct: when dataLen == 0 but there are non-null rows (numRows > nullCount), allocates 1-byte buffer to satisfy cuDF requirements. However, consider validating that this 1-byte allocation doesn't cause issues if we're in a tight memory situation (though this is unlikely given the small size).

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

_{4 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

thirtiseven · 2026-01-07T07:41:46Z

build

greptile-apps

Additional Comments (1)

sql-plugin/src/main/scala/com/nvidia/spark/rapids/HostColumnarBatchWithRowRange.scala, line 271-282 (link)

logic: Documentation claims split ownership transfer but implementation doesn't match - both splits increment reference count, so memory isn't transferred to last split as stated. Should the memory ownership transfer logic be implemented, or should the documentation be corrected to reflect the current reference counting approach?

_{3 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

thirtiseven · 2026-01-07T07:53:38Z

build

abellina · 2026-01-08T02:21:14Z

@thirtiseven I should have time to look at this tomorrow, if it can wait until then.

abellina · 2026-01-09T16:44:52Z

+    totalOutputBytes += GpuColumnVector.getTotalDeviceMemoryUsed(batch)
+    totalOutputRows += batch.numRows()
+    if (totalOutputRows > 0 && totalOutputBytes > 0) {
+      targetRows =


should we call this "nextBatchTargetRows"?

abellina · 2026-01-09T16:47:27Z

+          // Return the first batch now and keep the iterator for subsequent output batches.
+          // This ensures we only transfer one split at a time (avoid multiple device allocations).
+          closeOnExcept(it.next()) { first =>
+            pendingBatchIter = it


should we check if it.hasNext before we set pendingBatchIter? e.g. set to None otherwise?

yes good catch

abellina · 2026-01-09T18:47:08Z

+ *
+ * Memory management uses reference counting: each instance increments the reference count
+ * of the host columns on construction and decrements it on close. The host columns are
+ * freed when the last reference is closed.


we should add that validity and offset buffers are copied, not logically sliced, in this description.

greptile-apps

_{No files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

thirtiseven · 2026-01-20T06:51:38Z

build

thirtiseven · 2026-01-22T06:42:16Z

Hi @abellina could you take another look?

nvauto · 2026-01-26T02:05:11Z

NOTE: release/26.02 has been created from main. Please retarget your PR to release/26.02 if it should be included in the release.

thirtiseven · 2026-02-02T09:55:28Z

build

abellina · 2026-02-02T16:21:59Z

-        if (totalOutputRows > 0 && totalOutputBytes > 0) {
-          targetRows =
-            GpuBatchUtils.estimateRowCount(targetSizeBytes, totalOutputBytes, totalOutputRows)
+        val dataTypes = localSchema.fields.map(_.dataType)


it would be better if this was a class val.

abellina · 2026-02-02T16:22:58Z

+          HostColumnarBatchWithRowRange(hostColumns, rowCount, dataTypes)
        }

+        if (localGoal.isInstanceOf[RequireSingleBatchLike]) {


localGoal match { case RequireSingleBatchLike => ... case => // other goals ... }

abellina · 2026-02-02T16:25:29Z

+          // Return the first batch now and keep the iterator for subsequent output batches.
+          // This ensures we only transfer one split at a time (avoid multiple device allocations).
+          closeOnExcept(it.next()) { first =>
+            pendingBatchIter = if (it.hasNext) it else Iterator.empty


nit, could just set pendingBatchIter = it, without checking, since we should be gating all the other logic in that it.hasNext returns false

thanks for the review, done.

greptile-apps

_{3 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

thirtiseven · 2026-03-18T05:35:58Z

Let's waiting #14428 merging in first

nvauto · 2026-03-30T06:29:31Z

NOTE: release/26.04 has been created from main. Please retarget your PR to release/26.04 if it should be included in the release.

Rebased on top of NVIDIA#14428's per-batch retry mechanism. Adds GPU OOM split retry for the host-to-GPU transfer step using HostColumnarBatchWithRowRange, which supports logical slicing of host columns without copying underlying memory. Signed-off-by: Haoyang Li <haoyangl@nvidia.com>

Signed-off-by: Haoyang Li <haoyangl@nvidia.com>

nvauto · 2026-05-25T05:03:33Z

NOTE: release/26.06 has been created from main. Please retarget your PR to release/26.06 if it should be included in the release.

…om-split-retry

Signed-off-by: Haoyang Li <haoyangl@nvidia.com>

nvauto · 2026-07-27T06:54:19Z

NOTE: release/26.08 has been created from main. Please retarget your PR to release/26.08 if it should be included in the release.

thirtiseven requested a review from Copilot December 31, 2025 02:19

Copilot started reviewing on behalf of thirtiseven December 31, 2025 02:19 View session

Copilot AI reviewed Dec 31, 2025

View reviewed changes

greptile-apps Bot reviewed Dec 31, 2025

View reviewed changes

thirtiseven requested a review from Copilot December 31, 2025 12:37

Copilot started reviewing on behalf of thirtiseven December 31, 2025 12:38 View session

Copilot AI reviewed Dec 31, 2025

View reviewed changes

sameerz added the bug Something isn't working label Jan 4, 2026

thirtiseven self-assigned this Jan 5, 2026

thirtiseven marked this pull request as ready for review January 5, 2026 07:28

greptile-apps Bot reviewed Jan 5, 2026

View reviewed changes

wjxiz1992 changed the title ~~Prevent potiential GPU OOM in R2C with split retry~~ Prevent potential GPU OOM in R2C with split retry Jan 5, 2026

greptile-apps Bot reviewed Jan 5, 2026

View reviewed changes

firestarman reviewed Jan 5, 2026

View reviewed changes

greptile-apps Bot reviewed Jan 6, 2026

View reviewed changes

thirtiseven requested a review from firestarman January 6, 2026 07:53

firestarman reviewed Jan 7, 2026

View reviewed changes

Comment thread tests/src/test/scala/com/nvidia/spark/rapids/RowToColumnarIteratorRetrySuite.scala

Comment thread sql-plugin/src/main/scala/com/nvidia/spark/rapids/HostColumnarBatchWithRowRange.scala Outdated

firestarman reviewed Jan 7, 2026

View reviewed changes

Comment thread sql-plugin/src/main/scala/com/nvidia/spark/rapids/HostColumnarBatchWithRowRange.scala Outdated

firestarman reviewed Jan 7, 2026

View reviewed changes

Comment thread sql-plugin/src/main/scala/com/nvidia/spark/rapids/HostColumnarBatchWithRowRange.scala Outdated

greptile-apps Bot reviewed Jan 7, 2026

View reviewed changes

firestarman reviewed Jan 7, 2026

View reviewed changes

Comment thread sql-plugin/src/main/scala/com/nvidia/spark/rapids/HostColumnarBatchWithRowRange.scala Outdated

Comment thread tests/src/test/scala/com/nvidia/spark/rapids/RowToColumnarIteratorRetrySuite.scala

abellina reviewed Jan 9, 2026

View reviewed changes

greptile-apps Bot reviewed Jan 12, 2026

View reviewed changes

thirtiseven requested a review from abellina January 23, 2026 01:13

thirtiseven changed the base branch from main to release/26.02 January 26, 2026 06:52

abellina reviewed Feb 2, 2026

View reviewed changes

thirtiseven changed the base branch from release/26.02 to main February 3, 2026 02:09

greptile-apps Bot reviewed Feb 25, 2026

View reviewed changes

thirtiseven marked this pull request as draft March 18, 2026 05:35

thirtiseven added 2 commits April 10, 2026 10:48

Update stale comment about tryBuild retry

c821e95

Signed-off-by: Haoyang Li <haoyangl@nvidia.com>

thirtiseven force-pushed the fix/14018-r2c-gpu-oom-split-retry branch from a6ea3b9 to c821e95 Compare April 10, 2026 03:00

thirtiseven marked this pull request as ready for review April 10, 2026 05:53

address comments

3cc8e20

Signed-off-by: Haoyang Li <haoyangl@nvidia.com>

greptile-apps Bot reviewed Apr 10, 2026

View reviewed changes

Comment thread sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuRowToColumnarExec.scala Outdated

remove benchmark code

ff93eae

Signed-off-by: Haoyang Li <haoyangl@nvidia.com>

thirtiseven requested a review from abellina April 27, 2026 09:58

Merge remote-tracking branch 'upstream/main' into fix/14018-r2c-gpu-o…

fe63dcb

…om-split-retry

greptile-apps Bot reviewed Jul 9, 2026

View reviewed changes

Comment thread sql-plugin/src/main/scala/com/nvidia/spark/rapids/HostColumnarBatchWithRowRange.scala

address new comments

56bd5fd

Signed-off-by: Haoyang Li <haoyangl@nvidia.com>

Uh oh!

Conversation

thirtiseven commented Dec 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklists

Uh oh!

thirtiseven commented Dec 31, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Key Changes

Reviewed changes

Uh oh!

greptile-apps Bot commented Dec 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Sequence Diagram

Uh oh!

greptile-apps Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Additional Comments (1)

Uh oh!

thirtiseven commented Dec 31, 2025

Uh oh!

thirtiseven commented Dec 31, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Additional Comments (1)

Uh oh!

greptile-apps Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Additional Comments (1)

Uh oh!

greptile-apps Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Additional Comments (1)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

firestarman commented Jan 6, 2026

Uh oh!

greptile-apps Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Additional Comments (1)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

thirtiseven commented Jan 7, 2026

Uh oh!

greptile-apps Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Additional Comments (1)

Uh oh!

thirtiseven commented Dec 31, 2025 •

edited

Loading

greptile-apps Bot commented Dec 31, 2025 •

edited

Loading

greptile-apps Bot left a comment •

edited

Loading

greptile-apps Bot left a comment •

edited

Loading

greptile-apps Bot left a comment •

edited

Loading

greptile-apps Bot left a comment •

edited

Loading

greptile-apps Bot left a comment •

edited

Loading

greptile-apps Bot left a comment •

edited

Loading