Strip a gzipped heap dump without unpacking it first

A heap dump that's been shared has almost always been gzipped on the way
— the Android heap dumps in our test resources compress 3.9x to 4.4x,
and 5.6x to 5.8x once stripped — so stripping one meant gunzipping it,
stripping it, and gzipping the result again by hand.

The file overload now reads gzipped content gzipped, deciding from the
first two bytes rather than from the name, and writes gzipped output when
the output name ends with ".gz", which is what the default output name of
a ".hprof.gz" input already produced. The two halves are separate
extensions on the source and sink providers, so the streaming overload
can take either one on its own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: pyricau

docs/changelog.md

@@ -110,6 +110,8 @@ Releases before 2.8.1 predate these markers.
 * 💥 [#2777](https://github.com/square/leakcanary/issues/2777) `HprofPrimitiveArrayStripper` failed with a `NegativeArraySizeException` on a heap dump holding a primitive array whose content takes more than 2 GB, which is a `long` or `double` array of more than 268435455 elements. Fixing the parser for that issue covered the object array side of the stripper and left the primitive array side alone: the bytes written over an array's content were allocated as one array sized `arrayLength * elementByteSize` in `Int` arithmetic, which wraps negative that far out, so a `long[268435456]` asked for an array of -2147483648 bytes. Those bytes are now written by repeating an 8192 byte pattern over the content, so nothing is sized from the array at all. Verified on a 2.15 GB heap dump taken from a JVM holding a `long[268435456]`, which strips in 3 seconds.
     * 🔨 Not allocating a copy of every array also takes the memory stripping needs from a function of the largest array in the heap dump down to a constant, which matters on Android, where `HeapAnalysisConfig(stripHeapDump = true)` strips in the app's own process. Stripping a 110 MB heap dump holding a `byte[104857600]` used to need a 216 MB heap and completes in 8 MB now. On a 294 MB heap dump pulled from a real app, the bytes allocated go from 374 MB to 34 MB, and stripping is about 5% faster.
 * 💥 Reading or stripping a heap dump with 8 byte identifiers that holds a heap dump info record failed with *"Unknown tag 0x00"*. That record holds an `Int` heap id and then a string id, and both the reader skipping it and the stripper copying it over treated the heap id as an id too, so with 8 byte identifiers they moved 4 bytes too far and read the middle of whatever came next as a record tag. Heap dump info records only appear in heap dumps written by Android, which always uses 4 byte identifiers — where the two sizes are the same and the bug can't show — so this is about a heap dump written by `HprofWriter`.
+* ✨ `shark-cli strip-hprof` and `HprofPrimitiveArrayStripper.stripPrimitiveArrays(File)` now handle gzip on both ends: a heap dump whose content is gzipped is read gzipped whatever it's named, and the output is written gzipped when its name ends with ".gz", which is what the default output name of a ".hprof.gz" input already gave you — so "app.hprof.gz" strips to a gzipped "app-stripped.hprof.gz". The Android heap dumps in our test resources compress 3.9x to 4.4x, and 5.6x to 5.8x once stripped, so a heap dump that's been shared is usually gzipped by the time you get it, and stripping it meant gunzipping it first and gzipping the result again.
+    * ✨ `StreamingSourceProvider.gunzipIfGzipped()` and `StreamingSinkProvider.gzip()` are what do that. They compose onto any source or sink, so a caller of the `stripPrimitiveArrays` overload that takes a source and a sink can opt into the same behavior, or into only one half of it.
 * ✨ `HprofPrimitiveArrayStripper` and `shark-cli strip-hprof` now say what stripping leaves behind. Everything that isn't a primitive array or a wrapped primitive is copied over unchanged, and that includes the string records holding the class, field and method names the rest of the heap dump refers to. Those hold no runtime data in a heap dump from Android, but a heap dump from a JVM also holds every string constant of every loaded class in them, so stripping a JVM heap dump leaves the constants written in the code behind.
 
 ## Version 3.0 Alpha 9 (2026-06-25)

shark/shark-cli/src/main/java/shark/StripHprofCommand.kt

@@ -14,6 +14,9 @@ class StripHprofCommand : CliktCommand(
     |of the heap dump refers to. Those hold no runtime data in a heap dump from Android, but a heap
     |dump from a JVM also holds every string constant of every loaded class in them, so stripping a
     |JVM heap dump leaves the constants written in the code behind.
+    |
+    |A gzipped heap dump is read gzipped and written back gzipped, so "app.hprof.gz" gives you
+    |"app-stripped.hprof.gz".
     """.trimMargin()
 ) {
 

shark/shark-hprof/api/shark-hprof.api

@@ -629,10 +629,18 @@ public abstract interface class shark/StreamingSinkProvider {
 	public abstract fun openStreamingSink ()Lokio/BufferedSink;
 }
 
+public final class shark/StreamingSinkProviderKt {
+	public static final fun gzip (Lshark/StreamingSinkProvider;)Lshark/StreamingSinkProvider;
+}
+
 public abstract interface class shark/StreamingSourceProvider {
 	public abstract fun openStreamingSource ()Lokio/BufferedSource;
 }
 
+public final class shark/StreamingSourceProviderKt {
+	public static final fun gunzipIfGzipped (Lshark/StreamingSourceProvider;)Lshark/StreamingSourceProvider;
+}
+
 public abstract class shark/ValueHolder {
 	public static final field Companion Lshark/ValueHolder$Companion;
 	public static final field NULL_REFERENCE J

shark/shark-hprof/src/main/java/shark/HprofPrimitiveArrayStripper.kt

@@ -54,7 +54,13 @@ import shark.PrimitiveType.SHORT
  */
 class HprofPrimitiveArrayStripper {
 
-  /** @see HprofPrimitiveArrayStripper */
+  /**
+   * [inputHprofFile] is read gzipped when its content is gzipped, and [outputHprofFile] is written
+   * gzipped when its name ends with ".gz", so stripping "app.hprof.gz" writes a gzipped
+   * "app-stripped.hprof.gz" by default.
+   *
+   * @see HprofPrimitiveArrayStripper
+   */
   fun stripPrimitiveArrays(
     inputHprofFile: File,
     /**
@@ -75,9 +81,17 @@ class HprofPrimitiveArrayStripper {
       ),
     deleteInputHprofFile: Boolean = false
   ): File {
+    val fileSinkProvider = StreamingSinkProvider {
+      outputHprofFile.outputStream().sink().buffer()
+    }
     stripPrimitiveArrays(
-      hprofSourceProvider = FileSourceProvider(inputHprofFile),
-      hprofSinkProvider = { outputHprofFile.outputStream().sink().buffer() },
+      hprofSourceProvider = FileSourceProvider(inputHprofFile).gunzipIfGzipped(),
+      hprofSinkProvider =
+        if (outputHprofFile.name.endsWith(GZIP_FILE_EXTENSION)) {
+          fileSinkProvider.gzip()
+        } else {
+          fileSinkProvider
+        },
       onDoneOpeningNewSources = {
         if (deleteInputHprofFile) {
           // Using the Unix trick of deleting the file as soon as all readers have opened it.
@@ -433,3 +447,5 @@ private const val REPLACEMENT_PATTERN_BYTE_SIZE = 8192
 
 /** '?', in UTF-8 and in the low byte of a UTF-16BE character alike. */
 private const val QUESTION_MARK_BYTE: Byte = 63
+
+private const val GZIP_FILE_EXTENSION = ".gz"

shark/shark-hprof/src/main/java/shark/StreamingSinkProvider.kt

@@ -1,10 +1,21 @@
 package shark
 
 import okio.BufferedSink
+import okio.GzipSink
+import okio.buffer
 
 /**
  * Can open [BufferedSink] instances.
  */
 fun interface StreamingSinkProvider {
   fun openStreamingSink(): BufferedSink
 }
+
+/**
+ * Returns a [StreamingSinkProvider] that gzips what's written to it before handing it to a sink this
+ * one opens. The gzip trailer is written when the sink is closed, so what's written to it has to be
+ * closed for the result to be a complete gzip stream.
+ */
+fun StreamingSinkProvider.gzip(): StreamingSinkProvider = StreamingSinkProvider {
+  GzipSink(openStreamingSink()).buffer()
+}

shark/shark-hprof/src/main/java/shark/StreamingSourceProvider.kt

@@ -1,11 +1,36 @@
 package shark
 
 import okio.BufferedSource
+import okio.GzipSource
 import okio.Source
+import okio.buffer
 
 /**
  * Can open [Source] instances.
  */
 fun interface StreamingSourceProvider {
   fun openStreamingSource(): BufferedSource
 }
+
+/**
+ * Returns a [StreamingSourceProvider] that hands out the content of the sources this one opens,
+ * decompressed when that content is gzipped and as is when it isn't. Which one it is comes from the
+ * content itself rather than from a file name, so a heap dump that was gzipped after it was written
+ * reads the same as one that never was.
+ */
+fun StreamingSourceProvider.gunzipIfGzipped(): StreamingSourceProvider = StreamingSourceProvider {
+  val source = openStreamingSource()
+  if (source.startsWithGzipMagicNumber()) {
+    GzipSource(source).buffer()
+  } else {
+    source
+  }
+}
+
+private fun BufferedSource.startsWithGzipMagicNumber(): Boolean {
+  return request(GZIP_MAGIC_NUMBER.size.toLong()) &&
+    GZIP_MAGIC_NUMBER.withIndex().all { (index, byte) -> buffer[index.toLong()] == byte }
+}
+
+/** The two bytes every gzip stream starts with. */
+private val GZIP_MAGIC_NUMBER = byteArrayOf(0x1f, 0x8b.toByte())

shark/shark-hprof/src/test/java/shark/GzipStripHprofTest.kt

@@ -0,0 +1,61 @@
+package shark
+
+import java.io.File
+import okio.Buffer
+import okio.GzipSink
+import okio.GzipSource
+import okio.buffer
+import okio.sink
+import okio.source
+import org.assertj.core.api.Assertions.assertThat
+import org.junit.Rule
+import org.junit.Test
+import org.junit.rules.TemporaryFolder
+import shark.HprofRecord.HeapDumpRecord.ObjectRecord.PrimitiveArrayDumpRecord.LongArrayDump
+
+class GzipStripHprofTest {
+
+  @get:Rule
+  var testFolder = TemporaryFolder()
+
+  @Test fun `gzipped heap dump strips to the same bytes as that heap dump uncompressed`() {
+    val strippedFromHprof = HprofPrimitiveArrayStripper().stripPrimitiveArrays(hprofFile())
+    val strippedFromGzip =
+      HprofPrimitiveArrayStripper()
+        .stripPrimitiveArrays(gzippedHprofFile(), File(testFolder.root, "from-gzip.hprof"))
+
+    assertThat(strippedFromGzip.readBytes()).isEqualTo(strippedFromHprof.readBytes())
+  }
+
+  @Test fun `stripping a gzipped heap dump writes a gzipped heap dump`() {
+    val strippedFromHprof = HprofPrimitiveArrayStripper().stripPrimitiveArrays(hprofFile())
+
+    val strippedFromGzip = HprofPrimitiveArrayStripper().stripPrimitiveArrays(gzippedHprofFile())
+
+    assertThat(strippedFromGzip.name).isEqualTo("app-stripped.hprof.gz")
+    assertThat(strippedFromGzip.length()).isLessThan(strippedFromHprof.length())
+    val gunzipped =
+      GzipSource(strippedFromGzip.source()).buffer().use { source -> source.readByteArray() }
+    assertThat(gunzipped).isEqualTo(strippedFromHprof.readBytes())
+  }
+
+  private fun hprofFile() =
+    File(testFolder.newFolder(), "app.hprof").apply { writeBytes(heapDumpWithALongArray()) }
+
+  private fun gzippedHprofFile() =
+    File(testFolder.newFolder(), "app.hprof.gz").apply {
+      GzipSink(sink()).buffer().use { sink -> sink.write(heapDumpWithALongArray()) }
+    }
+
+  /** Holds a long array of values that all differ from the zeroes stripping replaces them with. */
+  private fun heapDumpWithALongArray(): ByteArray {
+    val buffer = Buffer()
+    HprofWriter.openWriterFor(buffer, hprofHeader = HprofHeader(heapDumpTimestamp = 42)).use { writer
+      ->
+      writer.write(
+        LongArrayDump(id = 1, stackTraceSerialNumber = 0, array = LongArray(1024) { 0xCAFE })
+      )
+    }
+    return buffer.readByteArray()
+  }
+}