Typed Stack exports/refs: secret preservation, Output field semantics, and Exports/Outputs duality

[lbialy]

Problem

When using typed Stack.exports(MyCaseClass(...)) and StackReference[MyCaseClass] together, there are several interconnected issues around how fields are serialized, deserialized, and how secret status is preserved across the stack boundary.

1. Exports/Outputs duality — cannot use one case class for both sides

On the write side (Stack.exports), field values are Output[A] because they're computed asynchronously. The Encoder correctly resolves Output[A] to A during serialization.

On the read side (StackReference[T]), Pulumi delivers non-secret values as plain JSON (e.g., "hello", [1,2,3]) and secret values wrapped in an envelope:

{
  "4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270",
  "value": "the-secret-value"
}

The current JsonReader[Output[A]] (from JsonReaderInstances) only handles the envelope format — it fails with "Invalid JSON" on plain values. This means:

• Non-secret fields must be plain types (String, List[String], etc.) on the read side
• Secret fields could theoretically use Output[String] on the read side, but...

Result: You cannot define a single case class that works for both Stack.exports and StackReference[T]. You need paired types:

// Write side — fields are Output[A] for Encoder
case class MyExports(
  name: Output[String],
  secret: Output[String]  // .asSecret
) derives Encoder

// Read side — fields are plain for JsonReader
case class MyOutputs(
  name: String,
  secret: String  // secret status is LOST
) derives JsonReader

This is cumbersome and error-prone — the two types must be kept in sync manually.

2. Secret status is lost on the read side

When a secret value crosses the StackReference boundary and is deserialized as a plain String, the consuming stack has no way to know it was a secret. If the consuming stack re-exports that value, it will be stored in plaintext in Pulumi state.

Example:

// Stack A exports:
Stack(...).exports(MyExports(
  dbPassword = dbPassword.asSecret  // marked as secret
))

// Stack B reads:
val ref = StackReference[MyOutputs](...)
val password = ref.map(_.outputs.dbPassword)  // plain String, secret status lost

// If Stack B re-exports this, it's no longer encrypted
Stack(...).exports(password = password)  // PLAINTEXT in state!

The correct behavior would be for StackReference[T] to preserve secret status. Fields that were exported as secrets should be readable as Output[String] with the secret flag set, so that downstream re-exports remain encrypted.

3. Pulumi's output representation in stack references

For context, here's how Pulumi represents stack outputs when read via StackReference:

Export type	JSON representation in StackReference
Plain value	Raw JSON: "hello", 42, ["a","b"]
Secret value	Envelope: {"4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270", "value": "secret"}
Output (non-secret)	Same as plain value (Output resolves to raw JSON)
Output (secret)	Same as secret envelope

The magic string 4dabf18193072939515e22adb298388d is Pulumi's SecretSig constant.

Proposed solution direction

The JsonReader[Output[A]] should be made more resilient:

1. If the JSON has the secret envelope → read as Output.secret(value)
2. If the JSON is a plain value → read as Output.pure(value) (non-secret)

This would allow a single case class with Output[A] fields to work for both sides:

case class MyContract(
  name: Output[String],        // plain on read, Output on write
  secret: Output[String]       // secret on read (preserved), Output on write
) derives JsonReader, Encoder

On read:

• name sees plain JSON → Output.pure("hello")
• secret sees secret envelope → Output.secret("the-value")

On write:

• name is Output[String] → Encoder resolves to "hello"
• secret is Output[String] created via .asSecret → Encoder resolves to secret-marked value

This preserves secret status across the boundary and eliminates the need for dual types.

4. Additional consideration: derives JsonReader macro constraint

The derives JsonReader macro (in ProductFormatsMacro.jsonReaderImpl) summons JsonReader[T] for each field but reports the error as "Missing given instance of JsonFormat[T]". This is a copy-paste from prepareFormatInstances — the error message should say JsonReader not JsonFormat. Minor but confusing during debugging.

File: besom-json/src/main/scala/besom/json/ProductFormats.scala, line 88.

Current workaround

Use paired *Exports / *Outputs case classes:

• *Exports with Output[A] fields + derives Encoder (write side)
• *Outputs with plain fields + derives JsonReader (read side)

Secret status is lost on the read side. Consumers must manually re-mark values as secret if they re-export them.

Impact

This affects any multi-stack Besom project using typed stack references. The Rig project (multi-layer platform orchestrator) hits this on every layer boundary.