fix(marshal): marshal ByteArrays to hex by erights · Pull Request #2871 · endojs/endo

erights · 2025-06-27T01:49:56Z

Closes: #XXXX
Refs: #XXXX

Description

Provides for serialization and unserialization of ByteArrays (passable frozen Immutable ArrayBuffers) to our various textual encodings (capData, smallcaps, justin).

It does not yet add any encoding for ByteArrays to encodePassable. Attn @gibson042

Alternative to #2844 using hex instead of base64

Security Considerations

Assuming this PR is correct, none

Scaling Considerations

Hex is a terribly inefficient encoding of binary, but that's the price of using a textual format.

Further, with https://github.com/tc39/proposal-arraybuffer-base64 not yet being part of the platform, Doing this conversion in JS vs native is horribly inefficient, especially on XS.

Hopefully, most of the time when we actually want to marshal large ByteArrays, we'll have switched to a binary format that needs no translation.

Documentation Considerations

Just an extension of the passable types that marshal should accept.

Testing Considerations

Includes tests

Compatibility Considerations

If a system post this PR sends a ByteArray through marshal to a receiver that is pre this PR, the receiver will error since this is a case it does not know about.

Upgrade Considerations

When storing a ByteArray to durable storage, you'll actually be storing this hex-based encoding. That durable storage can only be understood post this PR, which is a problem if there is a co-existence of code in one vat, some of which has bundled pre this PR and some post this PR.

erights · 2025-06-27T02:23:20Z

+// Failing because we now need to encode byteArrays
+test.failing(
  'original passable encoding corresponds to rankOrder',
  testOrderInvariants,
  pickLegacy,
 );
-test(
+// Failing because we now need to encode byteArrays
+test.failing(


Attn @gibson042

Hmm, I'm of the opinion that this should not land without reënabling these tests, either by supporting byteArrays in encodePassable or by filtering them out of the inputs here.

kriskowal · 2025-07-11T03:06:11Z

+  const len = data.length;
+  const out = [];
+  for (let i = 0; i < len; i += 1) {
+    out.push(data[i].toString(16).padStart(2, '0'));


It would not be awful to precompute or even embed a 256 entry table.

This should probably just be copied from agoric-sdk: https://github.com/Agoric/agoric-sdk/blob/master/packages/internal/src/hex.js

Well, not so done. After the switch, tests that used to succeed are now failing. This might be because the old code was incorrect and the tests followed along, or that the new hex.js code is incorrect. I'll track this down.

Unresolving

I would like to avoid entraining a dependency on the BufferConstructor type, if it can be helped.

Even though the dependency is gated by a feature test?

gibson042 · 2025-07-11T16:42:02Z

+// Failing because we now need to encode byteArrays
+test.failing(
  'original passable encoding corresponds to rankOrder',
  testOrderInvariants,
  pickLegacy,
 );
-test(
+// Failing because we now need to encode byteArrays
+test.failing(


Hmm, I'm of the opinion that this should not land without reënabling these tests, either by supporting byteArrays in encodePassable or by filtering them out of the inputs here.

gibson042 · 2025-07-11T16:52:19Z

+  const len = data.length;
+  const out = [];
+  for (let i = 0; i < len; i += 1) {
+    out.push(data[i].toString(16).padStart(2, '0'));


This should probably just be copied from agoric-sdk: https://github.com/Agoric/agoric-sdk/blob/master/packages/internal/src/hex.js

gibson042 · 2025-07-11T22:19:31Z

+/**
+ * Export a hex codec that can work with standard JS engines, but takes
+ * advantage of optimizations on some platforms (like Node.js's Buffer API).
+ */
+export const { uint8ArrayToHex, encodeHex, hexToUint8Array, decodeHex } =
+  typeof Buffer === 'undefined'
+    ? makePortableHexCodec()
+    : makeBufferishHexCodec(Buffer);


Should this file also anticipate proposal-arraybuffer-base64, which is currently at Stage 3?

Details

Suggested change

/**

* Export a hex codec that can work with standard JS engines, but takes

* advantage of optimizations on some platforms (like Node.js's Buffer API).

*/

export const { uint8ArrayToHex, encodeHex, hexToUint8Array, decodeHex } =

typeof Buffer === 'undefined'

? makePortableHexCodec()

: makeBufferishHexCodec(Buffer);

/**

* @param {Pick<HexCodec, 'uint8ArrayToHex' | 'hexToUint8Array'>} hexCodec

* @returns {HexCodec}

*/

const addDeprecatedFunctions = hexCodec => {

const { uint8ArrayToHex, hexToUint8Array } = hexCodec;

/**

* @deprecated Use `uint8ArrayToHex` instead

* @type {HexCodec['uint8ArrayToHex']}

*/

const encodeHex = u8 => uint8ArrayToHex(u8);

/**

* @deprecated Use `hexToUint8Array` instead

* @type {HexCodec['hexToUint8Array']}

*/

const decodeHex = hex => hexToUint8Array(hex);

return harden({

uint8ArrayToHex,

encodeHex,

hexToUint8Array,

decodeHex,

});

};

/**

* Create the most performant hex codec from the available global environment

* (possibly leveraging platform-specific optimizations such as the Node.js

* Buffer).

*

* @returns {Pick<HexCodec, 'uint8ArrayToHex' | 'hexToUint8Array'>}

*/

const makeBestHexCodec = () => {

if (Uint8Array.fromHex) {

return harden({

uint8ArrayToHex: u8 => u8.toHex(),

hexToUint8Array: hex => Uint8Array.fromHex(hex);

});

}

return typeof Buffer === 'function'

? makeBufferishHexCodec(Buffer)

: makePortableHexCodec();

};

export const { uint8ArrayToHex, encodeHex, hexToUint8Array, decodeHex } =

addDeprecatedFunctions(makeBestHexCodec());

gibson042 · 2025-07-11T22:22:13Z

+      buf.byteOffset,
+      buf.byteLength / Uint8Array.BYTES_PER_ELEMENT,
+    );
+    return u8a;


I just noticed that the makePortableHexCodec hexToUint8Array hardens its output but the makeBufferishHexCodec analog does not, which is presumably an unintentional difference.

Good catch. Yes, unintentional.

Copilot

Pull Request Overview

This PR adds support for marshaling ByteArrays (frozen Immutable ArrayBuffers) to textual formats using hex encoding. This enables serialization and deserialization of ByteArrays across all marshal text formats (capData, smallcaps, justin).

Key changes:

Implements hex encoding/decoding utilities for ByteArray serialization
Adds ByteArray support to all marshal text formats with hex-based encoding
Updates rank ordering system to use tied comparison for remotables by default

Reviewed Changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
packages/pass-style/src/hex.js	New hex codec implementation with portable and optimized variants
packages/pass-style/src/byteArray.js	Adds hex conversion functions for ByteArray marshaling
packages/marshal/src/encodeToCapData.js	Implements ByteArray encoding/decoding for capData format
packages/marshal/src/encodeToSmallcaps.js	Implements ByteArray encoding/decoding for smallcaps format
packages/marshal/src/marshal-justin.js	Adds ByteArray support to Justin format
packages/pass-style/test/byteArray.test.js	Test coverage for ByteArray hex conversion functions
packages/marshal/tools/marshal-test-data.js	Test data including ByteArray examples
packages/marshal/test/encodePassable.test.js	Marks tests as failing due to missing ByteArray support

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-09-16T08:03:28Z

+    ['a', '"a" must be an even number of characters'],
+    ['a00', '"a00" must be an even number of characters'],
+
+    // non-zero padding bits (MAY reject): ['Qf==', ...],


The comment references base64 padding 'Qf==' which is not relevant for hex encoding tests. This appears to be copied from base64 test code and should be removed or updated to reflect hex-specific invalid inputs.

Suggested change

// non-zero padding bits (MAY reject): ['Qf==', ...],

// (additional invalid hex inputs could be added here)

Co-authored-by: Richard Gibson <richard.gibson@gmail.com>

kriscendobot · 2026-04-30T20:20:23Z

FYI #3226 has been refreshed to subsume the codec-admission core of this PR, routing hex through the merged @endo/hex package (#3208) and adding the encodePassable encoding that this PR explicitly defers. Salvaged: harden(makeDecodeFromCapData) and harden(makeDecodeFromSmallcaps), plus the byteArray fixture in tools/marshal-test-data.js.

The rankOrder.js compareRankRemotablesTied refactor here is orthogonal to codec admission and is being broken out into its own PR.

Reserving the option to flip primaries later — happy to close #3226 in favor of this if you'd rather rebase here on @endo/hex.

erights · 2026-05-01T07:03:15Z


+/**
+ * Like `compareRank` and `compareAntiRank` and unlike `fullCompare`,
+ * `compareRankRemotablesTied` and `compareAntiRankRemotablesTied`


This change not propagated to #3226

erights · 2026-05-01T07:04:02Z

@@ -1,11 +1,19 @@
-import { makeTagged, passableSymbolForName } from '@endo/pass-style';
+import {


Changes to this file not propagated to #3226

erights · 2026-05-01T07:05:22Z

@@ -0,0 +1,210 @@
+// /* global Buffer */


Changes to this file not propagated to #3226, or at least, not obviously.

erights · 2026-05-01T07:06:42Z

+export { uint8ArrayToHex, hexToUint8Array } from './src/hex.js';
+
+export {
+  byteArrayToUint8Array,
+  uint8ArrayToByteArray,
+  byteArrayToHex,
+  hexToByteArray,
+} from './src/byteArray.js';


exports 6 whereas #3226 exports 4. Does #3226 drop two correctly?

@gibson042

…codecs Apply the relevant feedback from #2871 to subsume that PR: - @gibson042 #2871:byteArray.js:127 — `uint8ArrayToByteArray` now honors the source `Uint8Array`'s `byteOffset` and `length`, so a view over a slice of a larger buffer round-trips correctly: `uint8ArrayToByteArray(new Uint8Array(buf, byteOffset, length))` represents exactly the bytes in `[byteOffset, byteOffset + length)`. Documented the zero-copy / copy tradeoff with `sliceToImmutable`. - @gibson042 #2871:byteArray.js:130 — comment on `byteArrayToHex` / `hexToByteArray` is now phrased as "Like `encodeHex` but…" / "Like `decodeHex` but…", matching gibson's suggested wording for the byteArray-flavored wrappers. - @erights #2871:encodeToCapData.js:453 / encodeToSmallcaps.js — the factory functions `makeDecodeFromCapData` and `makeDecodeFromSmallcaps` are now `harden`-ed at module scope, matching the existing harden of their encoder counterparts (`makeEncodeToCapData`, `makeEncodeToSmallcaps`). - @erights #2871:marshal-test-data.js — propagate Mark's full byteArray fixture set: `roundTripPairs`, `jsonJustinPairs`, `unsortedSample`, and `sortedSample`. The Justin test compartment exposes `hexToByteArray` so the rendered Justin expression evaluates. Not propagated, deliberately: - `packages/marshal/src/rankOrder.js` — Mark's `compareRankRemotablesTied` refactor is orthogonal to codec admission and lands as its own PR (bots #73 today, awaiting an upstream PR). - `packages/pass-style/src/hex.js` — Mark vendored a private hex implementation in pass-style; we route through the merged `@endo/hex` package (#3208) instead. As a result, `packages/pass-style/index.js` exports `byteArrayToHex` / `hexToByteArray` but not `uint8ArrayToHex` / `hexToUint8Array`; consumers needing raw hex helpers should `import { encodeHex, decodeHex } from '@endo/hex'`. Tests: `@endo/marshal` 79 + 1 skip, `@endo/pass-style` 29. Lint clean.

erights · 2026-05-12T21:40:52Z

Consider closing in favor of #3226 and endojs/endo-but-for-bots#73 in anticipation of it being an endo PR.

Refs: #2871 Refs: #2883 Refs: #3226 ## Description Splits the rank-comparison refactor out from #2871 so that the codec-admission core of that PR can land in #3226 without depending on this orthogonal change. Introduces `compareRankRemotablesTied` (and its anti-rank twin), which behaves like `compareRank` but treats all remotables as tied for the same rank rather than short-circuiting on the first remotable encountered. Makes this the default `compare` for the rank-cover operations: - `isRankSorted`, `assertRankSorted`, `sortByRank`, `getIndexCover` (commit 1) - `unionRankCovers`, `intersectRankCovers` (commit 2) `compare` becomes optional and is moved to the trailing position on each. `getIndexCover`'s argument order shifts from `(sorted, compare, rankCover)` to `(sorted, rankCover, compare?)`; the only callers are within the Endo monorepo and are updated. `unionRankCovers` and `intersectRankCovers` likewise shift from `(compare, covers)` to `(covers, compare?)`, and `@endo/patterns`'s `patternMatchers.js` (the only caller) is updated. `compareRankRemotablesTied` and `compareAntiRankRemotablesTied` are re-exported from `@endo/marshal`'s `index.js`. The first commit is salvaged from #2871 with original authorship preserved (Mark S. Miller). ### Security Considerations No change to trust boundaries, no new authorities. The comparator change is internal to marshal's rank ordering; callers that pass an explicit `compare` keep their existing behavior. ### Scaling Considerations The tied comparator visits every remotable rather than short-circuiting on the first, so rank-cover operations over inputs containing many remotables do more work per call than the prior `compareRank` default. This matches the intent of the operations (they want a stable cover, not an early exit), and the prior behavior is still available by passing `compareRank` explicitly. ### Documentation Considerations The argument-order changes on `getIndexCover`, `unionRankCovers`, and `intersectRankCovers` are caller-visible. No internal callers remain on the old signatures. External callers, if any, would need to swap argument order; this is captured in the changeset. ### Testing Considerations `packages/marshal/test/rankOrder.test.js` is extended to: - Exercise the new optional/default-arg behavior of all six rank-cover operations. - Resolve two pre-existing TODOs that asked for the `totally orders ranks` and `is transitive` property tests to run against both `compareRank` and `compareRankRemotablesTied`. Both are total preorders, so both properties hold for both, and we now parameterize the tests to exercise each. ### Compatibility Considerations Argument-order shifts on three exported functions (`getIndexCover`, `unionRankCovers`, `intersectRankCovers`) and one new default comparator on six. Captured in a changeset. The `compareRankRemotablesTied` and `compareAntiRankRemotablesTied` exports are additions; nothing is removed. ### Upgrade Considerations No live-system upgrade concerns. The changeset covers the package-level migration for external consumers.

@kriskowal

This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and the packages will be published to npm automatically. If you're not ready to do a release yet, that's fine, whenever you add more changesets to master, this PR will be updated. # Releases ## @endo/bytes@1.0.0 ### Major Changes - [#3257](#3257) [`dd45f4a`](dd45f4a) Thanks [@kriskowal](https://github.com/kriskowal)! - Add `@endo/bytes` package providing `concatBytes`, `bytesEqual`, `bytesFromText`, `bytesToText`, `bytesToImmutable`, and `bytesFromImmutable` for platform-neutral byte handling. Built on `Uint8Array` with `TextEncoder`/`TextDecoder` captured once at module load. `bytesToImmutable` and `bytesFromImmutable` cross between mutable views and the immutable `ArrayBuffer` shape produced by the proposal-immutable-arraybuffer shim, so passable bytes can be carried across vat boundaries. Hardened, SES-safe; usable across Node, XS, and browser realms. The release is the first publish, going out as `1.0.0` from the `0.1.0` workspace floor (major bump from a `0.x.y` baseline lands at `1.0.0`). ## @endo/base64@1.1.0 ### Minor Changes - [#3216](#3216) [`7325bbe`](7325bbe) Thanks [@kriskowal](https://github.com/kriskowal)! - `@endo/base64`'s named exports (`encodeBase64`, `decodeBase64`, `atob`, `btoa`) are now frozen. Consumers that previously assigned to or extended these exports will see a `TypeError` under SES; read-only consumers are unaffected. The shim entry point `@endo/base64/shim.js` (which `@endo/init/pre.js` uses to install `globalThis.atob` / `globalThis.btoa` before `lockdown()`) is unchanged and continues to be safe to load pre-lockdown. ### Patch Changes - [#3216](#3216) [`7325bbe`](7325bbe) Thanks [@kriskowal](https://github.com/kriskowal)! - `encodeBase64` and `decodeBase64` now dispatch to the native `Uint8Array.prototype.toBase64` and `Uint8Array.fromBase64` intrinsics (TC39 proposal-arraybuffer-base64, Stage 4) on platforms that ship them, falling back to the existing pure-JavaScript implementation otherwise. Existing call sites pick up the speedup with no API change, and `decodeBase64`'s error messages (including any caller-supplied `name` and the failing offset) are unchanged. ## @endo/compartment-mapper@2.2.0 ### Minor Changes - [#3150](#3150) [`75253ad`](75253ad) Thanks [@boneskull](https://github.com/boneskull)! - This adds a new option, `additionalLocations`, to `mapNodeModules`. This option enables addition of arbitrary packages to the resulting Compartment Map. It enables execution of packages which need to dynamically load files from Compartments which would otherwise not appear in the map. Specific use-cases include tooling like Webpack and ESLint. ### Patch Changes - [#2901](#2901) [`69ca27c`](69ca27c) Thanks [@kriskowal](https://github.com/kriskowal)! - Sweep `makeFunctorFromMap` in `bundle.js` and `bundle-lite.js` to use optional chaining and nullish coalescing for the alias lookup now that [#1514](#1514) has completed. The "unable to locate module" diagnostic now distinguishes the alias-resolved case from the direct-key case, naming both the alias's resolved key and the original import key when the miss occurs via an alias. - [#3201](#3201) [`67ed1ce`](67ed1ce) Thanks [@boneskull](https://github.com/boneskull)! - Compatibility fix for passthrough-style wildcards in subpath exports. - Updated dependencies \[[`ad7a177`](ad7a177), [`f2aa55a`](f2aa55a), [`fa0b6a9`](fa0b6a9), [`459347b`](459347b)]: - @endo/hex@1.1.0 - ses@2.1.0 ## @endo/eslint-plugin@2.5.0 ### Minor Changes - [#3255](#3255) [`638306e`](638306e) Thanks [@kriskowal](https://github.com/kriskowal)! - Migrate the bundled `@endo/imports` ESLint config off the unmaintained `eslint-plugin-import` and onto the actively-maintained `eslint-plugin-import-x` soft fork. This is done via a Yarn package alias (`eslint-plugin-import: 'npm:eslint-plugin-import-x@4.16.2'` in the `dev` catalog), so the package on disk is still named `eslint-plugin-import` and ESLint continues to register its rules under the existing `import/*` namespace. The import-x implementation ships its own `unrs-resolver`, which natively honours the `package.json` `exports` field, so the explicit `import/resolver` settings block is no longer required and has been removed. Downstream consumers do not need to rename any `import/*` rule references; existing `eslintrc` snippets continue to work. ### Patch Changes - [#3274](#3274) [`e153a5a`](e153a5a) Thanks [@kriskowal](https://github.com/kriskowal)! - `harden-exports` now collects export names from all binding pattern shapes that may appear on the left-hand side of `export const ... = ...`: aliased object destructuring (`{ propName: aliasName }`), object and array rest, nested patterns, sparse holes, and default-value assignment patterns. A new `unknownBindingPattern` report surfaces any pattern type the helper does not recognize, in place of silent passthrough. ## @endo/evasive-transform@2.2.0 ### Minor Changes - [#3131](#3131) [`5c098c4`](5c098c4) Thanks [@naugtur](https://github.com/naugtur)! - - Add `customVisitor` option - a visitor function to be called on each node of the Babel traverse, in addition to the standard transforms in evasive-transform. Receives the same path argument as a normal Babel visitor. - Add an evasion for a class or object method being named `import` or `eval` ## @endo/hex@1.1.0 ### Minor Changes - [#3208](#3208) [`ad7a177`](ad7a177) Thanks [@kriskowal](https://github.com/kriskowal)! - Add `@endo/hex` package providing `encodeHex` and `decodeHex` as a ponyfill for the TC39 `Uint8Array.prototype.toHex` and `Uint8Array.fromHex` intrinsics (proposal-arraybuffer-base64, Stage 4). Dispatches to the native intrinsics at module load when available, falling through to a portable pure-JavaScript implementation elsewhere. ## @endo/marshal@1.10.0 ### Minor Changes - [#3265](#3265) [`45d06cd`](45d06cd) Thanks [@kriskowal](https://github.com/kriskowal)! - Add `compareRankRemotablesTied` and `compareAntiRankRemotablesTied`, which behave like `compareRank` and `compareAntiRank` except that they treat all remotables as tied for the same rank instead of short-circuiting the comparison on encountering a remotable. Make the `compare` parameter optional on `isRankSorted`, `assertRankSorted`, `sortByRank`, `getIndexCover`, `unionRankCovers`, and `intersectRankCovers`, defaulting to `compareRankRemotablesTied`. The following parameter orders change so the optional comparator lands at the end: - `getIndexCover(sorted, compare, rankCover)` → `getIndexCover(sorted, rankCover, compare?)` - `unionRankCovers(compare, covers)` → `unionRankCovers(covers, compare?)` - `intersectRankCovers(compare, covers)` → `intersectRankCovers(covers, compare?)` These are breaking changes to helpers that have no known external callers within the Endo monorepo. Within the monorepo, the only callers of `unionRankCovers` and `intersectRankCovers` are in `@endo/patterns`'s `patternMatchers.js`, which has been updated to match the new parameter order. Salvaged from <#2871> so that the codec-admission core of that PR (now in [#3226](#3226)) can land independently of this rank-comparison refactor. ## @endo/ocapn@1.1.0 ### Minor Changes - [#3256](#3256) [`bdb9ddc`](bdb9ddc) Thanks [@kriskowal](https://github.com/kriskowal)! - - Add a `framing` option to `makeTcpNetLayer` (`@endo/ocapn/netlayer/tcp-testing`). The default is `'syrup'`, which wraps each message in the `<length>:<payload>` framing implemented by `@endo/syrup-frame`. This is the framing the OCapN TCP-for-testing netlayer is moving toward (cf. the 2025-12-09 OCapN plenary, <https://github.com/ocapn/ocapn/blob/main/meeting-minutes/2025-12-09.md>), and is robust to TCP chunk boundaries that split a single OCapN message. Pass `framing: 'none'` to interoperate with the existing `ocapn/ocapn-test-suite` Python `testing_only_tcp` netlayer, which writes a syrup-encoded record with `sendall` and reads one back with `syrup.syrup_read` (no length prefix on the wire). The `'none'` option exists only for that suite's sake and goes away once the suite either adopts syrup framing or is retired. - [#3192](#3192) [`08b077d`](08b077d) Thanks [@kumavis](https://github.com/kumavis)! - Sync `@endo/ocapn` with [ocapn-test-suite](https://github.com/ocapn/ocapn-test-suite) at commit [74db78f08a40efba1e2b975d809374ff0e7acf60](ocapn/ocapn-test-suite@74db78f) (2026-02-25). - GC operations use list payloads (`exportPositions` / `wireDeltas`, `answerPositions`); wire labels `op:gc-exports` and `op:gc-answers`. - Remove `op:deliver-only`; fire-and-forget delivery uses `op:deliver` with `answerPosition` and `resolveMeDesc` set to `false`. - Codec refactors: `makeOcapnFalseForOptionalCodec` for optional `false` branches, homogeneous Syrup lists via `makeListCodecFromEntryCodec`, and related cleanup in operations and peer location hints. CI integration for the Python test suite is pinned to the same commit. - [#3209](#3209) [`20f9e21`](20f9e21) Thanks [@kumavis](https://github.com/kumavis)! - - Add a WebSocket netlayer exported as `@endo/ocapn/netlayer/ws` (`makeWebSocketNetLayer`). Used for interop with Guile-Goblins peers and for any other transport that prefers a framed WebSocket over the raw TCP test netlayer. - Add `@endo/ocapn/netlayer/tcp-testing` to the package's `exports` map so consumers can import the existing test netlayer without reaching into `src/`. - The main entry (`@endo/ocapn`) now re-exports `makeClient` and the swissnum helpers `swissnumFromBytes` / `swissnumToBytes` so consumers don't need a deep `src/client/...` import for the common case. - `makeClient` accepts a new `logger` option; when omitted the existing console-based logger is used, so this is backwards-compatible. - The CapTP version-mismatch log on `start-session` now includes both the received and expected version strings. ### Patch Changes - Updated dependencies \[[`ad7a177`](ad7a177), [`dd45f4a`](dd45f4a), [`45d06cd`](45d06cd), [`38fe678`](38fe678)]: - @endo/hex@1.1.0 - @endo/bytes@1.0.0 - @endo/marshal@1.10.0 - @endo/syrup-frame@0.1.1 ## ses@2.1.0 ### Minor Changes - [#3284](#3284) [`f2aa55a`](f2aa55a) Thanks [@boneskull](https://github.com/boneskull)! - Allows `Map.prototype.getOrInsert`, `Map.prototype.getOrInsertComputed`, `WeakMap.prototype.getOrInsert` and `WeakMap.prototype.getOrInsertComputed`. ### Patch Changes - [#3223](#3223) [`fa0b6a9`](fa0b6a9) Thanks [@gibson042](https://github.com/gibson042)! - Addresses infidelities and inefficiencies of [#3214](#3214) - [#3214](#3214) [`459347b`](459347b) Thanks [@erights](https://github.com/erights)! - Fixes bug [#3202](#3202) repairing momentary breakage of setFloat16 and setFloat32 ## @endo/bundle-source@4.3.1 ### Patch Changes - [#3261](#3261) [`7309d69`](7309d69) Thanks [@turadg](https://github.com/turadg)! - Replace `ts-blank-space` with Amaro for TypeScript type erasure, matching the parser Node.js uses for type stripping. - [#3216](#3216) [`7325bbe`](7325bbe) Thanks [@kriskowal](https://github.com/kriskowal)! - `@endo/base64`'s named exports (`encodeBase64`, `decodeBase64`, `atob`, `btoa`) are now frozen. Consumers that previously assigned to or extended these exports will see a `TypeError` under SES; read-only consumers are unaffected. The shim entry point `@endo/base64/shim.js` (which `@endo/init/pre.js` uses to install `globalThis.atob` / `globalThis.btoa` before `lockdown()`) is unchanged and continues to be safe to load pre-lockdown. - [#3237](#3237) [`b845f85`](b845f85) Thanks [@kriskowal](https://github.com/kriskowal)! - Update `BundleOptions` and the `powers` parameter types to reflect what `bundleSource` already accepts at runtime: `cacheSourceMaps` is now documented on `BundleOptions`, `commonDependencies` and `importHook` (only for the `endoZipBase64` format) are surfaced, and the optional `powers` parameter accepts the wider `BundlePowers` shape (including `pathResolve`, `userInfo`, `env`, `platform`, and `computeSha512`). This is a JSDoc/typings-only change; no runtime behavior changes. - Updated dependencies \[[`7325bbe`](7325bbe), [`7325bbe`](7325bbe), [`69ca27c`](69ca27c), [`5c098c4`](5c098c4), [`67ed1ce`](67ed1ce), [`75253ad`](75253ad)]: - @endo/base64@1.1.0 - @endo/compartment-mapper@2.2.0 - @endo/evasive-transform@2.2.0 ## @endo/captp@4.5.1 ### Patch Changes - [#2901](#2901) [`69ca27c`](69ca27c) Thanks [@kriskowal](https://github.com/kriskowal)! - Sweep `makeFinalizingMap`'s `get` operator to use optional chaining (`keyToRef.get(key)?.deref()`) now that [#1514](#1514) has completed, replacing the prior explicit conditional. Behavior is unchanged. - Updated dependencies \[[`45d06cd`](45d06cd)]: - @endo/marshal@1.10.0 ## @endo/patterns@1.9.1 ### Patch Changes - [#3265](#3265) [`45d06cd`](45d06cd) Thanks [@kriskowal](https://github.com/kriskowal)! - Add `compareRankRemotablesTied` and `compareAntiRankRemotablesTied`, which behave like `compareRank` and `compareAntiRank` except that they treat all remotables as tied for the same rank instead of short-circuiting the comparison on encountering a remotable. Make the `compare` parameter optional on `isRankSorted`, `assertRankSorted`, `sortByRank`, `getIndexCover`, `unionRankCovers`, and `intersectRankCovers`, defaulting to `compareRankRemotablesTied`. The following parameter orders change so the optional comparator lands at the end: - `getIndexCover(sorted, compare, rankCover)` → `getIndexCover(sorted, rankCover, compare?)` - `unionRankCovers(compare, covers)` → `unionRankCovers(covers, compare?)` - `intersectRankCovers(compare, covers)` → `intersectRankCovers(covers, compare?)` These are breaking changes to helpers that have no known external callers within the Endo monorepo. Within the monorepo, the only callers of `unionRankCovers` and `intersectRankCovers` are in `@endo/patterns`'s `patternMatchers.js`, which has been updated to match the new parameter order. Salvaged from <#2871> so that the codec-admission core of that PR (now in [#3226](#3226)) can land independently of this rank-comparison refactor. - Updated dependencies \[[`45d06cd`](45d06cd)]: - @endo/marshal@1.10.0 ## @endo/syrup-frame@0.1.1 ### Patch Changes - [#3256](#3256) [`38fe678`](38fe678) Thanks [@kriskowal](https://github.com/kriskowal)! - - New package `@endo/syrup-frame`: a sibling of `@endo/netstring` that drops the trailing `,` separator, so each framed payload on the wire is literally a Syrup byte-string record (`<length>:<payload>`). - Provides `makeSyrupReader` and `makeSyrupWriter` with the same shape as the netstring equivalents, including the `chunked` zero-copy writer mode. - Intended for testing interoperability with the OCapN TCP-for-testing protocol, which has moved to adopt this framing. The framing is among the protocols under consideration in the OCapN pre-standards group at time of writing; the 2025-12-09 OCapN plenary recorded the consensus that the TCP-for-testing netlayer should carry messages as length-prefixed Syrup byte strings (<https://github.com/ocapn/ocapn/blob/main/meeting-minutes/2025-12-09.md> and the parallel discussion on ocapn/ocapn#104). ## @endo/daemon@3.0.0 ### Major Changes - [#3204](#3204) [`291e224`](291e224) Thanks [@gibson042](https://github.com/gibson042)! - Detect truncation of Unix domain socket paths ### Patch Changes - [#3171](#3171) [`c372670`](c372670) Thanks [@turadg](https://github.com/turadg)! - TypeScript 6 conformance: public types in `types.d.ts` are now more precise. - `Context.thisDiesIfThatDies` and `thatDiesIfThisDies` parameters tightened from `string` to `FormulaIdentifier` (a branded string type). - `RemoteControl.accept`/`connect` and `RemoteControlState.accept`/`connect` take `remoteGateway: ERef<EndoGateway>` (was `Promise<EndoGateway>`); `RemoteControl.connect` now returns `ERef<EndoGateway>` (was `Promise<EndoGateway>`); `RemoteControlState.connect` returns `{ state; remoteGateway: ERef<EndoGateway> }`. - `EndoInspector` generic type parameter renamed from `Record` to `RecordT` to avoid shadowing the built-in `Record` utility type; `lookup` and `list` now use method syntax so that `EndoInspector<'some literal'>` remains assignable to `EndoInspector<string>` under `strictFunctionTypes`. TypeScript consumers implementing or calling these interfaces may need to update their types accordingly. - Updated dependencies \[[`ad7a177`](ad7a177), [`7325bbe`](7325bbe), [`7325bbe`](7325bbe), [`f2aa55a`](f2aa55a), [`69ca27c`](69ca27c), [`fa0b6a9`](fa0b6a9), [`69ca27c`](69ca27c), [`dd45f4a`](dd45f4a), [`45d06cd`](45d06cd), [`67ed1ce`](67ed1ce), [`459347b`](459347b), [`75253ad`](75253ad)]: - @endo/hex@1.1.0 - @endo/base64@1.1.0 - ses@2.1.0 - @endo/captp@4.5.1 - @endo/compartment-mapper@2.2.0 - @endo/bytes@1.0.0 - @endo/marshal@1.10.0 - @endo/patterns@1.9.1 ## @endo/cli@2.3.13 ### Patch Changes - Updated dependencies \[[`ad7a177`](ad7a177), [`7309d69`](7309d69), [`7325bbe`](7325bbe), [`b845f85`](b845f85), [`c372670`](c372670), [`291e224`](291e224), [`dd45f4a`](dd45f4a), [`45d06cd`](45d06cd)]: - @endo/hex@1.1.0 - @endo/bundle-source@4.3.1 - @endo/daemon@3.0.0 - @endo/bytes@1.0.0 - @endo/patterns@1.9.1 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Salvaged from endojs#2871 so that the codec-admission core of that PR (now in endojs#3226) can land without depending on this orthogonal refactor. - Add `compareRankRemotablesTied` and `compareAntiRankRemotablesTied`, which behave like `compareRank` / `compareAntiRank` but consider all remotables tied for the same rank instead of short-circuiting on encountering one. - Make the `compare` parameter optional on `isRankSorted`, `assertRankSorted`, `sortByRank`, and `getIndexCover`, defaulting to `compareRankRemotablesTied`. Reorder `getIndexCover`'s parameters from `(sorted, compare, rankCover)` to `(sorted, rankCover, compare?)` so the optional parameter lands at the end. No known external callers within the Endo monorepo. - Update `rankOrder.test.js` to exercise the new defaults and to reference `compareRankRemotablesTied` where intent is preserve-order rather than short-circuit. Refs: endojs#2883

erights self-assigned this Jun 27, 2025

erights requested review from gibson042 and kriskowal June 27, 2025 01:51

erights mentioned this pull request Jun 27, 2025

fix(marshal): marshal byteArrays #2844

Closed

erights force-pushed the markm-marshal-hex-byteArrays branch from 7c91967 to 5e1d4d1 Compare June 27, 2025 01:56

erights marked this pull request as ready for review June 27, 2025 02:14

erights commented Jun 27, 2025

View reviewed changes

Comment thread packages/marshal/src/encodeToCapData.js

erights commented Jun 27, 2025

View reviewed changes

Comment thread packages/marshal/src/rankOrder.js

erights commented Jun 27, 2025

View reviewed changes

Comment thread packages/marshal/src/rankOrder.js

erights commented Jun 27, 2025

View reviewed changes

erights force-pushed the markm-marshal-hex-byteArrays branch from 5e1d4d1 to 284cedd Compare June 29, 2025 00:44

erights added next-release ocapn labels Jun 29, 2025

erights force-pushed the markm-marshal-hex-byteArrays branch 2 times, most recently from 4eba653 to 08097b1 Compare July 11, 2025 01:42

kriskowal approved these changes Jul 11, 2025

View reviewed changes

gibson042 reviewed Jul 11, 2025

View reviewed changes

erights force-pushed the markm-marshal-hex-byteArrays branch 2 times, most recently from 54935b9 to 7b8a5fb Compare July 11, 2025 19:51

This was referenced Jul 11, 2025

Eventually replace agoric-sdk's hex.js with the one in endo #2871 Agoric/agoric-sdk#11592

Open

When to use compareRank vs compareRankRemotablesTied from #2871 #2883

Open

erights force-pushed the markm-marshal-hex-byteArrays branch from 40e2821 to 42cd111 Compare July 11, 2025 21:47

gibson042 requested changes Jul 11, 2025

View reviewed changes

Comment thread packages/pass-style/src/byteArray.js Outdated

erights force-pushed the markm-marshal-hex-byteArrays branch 2 times, most recently from c54b5d7 to 18f888a Compare July 11, 2025 21:53

gibson042 reviewed Jul 11, 2025

View reviewed changes

erights removed the next-release label Jul 11, 2025

erights force-pushed the markm-marshal-hex-byteArrays branch from e3eb1e7 to e9296e8 Compare July 14, 2025 01:45

erights force-pushed the markm-marshal-hex-byteArrays branch from e9296e8 to a4725d9 Compare July 21, 2025 21:28

erights requested a review from Copilot September 16, 2025 08:02

Copilot AI reviewed Sep 16, 2025

View reviewed changes

erights force-pushed the markm-marshal-hex-byteArrays branch from a4725d9 to 0399183 Compare September 18, 2025 03:36

erights and others added 5 commits September 18, 2025 17:35

fix(marshal): marshal ByteArrays to hex

73f3860

fixup! format

5198915

fixup! review suggestion

572eea6

Co-authored-by: Richard Gibson <richard.gibson@gmail.com>

fixup! avoid using Buffer even if present

8259047

fixup! comment out global Buffer decl

90fe86f

erights force-pushed the markm-marshal-hex-byteArrays branch from 0399183 to 90fe86f Compare September 19, 2025 00:36

erights mentioned this pull request Feb 26, 2026

feat(exo-stream): Introduce Exo streams #3036

Open

kriscendobot mentioned this pull request Apr 30, 2026

feat(marshal,pass-style): admit immutable ArrayBuffer through codecs #3226

Closed

kriscendobot mentioned this pull request Apr 30, 2026

refactor(marshal): compareRankRemotablesTied for rank-cover ops endojs/endo-but-for-bots#73

Closed

3 tasks

erights commented May 1, 2026

View reviewed changes

erights mentioned this pull request May 12, 2026

feat(marshal,pass-style): admit immutable ArrayBuffer through codecs endojs/endo-but-for-bots#57

Closed

kriskowal mentioned this pull request May 15, 2026

refactor(marshal): compareRankRemotablesTied for rank-cover ops #3265

Merged

kriskowal mentioned this pull request May 15, 2026

Version Packages #3198

Merged

-/**
- * Export a hex codec that can work with standard JS engines, but takes
- * advantage of optimizations on some platforms (like Node.js's Buffer API).
- */
-export const { uint8ArrayToHex, encodeHex, hexToUint8Array, decodeHex } =
-  typeof Buffer === 'undefined'
-    ? makePortableHexCodec()
-    : makeBufferishHexCodec(Buffer);
+/**
+ * @param {Pick<HexCodec, 'uint8ArrayToHex' | 'hexToUint8Array'>} hexCodec
+ * @returns {HexCodec}
+ */
+const addDeprecatedFunctions = hexCodec => {
+  const { uint8ArrayToHex, hexToUint8Array } = hexCodec;
+  /**
+   * @deprecated Use `uint8ArrayToHex` instead
+   * @type {HexCodec['uint8ArrayToHex']}
+   */
+  const encodeHex = u8 => uint8ArrayToHex(u8);
+  /**
+   * @deprecated Use `hexToUint8Array` instead
+   * @type {HexCodec['hexToUint8Array']}
+   */
+  const decodeHex = hex => hexToUint8Array(hex);
+  return harden({
+    uint8ArrayToHex,
+    encodeHex,
+    hexToUint8Array,
+    decodeHex,
+  });
+};
+/**
+ * Create the most performant hex codec from the available global environment
+ * (possibly leveraging platform-specific optimizations such as the Node.js
+ * Buffer).
+ *
+ * @returns {Pick<HexCodec, 'uint8ArrayToHex' | 'hexToUint8Array'>}
+ */
+const makeBestHexCodec = () => {
+  if (Uint8Array.fromHex) {
+    return harden({
+      uint8ArrayToHex: u8 => u8.toHex(),
+      hexToUint8Array: hex => Uint8Array.fromHex(hex);
+    });
+  }
+  return typeof Buffer === 'function'
+    ? makeBufferishHexCodec(Buffer)
+    : makePortableHexCodec();
+};
+export const { uint8ArrayToHex, encodeHex, hexToUint8Array, decodeHex } =
+  addDeprecatedFunctions(makeBestHexCodec());

	// non-zero padding bits (MAY reject): ['Qf==', ...],
	// (additional invalid hex inputs could be added here)

		@@ -1,11 +1,19 @@
		import { makeTagged, passableSymbolForName } from '@endo/pass-style';
		import {

Uh oh!

Conversation

erights commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Security Considerations

Scaling Considerations

Documentation Considerations

Testing Considerations

Compatibility Considerations

Upgrade Considerations

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Sep 16, 2025

Choose a reason for hiding this comment

Uh oh!

kriscendobot commented Apr 30, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

erights commented May 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

erights commented Jun 27, 2025 •

edited

Loading