Make ssl/crypto report the failures it was swallowing (#92)

The package had two conventions and no rule. RandBytes and Pbkdf2Sha256 checked
what OpenSSL returned and raised. Digest and HmacSha256 checked nothing and
handed back whatever was in the output buffer.

HmacSha256 returned thirty-two zero bytes when HMAC failed. That is not a
degenerate hash, it is a code an attacker can send: a verifier comparing a
computed code against one it was given accepts when the computation failed.
HMAC returns NULL when the key and the message pointers are both null, and an
empty Array[U8] has a null pointer, so the code was already reachable with
ordinary input. HmacSha256(empty, empty) returned zeros. Empty keys and empty
messages are well defined, so both now get a pointer to bytes HMAC will not
read, and a NULL return raises.

Digest.final returned its @pony_alloc output buffer without looking at what
EVP_DigestFinal_ex made of it. @pony_alloc does not zero, so a failed finalise
handed back whatever this actor freed last, as a hash. It raises now, and the
buffer is dropped.

EVP_MD_CTX_new returns NULL when it cannot allocate and EVP_DigestInit_ex
dereferences it. The constructors are partial and raise there, at the one point
that knows whether OpenSSL gave us a context. Deferring it would let someone
build an actor around a Digest that is doomed to raise on every final().

Six lengths were narrowed to a C int with no check. A length in [2^31, 2^32)
narrows negative and OpenSSL rejects it. A length past 2^32 narrows to a small
positive one: RandBytes(2^32 + 5) allocated four gigabytes, filled five bytes,
was told it succeeded, and returned the lot. Each length is checked against what
an int holds, before anything is allocated.

Digest's constructors, Digest.final and HmacSha256 are partial now. append was
already partial, so a streaming digest was always written inside a try.

Closes #85
Closes #86
Closes #88

Author: SeanTAllen

.release-notes/next-release.md

@@ -134,3 +134,45 @@ When a server selected an ALPN protocol, OpenSSL was left with a pointer to memo
 
 One thing changes for you: if you set your own resolver with `SSLContext.alpn_set_resolver` and it returns a protocol the client did not offer, the handshake now fails instead of continuing. `ALPNStandardProtocolResolver` is unaffected.
 
+
+## Make Digest and HmacSha256 raise on failure
+
+`Digest` and `HmacSha256` now raise when OpenSSL cannot do what you asked, where before they returned a wrong result with no error. Three calls gained a `?`: constructing a `Digest`, `Digest.final`, and `HmacSha256`. `Digest.append` was already partial.
+
+```pony
+// Before
+let digest = Digest.sha256()
+digest.append(data)?
+let hash = digest.final()
+
+let mac = HmacSha256(key, message)
+
+// After
+let digest = Digest.sha256()?
+digest.append(data)?
+let hash = digest.final()?
+
+let mac = HmacSha256(key, message)?
+```
+
+Constructing a `Digest` now raises if OpenSSL cannot allocate its context, rather than returning a digest that fails on every later call. When `HmacSha256` raises, reject the message. Do not fall back to a code of your own — a code you make up is one an attacker can send you.
+
+## Fix HmacSha256 returning an all-zero code when it fails
+
+`HmacSha256` returned thirty-two zero bytes when it could not compute the code, instead of failing. Thirty-two zero bytes is a value an attacker can send, so a program that checks a message by comparing a fresh code against a supplied one would accept a forgery whenever its own computation failed.
+
+It was reachable with ordinary input: computing the code of an empty key and an empty message returned all zeros. `HmacSha256` now returns the real code and raises when the computation actually fails.
+
+## Fix Digest returning a wrong hash or crashing when OpenSSL fails
+
+`Digest` could return a hash of the wrong bytes, or crash while being created, when an OpenSSL call inside it failed.
+
+`final` returned a block of memory as the hash without checking that OpenSSL had written it, so a failed call returned whatever that memory held — a wrong hash, and a leak of whatever was last in it. Creating a digest crashed when OpenSSL could not allocate its working context.
+
+A digest now raises rather than returning a wrong hash, and one that could not be created reports it at the first `append` or `final` rather than crashing.
+
+## Fix crypto functions truncating a length too large for an int
+
+`RandBytes`, `HmacSha256` and `Pbkdf2Sha256` silently truncated a length that did not fit the C `int` OpenSSL takes. `RandBytes`, asked for a number of bytes past four gigabytes, returned a buffer of that size with only a handful of random bytes in it and the rest zero, and reported success.
+
+These functions now raise on a length larger than an `int` can hold, before allocating anything.

CHANGELOG.md

@@ -14,6 +14,9 @@ All notable changes to this project will be documented in this file. This projec
 - Allow non-mutating methods to be called on a val receiver ([PR #89](https://github.com/ponylang/ssl/pull/89))
 - Fix leaks when loading Windows root certificates fails ([PR #90](https://github.com/ponylang/ssl/pull/90))
 - Fix a potential use-after-free in ALPN protocol selection ([PR #91](https://github.com/ponylang/ssl/pull/91))
+- Fix HmacSha256 returning an all-zero code when it fails ([PR #92](https://github.com/ponylang/ssl/pull/92))
+- Fix Digest returning a wrong hash or crashing when OpenSSL fails ([PR #92](https://github.com/ponylang/ssl/pull/92))
+- Fix crypto functions truncating a length too large for an int ([PR #92](https://github.com/ponylang/ssl/pull/92))
 
 ### Added
 
@@ -23,6 +26,7 @@ All notable changes to this project will be documented in this file. This projec
 - Add SSLDisposed to SSLState ([PR #68](https://github.com/ponylang/ssl/pull/68))
 - Require a val resolver for alpn_set_resolver ([PR #81](https://github.com/ponylang/ssl/pull/81))
 - Require a val context for SSLContext.client and server ([PR #81](https://github.com/ponylang/ssl/pull/81))
+- Make Digest and HmacSha256 raise on failure ([PR #92](https://github.com/ponylang/ssl/pull/92))
 
 ## [2.1.0] - 2026-04-20
 

examples/digest-example/digest-example.pony

@@ -4,23 +4,23 @@ use "../../ssl/crypto"
 
 actor Main
   new create(env: Env) =>
-    let sha256digest: Digest = Digest.sha256()
+    let sha256digest: Digest = Digest.sha256()?
     try
       sha256digest.append("Hello ")?
       sha256digest.append("World")?
-      let hash: Array[U8] val = sha256digest.final()
+      let hash: Array[U8] val = sha256digest.final()?
       env.out.print("SHA256: " + ToHexString(hash))
     else
       env.out.print("Error computing hash")
     end
 
     // SHAKE256 with variable-length output (OpenSSL 3.0.x or 4.0.x)
     ifdef "openssl_3.0.x" or "openssl_4.0.x" then
-      let shake: Digest = Digest.shake256(64)
+      let shake: Digest = Digest.shake256(64)?
       try
         shake.append("Hello ")?
         shake.append("World")?
-        let shake_hash: Array[U8] val = shake.final()
+        let shake_hash: Array[U8] val = shake.final()?
         env.out.print("SHAKE256 (64 bytes): " + ToHexString(shake_hash))
       else
         env.out.print("Error computing SHAKE hash")