snapshot testing (internal SDK developer feature)

[kuhe]

Describe the feature

This is a tracking issue for an internal test feature request.

Because of the large number of SDK operations and variations within each operation, we cannot hand-write tests for all of them.

Create a test framework that generates snapshot tests (with write and compare modes) for requests, responses, and error responses.

Features

Part 0: Coverage

• Smithy protocol test clients
• Subset of AWS SDK clients in various AWS Protocols
• All AWS SDK clients

Part 1: Request Snapshots

For each SDK client operation, generate a request snapshot.

To generate a snapshot, intercept an outgoing request and serialize it to a snapshot file which can be versioned.

• snapshot-write should be executed during the code generation process.
• snapshot-compare should be executed during the protocol test step.

Deterministic snapshot serialization

• normalize all UUIDs
• normalize all Timestamps, including those used for signatures
• redact authorization headers
• normalize user agents
• delete all AWS environment variables during the snapshot process
• the client must be given MOCK static credentials

This can be done at the final file-write step.

The basic format for a request snapshot is:

METHOD https://host
/path?query

header: value
header2: value

[body]

To make the request payload visually inspectable, the body should be normalized in a consistent way.

• For JSON, XML, Query strings, it can be formatted.
• For CBOR, it can be decoded and then formatted.
• For data streams, the stream should be flattened.

Example for data stream:

[async_iterable (Readable)]

[chunk (b64)]
MQo=
[chunk (b64)]
MAo=
[chunk (b64)]
MAo=
[chunk (b64)]
MQo=

• For event streams, the events should be flattened and formatted (challenging).

Example for event stream:

[async_iterable (PassThrough)]

[chunk (event-stream object view)]
  [total-size] 332 [header-size] 67 [prelude-crc] 3702041006
    [total-size] 249 [header-size] 231 [prelude-crc] 510171806
:date: Fri, Dec 31, 1999, 20:59:59 Pacific Standard Time (timestamp)
:chunk-signature: 109,12,189,143,31,120,66,246,189,208,118,112,208,197,214,85,160,150,22,155,231,205,26,54,81,56,103,194,241,96,137,250 (binary)

:event-type: headers
:message-type: event
:content-type: application/json
booleanHeader: false (boolean)
byteHeader: 0 (integer)
shortHeader: 0 (integer)
intHeader: 0 (integer)
longHeader: 0 (integer)
blobHeader: 1,0,0,1 (binary)
stringHeader: __stringHeader__
timestampHeader: Fri, Dec 31, 1999, 20:59:59 Pacific Standard Time (timestamp)

[json]
{}

  [message-crc] 329385819
[message-crc] 2711902348
============================================================

Input data generation

Input data must also be generated in a deterministic way. It's actually impossible to cover all permutations, but I have chosen the following data generation heuristic:

• Using the input shape,

  • for simple fields, generate a consistent placeholder value.
  • for strings specifically, default to the member name surrounded by __. E.g., Bucket: "__Bucket__"
  • For maps and lists, generate 3 members.
  • For unions, generate a deterministically selected union member. The selection is done by hashing the string path to the union value modulo the member size. In conjunction with the map and list policy of generating 3 members, this gives a fair degree of variance in the union member that is generated.
  • For event streams, generate an iterator that emits every union member once.

• Account for fields that require specific constraint values.

The following fields required overrides in JS to pass client side custom validation:

    PredictEndpoint: "https://localhost",
    ChecksumAlgorithm: "CRC64NVME",
    AccountId: "123456789012",
    OutpostId: "OutpostId",
    QueueUrl: "https://sqs.us-east-1.amazonaws.com/012345678901/MyQueue",

    MD5OfMessageBody: "07261c767f72c87a6c6e6e62c93c2664", // md5 of "__Body__"
    MD5OfBody: "07261c767f72c87a6c6e6e62c93c2664", // md5 of "__Body__"
    MessageBody: "__Body__",

Part 2: Response Snapshots

Response snapshots are more difficult to create. The process is the same, being to serialize an operation response to a file in write and compare modes, but the test framework is not going to be perfectly accurate in generating an mock of an AWS service's response message.

• If you have a server-side serializer for your various protocols, consider using that.
• Start by generating a snapshot input and request, same as in part 1.
• Also generate a response object, but serialize it to a response (e.g. HTTP Response)
• Using the SDK client under test, send and intercept the request, then respond with the generated response from the previous step.
• snapshot-write should be executed during the code generation process.
• snapshot-compare should be executed during the protocol test step.

Implementing the HTTPResponse serializer

To even do this, you have to get about 20% of the way to having a Server SDK, so if you were planning on doing that or have already done that, great 😐.

• For RPC, it's likely sufficient to response with status code 200, and two headers, namely:
  • content-type - use the default content type of the AWS Protocol under test
  • x-amzn-requestid - a UUID
  • For Smithy RPCv2 CBOR, also include "smithy-protocol=rpc-v2-cbor"
• For REST, also respond with status code 200, and the content-type and x-amzn-request-id headers
• Use member bindings for httpHeader(s)
• Use the member bound for httpPayload, or serialize unbound members to payload.
• Remember for AWS Query Protocol, use the XML ShapeSerializer for responses instead of the Query format.

Part 3: Error Response Snapshots

Snapshots of errors are similar to regular responses.

• for errors, use the @httpError trait to set the response code.
• for each AWS protocols, check the Smithy spec for response payload and header amendments specific to errors.
  • https://smithy.io/2.0/aws/protocols/aws-query-protocol.html#operation-error-serialization
  • https://smithy.io/2.0/aws/protocols/aws-ec2-query-protocol.html#operation-error-serialization
  • https://smithy.io/2.0/aws/protocols/aws-restxml-protocol.html#error-response-serialization
  • https://smithy.io/2.0/aws/protocols/aws-restjson1-protocol.html#operation-error-serialization
  • https://smithy.io/2.0/aws/protocols/aws-json-1_0-protocol.html#operation-error-serialization
    • https://smithy.io/2.0/aws/protocols/aws-json-1_1-protocol.html#differences-between-awsjson1-0-and-awsjson1-1

Use Case

release stability

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

[kuhe]

We enabled step one of request-side snapshot testing in #7777. This adds snapshots for protocol test clients internal to the repository (these are mock clients).

Earlier, we created https://www.npmjs.com/package/@smithy/snapshot-testing as the snapshot test runner, working within vitest.

For an example of a request snapshot, see https://github.com/aws/aws-sdk-js-v3/blob/main/private/aws-protocoltests-restjson-schema/test/snapshots/req/DuplexStream.txt

[kuhe]

This feature is now released, having implemented request, response, and error response snapshots.

• pending - expand snapshot testing to all services?