Serialization interface redesign

[schveiguy]

Thinking more about serialization, based on discussion in #64

We currently have the following model for deserialization:

input iopipe -> Tokenizer -> deserialize -> policy -> structured data

• iopipe is the raw data coming in
• Tokenizer is the thing that parses the input data into JSON tokens
• deserialize is the entry point for the deserialization of the JSON tokens into structured data.
• The policy is the customization point for how to specifically turn tokens into data.

The policy uses the tokenizer as a tool to deal at the token level. The Tokenizer enforces the input validity, and provides mechanisms (through UFCS functions and other things) to help transform the tokens into data (such as strings and numbers which might have escapes, etc.).

---

Now, what we have for serialization currently is a less complex system:

structured data -> serialize -> char[] delegate -> output iopipe

Adding in the policy, it looks like this:

structured data -> serialize -> policy -> char[] delegate -> output iopipe

This looks very similar, even symmetrical, to the chain for deserialization. However, notice the API for output is the simplest thing it can be -- a char[] accepting delegate.

In fact, the iopipe behind the delegate is not even important here. You could technically substitute any output range.

But we want options for customization. People want different formatting, they want customized handling of types. They want nice tools to enforce or help write valid JSON and JSON5.

Not only that, but we are an iopipe-library! the point of iopipe is to give buffer access to users. Hiding everything behind a char[] delegate defeats that purpose!

So I propose for serialization a new chain of control:

structured data -> serialize -> policy -> JSONFormatter -> output iopipe

The JSONFormatter object will assist like the Tokenizer, in generating valid JSON. It will have methods or UFCS functions like addMemberName(key), and automatically add the name and colon, expecting a value.

The API might look like this:

struct JSONFormatter(Chain) { // Chain is the output iopipe
    // aggregate output
    void beginObject();
    void beginArray();
    void endAggregate(); // ends object or array
    void addMember(T)(T key, Options); // options for member name

    // output non aggregates
    void beginString(char quote); // start a string
    void addStringData(bool validate = true, bool addEscapes = true, T)(T value); // will automatically escape string data as needed.
    void endString(); // end string, closing quote
    void addNumericData(T)(T value, string format = "%s"); // allow numeric types, or strings that are validated to be JSON numbers
    void addKeywordValue(KeywordValue value); // null, true, false, inf, etc.

    // for formatting/comments
    void addWhitespace(T)(T data); // Add whitespace data (validated)
    void addComment(T)(T commentData); // add a comment (JSON5 only), must be a complete comment (validated)

    // iopipe management
    void flushWritten(); // flush the currently written buffered data to the output.
    auto chain(); // return the underlying iopipe, for examination, fiddling with
}

Each serialization implementation function will be given a JSONFormatter, a Policy, and a Value to serialize, just like the deserializer gets that triple.

The policy can implement defaults for each type, similar to the deserializer. It has the additional chore of implementing formatting (like indentation).

Similar callbacks for outputting objects or arrays like serializeObject, or serializeArray.

We also might want to give a mechanism to directly write to the output iopipe. Maybe that's how the string data is written without validation? EDIT: did so.

---

A good test for whether we got the API right is if we can reimplement the formatjson example program with it.

ping @gulugulubing @benjones @Inkrementator @ahmetsait

[gulugulubing]

void addMember(T)(T key, Options); // options for member name

Is the Options here for something JSON5 support? like:
Quote style: "key" vs 'key'

[schveiguy]

That was my thinking yes.