New `enums` syntax to allow enum-level properties (`doc` / `doc-ref`, required underlying type)

[generalmimon]

Since the current enums syntax contains the enum values directly under the name of the enum, it doesn't allow for any "global" properties for the entire enum.

#358 discusses that this prevents support for doc / doc-ref keys to document the enum (if we disregard the ugly hack of mixing doc and integer keys on the same YAML level, as shown in #358 (comment), which IMO shouldn't even be considered). These would be optional properties that have no impact on the overall behavior of the language, though.

What's more important is that in #862 (comment), @GreyCat pointed out that the "untyped enums" that we have right now are not a very good idea. Pretty much all statically typed target languages require or encourage specifying the underlying integer type that the enum is based on. Sometimes you can technically omit specifying it explicitly, but that just means that some underlying type will be decided by the target language instead, which is often not what we want.

For example, #862 shows that C# uses int (a 32-bit signed integer type) by default, which effectively limits our C# enum support to signed 32-bit integers. In C++, the enum type can by default legally only represent integers with the minimum bit width that can fit all enum constants. If you try to convert a wider integer value to the enum type (#778 is related), you are "loading out-of-range values to enums without fixed underlying type", which is undefined behavior - see #959. The page https://pvs-studio.com/en/docs/warnings/v1016/ referenced in that issue has a good example:

Example 3.

enum EN { low = 2, high = 4 }; // Uses 3 bits, range: [0; 7]
EN a1 = static_cast<EN>(7);    // ok

According to the standard, the underlying type for this enum is 'int'. Inside this type, the compiler uses the minimum width of the bit field that can fit all the values of enum constants.

In this case, you will need at least 3 bits to fit all the values (2 = 0b010 and 4 = 0b100), so an EN variable can fit numbers from 0 (0b000) to 7 (0b111) inclusively. The number 8 already occupies four bits (0b1000), so it no longer fits in the EN type:

EN a2 = static_cast<EN>(8);    // UB

UndefinedBehaviorSanitizer also finds an error in this example: https://godbolt.org/z/GGYo7z.

Therefore, "untyped enums" are not really a thing - the compiler should always know the exact underlying type of each enum and declare it in the generated code, if the target language supports that. Another reason that illustrates why this is important is the to_i method of enums, see the User Guide:

6.4.5. Enums

Method name	Return type	Description
to_i	Integer	Converts an enum into the corresponding integer representation

It says that the return type is "integer", but that's not specific enough - the compiler must know the exact type. In theory, this should be the enum's underlying type, but since the compiler doesn't know that at the moment, it uses CalcIntType, i.e. signed 32-bit integer:

TypeDetector.scala:220-222

      case et: EnumType =>
        attr.name match {
          case "to_i" => CalcIntType

So in statically typed languages, the result of the enum.to_i method is currently truncated to a signed 32-bit integer, regardless of the actual underlying type.

---

Therefore, I propose a new syntax for enum definitions (which combines @GreyCat's suggestions in #358 (comment) and #862 (comment)):

enums:
  animals:
    doc: Animals you might find at the office.
    type: u4
    values:
      1: dogs
      2: cats

Obviously, the enum values themselves need to be moved to the values key to allow specifying other enum properties. That's a relatively trivial change. A bigger change is the new type key, which will be REQUIRED and will explicitly declare the enum's underlying type.

To address @GreyCat's concerns in #862 (comment), the underlying type will be enforced. First, this means that only integer constants representable by the underlying type will be allowed in the enum definition. Second, it will no longer be possible to directly convert an integer to an enum whose underlying type has a different width or signedness. So for the u4-based enum animals defined above, trying to use it on a s4 field would be a compile-time error:

seq:
  - id: animal
    type: s4  # error: 's4' does not match the underlying type 'u4' of the enum 'animals'
    enum: animals

Of course, this is a breaking change, but I believe it's the only future-proof solution that ensures consistent behavior across languages. What do you think?

[GreyCat]

Generally, this looks like a good step forward to me.

What I would still suggest to do is to keep this compatible on 0.x branch, and then break compatibility after 0.x — probably we should just release whatever we've got already (it's been 10 years, isn't it?) as 1.x, and then starting with 2.x introduce a bunch of these breaking changes (maybe remembering #55 too) + ideally, some kind of automatic conversion tool.

Theoretically, it's relatively easy to autodetect which of two formats are there (map of int keys = old, anything else = new). I totally understand that we'll likely not reap immediate benefits of precise int type definition, having to kind of be in two worlds simultaneously...

[Mingun]

Perhaps it is better to implement this option: if there is a values key, then consider that the enumeration is written in a new format (produce error for unknown keys at the same level as values). If it is not there, then consider that we are in the old format (produce error for all non-numeric keys):

enums:
  old:
    1: one
    2: two
    doc: # error, expected numeric key
  new:
    1: one # error, expected values, doc, type or to-string
    values:
      1: one
      2: two
      doc: # error, expected numeric key

Although I personally do not see any reason why new, non-numeric keys cannot be added at the top level, they never overlap with enumeration values.

Also, it seems to me that the following case should be taken into account. Imagine a situation where at first the enumeration was used in a field, but as the protocol evolved, the field size became insufficient and a new field of a larger size was added to the protocol with the same data, which may contain new values from the enumeration. I would like to use the same enumeration so as not to duplicate it, but the compiler may not allow this, since now not all values fit into the old field:

seq:
  - id: old
    type: b2
    enum: enumeration
  - id: new # added in version 2
    type: b4
    enum: enumeration
enums:
  enumeration:
    type: b4
    values:
      0b00: value1
      0b01: value2
      0b1000: value3 # added in version 2

[generalmimon]

@Mingun:

Also, it seems to me that the following case should be taken into account. Imagine a situation where at first the enumeration was used in a field, but as the protocol evolved, the field size became insufficient and a new field of a larger size was added to the protocol with the same data, which may contain new values from the enumeration. I would like to use the same enumeration so as not to duplicate it, but the compiler may not allow this, since now not all values fit into the old field:

seq:
  - id: old
    type: b2
    enum: enumeration
  - id: new # added in version 2
    type: b4
    enum: enumeration
enums:
  enumeration:
    type: b4
    values:
      0b00: value1
      0b01: value2
      0b1000: value3 # added in version 2

Makes sense, thanks for pointing this out - we can allow that. In my current draft implementation, only integer types equivalent to the enum's underlying type are allowed to be converted to the enum. In theory, this guarantees that bi-directional conversion between the integer type and the enum type is always well-defined. When parsing, no truncation occurs when converting the parsed integer type to the enum type, because the enum's underlying type can hold all values of the parsed integer type (i.e. the underlying type is at least as large as the parsed integer type). And during serialization, any enum value can be converted to the serialized integer type without truncation, since the serialized integer type can hold all possible values of the enum's underlying type.

However, this is probably unnecessarily restrictive. We can relax this requirement to allow converting any integer type to an enum if and only if all of its values are representable by the enum's underlying type. This guarantees that the conversion from an integer value to an enum type is always lossless (the integer type must be a subset of enum's underlying type), whereas not every enum value might necessarily fit into the in-stream integer during serialization. Therefore, it might not be possible to serialize every defined enum constant if the actual integer field is too small. Nevertheless, this could happen even now when we only allow integer types that match the enum's underlying type, for example in this scenario:

seq:
  - id: foo
    type:
      switch-on: false
      cases:
        false: u4
        _: u1
    enum: my_enum
enums:
  my_enum:
    type: u4
    values:
      0xffff_ffff: blah

Hmm, this apparently doesn't compile at the moment - enum cannot be used in combination with type switching. I'd say this is a bug.

Anyway, the idea is that the extension of a smaller integer type to a larger one is generally silently allowed, so even if the resulting type matches the enum's underlying type exactly, that's not a guarantee that all enum values can be actually serialized. On a similar note, both Java and Python (the only two target languages with serialization support) actually often use a larger integer type than was defined (Java uses a 64-bit long for 32-bit u4 because of the unsignedness; Python has arbitrary-precision integers, so their range is practically unbounded), so technically, you can already attempt to serialize an integer value that doesn't fit, even if no enums are involved. Of course, that's just another case of inconsistent data, so ideally _check should catch it (which is currently not the case), but I digress.

Let me analyze the case in #862 (comment) presented by @GreyCat, which was the reason I thought that applying enums to smaller integer types probably shouldn't be allowed. After converting it to the new syntax, it might look like this:

seq:
  - id: one
    type: s1
    enum: constants

enums:
  constants:
    type: s8
    values:
      0: zero
      255: foo
      -1: bar
      4294967295: baz

s8 is the smallest underlying type that can hold all defined enum constants. Since s1 is a subset of s8, applying the constants enum to field one is allowed. This should effectively turn the memory representation of one into an enum type backed by s8 (not s1) in target languages where that matters. Therefore, there won't be any doubt about what enum member one corresponds to, as #862 (comment) feared:

If we parse a single byte FF with this one — what it should map to, or which of these comparisons should be true — foo, bar, baz?

Depending on implementation, it can be:

• only bar (if we resolve this straight away into a string/symbol and only keep one which matched at time of reading)
• foo + bar (if we store it as a single-byte in memory, and compare it as a byte)
• bar + baz (if we keep it strictly as signed 1-byte, and whenever we compare one == constants::baz, both sides are promoted to default integer size of a platform and compared as (uint32_t) -1 == (uint32_t) 4294967295)
• etc.

As I mentioned in #1288 (comment), I think that in all statically typed languages that we currently support, enum types always have one specific underlying type (which in this case is s8). So one == constants::baz doesn't implicitly convert either side to any other type: both sides are effectively represented as s8 already, and we don't even get to choose otherwise.

Perhaps only in some dynamically typed languages with multiple integer types, where enum values are stored as plain integers (#1289), we might have to decide what type one should be. I'm specifically thinking of JavaScript once the support for exact 64-bit integers is implemented (#183), because then there would be two integer types: Number and BigInt. In theory, one could be either a Number (which seems to be enough, since it only needs to represent s1 values) or a BigInt (generally needed to accurately represent all values of type s8). Howewer, it's probably much safer to store it as a BigInt, because that guarantees that it will be compatible (without the need for explicit conversions) with other values of the constants enum, which would be BigInts (since s8 would map to BigInt).