DocLang backlog

[vagenas]

• ❗ High prio ❗

• training-relevant

  • decide on heading token (XML prolog, version)
  • align w.r.t. tokenizer

• update draft

  • make simpler (no redundant listings!) & stricter
  • align to recent updates (content, base64, layer classes, picture classes etc)
  • decide on ambiguous parts (e.g. group, br)

• update schema (XSD + Schematron)

  • extend schematron rules to address plain text in OTSL and others

• 🔸 Medium prio 🔸

• consider serialization extensions

  • alternative export mode where KVs are shown as just text

• consider standard extensions

  • support extended bounding polygons (beyond non-rotated rectangles) (see feat: extend bounding box #348)
    • (on draft level, could just be by generalizing from 4=22 location tokens (interpreted as rectangle) to N2 tokens for any arbitrary polygon with N vertices)

• update deserialization

• 🔹 Low prio 🔹

• DocLang (embedded) archive

• page breaks

• check formatting:hyperlink (feat(DocLang): add hyperlink support #578)

• list languages in draft (as subset of Linguist vX.Y.Z)

  • extend list to support e.g. Markdown?
  • define clearly how non-programming log/console-like content is to be captured (e.g. code with unset language or Shell?)

• cross-provenance content

• cross-page content

• misc

  • clarify dropping of include_formatting serializer switch

• ✅ Done ✅

• address key value regions (consolidate FormItem / KeyValueItem (KEY_VALUE_REGION) (feat: introduce field data model incl. Doclang serialization #519)

• fix table cell content deserialization (fix(DocLang): fix table cell <content> deserialization #512)

• fix document re-indexing (fix: fix document re-indexing #510)

• fix DocTags picture meta deserialization (fix(DocTags): fix deserialization to populate picture meta fields #505)

• review checkboxes (fix(Doclang): fix checkbox serialization #503)

• review chart serialization (compared to table serialization) (test(Doclang): add chart serialization test #498)

• align handling of default resolution (location) (fix(IDocTags): fix default location resolution handling #492)

• rich tables / nested tables (feat(IDocTags): add rich table support #491)

• leading / trailing whitespaces (feat(IDocTags): add content wrapping for handling whitespace #489)

• finalize outmost element name (chore: rename IDocTags to Doclang #494)

• add radio button support (update: not for now)

• XML Schema / DTD?

• converge between <content xml:space="preserve"> and {<content>+ other way of signalizing whitespace preservation e.g. via ATTLIST, XML schema, convention etc.}

• line breaks (feat(Doclang): add newline handling #575)

• content layer (e.g. furniture) not captured in DocLang (feat(Doclang): add content layer support #568)

• map HANDWRITTEN label to new DocLang element (similar to bold, italics) (feat: Add HANDWRITTEN_TEXT label support #561)

• inline code issue: conflict between <inline class="code"> and <code class="Python"> (chore(Doclang): remove inline element #517)

• use defusedxml? (fix: switch xml parsing #509)

• align image ref mode on DocLang serialization (fix(Doclang): align image mode, defaulting to placeholder #506)

• image URI serialization (fix(Doclang): fix image URI serialization #504)

• track CVAT test output (fix: populate picture meta, track Doclang output docling-cvat-tools#8)