Rework serialization and model dumping for v2 models

docs/MIGRATION.md

@@ -10,6 +10,7 @@ This project ships two QCSchema families:
   - Make subschema more re-useable and composible
   - Standardize naming and field availability to be more predictable
   - Bring visible change to accompany the Pydantic v2 transition
+  - Make models more compatible with standard Pydantic v2 paradigms and workflows
 
 - This guide is AI generated and edited with some care. But a better guide is the [cheat sheet](docs/qcschema_cheatsheet_9Jan2026.pdf).
 
@@ -27,6 +28,11 @@ This project ships two QCSchema families:
   qcelemental.models.v1 import SomeQCSchemaClass`. Never use `somefile`-style imports again.
 - Change your `SomeQCSchemaClass.dict()`, `.json()`, and `.copy()` to `.model_dump()`, `.model_dump_json()`, and `.model_copy()`.
   Both v1 and v2 classes define both sets, but the latter set is the Pydantic v2 way and will quench a lot of warnings.
+  - Replace `SomeQCSchemaClass.dict(encoding="json")` with `SomeQCSchemaClass.model_dump(mode="json")`
+- Models will dump all fields by default (except some models will exclude fields with a value of `None`). If you want to
+  exclude fields with default or unset values, use `.model_dump(exclude_unset=True)` and/or `.model_dump_json(exclude_unset=True)`.
+  - Note that this will cause `schema_name` and `schema_version` to be excluded, which may cause problems with
+    validation later on.
 
 ## Migrating to v2
 
@@ -72,6 +78,13 @@ v2, return control to the schema wrapper, call `convert_v(return_version)`, and
   - The package provides visible, non-destructive warnings and placeholder behavior such that importing `qcelemental.models.v1` succeeds, but
     instantiating v1 models raises a clear `RuntimeError` on Python versions >=3.14.
 
+- Serialization is handled in a more standard pydantic way. This makes the typical pydantic functions like `model_dump()`
+  `model_dump_json()` behave as expected, removing the custom serialize functions we previously had.
+  - Numpy serialization is handled by using a custom type annotation (`Array`) which will serialize to a list of floats
+    in "json" mode, and as a numpy array in "python" mode. Validation is handled here as well.
+  - `model_dump(mode="python")` will return a dict in Python format, where numpy arrays are kept as numpy arrays.
+  - `model_dump(mode="json")` will return a dict in JSON format, where numpy arrays are converted to lists.
+  - The above will make features like `model_dump_json()` work without any other custom code.
 
 ---
 

docs/changelog.rst

@@ -24,6 +24,33 @@ Changelog
 .. +++++
 
 
+.. _`sec:cl0500rc4`:
+
+0.50.0rc4 / 2026-MM-DD (Unreleased)
+-----------------------------------
+
+:docs:`v0.50.0rc4` for current. :docs:`v0.30.2` for QCSchema v1.
+
+Breaking Changes
+++++++++++++++++
+
+- (:pr:`393`) Reworks serialization and model dumping for v2 models. Many models will now include
+  unset and default fields by default. Also removes buggy custom serialization logic, leaning more on standard
+  pydantic recommendations. Also makes more models tolerant of `None` values being passed in for certain fields.
+
+New Features
+++++++++++++
+
+Enhancements
+++++++++++++
+
+Bug Fixes
++++++++++
+
+Misc.
++++++
+
+
 .. _`sec:cl0500rc3`:
 
 0.50.0rc3 / 2026-03-10

qcelemental/datum.py

@@ -6,6 +6,7 @@
 from typing import Any, Dict, Optional, Union
 
 import numpy as np
+from numpy.typing import NDArray
 from pydantic import (
     BaseModel,
     ConfigDict,
@@ -26,9 +27,7 @@ def reduce_complex(data):
     return data
 
 
-def keep_decimal_cast_ndarray_complex(
-    v: Any, nxt: SerializerFunctionWrapHandler, info: SerializationInfo
-) -> Union[list, Decimal, float]:
+def keep_decimal_cast_ndarray_complex(v: Any, nxt: SerializerFunctionWrapHandler, info: SerializationInfo) -> Any:
     """
     Ensure Decimal types are preserved on the way out
 

qcelemental/models/_v1v2/atomic.py

@@ -10,7 +10,7 @@
 #   from v1, they'll need redefining here
 from ..v2.atomic import AtomicProperties as AtomicProperties_v2
 from ..v2.atomic import WavefunctionProperties as WavefunctionProperties_v2
-from ..v2.basemodels import ExtendedConfigDict, ProtoModel
+from ..v2.basemodels import ProtoModel
 from ..v2.common_models import DriverEnum, Model, Provenance
 from ..v2.failed_operation import ComputeError
 from ..v2.types import Array
@@ -83,8 +83,6 @@ class AtomicResultProtocols(ProtoModel):
     error_correction: ErrorCorrectionProtocol = Field(default_factory=ErrorCorrectionProtocol)
     native_files: NativeFilesProtocolEnum = Field(NativeFilesProtocolEnum.none)
 
-    model_config = ExtendedConfigDict(force_skip_defaults=True)
-
 
 # ====  Inputs (Kw/Spec/In)  ====================================================
 

qcelemental/models/v2/align.py

@@ -4,7 +4,7 @@
 from pydantic import Field, field_validator
 
 from ...util import blockwise_contract, blockwise_expand
-from .basemodels import ExtendedConfigDict, ProtoModel
+from .basemodels import ProtoModel
 from .types import Array
 
 __all__ = ["AlignmentMill"]
@@ -26,8 +26,6 @@ class AlignmentMill(ProtoModel):
     atommap: Optional[Array[int]] = Field(None, description="Atom exchange map (nat,) for coordinates.")  # type: ignore
     mirror: bool = Field(False, description="Do mirror invert coordinates?")
 
-    model_config = ExtendedConfigDict(force_skip_defaults=True)
-
     @field_validator("shift")
     @classmethod
     def _must_be_3(cls, v):

qcelemental/models/v2/atomic.py

@@ -1,17 +1,20 @@
+from __future__ import annotations
+
 from enum import Enum
 from functools import partial
 from typing import TYPE_CHECKING, Any, Dict, Literal, Optional, Set, Union
 
 import numpy as np
-from pydantic import Field, field_validator
+from pydantic import Field, field_validator, model_serializer
+from pydantic_core.core_schema import SerializerFunctionWrapHandler
 
 from ...models import QCEL_V1V2_SHIM_CODE
 from ...util import provenance_stamp
-from .basemodels import ExtendedConfigDict, ProtoModel, check_convertible_version, qcschema_draft
+from .basemodels import ProtoModel, check_convertible_version, qcschema_draft
 from .basis_set import BasisSet
 from .common_models import DriverEnum, Model, Provenance
 from .molecule import Molecule
-from .types import Array
+from .types import Array, NestedData
 
 if TYPE_CHECKING:
     import qcelemental
@@ -249,10 +252,8 @@ class AtomicProperties(ProtoModel):
         None, description="The number of CCSDTQ iterations taken before convergence."
     )
 
-    model_config = ProtoModel._merge_config_with(force_skip_defaults=True)
-
     def __repr_args__(self) -> "ReprArgs":
-        return [(k, v) for k, v in self.dict().items()]
+        return [(k, v) for k, v in self.model_dump(exclude_unset=True).items()]
 
     @field_validator(
         "scf_dipole_moment",
@@ -300,12 +301,11 @@ def _validate_derivs(cls, v, info):
             raise ValueError(f"Derivative must be castable to shape {shape}!")
         return v
 
-    def dict(self, *args, **kwargs):
-        # pure-json dict repr for QCFractal compliance, see https://github.com/MolSSI/QCFractal/issues/579
-        # Sep 2021: commenting below for now to allow recomposing AtomicResult.properties for qcdb.
-        #   This will break QCFractal tests for now, but future qcf will be ok with it.
-        # kwargs["encoding"] = "json"
-        return super().model_dump(*args, **kwargs)
+    @model_serializer(mode="wrap")
+    def _remove_none(self, handler: SerializerFunctionWrapHandler) -> Dict[str, Any]:
+        # Removes fields with a value of None from the serialized output
+        serialized = handler(self)
+        return {k: v for k, v in serialized.items() if v is not None}
 
     def convert_v(
         self, target_version: int, /
@@ -536,11 +536,6 @@ class WavefunctionProperties(ProtoModel):
         None, description="Index to the beta-spin orbital occupations of the primary return."
     )
 
-    # Note that serializing WfnProp skips unset fields (and indeed the validator will error upon None values)
-    #   while including all fields for the submodel BasisSet. This is the right behavior, imo, but note that
-    #   v1 skips unset fields in BasisSet as well as the top-level model.
-    model_config = ProtoModel._merge_config_with(force_skip_defaults=True)
-
     @field_validator("scf_eigenvalues_a", "scf_eigenvalues_b", "scf_occupations_a", "scf_occupations_b")
     @classmethod
     def _assert1d(cls, v):
@@ -619,6 +614,12 @@ def _assert_exists(cls, v, info):
             raise ValueError(f"Return quantity {v} does not exist in the values.")
         return v
 
+    @model_serializer(mode="wrap")
+    def _remove_none(self, handler: SerializerFunctionWrapHandler) -> Dict[str, Any]:
+        # Removes fields with a value of None from the serialized output
+        serialized = handler(self)
+        return {k: v for k, v in serialized.items() if v is not None}
+
     def convert_v(
         self, target_version: int, /
     ) -> Union["qcelemental.models.v1.WavefunctionProperties", "qcelemental.models.v2.WavefunctionProperties"]:
@@ -628,9 +629,9 @@ def convert_v(
         if check_convertible_version(target_version, error="WavefunctionProperties") == "self":
             return self
 
-        dself = self.model_dump()
+        dself = self.model_dump(exclude_unset=True, exclude_none=True)  # v1 models don't handle None
         if target_version in [1, QCEL_V1V2_SHIM_CODE]:
-            dself["basis"] = self.basis.convert_v(target_version).dict()
+            dself["basis"] = self.basis.convert_v(target_version).model_dump()
 
             if target_version == 1:
                 self_vN = qcel.models.v1.WavefunctionProperties(**dself)
@@ -702,8 +703,6 @@ class AtomicProtocols(ProtoModel):
         description="Policies for keeping processed files from the computation",
     )
 
-    model_config = ExtendedConfigDict(force_skip_defaults=True)
-
     def convert_v(
         self, target_version: int, /
     ) -> Union["qcelemental.models.v1.AtomicResultProtocols", "qcelemental.models.v2.AtomicProtocols"]:
@@ -877,7 +876,7 @@ class AtomicResult(ProtoModel):
     properties: AtomicProperties = Field(..., description=str(AtomicProperties.__doc__))
     wavefunction: Optional[WavefunctionProperties] = Field(None, description=str(WavefunctionProperties.__doc__))
 
-    return_result: Union[float, Array[float], Dict[str, Any]] = Field(
+    return_result: NestedData = Field(
         ...,
         description="The primary return specified by the :attr:`~qcelemental.models.AtomicInput.driver` field. Scalar if energy; array if gradient or hessian; dictionary with property keys if properties.",
     )  # type: ignore
@@ -893,7 +892,7 @@ class AtomicResult(ProtoModel):
         True, description="The success of program execution. If False, other fields may be blank."
     )
     provenance: Provenance = Field(..., description=str(Provenance.__doc__))
-    extras: Dict[str, Any] = Field(
+    extras: NestedData = Field(
         {},
         description="Additional information to bundle with the computation. Use for schema development and scratch space.",
     )

qcelemental/models/v2/basemodels.py

@@ -1,9 +1,8 @@
-import json
 import warnings
 from pathlib import Path
 from typing import Any, Dict, Optional, Set, Union
 
-from pydantic import BaseModel, ConfigDict, model_serializer
+from pydantic import BaseModel, ConfigDict
 
 from qcelemental.models import QCEL_V1V2_SHIM_CODE
 
@@ -14,23 +13,12 @@ def _repr(self) -> str:
     return f'{self.__repr_name__()}({self.__repr_str__(", ")})'
 
 
-class ExtendedConfigDict(ConfigDict, total=False):
-    serialize_skip_defaults: bool
-    """When serializing, ignore default values (i.e. those not set by user)"""
-
-    force_skip_defaults: bool
-    """Manually force defaults to not be included in output dictionary"""
-
-
 class ProtoModel(BaseModel):
     """QCSchema extension of pydantic.BaseModel."""
 
-    model_config = ExtendedConfigDict(
+    model_config = ConfigDict(
         frozen=True,
         extra="forbid",
-        populate_by_name=True,  # Allows using alias to populate
-        serialize_skip_defaults=False,
-        force_skip_defaults=False,
     )
 
     def __init_subclass__(cls, **kwargs) -> None:
@@ -119,56 +107,13 @@ def parse_file(cls, path: Union[str, Path], *, encoding: Optional[str] = None) -
 
         return cls.parse_raw(path.read_bytes(), encoding=encoding)
 
-    # UNCOMMENT IF NEEDED FOR UPGRADE
-    #   defining this is maybe bad idea as dict(v2) does non-recursive dictionary, whereas model_dump does nested
-    # def dict(self, **kwargs) -> Dict[str, Any]:
-    #    warnings.warn("The `dict` method is deprecated; use `model_dump` instead.", DeprecationWarning)
-    #    return self.model_dump(**kwargs)
+    def dict(self, **kwargs) -> Dict[str, Any]:
+        warnings.warn("The `dict` method is deprecated; use `model_dump` instead.", DeprecationWarning, stacklevel=2)
 
-    @model_serializer(mode="wrap")
-    def _serialize_model(self, handler) -> Dict[str, Any]:
-        """
-        Customize the serialization output. Does duplicate with some code in model_dump, but handles the case of nested
-        models and any model config options.
-
-        Encoding is handled at the `model_dump` level and not here as that should happen only after EVERYTHING has been
-        dumped/de-pydantic-ized.
+        if "encoding" in kwargs:
+            kwargs["mode"] = kwargs.pop("encoding")
 
-        DEVELOPER WARNING: If default values for nested ProtoModels are not validated and are also not the expected
-        model (e.g. Provenance fields are dicts by default), then this function will throw an error because the self
-        field becomes the current value, not the model.
-        """
-
-        # Get the default return, let the model_dump handle kwarg
-        default_result = handler(self)
-        force_skip_default = self.model_config["force_skip_defaults"]
-        output_dict = {}
-        # Could handle this with a comprehension, easier this way
-        for key, value in default_result.items():
-            # Skip defaults on config level (skip default must be on and k has to be unset)
-            # Also check against exclusion set on a model_config level
-            if force_skip_default and key not in self.model_fields_set:
-                continue
-            output_dict[key] = value
-        return output_dict
-
-    def model_dump(self, **kwargs) -> Dict[str, Any]:
-        encoding = kwargs.pop("encoding", None)
-
-        # kwargs.setdefault("exclude_unset", self.model_config["serialize_skip_defaults"])  # type: ignore
-        # if self.model_config["force_skip_defaults"]:  # type: ignore
-        #     kwargs["exclude_unset"] = True
-
-        # Model config defaults will be handled in the @model_serializer function
-        # The @model_serializer function will be called AFTER this is called
-        data = super().model_dump(**kwargs)
-
-        if encoding is None:
-            return data
-        elif encoding == "json":
-            return json.loads(serialize(data, encoding="json"))
-        else:
-            raise KeyError(f"Unknown encoding type '{encoding}', valid encoding types: 'json'.")
+        return self.model_dump(**kwargs)
 
     def serialize(
         self,
@@ -222,12 +167,11 @@ def serialize(
     # UNCOMMENT IF NEEDED FOR UPGRADE REDO!!!
     def json(self, **kwargs):
         # Alias JSON here from BaseModel to reflect dict changes
-        warnings.warn("The `json` method is deprecated; use `model_dump_json` instead.", DeprecationWarning)
+        warnings.warn(
+            "The `json` method is deprecated; use `model_dump_json` instead.", DeprecationWarning, stacklevel=2
+        )
         return self.model_dump_json(**kwargs)
 
-    def model_dump_json(self, **kwargs):
-        return self.serialize("json", **kwargs)
-
     def compare(self, other: Union["ProtoModel", BaseModel], **kwargs) -> bool:
         r"""Compares the current object to the provided object recursively.
 
@@ -252,7 +196,7 @@ def _merge_config_with(cls, *args, **kwargs):
         """
         Helper function to merge protomodel's config with other args
 
-        args: other ExtendedConfigDict instances or equivalent dicts
+        args: other ConfigDict instances or equivalent dicts
         kwargs: Keys to add into the dictionary raw
         """
         output_dict = {**cls.model_config}
@@ -261,7 +205,7 @@ def _merge_config_with(cls, *args, **kwargs):
         # Update any specific keywords
         output_dict.update(kwargs)
         # Finally, check against the Extended Config Dict
-        return ExtendedConfigDict(**output_dict)
+        return ConfigDict(**output_dict)
 
 
 def check_convertible_version(ver: int, error: str):