Stnode Functionality
As seen throughout Using Roman Datamodels, “node” objects are used to actually
handle and store the data for a datamodel. Indeed, if one opens an ASDF file
with a roman datamodel directly with asdf.open
instead of
roman_datamodels.datamodels.open
, the resulting object stored under the
roman
attribute will be a DNode
object rather
than a DataModel
object. Thus the stnode, “node”
objects form the data storage and manipulation backbone of the roman datamodels.
Dynamic Generation
Unfortunately, the stnode objects are not directly written in the roman datamodels codebase. Instead, they are dynamically generated from the schema files defined in the RAD schema package. The reason for this is that the schema files are treated as the “truth” for how the data should be stored rather than any Python code. The dynamically generated object types are a way to facilitate this without massive code duplication. This of course means that issues concerning the stnode objects such as schema violations can be difficult to debug, without an understanding of the schema files and their connection with the stnode objects.
Node Types
There are two main container node types DNode
and
LNode
which correspond to “dictionary” and
“list”-like data structures, respectively. As we will see, all the specific
stnode objects construct from a schema file will be built off these two classes.
Hence, general functionality of an stnode object should be implemented in these
two classes so that all stnode objects can inherit from them.
Currently, these two objects are implemented so that they follow the
dictionary or list interface; meaning that, they can be accessed via the []
operator (node["keyword"]
or node[0]
). However, for the case of the
DNode
objects, keys can also be used to directly
access the data attributes of the object via the Python .
operator
(node.keyword
). This is so that the DNode
objects “look” like they are nice Python derived types.
Warning
Because the DNode
“attributes” are actually like
they are Python dictionary key, using the __getattr__
to enable .
access, things like dir(node)
, IDE autocompletion, and some other Python
introspection tools will not work as expected. In some cases this may result
in spurious warnings about accessing undefined attributes. It also means
that one should be referencing the schema files to understand what
attributes are available for a given stnode object.
This information can be found using the .info()
method. This method will
be a pass through to the asdf.AsdfFile.info
method.
Dynamic Node Construction
A specialized “node” class, that is a node class with a specific name which maps
to a corresponding schema name, will be created and registered by
stnode
when the module is first imported. The schemas which
get this treatment are the “tagged” schemas defined within the datamodels-*
manifest in the RAD package. Any “un-tagged” schemas in RAD will not have a
corresponding stnode object. Instead, the information they contain will be
stored in a DNode
or LNode
object, depending on the schema in question.
Note
The creation of stnode “node” types might occur when a user opens an ASDF file containing Roman data, as ASDF will load stnode as part of its de-serialization process. However, due to how Python imports work this should only happen once.
The specific stnode objects will be subclasses of the
TaggedObjectNode
or
TaggedListNode
classes. These classes are extensions
of the DNode
and LNode
classes which have extensions to handle looking up the schema information for on
the fly data validation. In particular, they will track the tag
information
contained within the manifest from RAD.
These “tagged-nodes” are then turned into specific stnode objects via the
factories in roman_datamodels.stnode._factories
. The way these factories work
is they process the tag
value and strip out the unique name for the schema,
which gets turned into a name for the type that the factory will create.
Note
If special methods are needed for a specific stnode object, then one needs
to add class to roman_datamodels.stnode._mixins
with the appropriate
methods/properties under the name <expected-class-name>Mixin
. The
factories will automatically pick up these mixins and mix them into the
stnode object correctly when it is created.
These factories are looped over and invoked by the
roman_datamodels.stnode._stnode
module which will be imported whenever
roman_datamodels.stnode
is imported which will generate the stnode objects and
register them during that import. Note that this module is imported as part of
the roman_datamodels.datamodels
module.
Scalar Nodes
In addition to the objects described above, there are the “scalar node”
objects, which are created from multiple inheritance of
TaggedScalarNode
and a scalar type. These objects are
used to represent the schemas under the tagged_scalars
directory in RAD.
Those schemas are used to decorate a few common scalar meta
fields with
additional information for the archive and sdp. Due to how the meta
keyword
is assembled (via multiple combiners), ASDF has a hard time traversing the
schemas to look for this information. Thus, these scalar nodes are tagged so that
ASDF has a hook to find them without trying a recursive search of the schema
files. If this issue is resolved in the future, or the metadata under meta
is reorganized, then scalar node concept can be removed from the codebase.
Note
The scalar nodes determine the type they mix together with
TaggedScalarNode
via, the SCALAR_TYPE
constant dictionary defined in roman_datamodels.stnode._factories
. This
dictionary keys off the type
keyword that all schemas have to define. If
a new type needs to be added, then one needs to add a new entry to this
dictionary.
Validation
The stnode objects are designed to attempt to validate the data they contain or
have set in them against the schema information in RAD. This is typically done
on the fly when the data is set via the .
interface; however, ASDF by
default will validate the data stored in the node against its schema during both
serialization and de-serialization.
In order to avoid the overhead of re-validating all of the data in a node when one thing is updated (this can induce a lot of overhead), the stnode objects will attempt to parse a given “tagged-node’s” schema down so that it is only validating the field being updated. It performs the validation by attempting to construct in-memory an ASDF-schema representing just the portion of the schema it needs to validate just that single field against. It then passes that schema into the ASDF validation routines to check the data. Unfortunately, this is not a perfect process nor is it particularly robust. It is possible for a schema to have fields that the parse down process cannot handle, or use JSON-schema constructs which the parser is unaware of. In these cases, validation might raise an error or pass invalid data.
Warning
The only validation process that is guaranteed to validate the data correctly is the full ASDF validation process. This is because ASDF will be using the full schema and be checking everything against it. The on-the-fly validation’s parsing may create unreliable validation scenarios.
Note
In order to avoid the “on-the-fly” validation process, one can set values in
a node/datamodel via the dictionary, []
, interface instead of the .
interface. This is because the []
purposely bypasses the on-the-fly
validation process. Thus in general it is recommend that one uses the .
interface for setting values in a node/datamodel, and only using the []
when one needs to store temporary invalid data in a node/datamodel. The use
[]
runs the risk of placing the node/datamodel in a state where it
cannot be serialized to ASDF.
ASDF
The stnode objects are designed to be serializable to and from ASDF files. As
noted above, the stnode objects wrapped by the
DataModel
are the actual objects which are
serialized to ASDF not the DataModel
object
itself.
roman_datamodels
provides a custom ASDF extension so that ASDF can handle
the stnode objects. This extension does not include the schemas used to build
the stnode objects, as the schemas are already included in extension provided by
the RAD package. The ASDF extension itself is defined in the
roman_datamodels.stnode._converters
module. As part of this module, the
serialization and de-serialization logic is defined in the “converters” for each
of the three “tagged” object base classes. The extension is then integrated into
ASDF by the roman_datamodels.stnode._integration
module, as this module allows
the ASDF extension to be registered with ASDF without having to always import
roman_datamodels
whether or not it is used for a particular case. This is
a recommendation from ASDF so that the extension will have minimal impact on the
general ASDF performance for a given user.