DataNodeConfig class

Bases: Section

Configuration fields needed to instantiate a DataNode.

A Data Node config is made to be used as a generator for actual data nodes. It holds configuration information needed to create an actual data node.

Attributes:

Name	Type	Description
**properties	dict[str, any]	A dictionary of additional properties.

Attributes

id instance-attribute

id: str = _validate_id(id)

A valid python identifier that uniquely identifies the section.

properties property writable

properties

A dictionary of additional properties.

scope property writable

scope: Scope

The Scope of the data nodes instantiated from the data node config.

storage_type property writable

storage_type: str

Storage type of the data nodes created from the data node config.

The possible values are : "csv", "excel", "pickle", "sql_table", "sql", "mongo_collection", "generic", "json", "parquet", "in_memory and "s3_object".

The default value is "pickle".

Note that the "in_memory" value can only be used when JobConfig mode is "development".

validity_period property writable

validity_period: Optional[timedelta]

The validity period of the data nodes instantiated from the data node config.

It corresponds to the duration since the last edit date for which the data node can be considered valid. Once the validity period has passed, the data node is considered stale and relevant tasks that are submitted will run even if they are skippable.

If the validity period is set to None (the default value), the data node is always up-to-date.

Methods

default_config() classmethod

default_config() -> DataNodeConfig

Get a data node configuration with all the default values.

Returns:

Type	Description
DataNodeConfig	The default data node configuration.