Pipeline configs
A pipeline configuration is necessary to instantiate a Pipeline. To create a
PipelineConfig
, you can use the Config.configure_pipeline()
method with the following parameters:
- id: The id of this new pipeline configuration. This id is mandatory and must be a unique and valid Python identifier.
- tasks: The list of task configurations.
- properties: A dictionary of additional properties.
Reserved keys
Note that we cannot use the word "_entity_owner" as a key in the properties as it has been reserved for internal use.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
|
In the previous code example, in line 16, we create a pipeline configuration with the id "my_pipeline" and made of a
single task configuration task_config
.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
|
In this second code example, in line 21, we create a pipeline configuration with the id "another_pipeline" and made
of the two task configurations created in lines 12 and 16 first_task_cfg
and second_task_cfg
.
Note
Note that the order of the task_config in the list does not matter. The following lines are equivalent.
pipeline_cfg = Config.configure_pipeline("pipeline",
[first_task_cfg, second_task_cfg])
pipeline_cfg = Config.configure_pipeline("pipeline",
[second_task_cfg, first_task_cfg])