Scenarios for task orchestration
In this section, we explore how to benefit from data processing and task orchestration submitting Taipy Scenarios.
What is a Scenario?¶
Taipy provides the key concept of Scenario
. Among other functionalities,
a scenario provides the functionalities for modeling, executing and monitoring algorithms.
It models an algorithm as an execution graph (a Directed Acyclic Graph or DAG) to solve
a data science problem. The execution graph can be seen as a succession of functions
(or Task
) that exchange data (or DataNode
). Scenarios can range from simple to
highly complex.
Let's take some examples.
The following picture represents a simple scenario made of a single cleaning task processing a single input data node, the raw data, and returning a single output data node, the cleaned data.
The second example below is slightly more complex. The first task cleaning processes a single input data node, the raw data, and returns some intermediate data node named cleaned data. The second task filtering reads the same intermediate data node cleaned data and returns a single output data node named filtered data.
The third example below introduces some complexity. As you can see in the picture below, the task generating does not have any input. On the contrary, the task aggregating takes multiple inputs and returns multiple outputs.
- A
DataNode
(the dark blue boxes) represents a reference to a dataset or a parameter set. For more details, please refer to the data integration section.
A data node can be shared by multiple tasks as input or output. - A
Task
(the orange boxes) can be seen as a function receiving data node(s) as input and returning data node(s) as output.
Why use Scenarios?¶
Using Taipy for data processing and task execution offers several distinct advantages that cater to the needs of data scientists and developers aiming to build robust, production-ready applications with ease for their end-users. Here are some key benefits:
-
Easy to configure: No matter how complex your problem may be, defining execution graphs as scenarios, tasks, and data nodes is simple. For more details on how to configure scenarios, see the scenario configuration page.
Taipy Studio, an extension for Visual Studio Code, enhances the user experience by providing a graphical editor to build a scenario configuration.
For more details, see the Taipy Studio page. -
Efficient Orchestration and execution: Taipy already implements utility methods to create, manage, submit your scenarios. With support for parallelism, remote execution, horizontal scalability, and skippable tasks, your end-users can run their data processing pipelines smoothly and quickly while minimizing resource costs.
For more details on how to create, submit and manage scenarios, see the scenario creation, scenario submission, scenario management pages. -
Taipy visual elements: Benefit from a comprehensive set of visual elements to empower end users just in one line of code. Manage, display, edit, and submit scenarios in a user-friendly graphical interface.
For more details, see the visual elements page. -
Integration with existing Tools: Integrate seamlessly with any Python library or any external model such as Scikit learn, MLlib, XGBoost, TensorFlow, OpenML, Hugging Face, etc. Any Python function can be used as a Taipy task, and any data source can be used as a data node.
How to use Scenarios?¶
A Scenario
is instantiated from a ScenarioConfig
, which encapsulates the execution graph
definition. To create and submit scenarios, you need to:
-
Configure a scenario: Configuring a scenario for Task execution is done using the
Config.configure_scenario()
method and consists in defining the scenario structure and the execution graph. For that the data nodes and tasks must have been configured usingConfig.configure_data_node()
andConfig.configure_task()
methods.
For more details, see the scenario configuration page. -
Create a scenario: Instantiating a scenario from its configuration is done programmatically with the
create_scenario()
function. When a scenario is instantiated, related tasks and data nodes are instantiated from their configurations as well. For more details, see the scenario creation page.
Typically, this step is done by an end-user using the graphical interface built with Taipy GUI. In particular, the scenario selector natively includes a scenario creation capability. -
Submit a scenario for execution: Submitting a scenario for an execution is done programmatically through the
submit()
method.
For more details, see the scenario submission page.
Typically, this step is done by an end-user using the graphical interface built with Taipy GUI. The scenario viewer is designed for this purpose.
Example
Please refer to the hello world page to get a didactic example.
The following code shows a complete example of how to create a scenario configuration and submit it for execution. It consists of creating a dumb function named "do_nothing" and used to configure a scenario. The scenario configuration is then used to instantiate a scenario and submit it for execution.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
|
The following code shows a complete example of how to have a quick user interface for scenario execution. It consists of creating a dumb function named "identity" and used to create a scenario configuration. A user interface is created with three controls allowing to create and execute scenarios.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
|