Skip to content

Scenarios for task orchestration

In this section, we explore how to benefit from data processing and task orchestration submitting Taipy Scenarios.

What is a Scenario?

Taipy provides the key concept of Scenario. Among other functionalities, a scenario provides the functionalities for modeling, executing and monitoring algorithms. It models an algorithm as an execution graph (a Directed Acyclic Graph or DAG) to solve a data science problem. The execution graph can be seen as a succession of functions (or Task) that exchange data (or DataNode). Scenarios can range from simple to highly complex.

Let's take some examples.

The following picture represents a simple scenario made of a single cleaning task processing a single input data node, the raw data, and returning a single output data node, the cleaned data.

Simple scenario

The second example below is slightly more complex. The first task cleaning processes a single input data node, the raw data, and returns some intermediate data node named cleaned data. The second task filtering reads the same intermediate data node cleaned data and returns a single output data node named filtered data.

Linear scenario

The third example below introduces some complexity. As you can see in the picture below, the task generating does not have any input. On the contrary, the task aggregating takes multiple inputs and returns multiple outputs.

Branching scenario

  • A DataNode (the dark blue boxes) represents a reference to a dataset or a parameter set. For more details, please refer to the data integration section.
    A data node can be shared by multiple tasks as input or output.
  • A Task (the orange boxes) can be seen as a function receiving data node(s) as input and returning data node(s) as output.

Why use Scenarios?

Using Taipy for data processing and task execution offers several distinct advantages that cater to the needs of data scientists and developers aiming to build robust, production-ready applications with ease for their end-users. Here are some key benefits:

  1. Easy to configure: No matter how complex your problem may be, defining execution graphs as scenarios, tasks, and data nodes is simple. For more details on how to configure scenarios, see the scenario configuration page.
    Taipy Studio, an extension for Visual Studio Code, enhances the user experience by providing a graphical editor to build a scenario configuration.
    For more details, see the Taipy Studio page.

  2. Efficient Orchestration and execution: Taipy already implements utility methods to create, manage, submit your scenarios. With support for parallelism, remote execution, horizontal scalability, and skippable tasks, your end-users can run their data processing pipelines smoothly and quickly while minimizing resource costs.
    For more details on how to create, submit and manage scenarios, see the scenario creation, scenario submission, scenario management pages.

  3. Taipy visual elements: Benefit from a comprehensive set of visual elements to empower end users just in one line of code. Manage, display, edit, and submit scenarios in a user-friendly graphical interface.
    For more details, see the visual elements page.

  4. Integration with existing Tools: Integrate seamlessly with any Python library or any external model such as Scikit learn, MLlib, XGBoost, TensorFlow, OpenML, Hugging Face, etc. Any Python function can be used as a Taipy task, and any data source can be used as a data node.

How to use Scenarios?

A Scenario is instantiated from a ScenarioConfig, which encapsulates the execution graph definition. To create and submit scenarios, you need to:

  1. Configure a scenario: Configuring a scenario for Task execution is done using the Config.configure_scenario() method and consists in defining the scenario structure and the execution graph. For that the data nodes and tasks must have been configured using Config.configure_data_node() and Config.configure_task() methods.
    For more details, see the scenario configuration page.

  2. Create a scenario: Instantiating a scenario from its configuration is done programmatically with the create_scenario() function. When a scenario is instantiated, related tasks and data nodes are instantiated from their configurations as well. For more details, see the scenario creation page.
    Typically, this step is done by an end-user using the graphical interface built with Taipy GUI. In particular, the scenario selector natively includes a scenario creation capability.

  3. Submit a scenario for execution: Submitting a scenario for an execution is done programmatically through the submit() method.
    For more details, see the scenario submission page.
    Typically, this step is done by an end-user using the graphical interface built with Taipy GUI. The scenario viewer is designed for this purpose.

Example

Please refer to the hello world page to get a didactic example.

The following code shows a complete example of how to create a scenario configuration and submit it for execution. It consists of creating a dumb function named "do_nothing" and used to configure a scenario. The scenario configuration is then used to instantiate a scenario and submit it for execution.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
import taipy as tp
from taipy import Config


def do_nothing(value):
    print(f"do_nothing but printing the value: {value}")

if __name__ == "__main__":
    # Configure a scenario
    value_cfg = Config.configure_data_node("my_value", default_data=42)
    task_cfg = Config.configure_task("my_task", do_nothing, input=[value_cfg])
    scenario_cfg = Config.configure_scenario("my_scenario", [task_cfg])

    # Run the Orchestrator service
    tp.Orchestrator().run()

    # Create a scenario
    scenario = tp.create_scenario(scenario_cfg)

    # Submit the scenario
    tp.submit(scenario)
Here is the complete python code corresponding to the example.

The following code shows a complete example of how to have a quick user interface for scenario execution. It consists of creating a dumb function named "identity" and used to create a scenario configuration. A user interface is created with three controls allowing to create and execute scenarios.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
from taipy import Config, Orchestrator, Gui


# Implementing a function to be used as a Taipy task in a scenario
def identity(*values):
    return values

if __name__ == "__main__":
    # Creating a scenario variable and a job variable to be bound to the visual elements
    scenario = None
    job = None

    # Configure a scenario
    in_cfg = Config.configure_data_node(id="inpt", default_data="IN")
    out_cfg = Config.configure_data_node(id="outpt")
    task_cfg = Config.configure_task(id="fct", function=identity, input=[in_cfg], output=[out_cfg])
    scenario_cfg = Config.configure_scenario(id="scenario", task_configs=[task_cfg])

    # Run the Orchestrator service
    Orchestrator().run()

    # Run the GUI service with a scenario selector, a scenario viewer, and a job selector
    page = ("""<|{scenario}|scenario_selector|>
<|{scenario}|scenario|>
<|{job}|job_selector|>
""")
    Gui(page=page).run()
Here is the complete python code corresponding to the example.