You can download the code of this step here or all the steps here.
For Notebooks
The "Getting Started" Notebook is available here.
Step 12: Compare Scenarios¶
Cycles are helpful to keep track of KPI over time. The goal of this step is to compare the primary scenario of every cycle and its pipelines over time.
To achieve this:
-
A new dataframe has to be initialized. It will store the metrics for the baseline and ml pipeline.
-
Then, a part will use a boolean to show or not the comparison.
-
Finally, a selector will change the displayed metrics of the graph.
# Initial dataset for comparison
comparison_scenario = pd.DataFrame(columns=["Scenario Name",
"RMSE baseline",
"MAE baseline",
"RMSE ML",
"MAE ML"])
# Selector for metrics
metric_selector = ["RMSE", "MAE"]
selected_metric = metric_selector[0]
First, a function has to be created to compare the primary scenario of all the cycles.
tp.get_primary_scenarios()
is the useful function to use for this effect. compare()
goes through all of these
scenarios and pipelines and add the metrics in lists. In the end, state.comparison_scenario is updated and
comparison_scenario_done set to True
.
from sklearn.metrics import mean_absolute_error, mean_squared_error
def compute_metrics(historical_data, predicted_data):
rmse = mean_squared_error(historical_data, predicted_data)
mae = mean_absolute_error(historical_data, predicted_data)
return rmse, mae
def compare(state):
print("Comparing...")
# Initial lists for comparison
scenario_names = []
rmses_baseline = []
maes_baseline = []
rmses_ml = []
maes_ml = []
# Go through all the primary scenarios
all_scenarios = tp.get_primary_scenarios()
all_scenarios_ordered = sorted(all_scenarios, key=lambda x: x.creation_date.timestamp())
for scenario in all_scenarios_ordered:
print(f"Scenario {scenario.name}")
# Go through all the pipelines
for pipeline in scenario.pipelines.values():
print(f" Pipeline {pipeline.config_id}")
# Get the predictions dataset with the historical data
only_prediction_dataset = create_predictions_dataset(pipeline)[-pipeline.n_predictions.read():]
# Series to compute the metrics (true values and predicted values)
historical_values = only_prediction_dataset["Historical values"]
predicted_values = only_prediction_dataset["Predicted values"]
# Compute the metrics for this pipeline and primary scenario
rmse, mae = compute_metrics(historical_values, predicted_values)
# Add values to the appropriate lists
if "baseline" in pipeline.config_id:
rmses_baseline.append(rmse)
maes_baseline.append(mae)
elif "ml" in pipeline.config_id:
rmses_ml.append(rmse)
maes_ml.append(mae)
scenario_names.append(scenario.creation_date.strftime("%A %d %b"))
# Update comparison_scenario
state.comparison_scenario = pd.DataFrame({"Scenario Name":scenario_names,
"RMSE baseline":rmses_baseline,
"MAE baseline":maes_baseline,
"RMSE ML":rmses_ml,
"MAE ML":maes_ml})
# When comparison_scenario_done will be set to True,
# the part with the graphs will be finally rendered
state.comparison_scenario_done = True
Let's create a page related to this comparison. As said before, this page will contain a graph to compare scenarios
and pipelines; and a selector to choose the metric on which to compare. When pressed the button at the bottom of the
page calls the compare()
function. When finished, the render property of the part will render the rest of the
page. Also, a new Taipy's block is present in the Markdown:
expandable.
# Performance page
page_performance = """
<|part|render={len(comparison_scenario)>0}|
<|Table|expanded=False|expandable|
<|{comparison_scenario}|table|width=100%|>
|>
<|{selected_metric}|selector|lov={metric_selector}|dropdown|>
<|part|render={selected_metric=="RMSE"}|
<|{comparison_scenario}|chart|type=bar|x=Scenario Name|y[1]=RMSE baseline|y[2]=RMSE ML|>
|>
<|part|render={selected_metric=="MAE"}|
<|{comparison_scenario}|chart|type=bar|x=Scenario Name|y[1]=MAE baseline|y[2]=MAE ML|>
|>
|>
<center><|Compare primarys|button|on_action=compare|></center>
"""
# Add the page_performance section to the menu
lov_menu = [("Data-Visualization", "Data Visualization"),
("Scenario-Manager", "Scenario Manager"),
("Performance", "Performance")]
# Create a menu with our pages
root_md = "<|menu|label=Menu|lov={lov_menu}|on_action=menu_fct|>"
pages = {"/":root_md,
"Data-Visualization":page_data_visualization,
"Scenario-Manager":page_scenario_manager,
"Performance":page_performance}
# Run of the Taipy Core service
tp.Core().run()
selected_scenario = None
scenario_selector = tp.get_scenarios()
Gui(pages=pages).run(dark_mode=False)