Updated example 2

config_files/config.yml

@@ -56,9 +56,13 @@ metrics: ["accuracy_score", "f1_score"]
 # The metric that will be used in the hyper-parameter optimization process
 metric_princ: "f1_score"
 # The type of hyper-parameter optimization method
-hps_type: "randomized_search"
-# The number of iteration in the hyper-parameter optimization process
-hps_iter: 2
+hps_type: "Random"
+# The arguments of the hyper-parameter optimization method
+hps_args:
+  # The number of iteration of the optimization process
+  n_iter: 4
+  # If True, for multiview algoriithm, will use n_iter*n_views iterations to optimize.
+  equivalent_draws: True
 
 
 # The following arguments are classifier-specific, and are documented in each

docs/source/tutorials/example2.rst

@@ -2,25 +2,31 @@
 Example 2 : Understanding the hyper-parameter optimization
 ==========================================================
 
-Intuitive explanation on hyper-parameters
+Hyper-parameters intuition
 -----------------------------------------
 
-Hyper-parameters are parameters of a classifier (monoview or multiview) that are task-dependant and have a huge part in the performance of the algorithm for a given task.
+Hyper-parameters are parameters of a classifier (monoview or multiview) that are
+task-dependant and have a huge part in the performance of the algorithm for a given task.
 
-The simplest example is the decision tree. One of it's hyper-parameter is the depth of the tree. The deeper the tree is,
-the most it will fit on the learning data. However a tree too deep will most likely overfit and won't have any relevance on
+The simplest example is the decision tree. One of it's hyper-parameter is the
+depth of the tree. The deeper the tree is,
+the most it will fit on the learning data. However, a tree too deep will most
+likely overfit and won't have any relevance on
 unseen testing data.
 
-This platform proposes a randomized search and a grid search to optimize hyper-parameters. In this example,
-we first will analyze the theory and then how to use it.
+This platform proposes a randomized search and a grid search to optimize
+hyper-parameters. In this example, we first will analyze the theory and
+then how to use it.
 
 
 Understanding train/test split
 ------------------------------
 
-In order to provide robust results, this platform splits the dataset in a training set, that will be used by the
-classifier to optimize their hyper-parameter and learn a relevant model, and a testing set that will take no part in
-the learning process and serve as unseen data to estimate each model's generalization capacity.
+In order to provide robust results, this platform splits the dataset in a
+training set, that will be used by the classifier to optimize their
+hyper-parameter and learn a relevant model, and a testing set that will take
+no part in the learning process and serve as unseen data to estimate each
+model's generalization capacity.
 
 This split ratio is controlled by the config file's argument ``split:``. It uses a float to pass the ratio between the size of the testing set and the training set  :
 :math:`\text{split} = \frac{\text{test size}}{\text{dataset size}}`. In order to be as fair as possible, this split is made by keeping the ratio between each class in the training set and in the testing set.
@@ -213,10 +219,10 @@ TODO COMMENT
 
 **Conclusion**
 
-THe impact of split ratio : dataset related.
+The impact of split ratio : dataset related.
 
-Example 2.2 : Usage of hyper-parameter optimization :
-<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
+Example 2.2 : Usage of randomized hyper-parameter optimization :
+<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
 
 In the previous example, we have seen that the split ratio has an impact on the computational time.
 But the most time-consuming task is optimizing the hyper parameters.
@@ -227,7 +233,8 @@ However, most of the time, they are unknown to the user, and then have to be opt
 In this example, we will use the hyper-parameter optimization method implemented in the platform, to do so we will use three lines of the config file :
 
 - ``hps_type:``, controlling the type of hyper-parameter search,
-- ``hps_iter:``, controlling the number of random draws during the hyper-parameter search,
+- ``n_iter:``, controlling the number of random draws during the hyper-parameter search,
+- ``equivalent_draws``, controlling the number fo draws for multiview algorithms,
 - ``nb_folds:``, controlling the number of folds in the cross-validation process.
 
 So if you run ``example 2.2.1`` with :
@@ -237,16 +244,12 @@ So if you run ``example 2.2.1`` with :
    >>> from multiview_platform.execute import execute
    >>> execute("example2.2.1")
 
-The ``hps_type`` argument is set to ``"randomised_search"``, which is at the moment the only hyper-parameter optimization method implemented in the platform.
-The ``hps_iter`` argument is set to ``5``,
+The ``hps_type`` argument is set to ``"Random"``, which is at the moment the only hyper-parameter optimization method implemented in the platform.
+The ``n_iter`` argument is set to ``5``,
+The ``equivalent_draws`` argument  is set to ``True``,
 The ``nb_folds`` argument is set o ``5``.
 
-**WARNING : The "csv-table" directive's ":file:" and ":url:" options represent a potential security holes. They can be disabled with the "file_insertion_enabled" runtime setting.**
-
-.. csv-table::
-    :file: ./images/result_default_hp_high_train.csv
-
-Here, we used ``split: 0.2`` and the results are far better than with the preset of hyper paramters, as the classifiers are able to fit the task.
+Here, we used ``split: 0.2`` and the results are far better than with the preset of hyper parameters, as the classifiers are able to fit the task.
 
 
 The computing time should be longer than the previous examples. Let's see the pseudo code of the benchmark, while using the hyper-parameter optimization::
@@ -272,32 +275,61 @@ The computing time should be longer than the previous examples. Let's see the ps
         └
         learn on the whole training set
 
-The instructions inside the brackets are the one that the HP optimization adds. So for the monoview algorithms,
+The instructions inside the brackets are the one that the hyper-parameter
+optimization adds. So for the monoview algorithms,
 the computational impact of the HPO is bigger than for the multiview algorithms.
 
-The choice made here is to allow the same amount of draws for each HPO. However, as many of the multiview algorithms
+The choice made here is to allow the same amount of draws for each HPO. However,
+as many of the multiview algorithms
 are more complex and have bigger HP spaces, allowing them more draws, can be a defendable idea.
 
-However, for most of the tasks, using the HPO is a necessity to be able to get the most of each classifier in terms
-of performance.
+However, for most of the tasks, using the HPO is a necessity to be able to
+get the most of each classifier in terms of performance.
 
-The HPO is a matter of tradeoff between precision and computational demand. For most algorithms the more draws you
-allow, the closer to ideal the outputted HP will be, however, many draws mean much longer computational time.
+The HPO is a matter of trade-off between precision and computational demand.
+For most algorithms the more draws you allow, the closer to ideal the outputted
+HP will be, however, many draws mean much longer computational time.
 
-Similarly, the number of folds has a great importance in estimating the performance of a specific Hp combination,
-and the more folds the but more folds take also more time, as one has to train more times and on bigger parts of the
+Similarly, the number of folds has a great importance in estimating the
+performance of a specific HP combination, and the more folds the but more folds
+take also more time, as one has to train more times and on bigger parts of the
 dataset.
 
-The figure below represents the duration of the execution on a personal computer with different fold/draws settings :
+The figure below represents the duration of the execution on a personal computer
+with different fold/draws settings :
 
 .. raw:: html
     :file: ./images/durations.html
 
-The duration is in seconds, and we used 2,5,10,15,20 as values for ``nb_folds`` and 2,5,10,20,30,50,100 for ``hps_iter`` with two monoview classifiers and one multiview classifier on simulated data.
+The duration is in seconds, and we used 2,5,10,15,20 as values for ``nb_folds``
+and 2,5,10,20,30,50,100 for ``n_iter`` with two monoview classifiers and one
+multiview classifier on simulated data.
 
 .. note::
-    In order to compensate the fact that the multiview classifiers have more complex problems to solve, it is possible to use ``"randomized_search-equiv"`` as the HPS optimization method to allow
-    ``hps_iter`` draws for the monoview classifiers and ``hps_iter * nb_view`` draws for the ones that are multiview.
+    The hyper-parameter optimization process generates a report for each
+    classifier, providing each set of parameters and its cross-validation score,
+    to be able to extract the relevant parameters for a future benchmark on the
+    same dataset.
+
+
+Example 2.3 : Usage of grid search :
+<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
 
+In SuMMIT, it is possible to use a grid search if one has several possible
+hyper-parameter values in mind to test.
 
+In order to set up the grid search one has to provide in the ``hps_args:``
+argument the names, parameters and values to test. Let us say we want to try
+several depths for a decision tree, and several ``C`` values for a
+linear `SVM <ttps://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html>`_:
+
+.. code-block:: yaml
+
+    hps_type: "Grid"
+    hps_args:
+      decision_tree:
+        max_depth: [1,2,3,4,5]
+      svm_linear:
+        C: [0.1,0.2,0.3,0.4,0.5]
 
+TODO : a more complex example

docs/source/tutorials/example4.rst

@@ -5,7 +5,7 @@ Taking control : Use your own dataset
 .. role:: python(code)
     :language: python
 
-While developping this platform, the goal has been to bea able to use it relatively easily on different datasets.
+While developping this platform, the goal has been to be able to use it relatively easily on different datasets.
 In order to do so, a fixed input format is used, and we choosed HDF5 as it allows to store a multiview dataset and its metadata in a single file, while being able to load it partially.
 
 The bare necessities

docs/source/tutorials/images/durations.html

@@ -20,7 +20,7 @@
                     Plotly.newPlot(
                         'a34ab35c-7722-424b-b26f-05ce9018d050',
                         [{"contours": {"z": {"highlightcolor": "limegreen", "project": {"z": true}, "show": true, "usecolormap": true}}, "type": "surface", "x": [2, 5, 10, 20, 30, 50, 100], "y": [2, 5, 10, 15, 20], "z": [[12.30792236328125, 14.539308309555054, 25.204104900360107, 40.63256311416626, 57.830570936203, 94.06869697570801, 177.3781054019928], [23.060000896453857, 38.69007134437561, 55.38164758682251, 112.99745631217957, 154.44376754760742, 249.8409869670868, 444.60806703567505], [31.165709257125854, 75.43783736228943, 112.19536924362183, 184.5037076473236, 290.94096207618713, 461.04882979393005, 932.619647026062], [73.37707662582397, 100.57551407814026, 183.70872330665588, 331.44516921043396, 433.04977560043335, 637.6133468151093, 1335.7918057441711], [79.04550528526306, 153.53510761260986, 199.00436758995056, 341.0050745010376, 608.0813648700714, 918.9630422592163, 1716.7227251529694]]}],
-                        {"scene": {"xaxis": {"title": {"text": "x : Number of draws"}}, "yaxis": {"title": {"text": "y : Number of folds"}}, "zaxis": {"title": {"text": "z : Benchmark duration (s)"}}}, "template": {"data": {"bar": [{"error_x": {"color": "#2a3f5f"}, "error_y": {"color": "#2a3f5f"}, "marker": {"line": {"color": "#E5ECF6", "width": 0.5}}, "type": "bar"}], "barpolar": [{"marker": {"line": {"color": "#E5ECF6", "width": 0.5}}, "type": "barpolar"}], "carpet": [{"aaxis": {"endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f"}, "baxis": {"endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f"}, "type": "carpet"}], "choropleth": [{"colorbar": {"outlinewidth": 0, "ticks": ""}, "type": "choropleth"}], "contour": [{"colorbar": {"outlinewidth": 0, "ticks": ""}, "colorscale": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]], "type": "contour"}], "contourcarpet": [{"colorbar": {"outlinewidth": 0, "ticks": ""}, "type": "contourcarpet"}], "heatmap": [{"colorbar": {"outlinewidth": 0, "ticks": ""}, "colorscale": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]], "type": "heatmap"}], "heatmapgl": [{"colorbar": {"outlinewidth": 0, "ticks": ""}, "colorscale": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]], "type": "heatmapgl"}], "histogram": [{"marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "histogram"}], "histogram2d": [{"colorbar": {"outlinewidth": 0, "ticks": ""}, "colorscale": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]], "type": "histogram2d"}], "histogram2dcontour": [{"colorbar": {"outlinewidth": 0, "ticks": ""}, "colorscale": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]], "type": "histogram2dcontour"}], "mesh3d": [{"colorbar": {"outlinewidth": 0, "ticks": ""}, "type": "mesh3d"}], "parcoords": [{"line": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "parcoords"}], "scatter": [{"marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "scatter"}], "scatter3d": [{"line": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "scatter3d"}], "scattercarpet": [{"marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "scattercarpet"}], "scattergeo": [{"marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "scattergeo"}], "scattergl": [{"marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "scattergl"}], "scattermapbox": [{"marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "scattermapbox"}], "scatterpolar": [{"marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "scatterpolar"}], "scatterpolargl": [{"marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "scatterpolargl"}], "scatterternary": [{"marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "scatterternary"}], "surface": [{"colorbar": {"outlinewidth": 0, "ticks": ""}, "colorscale": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]], "type": "surface"}], "table": [{"cells": {"fill": {"color": "#EBF0F8"}, "line": {"color": "white"}}, "header": {"fill": {"color": "#C8D4E3"}, "line": {"color": "white"}}, "type": "table"}]}, "layout": {"annotationdefaults": {"arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1}, "colorscale": {"diverging": [[0, "#8e0152"], [0.1, "#c51b7d"], [0.2, "#de77ae"], [0.3, "#f1b6da"], [0.4, "#fde0ef"], [0.5, "#f7f7f7"], [0.6, "#e6f5d0"], [0.7, "#b8e186"], [0.8, "#7fbc41"], [0.9, "#4d9221"], [1, "#276419"]], "sequential": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]], "sequentialminus": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]]}, "colorway": ["#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52"], "font": {"color": "#2a3f5f"}, "geo": {"bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white"}, "hoverlabel": {"align": "left"}, "hovermode": "closest", "mapbox": {"style": "light"}, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": {"angularaxis": {"gridcolor": "white", "linecolor": "white", "ticks": ""}, "bgcolor": "#E5ECF6", "radialaxis": {"gridcolor": "white", "linecolor": "white", "ticks": ""}}, "scene": {"xaxis": {"backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white"}, "yaxis": {"backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white"}, "zaxis": {"backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white"}}, "shapedefaults": {"line": {"color": "#2a3f5f"}}, "ternary": {"aaxis": {"gridcolor": "white", "linecolor": "white", "ticks": ""}, "baxis": {"gridcolor": "white", "linecolor": "white", "ticks": ""}, "bgcolor": "#E5ECF6", "caxis": {"gridcolor": "white", "linecolor": "white", "ticks": ""}}, "title": {"x": 0.05}, "xaxis": {"automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "zerolinecolor": "white", "zerolinewidth": 2}, "yaxis": {"automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "zerolinecolor": "white", "zerolinewidth": 2}}}, "title": {"text": "Benchmark durations for multiple HPO settings"}},
+                        {"scene": {"xaxis": {"title": {"text": "n_draws"}}, "yaxis": {"title": {"text": "n_folds"}}, "zaxis": {"title": {"text": "Duration (s)"}}}, "template": {"data": {"bar": [{"error_x": {"color": "#2a3f5f"}, "error_y": {"color": "#2a3f5f"}, "marker": {"line": {"color": "#E5ECF6", "width": 0.5}}, "type": "bar"}], "barpolar": [{"marker": {"line": {"color": "#E5ECF6", "width": 0.5}}, "type": "barpolar"}], "carpet": [{"aaxis": {"endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f"}, "baxis": {"endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f"}, "type": "carpet"}], "choropleth": [{"colorbar": {"outlinewidth": 0, "ticks": ""}, "type": "choropleth"}], "contour": [{"colorbar": {"outlinewidth": 0, "ticks": ""}, "colorscale": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]], "type": "contour"}], "contourcarpet": [{"colorbar": {"outlinewidth": 0, "ticks": ""}, "type": "contourcarpet"}], "heatmap": [{"colorbar": {"outlinewidth": 0, "ticks": ""}, "colorscale": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]], "type": "heatmap"}], "heatmapgl": [{"colorbar": {"outlinewidth": 0, "ticks": ""}, "colorscale": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]], "type": "heatmapgl"}], "histogram": [{"marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "histogram"}], "histogram2d": [{"colorbar": {"outlinewidth": 0, "ticks": ""}, "colorscale": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]], "type": "histogram2d"}], "histogram2dcontour": [{"colorbar": {"outlinewidth": 0, "ticks": ""}, "colorscale": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]], "type": "histogram2dcontour"}], "mesh3d": [{"colorbar": {"outlinewidth": 0, "ticks": ""}, "type": "mesh3d"}], "parcoords": [{"line": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "parcoords"}], "scatter": [{"marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "scatter"}], "scatter3d": [{"line": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "scatter3d"}], "scattercarpet": [{"marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "scattercarpet"}], "scattergeo": [{"marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "scattergeo"}], "scattergl": [{"marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "scattergl"}], "scattermapbox": [{"marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "scattermapbox"}], "scatterpolar": [{"marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "scatterpolar"}], "scatterpolargl": [{"marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "scatterpolargl"}], "scatterternary": [{"marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "scatterternary"}], "surface": [{"colorbar": {"outlinewidth": 0, "ticks": ""}, "colorscale": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]], "type": "surface"}], "table": [{"cells": {"fill": {"color": "#EBF0F8"}, "line": {"color": "white"}}, "header": {"fill": {"color": "#C8D4E3"}, "line": {"color": "white"}}, "type": "table"}]}, "layout": {"annotationdefaults": {"arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1}, "colorscale": {"diverging": [[0, "#8e0152"], [0.1, "#c51b7d"], [0.2, "#de77ae"], [0.3, "#f1b6da"], [0.4, "#fde0ef"], [0.5, "#f7f7f7"], [0.6, "#e6f5d0"], [0.7, "#b8e186"], [0.8, "#7fbc41"], [0.9, "#4d9221"], [1, "#276419"]], "sequential": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]], "sequentialminus": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]]}, "colorway": ["#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52"], "font": {"color": "#2a3f5f"}, "geo": {"bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white"}, "hoverlabel": {"align": "left"}, "hovermode": "closest", "mapbox": {"style": "light"}, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": {"angularaxis": {"gridcolor": "white", "linecolor": "white", "ticks": ""}, "bgcolor": "#E5ECF6", "radialaxis": {"gridcolor": "white", "linecolor": "white", "ticks": ""}}, "scene": {"xaxis": {"backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white"}, "yaxis": {"backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white"}, "zaxis": {"backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white"}}, "shapedefaults": {"line": {"color": "#2a3f5f"}}, "ternary": {"aaxis": {"gridcolor": "white", "linecolor": "white", "ticks": ""}, "baxis": {"gridcolor": "white", "linecolor": "white", "ticks": ""}, "bgcolor": "#E5ECF6", "caxis": {"gridcolor": "white", "linecolor": "white", "ticks": ""}}, "title": {"x": 0.05}, "xaxis": {"automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "zerolinecolor": "white", "zerolinewidth": 2}, "yaxis": {"automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "zerolinecolor": "white", "zerolinewidth": 2}}}, "title": {"text": "Benchmark durations for multiple HPO settings"}},
                         {"responsive": true}
                     )
                 };