Fix flw_pairs loading. Prepare all new exps: omp_distillation, preds coherence, preds correlation, normalize_D when OMP, n_jobs=-1 in SOTA. In exps script, test both train+dev,train+dev and train,dev

Handle similarity_similarities and similarity_predictions in the pipeline and set lfw_pairs to binary classif (todo: change the labels for omp)

Fix hyperparams bugs in base and random. Fix extracted forest size used in random. Factorize random fitting

Integrate Paolo's code of method 'Ensemble selection from libraries of models' by Rich Caruana et al

Speedup similarity forest regressor and add parallelization at the extracted forest size level in the training

Fix parallelization and estimator default hyperparams in kmeans and similarity methods. Fix on resume mode in train.py. Fix stage5 saving (tmp) in compute_results.py

Add on resume mode for the experiment training (and set the overwrite of the resulting model of the experiment optional)

Add Paolo's first implementation of this paper: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2822360/

- Add new results for stage4.

- Add stage4 (results and experiments);
- Do not save model object.

- Remove extracted_forest_sizes_number parameter from compute_results.py and retreive the value instead;
- Add almost all remaining experiment config files of stages 1, 2 and 3;
- Add almost all remaining result plots of stages 1, 2 and 3;
- Add some temporary scripts to run all stages experiments.

- Add command lines example for stage 3;
- Add experiment_id option that is useful sometimes;
- Fix subsets_used param;
- Remove experiment_id in config experiment file names;
- Add config experiment files for stages 2 and 3;
- Add results for stages 2 and 3 (california_housing).

- Fix possible issues for extracted forest sizes computation: around to reduce possible zeroes and remove duplicates;
- Create output experiment stage dir if not exists;
- Add base_score_metric to model raw results class;
- Add best params for lfw_pairs (maybe try with a larger number of random seeds since the score is not that high).

- Even if hyperparameters file is ignore with skip_best_hyperparams option, still use the same forest_size to be comparable;
- Update experiment files for stage1 wo_param experiments (using the same forest size as the with_params experiments);
- In compute_results: remove useless folder creation; temporary add extracted_forest_sizes_number option to specify the extracted forest sizes number; temporary not plotting train and dev losses in stage1 loss values figure;
- In plotter, clean-up stage1 figure generation;
- Add first unbiased losses plot (stage1: best params vs default params in california housing dataset).

- Fix some variable names;
- Add exp files of stage1 for california housing

- Reduce the extracted forest sizes upper bound and number because OMP seems to converge only with small forest sizes;
- Add extraction_strategy parameter in order to save base forest and the forests trained with the same size as the extracted forest sizes used in the experiment that used OMP.

POC of possible wrong way to compute best hyperparams. Are there the best only before the application of OMP extraction?

- Definitely use the correct forest size (either the one from best hyperparameters or the one specified in parameter);
- Use a number of extracted forest sizes proportional as the forest size instead of fixed forest size;
- Add an option to save the current command line name instead of using the unamed directory;
- Add new california housing dataset best hyperparameters, and convert all value types that are number from string to int/float in other best hyperparameter files;
- Remove useless code from compute_results.py in prevision of the changes;
- Before best hyperparameters saving, save number as int or float instead of string;
- Add job_number option for parallelisation in both train.py and compute_hyperparameters.py scripts;
- Clean-up TODO list.

- Add new best params for 7 datasets.

- Ignore unamed experiment configuration file backups;
- Factorize default dataset loading parameters;
- Add missing return_X_y in basic dataset loaders.

- Update TODO list.

When training, look if there is bayesian search results, if yes use this. Exception: forest_size use the one given by parser if applicable

Big changes: Create intermediate classes OMPForest and SingleOmpForest for code factoring: share code between OmpForestRegressor and OmpForestBinaryClassifer. Remove set_wweights and set_forest which are not relevant anymore. load function from model_factory isn't trustfull now: raises an error. TODO: multiclass classifier

- Add experiment_configuration parameter to run an experiment from a json configuration file. If the experiment configuration are commnig from the arguments, save it to a file to keep trace of it;
- Add few comments in train.py.

- Use as much CPU as possible when training a random forest regressor.

Replace use_dev_subset by subsets_used parameter, in order to specify more clearly which combination of train dev to used to train the forest and OMP