diff --git a/README.md b/README.md
deleted file mode 100644
index 0e30eee30f235ed46fd67c03fdd9f9046fd96e50..0000000000000000000000000000000000000000
--- a/README.md
+++ /dev/null
@@ -1,93 +0,0 @@
-# SEA ICML 2024
-
-
-
-## Getting started
-
-To make it easy for you to get started with GitLab, here's a list of recommended next steps.
-
-Already a pro? Just edit this README.md and make it your own. Want to make it easy? [Use the template at the bottom](#editing-this-readme)!
-
-## Add your files
-
-- [ ] [Create](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#create-a-file) or [upload](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#upload-a-file) files
-- [ ] [Add files using the command line](https://docs.gitlab.com/ee/gitlab-basics/add-file.html#add-a-file-using-the-command-line) or push an existing Git repository with the following command:
-
-```
-cd existing_repo
-git remote add origin https://gitlab.lis-lab.fr/valentin.emiya/sea-icml-2024.git
-git branch -M main
-git push -uf origin main
-```
-
-## Integrate with your tools
-
-- [ ] [Set up project integrations](https://gitlab.lis-lab.fr/valentin.emiya/sea-icml-2024/-/settings/integrations)
-
-## Collaborate with your team
-
-- [ ] [Invite team members and collaborators](https://docs.gitlab.com/ee/user/project/members/)
-- [ ] [Create a new merge request](https://docs.gitlab.com/ee/user/project/merge_requests/creating_merge_requests.html)
-- [ ] [Automatically close issues from merge requests](https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically)
-- [ ] [Enable merge request approvals](https://docs.gitlab.com/ee/user/project/merge_requests/approvals/)
-- [ ] [Set auto-merge](https://docs.gitlab.com/ee/user/project/merge_requests/merge_when_pipeline_succeeds.html)
-
-## Test and Deploy
-
-Use the built-in continuous integration in GitLab.
-
-- [ ] [Get started with GitLab CI/CD](https://docs.gitlab.com/ee/ci/quick_start/index.html)
-- [ ] [Analyze your code for known vulnerabilities with Static Application Security Testing (SAST)](https://docs.gitlab.com/ee/user/application_security/sast/)
-- [ ] [Deploy to Kubernetes, Amazon EC2, or Amazon ECS using Auto Deploy](https://docs.gitlab.com/ee/topics/autodevops/requirements.html)
-- [ ] [Use pull-based deployments for improved Kubernetes management](https://docs.gitlab.com/ee/user/clusters/agent/)
-- [ ] [Set up protected environments](https://docs.gitlab.com/ee/ci/environments/protected_environments.html)
-
-***
-
-# Editing this README
-
-When you're ready to make this README your own, just edit this file and use the handy template below (or feel free to structure it however you want - this is just a starting point!). Thanks to [makeareadme.com](https://www.makeareadme.com/) for this template.
-
-## Suggestions for a good README
-
-Every project is different, so consider which of these sections apply to yours. The sections used in the template are suggestions for most open source projects. Also keep in mind that while a README can be too long and detailed, too long is better than too short. If you think your README is too long, consider utilizing another form of documentation rather than cutting out information.
-
-## Name
-Choose a self-explaining name for your project.
-
-## Description
-Let people know what your project can do specifically. Provide context and add a link to any reference visitors might be unfamiliar with. A list of Features or a Background subsection can also be added here. If there are alternatives to your project, this is a good place to list differentiating factors.
-
-## Badges
-On some READMEs, you may see small images that convey metadata, such as whether or not all the tests are passing for the project. You can use Shields to add some to your README. Many services also have instructions for adding a badge.
-
-## Visuals
-Depending on what you are making, it can be a good idea to include screenshots or even a video (you'll frequently see GIFs rather than actual videos). Tools like ttygif can help, but check out Asciinema for a more sophisticated method.
-
-## Installation
-Within a particular ecosystem, there may be a common way of installing things, such as using Yarn, NuGet, or Homebrew. However, consider the possibility that whoever is reading your README is a novice and would like more guidance. Listing specific steps helps remove ambiguity and gets people to using your project as quickly as possible. If it only runs in a specific context like a particular programming language version or operating system or has dependencies that have to be installed manually, also add a Requirements subsection.
-
-## Usage
-Use examples liberally, and show the expected output if you can. It's helpful to have inline the smallest example of usage that you can demonstrate, while providing links to more sophisticated examples if they are too long to reasonably include in the README.
-
-## Support
-Tell people where they can go to for help. It can be any combination of an issue tracker, a chat room, an email address, etc.
-
-## Roadmap
-If you have ideas for releases in the future, it is a good idea to list them in the README.
-
-## Contributing
-State if you are open to contributions and what your requirements are for accepting them.
-
-For people who want to make changes to your project, it's helpful to have some documentation on how to get started. Perhaps there is a script that they should run or some environment variables that they need to set. Make these steps explicit. These instructions could also be useful to your future self.
-
-You can also document commands to lint the code or run tests. These steps help to ensure high code quality and reduce the likelihood that the changes inadvertently break something. Having instructions for running tests is especially helpful if it requires external setup, such as starting a Selenium server for testing in a browser.
-
-## Authors and acknowledgment
-Show your appreciation to those who have contributed to the project.
-
-## License
-For open source projects, say how it is licensed.
-
-## Project status
-If you have run out of energy or time for your project, put a note at the top of the README saying that development has slowed down or stopped completely. Someone may choose to fork your project or volunteer to step in as a maintainer or owner, allowing your project to keep going. You can also make an explicit request for maintainers.
diff --git a/README.rst b/README.rst
new file mode 100644
index 0000000000000000000000000000000000000000..0940036c533afe85a10ab37642b58e6dcebb95ea
--- /dev/null
+++ b/README.rst
@@ -0,0 +1,243 @@
+Guaranteed sparse support recovery using the Straight-Through-Estimator
+=======================================================================
+
+Support Exploration Algorithm (SEA)
+
+Read our paper at : https://hal.science/hal-03964976/document
+
+Installation
+------------
+
+**All experiments were done with Python 3.8 on Linux.**
+**Compatibility tests were done with Python 3.8 on Windows.**
+**This project is also working with Python 3.9, 3.10 and 3.11 on Linux.**
+There are known issues with Python3.9 on Anaconda for the phase diagram experiment.
+We did not try running the experiments with other configurations.
+No compatibility test was done.
+
+Go to the ``code`` folder and install locally by running ::
+
+ pip install -e .
+
+Minimal working example
+-----------------------
+
+A minimal working example is available in `minimal_example.py`.
+
+Running experiments of the paper
+--------------------------------
+
+All plots are saved in svg file.
+Some plot have a draft version, which is a html file.
+You might want to zoom in or out in order to have a better view of the figure in the draft version of the figures.
+
+Phase diagram experiment
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+This experiment has a really long running time. You need to use a device with a lot of CPU cores to reduce the computation time.
+Known issue: Using Anaconda on Windows, the program can stop and display `Aborted!` in the terminal.
+If this happen, you can run again the command for launching the experiment, it will resume where it stopped.
+
+Noiseless experiment
+""""""""""""""""""""
+
+Go to the ``code/sksea`` folder and run ::
+
+ python run_exp_phase_transition.py -en noiseless -na 500 -cpu 5
+
+This will reproduce the phase diagram experiment with the parameters of the paper:
+
+- -en noiseless: The experiment will be named noiseless.
+- -na 500: Size of x*.
+- -f 256: SEA and IHT based algorithms will run with 256*sparsity iterations.
+- -cpu 10: No more than 10 cpus will be used to make computation (The code is fully parallelized, each column of the phase diagram is computed in parallel. For a point of the phase diagram, each problem is computed in batches in parallel).
+
+You can choose the number of problem solve by a single cpu process by using the ``-bs`` option.
+By default, the value is set to 200 (``-bs 200``).
+If you only want to run a subset of the algorithms, you can use the ``-af`` option.
+By default, the value is set to ``-af OMP -af ELS -af OMPR -af IHT -af IHT-els -af IHT-omp -af HTPFAST -af HTPFAST-els -af HTPFAST-omp -af SEAFAST -af SEAFAST-omp -af SEAFAST-els``.
+For instance, if you want to run only SEA and SEA-els, you can use ``-af SEAFAST -af SEAFAST-els``.
+For speeding up the computation, you can run independently one column of the phase diagram by using the -dv option.
+By default, the value is set to ``-dv 0.025 -dv 0.05 -dv 0.075 -dv 0.1 -dv 0.125 -dv 0.15 -dv 0.175 -dv 0.2 -dv 0.225 -dv 0.25 -dv 0.275 -dv 0.3 -dv 0.4 -dv 0.5 -dv 0.6 -dv 0.7 -dv 0.8 -dv 0.9``.
+So, for instance, if you want to run only the first column of the phase diagram, you can use ``-dv 0.025 -ncp``.
+Adding ``-ncp`` will disable the compilation of the results.
+When everything is computed, you can run ``python run_exp_phase_transition.py -en noiseless -cp`` to compile the results.
+
+More help is available using the ``python run_exp_phase_transition.py -h`` command.
+Raw computation files are located in ``code/sksea/data_results`` folder.
+
+When computations are done, adding ``-pl`` to the precedent command or running ::
+
+ python run_exp_phase_transition.py -en noiseless -na 500 -pl
+
+will plot the results of the experiment in ``code/sksea/figures/icml`` with the parameters of the paper:
+
+- -pl: Plot results instead of making computations.
+
+Thus you will find in ``code/sksea/figures/icml`` folder:
+
+- ``noiseless.svg`` which is the empirical support recovery phase transition diagram with the level curves of success probabilities 0.95 in the noiseless case shown in the paper.
+- ``noiseless_zoom.svg`` which is the empirical support recovery phase transition diagram with the level curves of success probabilities 0.95 in the noiseless case shown in the paper, zoomed in the region shown in the introduction of the paper.
+
+Also, you will find in ``code/sksea/figures/comparison`` folder, ``noiseless_0.0001_hm.html`` which is a draft version of the curve of the paper, with one curve by algorithm.
+You might want to zoom in order to see the curves properly.
+
+Noisy experiment
+""""""""""""""""
+
+Go to the ``code/sksea`` folder and run ::
+
+ python run_exp_phase_transition.py -en noisy -na 500 -cpu 5 -nf 0.01
+
+This will reproduce the phase diagram experiment with the parameters of the paper:
+
+- -en noisy: The experiment will be named noisy.
+- -nf 0.01: :math:`\epsilon` is equal to 0.01 for a noise drawn from a sphere of radius :math:`\epsilon||Ax*||`.
+- other parameters are explained above.
+
+More help is available using the ``python run_exp_phase_transition.py -h`` command.
+All techniques for speeding up the computation in the noiseless experiment are also available here.
+Raw computation files are located in ``code/sksea/data_results`` folder.
+
+When computations are done, adding ``-pl`` to the precedent command or running ::
+
+ python run_exp_phase_transition.py -en noisy -na 500 -pl
+
+will plot the results of the experiment in ``code/sksea/figures/icml`` with the parameters of the paper.
+
+Thus you will find in ``code/sksea/figures/icml`` folder:
+
+- ``noisy.svg`` which is the empirical support recovery phase transition diagram with the level curves of success probabilities 0.95 in the noisy case shown in the paper.
+- ``noisy_zoom.svg`` which is the empirical support recovery phase transition diagram with the level curves of success probabilities 0.95 in the noisy case shown in the paper, zoomed in the region shown in the introduction of the paper.
+
+Also, you will find in ``code/sksea/figures/comparison`` folder, ``noisy_0.0001_hm.html`` which is a draft version of the curve of the paper, with one curve by algorithm.
+You might want to zoom in order to see the curves properly.
+
+Deconvolution experiment
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+Extensive experiment (noiseless)
+""""""""""""""""""""""""""""""
+
+Go to the ``code/sksea`` folder and run ::
+
+ python run_exp_deconv.py -en noiseless
+
+This will reproduce the extensive deconvolution experiment with the parameters of the paper:
+
+- -en noiseless: The experiment will be named noiseless.
+
+As in the phase diagram experiment, you can choose to run only a subset of the algorithms by using the ``-af`` option.
+Here, by default, the value is set to ``-af OMPFAST -af ELSFAST -af OMPRFAST -af IHT -af IHT-els -af IHT-omp -af HTPFAST -af HTPFAST-els -af HTPFAST-omp -af SEAFAST -af SEAFAST-omp -af SEAFAST-els``.
+You can also run each sparsity level using the ``-sp`` option along ``-ncp`` option to disable compilation.
+For instance ``-sp 10 -sp 20 -sp 30 -ncp`` will run the experiment for k=10, k=20 and k=30 without compiling the results.
+Then, you can run ``python run_exp_deconv.py -en noiseless -cp -nr`` to compile the results without running again the experiments.
+More help is available using the ``python run_exp_deconv.py -h`` command.
+Raw computation files are located in ``code/sksea/results/deconv/noiseless`` folder.
+
+When computations are done, adding ``-pl`` to the precedent command or running ::
+
+ python run_exp_deconv.py -en noiseless -pl
+
+will plot the results of the experiment in ``code/sksea/results/deconv/noiseless/icml`` folder.
+You will find:
+
+- ``sup_dist.svg``: The mean of :math:`dist_{sup}` for all sparsity levels and all algorithms.
+- ``sota.svg``: The mean of :math:`dist_{sup}` for a subset of the algorithms to reproduce the figure of the introduction of the paper.
+- ``ws.svg``: The mean of the Wasserstein distance for all sparsity levels and all algorithms.
+- ``f_mse_y.svg``: The mean of :math:`\ell_{2, \text{rel}\_\text{loss}}` for all sparsity levels and all algorithms.
+- ``multi_n_supports_log.svg``: The mean of the number of explored supports for all sparsity levels and all algorithms in a log scale.
+- ``multi_n_supports.svg``: The mean of the number of explored supports for all sparsity levels and all algorithms in a linear scale.
+
+Extensive experiment (noisy)
+""""""""""""""""""""""""""
+
+Go to the ``code/sksea`` folder and run ::
+
+ python run_exp_deconv.py -nf 0.1 -en noisy
+
+This will reproduce the extensive deconvolution experiment with the parameters of the paper:
+
+- -nf 0.1: :math:`\epsilon` is equal to 0.1 for a noise is drawn from a sphere of radius :math:`\epsilon||Ax*||`.
+- -en noisy: The experiment will be named noisy.
+- other parameters are explained above.
+
+Raw computation files are located in ``code/sksea/results/deconv/noisy`` folder.
+When computations are done, adding ``-pl`` to the precedent command or running ::
+
+ python run_exp_deconv.py -en noisy -pl
+
+will plot the results of the experiment in ``code/sksea/results/deconv/noisy/icml`` folder.
+You will find the same files as in the noiseless experiment.
+
+Precise experiment (noiseless)
+""""""""""""""""""""""""""""""
+
+Go to the ``code/sksea`` folder and run ::
+
+ python exp_deconv.py -en noiseless
+
+This will reproduce the deconvolution experiment whe k=20 with the parameters of the paper:
+
+- -en noiseless: The experiment will be named noiseless.
+
+When computations are done, you will find in ``code/sksea/figures/expe_deconv/noiseless/icml`` folder:
+
+-``zoom_signal_light.svg``: The noiseless version of the cropped signal showed in the main part of the article.
+-``zoom_signal_complete.svg``: The noiseless version of the cropped signal showed in appendices.
+- ``full_signal.svg``: The noiseless version of the full signal in appendices.
+- ``iter.svg``: The evolution of the loss through iterations for all algorithms.
+- ``hist.svg``: The evolution of the loss through explored supports for all algorithms.
+
+
+Precise experiment (noisy)
+""""""""""""""""""""""""""""""
+
+Go to the ``code/sksea`` folder and run ::
+
+ python exp_deconv.py -en noisy -nf 0.1
+
+This will reproduce the deconvolution experiment when k=20 with the parameters of the paper:
+
+- -en noisy: The experiment will be named noisy.
+- -nf 0.1: :math:`\epsilon` is equal to 0.1.
+
+When computations are done, you will find in ``code/sksea/figures/expe_deconv/noisy/icml`` folder the same files as above
+
+Machine learning experiment
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Go to the ``code/sksea`` folder and run ::
+
+ python training_tasks.py
+
+This will reproduce the phase diagram experiment with the parameters of the paper
+
+By default, all datasets will used.
+For instance, if you want to work on a specific dataset
+(cal_housing, comp-activ-harder, slice, year, letter or ijcnn1),
+you can use ``-d cal_housing -d letter`` to only work with cal_housing and letter.
+You can also use ``-af`` to only run a subset of the algorithms.
+Raw computations file are located in ``code/sksea/results/training_tasks``.
+
+**IMPORTANT** The code download automatically the dataset used in `Sparse Convex Optimization via Adaptively Regularized Hard Thresholding`
+by Axiotis, Kyriakos and Sviridenko, Maxim from Google Drive.
+This download is needed only once.
+For some reason, the download can fail and the code throws the following error:
+
+FileNotFoundError: The dataset cannot be download automatically from Google Drive.
+Please download it manualy from https://drive.google.com/file/d/1RDu2d46qGLI77AzliBQleSsB5WwF83TF/view,
+and place it in code/sksea/downloads/datasets.zip.
+Then, launch this command again.
+
+In that case, please download manually the dataset from https://drive.google.com/file/d/1RDu2d46qGLI77AzliBQleSsB5WwF83TF/view
+and place it in `code/sksea/downloads/datasets.zip`.
+
+When computations are done, adding ``-pl`` to the precedent command or running ::
+
+ python training_tasks.py -pl
+
+will plot the results of the experiment in ``code/sksea/results/training_tasks_plot_icml``.
+Here again, you choose the dataset you want the result from by adding ``-d``.
+
+Running ``python training_tasks.py -pd`` will plot a draft version of the same figures in ``code/sksea/results/training_tasks_plot``.
diff --git a/code/setup.cfg b/code/setup.cfg
new file mode 100755
index 0000000000000000000000000000000000000000..16068d97197da95af3f7106f1bd3ba9998e9480f
--- /dev/null
+++ b/code/setup.cfg
@@ -0,0 +1,30 @@
+[tool:pytest]
+testpaths = sksea
+addopts = --verbose
+ --cov-report=term-missing
+ --cov-report=html
+ --cov=sksea
+ --doctest-modules
+
+[coverage:run]
+branch = True
+source = sksea
+include = */sksea/*
+omit = */tests/*
+
+[coverage:report]
+exclude_lines =
+ pragma: no cover
+ if self.debug:
+ if settings.DEBUG
+ raise AssertionError
+ raise NotImplementedError
+ if 0:
+ if __name__ == .__main__.:
+ if obj is None: return
+ if verbose > 0:
+ if self.verbose > 0:
+ if verbose > 1:
+ if self.verbose > 1:
+ pass
+ def __str__(self):
diff --git a/code/setup.py b/code/setup.py
new file mode 100644
index 0000000000000000000000000000000000000000..db9e4e784bd0f90bd754a6720538c065a6f26ae7
--- /dev/null
+++ b/code/setup.py
@@ -0,0 +1,63 @@
+# -*- coding: utf-8 -*-
+from setuptools import setup
+from sys import version_info
+
+if version_info.minor == 11:
+ setup(name='sksea',
+ version='0.1',
+ description='Support exploration algorithms',
+ license='GPL',
+ packages=['sksea'],
+ zip_safe=False,
+ python_requires='~=3.11',
+ extras_require={
+ 'dev': ['coverage', 'pytest', 'pytest-cov'], },
+ install_requires=[
+ "click==8.*,>=8.1.7",
+ "gdown==5.*,>=5.0.1",
+ "hyperopt==0.*,>=0.2.7",
+ "loguru==0.*,>=0.7.2",
+ "mat4py==0.*,>=0.6.0",
+ "matplotlib==3.*,>=3.8.2",
+ "numpy==1.*,>=1.26.3",
+ "pandas==2.*,>=2.2.0",
+ "plotly==5.*,>=5.18.0",
+ "pyarrow==15.*,>=15.0.0",
+ "pysindy==1.*,>=1.7.5",
+ "ray==2.*,>=2.9.1",
+ "requests==2.*,>=2.31.0",
+ "scikit-learn==1.*,>=1.4.0",
+ "scipy==1.*,>=1.12.0,<1.14.0",
+ "tabulate==0.*,>=0.9.0",
+ "tqdm==4.*,>=4.66.1",
+ ],
+ )
+else:
+ setup(name='sksea',
+ version='0.1',
+ description='Support exploration algorithms',
+ license='GPL',
+ packages=['sksea'],
+ zip_safe=False,
+ python_requires='>=3.8,<3.11',
+ extras_require={
+ 'dev': ['coverage', 'pytest', 'pytest-cov'], },
+ install_requires=[
+ 'numpy~=1.24.1',
+ 'scipy~=1.10.0',
+ 'matplotlib~=3.6.3',
+ 'scikit-learn~=1.2.0',
+ 'mat4py~=0.5',
+ 'click~=8.1.3',
+ 'ray~=2.2.0',
+ 'tqdm~=4.64.1',
+ 'loguru~=0.6.0',
+ 'plotly~=5.11.0',
+ 'gdown>=4.6.0',
+ 'pandas~=1.5.2',
+ 'tabulate~=0.9.0',
+ 'pysindy~=1.7.5',
+ 'requests~=2.27.1',
+ 'hyperopt~=0.2.7'
+ ],
+ )
diff --git a/code/sksea/__init__.py b/code/sksea/__init__.py
new file mode 100644
index 0000000000000000000000000000000000000000..40a96afc6ff09d58a702b76e3f7dd412fe975e26
--- /dev/null
+++ b/code/sksea/__init__.py
@@ -0,0 +1 @@
+# -*- coding: utf-8 -*-
diff --git a/code/sksea/algorithms.py b/code/sksea/algorithms.py
new file mode 100644
index 0000000000000000000000000000000000000000..a7270777b989eef830c07eda0b683d76bc1db99a
--- /dev/null
+++ b/code/sksea/algorithms.py
@@ -0,0 +1,1466 @@
+"""
+All algorithms used to solve problems.
+Part of the signature is common for all of them.
+ista and amp are depreciated.
+"""
+import math
+# -*- coding: utf-8 -*-
+from collections import defaultdict
+from itertools import combinations
+from pathlib import Path
+from typing import List, Tuple, Union, Optional
+import functools
+
+import numpy as np
+import pandas as pd
+import plotly.graph_objects as go
+from loguru import logger
+from scipy.sparse.linalg import cg
+from scipy.optimize import fmin_cg
+from sklearn.base import RegressorMixin
+from sklearn.linear_model._base import LinearModel
+from sklearn.utils.validation import check_X_y, check_random_state
+from tabulate import tabulate
+
+from sksea.sparse_coding import SparseSupportOperator, MatrixOperator
+from sksea.utils import find_support, soft_thresholding, soft_thresholding_der, hard_thresholding
+
+# gradient step size for SEA, need to be tested again before use
+PAS = {
+ 'Lstep': None,
+ '1': lambda x_s: 1,
+ 'mean': lambda x_s: np.linalg.norm(x_s, 1) / x_s.shape[0],
+ 'min': lambda x_s: np.min(x_s),
+ 'max': lambda x_s: np.max(x_s),
+ 'harm': lambda x_s: x_s.shape[0] * np.linalg.norm(x_s, -1),
+}
+
+
+def normalizer(func):
+ """
+ Decorator allowing usage of normalized operator in algorithms.
+ """
+
+ @functools.wraps(func)
+ def wrapper(linop, *args, normalize=True, **kwargs):
+ """
+ Function allowing linear operator normalization
+
+ :param (sksea.utils.AbstractLinearOperator) linop: Linear operator to normalize
+ :param (bool) normalize: If True, normalize the algorithm before running the algorithm
+ :return: The output of the original function, un-normalized if needed
+ """
+ if normalize:
+ normalized_linop, w_diag = linop.get_normalized_operator() # Normalize matrix before execution
+ # For handling both output of SEA/SEA_BEST
+ if kwargs.get('return_both', False):
+ if func.__name__ == 'sea_fast': # For handling the support history of SEA_FAST
+ (x_w_1, *other_out_1), (x_w_2, *other_out_2), *other_out = func(normalized_linop, *args, **kwargs)
+ else:
+ (x_w_1, *other_out_1), (x_w_2, *other_out_2) = func(normalized_linop, *args, **kwargs)
+
+ x_1 = x_w_1 * w_diag # Un-normalize the output of the algorithm
+ x_2 = x_w_2 * w_diag # Un-normalize the output of the algorithm
+ if func.__name__ == 'sea_fast': # For handling the output of sea with support history
+ return (x_1, *other_out_1), (x_2, *other_out_2), *other_out # noqa
+ else:
+ return (x_1, *other_out_1), (x_2, *other_out_2)
+
+ else:
+ x_w, *other_out = func(normalized_linop, *args, **kwargs)
+ x = x_w * w_diag # Un-normalize the output of the algorithm
+ return (x, *other_out) # noqa
+ else:
+ return func(linop, *args, **kwargs)
+
+ return wrapper
+
+
+def ista(linop, y, alpha, n_iter, rel_tol=-np.inf) -> Tuple[np.ndarray, List[float]]:
+ """
+ DEPRECIATED: Need to update its signature to match omp's in order to be used in experiments
+ Solve min_x 1/2 ||D * x - y||_2^2 + \alpha ||x||_1
+
+ :param (sksea.utils.AbstractLinearOperator) linop: Linear operator representing the D matrix
+ :param (np.ndarray) y: Target vector
+ :param (float) alpha: Regularisation coefficient for the l1-norm
+ :param (int) n_iter: Number of iteration of the algorithm
+ :param (float) rel_tol: The algorithm stops when the iterations relative difference is lower than rel_tol
+ :return: The solution vector `x`, the sequence of residuals `res_norm`
+ """
+ x_len = linop.shape[1]
+ x = np.zeros(x_len)
+ lip = linop.compute_lipschitz()
+ pas = 2 * 0.9 / lip
+ res_norm = []
+ res = y - linop @ x
+ res_norm.append(np.linalg.norm(res))
+
+ for it in range(n_iter):
+ g = linop.H @ -res # gradient
+ x -= pas * g # gradient step
+ x = soft_thresholding(x, alpha / lip) # projection
+ res = y - linop @ x
+ res_norm.append(np.linalg.norm(res))
+ if (res_norm[-2] - res_norm[-1]) / res_norm[-2] < rel_tol:
+ break
+
+ return x, res_norm
+
+
+@normalizer
+def iht(linop, y, n_nonzero, n_iter, rel_tol=-np.inf, f=None, grad_f=None, is_mse=False, algo_init=None, optimizer=None,
+ lip_fact=2 * 0.9
+ ) -> Tuple[np.ndarray, List[float]]:
+ """
+ Use IHT algorithm for solving: min_x f(x) w.r.t ||X||_0 <= n_nonzero
+
+ :param (sksea.utils.AbstractLinearOperator) linop: Linear operator representing the D matrix
+ :param (np.ndarray) y: Not used, left for signature compatibility of old experiments
+ :param (int) n_nonzero: Size of the wanted support
+ :param (int) n_iter: Number of iteration of the algorithm
+ :param (float) rel_tol: The algorithm stops when the iterations relative difference is lower than rel_tol
+ :param (Callable[[np.ndarray, Optional[np.ndarray]], float]) f: Loss to minimize.
+ The first argument is the vector to use for the evaluation. The second is the support of the evaluation.
+ :param (Callable[[np.ndarray, Optional[np.ndarray]], np.ndarray]) grad_f: Gradient of the loss to minimize.
+ The first argument is the vector to use for the evaluation. The second is the support of the evaluation.
+ :param (bool) is_mse: If True, use better optimization algorithms (linear conjugate gradient)
+ for solving min_x 1/2 ||D * x - y||_2^2 w.r.t ||X||_0 <= n_nonzero instead of using non-linear algorithms.
+ Only used by algo_init. IHT don't need inner optimization
+ :param (Callable or None) algo_init: Function to use for IHT initialization. If None, initialize IHT with 0
+ :param optimizer: For signature compatibility
+ :return: The solution vector `x`, the sequence of residuals `res_norm`
+ """
+ # Initializations
+ if algo_init is not None:
+ x, res_norm = algo_init(linop, y, n_nonzero, n_iter, rel_tol=-np.inf, normalize=False,
+ f=f, grad_f=grad_f, is_mse=is_mse)
+ else: # 0
+ x_len = linop.shape[1]
+ x = np.zeros(x_len)
+ lip = linop.compute_lipschitz()
+ pas = lip_fact / lip
+ res_norm = [f(x, linop)]
+ last_x = np.copy(x)
+
+ for _ in range(n_iter):
+ x -= pas * grad_f(x, linop) # gradient step
+ x = hard_thresholding(x, n_nonzero) # projection
+ res_norm.append(f(x, linop))
+ if (res_norm[-2] - res_norm[-1]) / res_norm[-2] < rel_tol or np.isclose(last_x, x).all():
+ break
+ np.copyto(last_x, x)
+
+ return x, res_norm
+
+
+@normalizer
+def amp(linop, y, alpha, n_iter, rel_tol=-np.inf, n_nonzero=None, return_both=False
+ ) -> Union[
+ Tuple[np.ndarray, List[float]],
+ Tuple[Tuple[np.ndarray, List[float]], Tuple[np.ndarray, List[float]]]]:
+ """
+ DEPRECIATED: Need to update its signature to match omp's in order to be used in experiments
+ Solve min_x 1/2 ||D * x - y||_2^2 + \alpha ||x||_1
+
+ :param (sksea.utils.AbstractLinearOperator) linop: Linear operator representing the D matrix
+ :param (np.ndarray) y: Target vector
+ :param (float) alpha: Regularisation coefficient for the l1-norm
+ :param (int) n_iter: Number of iteration of the algorithm
+ :param (float) rel_tol: The algorithm stops when the iterations relative difference is lower than rel_tol
+ :param (int or None) n_nonzero: Size of the wanted support. If None, return the non-sparse full solution
+ :param (bool) return_both: If True, return the non-sparse full solution and the sparse one.
+
+ :return: The solution vector `x`, the sequence of residuals `res_norm`
+ """
+ if return_both and n_nonzero is None:
+ raise ValueError("If return_both is True, n_nonzero should not be None")
+
+ x_len = linop.shape[1]
+ y_len = linop.shape[0]
+ x = np.zeros(x_len) # x^t
+ z = np.zeros(y_len) # z^t
+ delta = y_len / x_len
+ res_norm = []
+ res = y - linop @ x
+ res_norm.append(np.linalg.norm(res))
+
+ for it in range(n_iter):
+ thres = alpha * np.linalg.norm(z) / np.sqrt(y_len)
+ xt = linop.H @ z + x
+ x = soft_thresholding(xt, thres)
+ res = y - linop @ x
+ z = res + 1 / delta * z * np.count_nonzero(soft_thresholding_der(xt, thres)) / x_len
+ res_norm.append(np.linalg.norm(res))
+ if it > 1 and (res_norm[-2] - res_norm[-1]) / res_norm[-2] < rel_tol:
+ break
+
+ x_ht = hard_thresholding(x, n_nonzero)
+ res_norm_ht = res_norm + [np.linalg.norm(y - linop @ x_ht)]
+ if return_both:
+ return (x, res_norm), (x_ht, res_norm_ht)
+ elif n_nonzero is not None:
+ return x_ht, res_norm_ht
+ else:
+ return x, res_norm
+
+
+@normalizer
+def sea(linop, y, n_nonzero, n_iter, return_best=False, keep_nonzero_x=True, rel_tol=-np.inf,
+ algo_init=None, return_both=False, optimize_sea=None, f=None, grad_f=None, is_mse=True, pas=None,
+ supp_hist=False, optimizer='cg', full_explo=False
+ ) -> Union[
+ Tuple[np.ndarray, List[float]],
+ Tuple[Tuple[np.ndarray, List[float]], Tuple[np.ndarray, List[float]]],
+ Tuple[Tuple[np.ndarray, List[float]], Tuple[np.ndarray, List[float]], np.ndarray]]:
+ """
+ Use OMP algorithm for solving: min_x f(x) w.r.t ||X||_0 <= n_nonzero
+
+ :param (sksea.utils.AbstractLinearOperator) linop: Linear operator representing the D matrix
+ :param (np.ndarray) y: Target vector
+ :param (int) n_nonzero: Size of the wanted support
+ :param (int) n_iter: Number of iteration of SEA
+ :param (bool) return_best: If True, return SEA_BEST
+ :param (bool) keep_nonzero_x: If True, keep all coefficients in the support exploration variable
+ :param (float) rel_tol: The algorithm stops when the iterations relative difference is lower than rel_tol
+ :param (Callable or None) algo_init: Function to use for sea initialization. If None, initialize SEA with 0
+ :param (bool) return_both: If True, return SEA and SEA_BEST
+ :param (str or None) optimize_sea: If specified, run a full optimisation scheme for `all` iterations of SEA
+ or only for the `last` iteration (the last is the best if `return_best` is True)
+ :param (Callable[[np.ndarray, Optional[np.ndarray]], float]) f: Loss to minimize.
+ The first argument is the vector to use for the evaluation. The second is the support of the evaluation.
+ :param (Callable[[np.ndarray, Optional[np.ndarray]], np.ndarray]) grad_f: Gradient of the loss to minimize.
+ The first argument is the vector to use for the evaluation. The second is the support of the evaluation.
+ :param (bool) is_mse: If True, use better optimization algorithms (linear conjugate gradient)
+ for solving min_x 1/2 ||D * x - y||_2^2 w.r.t ||X||_0 <= n_nonzero instead of using non-linear algorithms
+ :param (str or None) pas: Gradient step size to use on the exploratory variable
+ :param (bool) supp_hist: If True, return sea, sea best and support history
+ :return: The solution vector `x`,
+ the sequence of residuals `res_norm` which includes the sequence of residuals of algo_init
+ """
+ x_len = linop.shape[1]
+ optimize_sea = optimize_sea.lower() if optimize_sea is not None else None
+ # Initializations
+ if algo_init is not None:
+ x_bar, res_norm = algo_init(linop, y, n_nonzero, n_iter, rel_tol=-np.inf, normalize=False,
+ f=f, grad_f=grad_f, is_mse=is_mse)
+ x = np.copy(x_bar)
+ else: # 0
+ x_bar = np.zeros(x_len)
+ x = np.zeros(x_len)
+ res_norm = [f(x, linop)]
+ offset = len(res_norm) # n_iter of initialization
+ lip = linop.compute_lipschitz()
+
+ # Step size selection
+ pas_func = PAS.get(pas)
+ if pas_func is None or algo_init is None:
+ pas = 2 * 0.9 / lip
+ else:
+ pas = pas_func(np.abs(x[x.nonzero()]))
+
+ if supp_hist: # Save support for all iterations
+ support_history = np.ones(n_iter - 1 + offset, dtype=int) * -1
+
+ # Iterative scheme
+ best_res_norm = res_norm[-1]
+ best_it = 0
+ best_x = np.copy(x)
+ best_s = np.zeros_like(x, dtype=bool)
+ last_s = np.zeros_like(x, dtype=bool)
+ for it in range(n_iter):
+ s = find_support(x_bar, n_nonzero)
+ if supp_hist and it != 0:
+ support_history[it - 1 + offset] = (last_s != s).sum() # noqa
+ if optimize_sea == 'all' and (last_s != s).any(): # Cg optimisation only on support change # noqa
+ x = optimize(linop, x, y, s=s, f=f, grad_f=grad_f, is_mse=is_mse, optimizer=optimizer)
+ last_s = s
+ g = grad_f(x * s, linop) # Gradient en x*s
+ if optimize_sea != 'all': # Simple gradient step on x if not doing entire optimization
+ if keep_nonzero_x:
+ x = x - pas * g * s
+ else:
+ x = (x - pas * g) * s
+
+ # Update exploration variable
+ if full_explo:
+ # Reduce the size of the exploration variable
+ min_x_bar = np.min(np.abs(x_bar))
+ x_bar[x_bar > 0] -= min_x_bar
+ x_bar[x_bar < 0] += min_x_bar
+ if n_nonzero != x_len: # If n_nonzero == x_len, there is no support change
+ while np.all(find_support(x_bar, n_nonzero) == s):
+ x_bar -= pas * g
+ else:
+ x_bar -= pas * g # Update exploration variable
+ res_norm.append(f(x * s, linop))
+ if res_norm[-1] < best_res_norm: # Keep best iteration
+ best_res_norm = res_norm[-1]
+ best_x = x * s
+ np.copyto(best_s, s)
+ best_it = it
+ if np.abs(res_norm[-2] - res_norm[-1]) / res_norm[-2] < rel_tol:
+ break
+
+ # Last optimization (if needed)
+ best_res_list = res_norm[:offset + best_it + 1]
+ if optimize_sea == 'last':
+ x = optimize(linop, x, y, s=s) # noqa
+ res_norm.append(f(x, linop))
+ best_x = optimize(linop, best_x, y, s=best_s, f=f, grad_f=grad_f, is_mse=is_mse)
+ best_res_list.append(f(best_x, linop))
+
+ # Returning results
+ sea_results = (x * s, res_norm)
+ sea_best_results = (best_x, best_res_list)
+ if supp_hist:
+ return sea_results, sea_best_results, support_history
+ elif return_both:
+ return sea_results, sea_best_results
+ elif return_best:
+ return sea_best_results
+ else:
+ return sea_results
+
+
+class ExplorationHistory:
+ """
+ Exploration history for SEA-like algorithms
+ """
+
+ def __init__(self):
+ self.x = dict()
+ self.grad = dict()
+ self.loss = dict()
+ self.it = defaultdict(lambda: [])
+ self.n_stable = defaultdict(lambda: [])
+ self.change_size = []
+ self.best_it = None
+ self.best_loss = np.inf
+ self.last_supp = None
+ self.is_open = True
+ self.old_it = []
+ self.old_n_stable = []
+ self.old_change_size = []
+
+ # def get_info(self, s):
+ # """
+ # Return all the information about a support
+ #
+ # :param (np.ndarray) s: Support
+ # :return: Sparse iterate, gradient evaluated at the sparse iterate, loss value, iterations or all histories
+ # """
+ # buffer = s.tobytes()
+ # out = []
+ # if buffer in self.it.keys():
+ # out.append((self.x[buffer], self.grad[buffer], self.loss[buffer], self.it[buffer]))
+ # for old_it in self.old_it:
+ # if buffer in old_it.keys():
+ # out.append((old_it[buffer], self.grad[buffer], self.loss[buffer], self.it[buffer]))
+ # return out
+
+ def get_last_support(self) -> np.ndarray:
+ """
+ Return the last support visited by the algorithm
+ """
+ return np.frombuffer(self.last_supp, dtype=bool).copy()
+
+ def _count_n_iter_stable(self, it):
+ """
+ Count and store the number of iterations spent in the last support.
+ /!\ USE ONLY ON SUPPORT CHANGE
+
+ :param (int) it: Iteration of the last support change
+ """
+ if self.last_supp is not None:
+ self.n_stable[self.last_supp].append(it - self.it[self.last_supp][-1])
+
+ def add(self, s, x, grad, loss, it, copy_x=True, copy_grad=True):
+ """
+ Add a support and its information to the history
+
+ :param (np.ndarray) s: Support
+ :param (np.ndarray) x: Sparse iterate
+ :param (np.ndarray or None) grad: Gradient evaluated at the sparse iterate
+ :param (int) loss: Loss value
+ :param (int) it: Current iteration
+ """
+ buffer = s.tobytes()
+
+ self.x[buffer] = x.copy() if copy_x else x
+ self.grad[buffer] = grad.copy() if copy_grad else grad
+ self.it[buffer].append(it)
+ self.loss[buffer] = loss
+
+ if self.last_supp is not None:
+ if self.last_supp != buffer:
+ self._count_n_iter_stable(it)
+ self.change_size.append(
+ np.sum(np.abs(np.frombuffer(buffer, dtype=bool) ^ np.frombuffer(self.last_supp, dtype=bool))))
+
+ self.last_supp = buffer
+
+ if loss < self.best_loss:
+ self.best_loss = loss
+ self.best_it = it
+
+ def get(self, s, copy_x=True, copy_grad=True, it=None) -> Optional[Tuple[np.ndarray, np.ndarray, int]]:
+ """
+ Get information from an already seen support
+
+ :param (np.ndarray) s: Current support
+ :return: Sparse iterate, gradient evaluated at the sparse iterate, loss value
+ """
+ buffer = s.tobytes()
+ if buffer in self.x.keys():
+ x = self.x[buffer].copy() if copy_x else self.x[buffer]
+ grad = self.grad[buffer].copy() if copy_grad and self.grad[buffer] is not None else self.grad[buffer]
+ if it is not None:
+ self.it[buffer].append(it)
+ return x, grad, self.loss[buffer]
+ else:
+ return None
+
+ def close_exploration(self, last_it):
+ """
+ Save best iteration and loss. Transform defaultDict into dict. To continue exploration, use relaunch_exploration
+
+ :param (int) last_it: Last iteration of the algorithm
+ """
+ # Count the number of iteration spent in the last support
+ self._count_n_iter_stable(last_it)
+
+ # Transform defaultDict into dict
+ self.it = dict(self.it)
+ self.n_stable = dict(self.n_stable)
+
+ self.is_open = False
+
+ def get_supports(self) -> List[np.ndarray]:
+ """
+ Return a list of all the supports visited by the algorithm
+ """
+ return [np.frombuffer(buffer, bool) for buffer in self.loss.keys()]
+
+ def get_n_supports(self, best=None) -> int:
+ """
+ Return the number of supports visited by the algorithm
+ """
+ if best and self.best_it is None:
+ raise ValueError("No best support found")
+ elif best or (best is None and self.best_it is not None):
+ n_supports = 0
+ for buffer, iterations in self.it.items():
+ if iterations[0] <= self.best_it:
+ n_supports += 1
+ return n_supports
+ else:
+ return len(self.it.keys())
+
+ def get_top(self, save_folder=None) -> Tuple[pd.DataFrame, list]:
+ """
+ Create a ranking with the top support
+
+ :param (Path) save_folder: Folder path for visualization
+ :return: Support ranking and size of support change
+ """
+ ranking = pd.DataFrame([
+ [idx + 1, loss, len(self.it[buff_supp]), self.it[buff_supp][-1], self.n_stable[buff_supp][-1]]
+ for idx, (buff_supp, loss) in enumerate(sorted(self.loss.items(), key=lambda item: item[1])) if buff_supp in self.it.keys()
+ ], columns=["rank", "loss", "n_visits", "last_visit", "n_iter"])
+ if save_folder is not None:
+ save_folder.mkdir(parents=True, exist_ok=True) # noqa
+ with open(save_folder / 'latex.txt', 'w') as f:
+ f.write(tabulate(ranking, headers=ranking.columns, showindex=False, tablefmt="latex"))
+ ranking.to_csv(save_folder / "ranking.csv")
+ fig = go.Figure()
+ fig.add_trace(go.Histogram(x=np.array(self.change_size), xbins=dict(start=0.75, end=12.25, size=0.5),
+ autobinx=False))
+ fig.update_layout( # bargap = 0.5,
+ xaxis_title=f"Size of support changes - Total = {len(self.change_size)}"
+ )
+ fig.write_html(save_folder / "change_size.html")
+ return ranking, self.change_size
+
+ def relaunch_exploration(self):
+ """
+ Undo the close_exploration method for allowing the exploration of new supports
+ """
+ if self.is_open:
+ raise ValueError("Exploration wasn't closed")
+ self.old_it.append(self.it)
+ self.it = defaultdict(lambda: [])
+ self.old_n_stable.append(self.n_stable)
+ self.n_stable = defaultdict(lambda: [])
+ self.old_change_size.append(self.change_size)
+ self.change_size = []
+ self.is_open = True
+ self.last_supp = None
+
+ def get_loss_by_explored_support(self):
+ buffer_it_loss = []
+ for buffer, it in self.it.items():
+ buffer_it_loss.append((buffer, it[0], self.loss[buffer]))
+ buffer_it_loss.sort(key=lambda x: x[1])
+ return [x[2] for x in buffer_it_loss]
+
+
+@normalizer
+def sea_fast(linop, y, n_nonzero, n_iter=None, return_best=False, rel_tol=-np.inf,
+ algo_init=None, return_both=False, f=None, grad_f=None, is_mse=True, return_history=True, optimizer='cg',
+ surpress_warning=False, lip_fact=2 * 0.9
+ ) -> Union[Tuple[np.ndarray, List[float]],
+Tuple[np.ndarray, List[float], ExplorationHistory],
+Tuple[Tuple[np.ndarray, List[float]], Tuple[np.ndarray, List[float]]],
+Tuple[Tuple[np.ndarray, List[float], ExplorationHistory],
+Tuple[np.ndarray, List[float], ExplorationHistory]]]:
+ """
+ Use SEA algorithm for solving: min_x f(x) w.r.t ||X||_0 <= n_nonzero
+
+ :param (sksea.utils.AbstractLinearOperator) linop: Linear operator representing the D matrix
+ :param (np.ndarray) y: Target vector
+ :param (int) n_nonzero: Size of the wanted support
+ :param (int) n_iter: Number of iteration of the gradient descent in the intra-support optimisation phase
+ :param (bool) return_best: If True, return SEA_BEST
+ :param (float) rel_tol: The algorithm stops when the iterations relative difference is lower than rel_tol
+ :param (Callable or None) algo_init: Function to use for sea initialization. If None, initialize SEA with 0
+ :param (bool) return_both: If True, return SEA and SEA_BEST
+ or only for the LAST iteration (the last is the best if `return_best` is True)
+ :param (Callable[[np.ndarray, Optional[np.ndarray]], float]) f: Loss to minimize.
+ The first argument is the vector to use for the evaluation. The second is the support of the evaluation.
+ :param (Callable[[np.ndarray, Optional[np.ndarray]], np.ndarray]) grad_f: Gradient of the loss to minimize.
+ The first argument is the vector to use for the evaluation. The second is the support of the evaluation.
+ :param (bool) is_mse: If True, use better optimization algorithms (linear conjugate gradient)
+ for solving min_x 1/2 ||D * x - y||_2^2 w.r.t ||X||_0 <= n_nonzero instead of using non-linear algorithms
+ :return: The solution vector `x`, the sequence of residuals `res_norm`
+ """
+ x_len = linop.shape[1]
+ # Initializations
+ if n_iter is None or n_iter == 0:
+ n_iter_is_n_support = True
+ n_iter_max = np.inf
+ else:
+ n_iter_is_n_support = False
+ n_iter_max = n_iter
+
+ if algo_init is not None:
+ x_bar, *others = algo_init(linop, y, n_nonzero, n_iter, rel_tol=-np.inf, normalize=False,
+ f=f, grad_f=grad_f, is_mse=is_mse)
+ x = np.copy(x_bar)
+ else: # 0
+ x_bar = np.zeros(x_len)
+ x = np.zeros(x_len)
+ others = ()
+ res_norm = [f(x, linop)]
+ L = linop.compute_lipschitz()
+ pas = lip_fact / L
+
+ # For keeping track of the best iterate
+ best_res_norm = np.inf
+ best_it = 0
+ best_x = np.copy(x)
+ best_s = np.zeros_like(x, dtype=bool)
+ last_s = np.zeros_like(x, dtype=bool)
+
+ history = None
+ for ot in others:
+ if isinstance(ot, ExplorationHistory):
+ history = ot
+ history.relaunch_exploration()
+ break
+ if history is None:
+ history = ExplorationHistory()
+
+ # Iterative scheme
+ it = 0
+ # old_n_supports = 0
+ while it < n_iter_max:
+ s = find_support(x_bar, n_nonzero)
+
+ hist = history.get(s, copy_x=False, copy_grad=False, it=it)
+ if (last_s != s).any() and hist is None: # Cg optimisation only on unexplored support change # noqa
+ x = optimize(linop, x, y, s=s, f=f, grad_f=grad_f, is_mse=is_mse, optimizer=optimizer,
+ surpress_warning=surpress_warning)
+ g = grad_f(x * s, linop) # gradient en x*s
+ loss = f(x * s, linop)
+ history.add(s, x, g, loss, it, copy_x=False, copy_grad=False)
+ last_s = s
+ else:
+ x, g, loss = hist
+ if g is None:
+ g = grad_f(x * s, linop)
+ history.add(s, x, g, loss, it, copy_x=False, copy_grad=False)
+
+ x_bar -= pas * g # Exploratory variable update
+ res_norm.append(loss)
+
+ if res_norm[-1] < best_res_norm: # Keep best iteration
+ best_res_norm = res_norm[-1]
+ best_x = x * s
+ np.copyto(best_s, s)
+ best_it = it
+ if np.abs(res_norm[-2] - res_norm[-1]) / res_norm[-2] < rel_tol:
+ break
+ elif n_iter_is_n_support and (
+ history.get_n_supports(best=False) > n_nonzero or # Stop when support quota reached
+ history.get_n_supports(best=False) >= math.comb(x_len, n_nonzero)):
+ break
+ # elif n_iter_is_n_support and False:
+ # print(f"n_iter = {history.get_n_supports(best=False)} / {n_nonzero, math.comb(x_len, n_nonzero)}")
+ if it >= 100000 and n_iter_is_n_support:
+ logger.warning(f"n_supports = {history.get_n_supports(best=False)} / "
+ f"{n_nonzero, math.comb(x_len, n_nonzero)}"
+ f" after 100000 iterations")
+ break
+ # if history.get_n_supports(best=False) > old_n_supports + 1:
+ # logger.error("Support miscounted")
+ # old_n_supports = history.get_n_supports(best=False)
+ it += 1
+
+ best_res_list = res_norm[:best_it + 2]
+ history.close_exploration(it) # noqa
+
+ # Adding history to output
+ if return_history:
+ sea_results = (x * s, res_norm, history) # noqa
+ sea_best_results = (best_x, best_res_list, history)
+ else:
+ sea_results = (x * s, res_norm) # noqa
+ sea_best_results = (best_x, best_res_list)
+
+ # Returning results
+ if return_both:
+ return sea_results, sea_best_results
+ elif return_best:
+ return sea_best_results
+ else:
+ return sea_results
+
+
+def optimize(linop, x, y, alpha=None, s=None, n_iter=0, rel_tol=-np.inf, optimizer="cg", f=None, grad_f=None,
+ is_mse=True, surpress_warning=False) -> np.ndarray:
+ """
+ Use an optimization algorithm in order to solve min_x f(x) on a chosen support
+
+ :param (sksea.utils.AbstractLinearOperator) linop: Linear operator representing the D matrix in Dx = y
+ :param (np.ndarray) x: Current solution
+ :param (np.ndarray) y: Target vector
+ :param (float or None) alpha: Step size in the gradient descent of intra-support optimization is `alpha / lipsh`
+ :param (np.ndarray or None) s: Support space. If not specified, use all available space
+ :param (int) n_iter: Number of iteration of the HandMade gradient descent
+ :param (float) rel_tol: The HandMade gradient descent stops when
+ the iterations relative difference is lower than rel_tol
+ :param (str) optimizer: Optimizer to use.
+ - If 'cg', use Conjugate Gradient descent algorithm.
+ - If 'hmgd', use a HandMade Gradient Descent algorithm
+ - If 'pi', use pseudo-inverse
+ - If 'chol' use Cholesky decomposition. Linop must be a SparseLinearOperator.
+ :param (Callable[[np.ndarray, Optional[np.ndarray]], float]) f: Loss to minimize.
+ The first argument is the vector to use for the evaluation. The second is the support of the evaluation.
+ :param (Callable[[np.ndarray, Optional[np.ndarray]], np.ndarray]) grad_f: Gradient of the loss to minimize.
+ The first argument is the vector to use for the evaluation. The second is the support of the evaluation.
+ :param (bool) is_mse: If True, use better optimization algorithms (linear conjugate gradient)
+ for solving min_x 1/2 ||D * x - y||_2^2 w.r.t ||X||_0 <= n_nonzero instead of using non-linear algorithms
+ :return: The solution vector `x`
+ """
+ if s is None:
+ s = np.ones_like(x, dtype=bool)
+
+ if optimizer != "chol":
+ linop_s = linop.get_operator_on_support(s)
+ x_s = x[s]
+ else:
+ linop_s = None
+ x_s = None
+
+ if optimizer == "hmgd": # Hand-made gradient descent
+ lipsch = linop_s.compute_lipschitz() if s.sum() > 1 else linop.compute_lipschitz()
+ res = linop_s @ x_s - y
+ res_norm = [np.linalg.norm(res)]
+ for _ in range(n_iter):
+ g = linop_s.H @ res
+ x_s -= alpha * (1 / lipsch) * g
+ res = linop_s @ x_s - y
+ res_norm.append(np.linalg.norm(res))
+ # Early stopping
+ if (res_norm[-2] - res_norm[-1]) / res_norm[-2] < rel_tol:
+ break
+
+ elif optimizer == "cg": # Conjugate gradient descent
+ if is_mse: # Linear
+ # https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.linalg.cg.html
+ b = linop_s.H @ y
+ a = linop_s.H @ linop_s
+ x_s, info = cg(a, b, x_s, atol=0, tol=1e-5)
+ if info > 0:
+ if not surpress_warning:
+ # logger.warning("Conjugate gradient descent did not converge")
+ pass
+ elif info < 0:
+ raise ValueError("Conjugate gradient descent failed")
+ if np.isnan(np.dot(x_s, x_s)).any():
+ # logger.warning("Conjugate gradient descent returned NaN"
+ # "\n Replacing the output of gradient descent by 0")
+ x_s[np.isnan(x_s)] = 0
+ else: # Non linear
+ # https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fmin_cg.html
+ x_s = fmin_cg(f, x_s, grad_f, (linop_s,), disp=False)
+
+ elif is_mse:
+ if optimizer == "pi":
+ assert isinstance(linop, MatrixOperator)
+ x_s = np.linalg.pinv(linop_s.matrix) @ y
+
+ elif optimizer == "chol":
+ assert isinstance(linop, SparseSupportOperator)
+ try:
+ linop.change_support(s)
+ x_s = linop.solve()[np.argsort(linop.support)]
+ except np.linalg.LinAlgError as inversion_error:
+ linop.reset()
+ if not surpress_warning:
+ logger.warning("Inversion issue with Cholesky decomposition: \n" + str(inversion_error) +
+ "\nUsing conjugate gradient descent instead")
+ return optimize(linop, x, y, alpha=alpha, s=s, n_iter=n_iter, rel_tol=rel_tol, optimizer="cg", f=f,
+ grad_f=grad_f, is_mse=is_mse)
+ except ValueError as value_error:
+ linop.reset()
+ if not surpress_warning:
+ logger.warning("Value Error with Cholesky decomposition: \n" + str(value_error) +
+ "\nUsing conjugate gradient descent instead")
+ return optimize(linop, x, y, alpha=alpha, s=s, n_iter=n_iter, rel_tol=rel_tol, optimizer="cg", f=f,
+ grad_f=grad_f, is_mse=is_mse)
+ except Exception as e:
+ logger.error("Not expected Error with Cholesky decomposition: \n" + str(e) +
+ "\nUsing conjugate gradient descent instead")
+ return optimize(linop, x, y, alpha=alpha, s=s, n_iter=n_iter, rel_tol=rel_tol, optimizer="cg", f=f,
+ grad_f=grad_f, is_mse=is_mse)
+
+ else:
+ raise ValueError("Bad value of optimizer when is_mse is True")
+ else:
+ raise ValueError("Bad value of optimizer")
+
+ del linop_s
+ # gc.collect() # Force linop_s to be removed from the memory
+ # (Done in run_experiment from training_task.py instead of here for performance purposes)
+ x_out = np.zeros_like(x)
+ x_out[s] = x_s
+ return x_out
+
+
+@normalizer
+def omp(linop, y, n_nonzero, n_iter, *args, alpha=0.9, rel_tol=-np.inf, optimizer='cg', f=None, grad_f=None,
+ is_mse=True, **kwargs
+ ) -> Tuple[np.ndarray, List[float]]:
+ """
+ Use OMP algorithm for solving: min_x f(x) w.r.t ||X||_0 <= n_nonzero
+
+ :param (sksea.utils.AbstractLinearOperator) linop: Linear operator representing the D matrix in Dx = y
+ :param (np.ndarray) y: Target vector
+ :param (int) n_nonzero: Size of the wanted support
+ :param (int) n_iter: Number of iteration of the gradient descent in the intra-support optimisation phase
+ :param (float) alpha: Step size in the gradient descent of intra-support optimisation is `alpha / l`
+ with `l` the Lipschitz constant of linop.
+ :param (float) rel_tol: The algorithm stops when the iterations relative difference is lower than rel_tol
+ :param (str) optimizer: Optimizer to use.
+ - If 'cg', use Conjugate Gradient descent algorithm.
+ - If 'hmgd', use a HandMade Gradient Descent algorithm
+ :param (Callable[[np.ndarray, Optional[np.ndarray]], float]) f: Loss to minimize.
+ The first argument is the vector to use for the evaluation. The second is the support of the evaluation.
+ :param (Callable[[np.ndarray, Optional[np.ndarray]], np.ndarray]) grad_f: Gradient of the loss to minimize.
+ The first argument is the vector to use for the evaluation. The second is the support of the evaluation.
+ :param (bool) is_mse: If True, use better optimisation algorithms (linear conjugate gradient)
+ for solving min_x 1/2 ||D * x - y||_2^2 w.r.t ||X||_0 <= n_nonzero instead of using non-linear algorithms
+ :return: The solution vector `x`, the sequence of residuals `res_norm`
+ """
+ x_len = linop.shape[1]
+ x = np.zeros(x_len)
+ res_norm = []
+ s = np.zeros(x_len, dtype=bool) # Support
+ res_norm.append(f(x, linop))
+
+ for _ in range(n_nonzero):
+ i = np.argmax(np.abs(grad_f(x, linop)) * ~s) # Non-explored direction with the highest gradient
+ s[i] = True # Add this direction in the support
+
+ # Optimisation in the support space ((conjugate) gradient descent)
+ # Using the pseudo inverse method doesn't work here
+ x = optimize(linop, x, y, alpha, s, n_iter, rel_tol, optimizer, f, grad_f, is_mse)
+
+ res_norm.append(f(x, linop))
+ if (res_norm[-2] - res_norm[-1]) / res_norm[-2] < rel_tol:
+ break
+
+ return x, res_norm
+
+
+@normalizer
+def omp_fast(linop, y, n_nonzero, n_iter, *args, alpha=0.9, rel_tol=-np.inf, optimizer='cg', f=None, grad_f=None,
+ is_mse=True, return_history=True, **kwargs
+ ) -> Union[Tuple[np.ndarray, List[float]], Tuple[np.ndarray, List[float], ExplorationHistory]]:
+ """
+ Use OMP algorithm for solving: min_x f(x) w.r.t ||X||_0 <= n_nonzero
+
+ :param (sksea.utils.AbstractLinearOperator) linop: Linear operator representing the D matrix in Dx = y
+ :param (np.ndarray) y: Target vector
+ :param (int) n_nonzero: Size of the wanted support
+ :param (int) n_iter: Number of iteration of the gradient descent in the intra-support optimisation phase
+ :param (float) alpha: Step size in the gradient descent of intra-support optimisation is `alpha / l`
+ with `l` the Lipschitz constant of linop.
+ :param (float) rel_tol: The algorithm stops when the iterations relative difference is lower than rel_tol
+ :param (str) optimizer: Optimizer to use.
+ - If 'cg', use Conjugate Gradient descent algorithm.
+ - If 'hmgd', use a HandMade Gradient Descent algorithm
+ :param (Callable[[np.ndarray, Optional[np.ndarray]], float]) f: Loss to minimize.
+ The first argument is the vector to use for the evaluation. The second is the support of the evaluation.
+ :param (Callable[[np.ndarray, Optional[np.ndarray]], np.ndarray]) grad_f: Gradient of the loss to minimize.
+ The first argument is the vector to use for the evaluation. The second is the support of the evaluation.
+ :param (bool) is_mse: If True, use better optimisation algorithms (linear conjugate gradient)
+ for solving min_x 1/2 ||D * x - y||_2^2 w.r.t ||X||_0 <= n_nonzero instead of using non-linear algorithms
+ :return: The solution vector `x`, the sequence of residuals `res_norm`
+ """
+ x_len = linop.shape[1]
+ x = np.zeros(x_len)
+ res_norm = []
+ s = np.zeros(x_len, dtype=bool) # Support
+ res_norm.append(f(x, linop))
+
+ history = ExplorationHistory()
+
+ for it in range(n_nonzero):
+ grad = grad_f(x, linop)
+ i = np.argmax(np.abs(grad) * ~s) # Non-explored direction with the highest gradient
+ s[i] = True # Add this direction in the support
+
+ hist = history.get(s, copy_x=False, copy_grad=False, it=it)
+ if hist is None:
+ x = optimize(linop, x, y, alpha, s, n_iter, rel_tol, optimizer, f=f, grad_f=grad_f,
+ is_mse=is_mse)
+ res_norm_gd = f(x, linop)
+ history.add(s, x, grad, res_norm_gd, it, copy_x=False, copy_grad=False)
+ else:
+ x_temp, _, res_norm_gd = hist
+
+ res_norm.append(res_norm_gd)
+ if (res_norm[-2] - res_norm[-1]) / res_norm[-2] < rel_tol:
+ break
+
+ history.close_exploration(it) # noqa
+ if return_history:
+ return x.copy(), res_norm, history
+ else:
+ return x, res_norm
+
+
+@normalizer
+def ompr(linop, y, n_nonzero, n_iter, *args, alpha=0.9, rel_tol=-np.inf, optimizer='cg', f=None, grad_f=None,
+ is_mse=True, **kwargs
+ ) -> Tuple[np.ndarray, List[float]]:
+ """
+ Use OMPR algorithm for solving: min_x f(x) w.r.t ||X||_0 <= n_nonzero
+
+ :param (sksea.utils.AbstractLinearOperator) linop: Linear operator representing the D matrix
+ :param (np.ndarray) y: Target vector
+ :param (int) n_nonzero: Size of the wanted support
+ :param (int) n_iter: Maximum number of iteration for the algorithm
+ :param (float) alpha: Step size in the gradient descent of intra-support optimisation is `alpha / l`
+ with `l` the Lipschitz constant of linop.
+ :param (float) rel_tol: The algorithm stops when the iterations relative difference is lower than rel_tol
+ :param (str) optimizer: Optimizer to use.
+ - If 'cg', use Conjugate Gradient descent algorithm.
+ - If 'hmgd', use a HandMade Gradient Descent algorithm
+ :param (Callable[[np.ndarray, Optional[np.ndarray]], float]) f: Loss to minimize.
+ The first argument is the vector to use for the evaluation. The second is the support of the evaluation.
+ :param (Callable[[np.ndarray, Optional[np.ndarray]], np.ndarray]) grad_f: Gradient of the loss to minimize.
+ The first argument is the vector to use for the evaluation. The second is the support of the evaluation.
+ :param (bool) is_mse: If True, use better optimization algorithms (linear conjugate gradient)
+ for solving min_x 1/2 ||D * x - y||_2^2 w.r.t ||X||_0 <= n_nonzero instead of using non-linear algorithms
+ :return: The solution vector `x`, the sequence of residuals `res_norm`
+ """
+ x, res_norm = omp(linop, y, n_nonzero, n_iter, alpha=alpha, rel_tol=rel_tol, normalize=False, f=f, grad_f=grad_f,
+ is_mse=is_mse)
+ s = find_support(x, n_nonzero) # x != 0
+ x_old = np.copy(x) # This variable allows us to undo the last iteration if necessary
+
+ for _ in range(n_iter):
+ i = np.argmax(np.abs(grad_f(x, linop)) * ~s) # Non-explored direction with the highest gradient
+ abs_x = np.abs(x)
+ j = np.where(abs_x == np.min(abs_x[s]))[0][0] # Smallest coefficient of x in s
+ s[i] = True
+ s[j] = False
+ x[j] = 0 # We need to remove the data outside the support
+
+ # Optimisation in the support space (gradient descent)
+ # Using the pseudo inverse method doesn't work here
+ x = optimize(linop, x, y, alpha, s, n_iter, rel_tol, optimizer, f=f, grad_f=grad_f, is_mse=is_mse)
+
+ res_norm.append(f(x, linop))
+ if res_norm[-1] >= res_norm[-2]:
+ # If the replacement increase the residual, we have to stop and cancel the last step
+ x = x_old
+ res_norm.pop()
+ break
+ if (res_norm[-2] - res_norm[-1]) / res_norm[-2] < rel_tol:
+ break
+ np.copyto(x_old, x)
+
+ return x, res_norm
+
+
+@normalizer
+def ompr_fast(linop, y, n_nonzero, n_iter, *args, alpha=0.9, rel_tol=-np.inf, optimizer='cg', f=None, grad_f=None,
+ is_mse=True, return_history=True, **kwargs
+ ) -> Union[Tuple[np.ndarray, List[float]], Tuple[np.ndarray, List[float], ExplorationHistory]]:
+ """
+ Use OMPR algorithm for solving: min_x f(x) w.r.t ||X||_0 <= n_nonzero
+
+ :param (sksea.utils.AbstractLinearOperator) linop: Linear operator representing the D matrix
+ :param (np.ndarray) y: Target vector
+ :param (int) n_nonzero: Size of the wanted support
+ :param (int) n_iter: Maximum number of iteration for the algorithm
+ :param (float) alpha: Step size in the gradient descent of intra-support optimisation is `alpha / l`
+ with `l` the Lipschitz constant of linop.
+ :param (float) rel_tol: The algorithm stops when the iterations relative difference is lower than rel_tol
+ :param (str) optimizer: Optimizer to use.
+ - If 'cg', use Conjugate Gradient descent algorithm.
+ - If 'hmgd', use a HandMade Gradient Descent algorithm
+ :param (Callable[[np.ndarray, Optional[np.ndarray]], float]) f: Loss to minimize.
+ The first argument is the vector to use for the evaluation. The second is the support of the evaluation.
+ :param (Callable[[np.ndarray, Optional[np.ndarray]], np.ndarray]) grad_f: Gradient of the loss to minimize.
+ The first argument is the vector to use for the evaluation. The second is the support of the evaluation.
+ :param (bool) is_mse: If True, use better optimization algorithms (linear conjugate gradient)
+ for solving min_x 1/2 ||D * x - y||_2^2 w.r.t ||X||_0 <= n_nonzero instead of using non-linear algorithms
+ :return: The solution vector `x`, the sequence of residuals `res_norm`
+ """
+ x, _, history = omp_fast(linop, y, n_nonzero, n_iter, alpha=alpha, rel_tol=rel_tol, normalize=False, f=f, grad_f=grad_f, is_mse=is_mse)
+ s = history.get_last_support() # x != 0
+ x_old = np.copy(x) # This variable allows us to undo the last iteration if necessary
+ res_norm = [f(x, linop)]
+
+ history.relaunch_exploration()
+
+ for it in range(n_iter):
+ i = np.argmax(np.abs(grad_f(x, linop)) * ~s) # Non-explored direction with the highest gradient
+ abs_x = np.abs(x)
+ j = np.where(abs_x == np.min(abs_x[s]))[0][0] # Smallest coefficient of x in s
+ s[i] = True
+ s[j] = False
+ x[j] = 0 # We need to remove the data outside the support
+
+ # Optimisation in the support space (gradient descent)
+ # Using the pseudo inverse method doesn't work here
+ hist = history.get(s, copy_x=False, copy_grad=False, it=it)
+ if hist is None:
+ x = optimize(linop, np.copy(x), y, alpha, s, n_iter, rel_tol, optimizer, f=f, grad_f=grad_f, is_mse=is_mse)
+ res_norm_gd = f(x, linop)
+ history.add(s.copy(), x, None, res_norm_gd, it, copy_x=False, copy_grad=False)
+ else:
+ _, _, res_norm_gd = hist
+
+ res_norm.append(res_norm_gd)
+ if res_norm[-1] >= res_norm[-2]:
+ # If the replacement increase the residual, we have to stop and cancel the last step
+ x = x_old
+ res_norm.pop()
+ break
+ if (res_norm[-2] - res_norm[-1]) / res_norm[-2] < rel_tol:
+ break
+ np.copyto(x_old, x)
+
+ history.close_exploration(it) # noqa
+ if return_history:
+ return x, res_norm, history
+ else:
+ return x, res_norm
+
+
+@normalizer
+def els(linop, y, n_nonzero, n_iter, *args, alpha=0.9, rel_tol=-np.inf, optimizer='cg', f=None, grad_f=None,
+ is_mse=True, **kwargs
+ ) -> Tuple[np.ndarray, List[float]]:
+ """
+ Use ELS algorithm for solving: min_x f(x) w.r.t ||X||_0 <= n_nonzero
+
+ :param (sksea.utils.AbstractLinearOperator) linop: Linear operator representing the D matrix
+ :param (np.ndarray) y: Target vector
+ :param (int) n_nonzero: Size of the wanted support
+ :param (int) n_iter: Maximum number of iteration for the algorithm
+ :param (float) alpha: Step size in the gradient descent of intra-support optimisation is `alpha / l`
+ with `l` the Lipschitz constant of linop.
+ :param (float) rel_tol: The algorithm stops when the iterations relative difference is lower than rel_tol
+ :param (str) optimizer: Optimizer to use.
+ - If 'cg', use Conjugate Gradient descent algorithm.
+ - If 'hmgd', use a HandMade Gradient Descent algorithm
+ :param (Callable[[np.ndarray, Optional[np.ndarray]], float]) f: Loss to minimize.
+ The first argument is the vector to use for the evaluation. The second is the support of the evaluation.
+ :param (Callable[[np.ndarray, Optional[np.ndarray]], np.ndarray]) grad_f: Gradient of the loss to minimize.
+ The first argument is the vector to use for the evaluation. The second is the support of the evaluation.
+ :param (bool) is_mse: If True, use better optimisation algorithms (linear conjugate gradient)
+ for solving min_x 1/2 ||D * x - y||_2^2 w.r.t ||X||_0 <= n_nonzero instead of using non-linear algorithms
+ :return: The solution vector `x`, the sequence of residuals `res_norm`
+ """
+ x, res_norm = omp(linop, y, n_nonzero, n_iter, alpha=alpha, rel_tol=rel_tol, normalize=False,
+ f=f, grad_f=grad_f, is_mse=is_mse)
+ if n_nonzero == linop.shape[1]:
+ logger.warning("ELS is equivalent to OMP when n_nonzero == linop.shape[1]")
+ return x, res_norm
+ s = find_support(x, n_nonzero) # x != 0
+ x_old = np.copy(x) # This variable allows us to undo the last iteration if necessary
+
+ for _ in range(n_iter):
+ abs_x = np.abs(x)
+ j = np.where((abs_x == np.min(abs_x[s])) & (s == 1))[0][0] # Smallest coefficient of x in s
+ s[j] = False
+ x[j] = 0
+ # We need to remove the data outside the support
+
+ # Search for the best replacement by looking at the result of the optimisation problem in each direction
+ min_i = 0
+ min_x = np.copy(x)
+ min_res = np.inf
+ for i in np.nonzero(1 - s)[0]:
+ if i == j:
+ continue # We don't want to take the direction we have just removed
+
+ s[i] = True # Add temporarily the current research direction in the support
+
+ # Optimisation in the support space (gradient descent)
+ # Using the pseudo inverse method doesn't work here
+ x_temp = optimize(linop, np.copy(x), y, alpha, s, n_iter, rel_tol, optimizer, f=f, grad_f=grad_f,
+ is_mse=is_mse)
+ res_norm_gd = f(x_temp, linop)
+
+ if min_res > res_norm_gd:
+ min_i = i
+ np.copyto(min_x, x_temp) # min_x = x_temp
+ min_res = res_norm_gd
+ s[i] = False # Remove temporary research direction in the support
+
+ # Get the best optimisation problem result
+ s[min_i] = True
+ np.copyto(x, min_x)
+
+ res_norm.append(f(x, linop))
+ if res_norm[-1] >= res_norm[-2]:
+ # If the replacement increase the residual, we have to stop and cancel the last step
+ x = x_old
+ res_norm.pop()
+ break
+ if (res_norm[-2] - res_norm[-1]) / res_norm[-2] < rel_tol:
+ break
+ np.copyto(x_old, x)
+
+ return x, res_norm
+
+
+@normalizer
+def els_fast(linop, y, n_nonzero, n_iter, *args, alpha=0.9, rel_tol=-np.inf, optimizer='cg', f=None, grad_f=None,
+ is_mse=True, return_history=True, **kwargs
+ ) -> Union[Tuple[np.ndarray, List[float]], Tuple[np.ndarray, List[float], ExplorationHistory]]:
+ """
+ Use ELS algorithm for solving: min_x f(x) w.r.t ||X||_0 <= n_nonzero
+
+ :param (sksea.utils.AbstractLinearOperator) linop: Linear operator representing the D matrix
+ :param (np.ndarray) y: Target vector
+ :param (int) n_nonzero: Size of the wanted support
+ :param (int) n_iter: Maximum number of iteration for the algorithm
+ :param (float) alpha: Step size in the gradient descent of intra-support optimisation is `alpha / l`
+ with `l` the Lipschitz constant of linop.
+ :param (float) rel_tol: The algorithm stops when the iterations relative difference is lower than rel_tol
+ :param (str) optimizer: Optimizer to use.
+ - If 'cg', use Conjugate Gradient descent algorithm.
+ - If 'hmgd', use a HandMade Gradient Descent algorithm
+ :param (Callable[[np.ndarray, Optional[np.ndarray]], float]) f: Loss to minimize.
+ The first argument is the vector to use for the evaluation. The second is the support of the evaluation.
+ :param (Callable[[np.ndarray, Optional[np.ndarray]], np.ndarray]) grad_f: Gradient of the loss to minimize.
+ The first argument is the vector to use for the evaluation. The second is the support of the evaluation.
+ :param (bool) is_mse: If True, use better optimisation algorithms (linear conjugate gradient)
+ for solving min_x 1/2 ||D * x - y||_2^2 w.r.t ||X||_0 <= n_nonzero instead of using non-linear algorithms
+ :return: The solution vector `x`, the sequence of residuals `res_norm`
+ """
+ x, res_norm, history = omp_fast(linop, y, n_nonzero, n_iter, alpha=alpha, rel_tol=rel_tol, normalize=False,
+ f=f, grad_f=grad_f, is_mse=is_mse)
+ if n_nonzero == linop.shape[1]:
+ logger.warning("ELS is equivalent to OMP when n_nonzero == linop.shape[1]")
+ return x, res_norm, history
+ s = history.get_last_support() # x != 0
+ x_old = np.copy(x) # This variable allows us to undo the last iteration if necessary
+ res_norm = [f(x, linop)]
+
+ history.relaunch_exploration()
+
+ for it in range(n_iter):
+ abs_x = np.abs(x)
+ j = np.where((abs_x == np.min(abs_x[s])) & (s == 1))[0][0] # Smallest coefficient of x in s
+ s[j] = False
+ x[j] = 0 # We need to remove the data outside the support
+
+ # Search for the best replacement by looking at the result of the optimisation problem in each direction
+ min_i = 0
+ min_res = np.inf
+ for i in np.nonzero(1 - s)[0]:
+ if i == j:
+ continue # We don't want to take the direction we have just removed
+
+ s[i] = True # Add temporarily the current research direction in the support
+
+ # Optimisation in the support space (gradient descent)
+ # Using the pseudo inverse method doesn't work here
+ hist = history.get(s, copy_x=False, copy_grad=False, it=it)
+ if hist is None:
+ x_temp = optimize(linop, np.copy(x), y, alpha, s, n_iter, rel_tol, optimizer, f=f, grad_f=grad_f,
+ is_mse=is_mse)
+ res_norm_gd = f(x_temp, linop)
+ history.add(s.copy(), x_temp, None, res_norm_gd, it, copy_x=False, copy_grad=False)
+ else:
+ _, _, res_norm_gd = hist
+
+ if min_res > res_norm_gd:
+ min_i = i
+ min_res = res_norm_gd
+ s[i] = False # Remove temporary research direction in the support
+
+ # Get the best optimization problem result
+ s[min_i] = True
+ min_x, _, loss = history.get(s, copy_x=False, copy_grad=False, it=it)
+ np.copyto(x, min_x)
+
+ res_norm.append(loss)
+ if res_norm[-1] >= res_norm[-2]:
+ # If the replacement increase the residual, we have to stop and cancel the last step
+ x = x_old
+ res_norm.pop()
+ break
+ if (res_norm[-2] - res_norm[-1]) / res_norm[-2] < rel_tol:
+ break
+ np.copyto(x_old, x)
+
+ history.close_exploration(it) # noqa
+ if return_history:
+ return x, res_norm, history
+ else:
+ return x, res_norm
+
+
+@normalizer
+def es(linop, y, n_nonzero, n_iter=0, alpha=0.9, rel_tol=-np.inf, optimizer='cg', f=None, grad_f=None, is_mse=True
+ ) -> Tuple[np.ndarray, List[float]]:
+ """
+ Use Exhaustive Search (ES) algorithm for solving: min_x f(x) w.r.t ||X||_0 <= n_nonzero
+
+ :param (sksea.utils.AbstractLinearOperator) linop: Linear operator representing the D matrix
+ :param (np.ndarray) y: Target vector
+ :param (int) n_nonzero: Size of the wanted support
+ :param (int) n_iter: Maximum number of iteration for the algorithm
+ :param (float) alpha: Step size in the gradient descent of intra-support optimisation is `alpha / l`
+ with `l` the Lipschitz constant of linop.
+ :param (float) rel_tol: The algorithm stops when the iterations relative difference is lower than rel_tol
+ :param (str) optimizer: Optimizer to use.
+ - If 'cg', use Conjugate Gradient descent algorithm.
+ - If 'hmgd', use a HandMade Gradient Descent algorithm
+ :param (Callable[[np.ndarray, Optional[np.ndarray]], float]) f: Loss to minimize.
+ The first argument is the vector to use for the evaluation. The second is the support of the evaluation.
+ :param (Callable[[np.ndarray, Optional[np.ndarray]], np.ndarray]) grad_f: Gradient of the loss to minimize.
+ The first argument is the vector to use for the evaluation. The second is the support of the evaluation.
+ :param (bool) is_mse: If True, use better optimization algorithms (linear conjugate gradient)
+ for solving min_x 1/2 ||D * x - y||_2^2 w.r.t ||X||_0 <= n_nonzero instead of using non-linear algorithms
+ :return: The solution vector `x`, the sequence of residuals `res_norm`
+ """
+ x_len = linop.shape[1]
+ x = np.zeros(x_len)
+ x_best = np.zeros_like(x)
+ res_norm_best = f(x, linop)
+ res_norm = []
+ s = np.zeros(x_len, dtype=bool) # Support
+ res_norm.append(f(x, linop))
+
+ for combination in combinations(range(x_len), n_nonzero):
+ # Select support
+ s[:] = False
+ s[np.array(combination)] = True
+ # Optimize
+ x = optimize(linop, x, y, alpha, s, n_iter, rel_tol, optimizer, f, grad_f, is_mse)
+
+ # Store best result
+ res_norm_tmp = f(x, linop)
+ res_norm.append(res_norm_tmp)
+ if res_norm_tmp < res_norm_best:
+ np.copyto(x_best, x)
+ res_norm_best = res_norm_tmp
+ return x_best, res_norm
+
+
+@normalizer
+def htp(linop, y, n_nonzero, n_iter, alpha=0.9, rel_tol=-np.inf, optimizer='cg', f=None, grad_f=None, is_mse=True,
+ algo_init=None) -> Tuple[np.ndarray, List[float]]:
+ """
+ Use HTP algorithm for solving: min_x f(x) w.r.t ||X||_0 <= n_nonzero
+
+ :param (sksea.utils.AbstractLinearOperator) linop: Linear operator representing the D matrix in Dx = y
+ :param (np.ndarray) y: Target vector
+ :param (int) n_nonzero: Size of the wanted support
+ :param (int) n_iter: Number of iteration of the gradient descent in the intra-support optimisation phase
+ :param (float) alpha: Step size in the gradient descent of intra-support optimisation is `alpha / l`
+ with `l` the Lipschitz constant of linop.
+ :param (float) rel_tol: The algorithm stops when the iterations relative difference is lower than rel_tol
+ :param (str) optimizer: Optimizer to use.
+ - If 'cg', use Conjugate Gradient descent algorithm.
+ - If 'hmgd', use a HandMade Gradient Descent algorithm
+ :param (Callable[[np.ndarray, Optional[np.ndarray]], float]) f: Loss to minimize.
+ The first argument is the vector to use for the evaluation. The second is the support of the evaluation.
+ :param (Callable[[np.ndarray, Optional[np.ndarray]], np.ndarray]) grad_f: Gradient of the loss to minimize.
+ The first argument is the vector to use for the evaluation. The second is the support of the evaluation.
+ :param (bool) is_mse: If True, use better optimization algorithms (linear conjugate gradient)
+ for solving min_x 1/2 ||D * x - y||_2^2 w.r.t ||X||_0 <= n_nonzero instead of using non-linear algorithms
+ :param (Callable or None) algo_init: Function to use for IHT initialization. If None, initialize IHT with 0
+ :return: The solution vector `x`, the sequence of residuals `res_norm`
+ """
+ # Initialisation
+ if algo_init is not None:
+ x, _ = algo_init(linop, y, n_nonzero, n_iter, rel_tol=-np.inf, normalize=False,
+ f=f, grad_f=grad_f, is_mse=is_mse)
+ else: # 0
+ x_len = linop.shape[1]
+ x = np.zeros(x_len)
+
+ lip = linop.compute_lipschitz()
+ pas = 2 * 0.9 / lip
+ res_norm = [f(x, linop)] # First residual
+ last_s = np.zeros_like(x, dtype=bool)
+
+ for _ in range(n_iter):
+ x -= pas * grad_f(x, linop) # gradient step
+ s = find_support(x, n_nonzero) # Support selection
+
+ # Optimisation in the support space ((conjugate) gradient descent)
+ x = optimize(linop, x, y, alpha, s, n_iter, rel_tol, optimizer, f, grad_f, is_mse)
+
+ res_norm.append(f(x, linop))
+ # Stop when the algorithm is stuck in a local minimum
+ if (res_norm[-2] - res_norm[-1]) / res_norm[-2] < rel_tol or np.all(s == last_s):
+ break
+ np.copyto(last_s, s)
+
+ return x, res_norm
+
+
+@normalizer
+def htp_fast(linop, y, n_nonzero, n_iter, alpha=0.9, rel_tol=-np.inf, optimizer='cg', f=None, grad_f=None, is_mse=True,
+ algo_init=None, return_history=True, return_best=True, return_both=False, lip_fact=2 * 0.9
+ ) -> Union[Tuple[np.ndarray, List[float]],
+Tuple[np.ndarray, List[float], ExplorationHistory],
+Tuple[Tuple[np.ndarray, List[float]], Tuple[np.ndarray, List[float]]],
+Tuple[Tuple[np.ndarray, List[float], ExplorationHistory],
+Tuple[np.ndarray, List[float], ExplorationHistory]]]:
+ """
+ Use HTP algorithm for solving: min_x f(x) w.r.t ||X||_0 <= n_nonzero
+
+ :param (sksea.utils.AbstractLinearOperator) linop: Linear operator representing the D matrix in Dx = y
+ :param (np.ndarray) y: Target vector
+ :param (int) n_nonzero: Size of the wanted support
+ :param (int) n_iter: Number of iteration of the gradient descent in the intra-support optimisation phase
+ :param (float) alpha: Step size in the gradient descent of intra-support optimisation is `alpha / l`
+ with `l` the Lipschitz constant of linop.
+ :param (float) rel_tol: The algorithm stops when the iterations relative difference is lower than rel_tol
+ :param (str) optimizer: Optimizer to use.
+ - If 'cg', use Conjugate Gradient descent algorithm.
+ - If 'hmgd', use a HandMade Gradient Descent algorithm
+ :param (Callable[[np.ndarray, Optional[np.ndarray]], float]) f: Loss to minimize.
+ The first argument is the vector to use for the evaluation. The second is the support of the evaluation.
+ :param (Callable[[np.ndarray, Optional[np.ndarray]], np.ndarray]) grad_f: Gradient of the loss to minimize.
+ The first argument is the vector to use for the evaluation. The second is the support of the evaluation.
+ :param (bool) is_mse: If True, use better optimisation algorithms (linear conjugate gradient)
+ for solving min_x 1/2 ||D * x - y||_2^2 w.r.t ||X||_0 <= n_nonzero instead of using non-linear algorithms
+ :param (Callable or None) algo_init: Function to use for IHT initialization. If None, initialize IHT with 0
+ :return: The solution vector `x`, the sequence of residuals `res_norm`
+ """
+ # Initialisation
+ if algo_init is not None:
+ x, *others = algo_init(linop, y, n_nonzero, n_iter, rel_tol=-np.inf, normalize=False,
+ f=f, grad_f=grad_f, is_mse=is_mse)
+
+ else: # 0
+ x_len = linop.shape[1]
+ x = np.zeros(x_len)
+ others = ()
+
+ lip = linop.compute_lipschitz()
+ pas = lip_fact / lip
+ res_norm = [f(x, linop)] # First residual
+ g = grad_f(x, linop)
+
+ history = None
+ for ot in others:
+ if isinstance(ot, ExplorationHistory):
+ history = ot
+ history.relaunch_exploration()
+ break
+ if history is None:
+ history = ExplorationHistory()
+
+ # For keeping track of the best iterate
+ best_res_norm = np.inf
+ best_it = 0
+ best_x = np.copy(x)
+ best_s = np.zeros_like(x, dtype=bool)
+ loop = False
+ for it in range(n_iter):
+ x -= pas * g # gradient step
+ s = find_support(x, n_nonzero) # Support selection
+
+ hist = history.get(s, it=it)
+ if hist is None: # Optimization in support space
+ x = optimize(linop, x, y, alpha, s, n_iter, rel_tol, optimizer, f, grad_f, is_mse)
+ g = grad_f(x, linop) # gradient in x
+ loss = f(x, linop)
+ history.add(s, x, g, loss, it, copy_grad=False)
+ else:
+ x, g, loss = hist
+ loop = True # If we come back to an already visited support, we are looping
+ if g is None:
+ g = grad_f(x * s, linop)
+ history.add(s, x, g, loss, it, copy_x=False, copy_grad=False)
+
+ res_norm.append(loss)
+
+ if res_norm[-1] < best_res_norm: # Keep best iteration
+ best_res_norm = res_norm[-1]
+ np.copyto(best_x, x)
+ np.copyto(best_s, s)
+ best_it = it
+
+ # Stop when the algorithm is stuck in a local minimum
+ if (res_norm[-2] - res_norm[-1]) / res_norm[-2] < rel_tol or loop:
+ break
+
+ best_res_list = res_norm[:best_it + 2]
+ history.close_exploration(it) # noqa
+
+ # Adding history to output
+ if return_history:
+ results = (x, res_norm, history) # noqa
+ best_results = (best_x, best_res_list, history)
+ else:
+ results = (x, res_norm) # noqa
+ best_results = (best_x, best_res_list)
+
+ # Returning results
+ if return_both:
+ return results, best_results
+ elif return_best:
+ return best_results
+ else:
+ return results
+
+
+@normalizer
+def rea(linop, y, n_nonzero, n_iter=0, alpha=0.9, rel_tol=-np.inf, optimizer='cg', f=None, grad_f=None, is_mse=True,
+ random_seed=0
+ ) -> Tuple[np.ndarray, List[float]]:
+ """
+ Use Random Exploration Algorithm (REA) algorithm for solving: min_x f(x) w.r.t ||X||_0 <= n_nonzero
+
+ :param (sksea.utils.AbstractLinearOperator) linop: Linear operator representing the D matrix
+ :param (np.ndarray) y: Target vector
+ :param (int) n_nonzero: Size of the wanted support
+ :param (int) n_iter: Maximum number of iteration for the algorithm
+ :param (float) alpha: Step size in the gradient descent of intra-support optimisation is `alpha / l`
+ with `l` the Lipschitz constant of linop.
+ :param (float) rel_tol: The algorithm stops when the iterations relative difference is lower than rel_tol
+ :param (str) optimizer: Optimizer to use.
+ - If 'cg', use Conjugate Gradient descent algorithm.
+ - If 'hmgd', use a HandMade Gradient Descent algorithm
+ :param (Callable[[np.ndarray, Optional[np.ndarray]], float]) f: Loss to minimize.
+ The first argument is the vector to use for the evaluation. The second is the support of the evaluation.
+ :param (Callable[[np.ndarray, Optional[np.ndarray]], np.ndarray]) grad_f: Gradient of the loss to minimize.
+ The first argument is the vector to use for the evaluation. The second is the support of the evaluation.
+ :param (bool) is_mse: If True, use better optimisation algorithms (linear conjugate gradient)
+ for solving min_x 1/2 ||D * x - y||_2^2 w.r.t ||X||_0 <= n_nonzero instead of using non-linear algorithms
+ :return: The solution vector `x`, the sequence of residuals `res_norm`
+ """
+ x_len = linop.shape[1]
+ x = np.zeros(x_len)
+ x_best = np.zeros_like(x)
+ res_norm_best = f(x, linop)
+ res_norm = []
+ s = np.zeros(x_len, dtype=bool) # Support
+ res_norm.append(f(x, linop))
+
+ rand = np.random.RandomState(seed=random_seed)
+
+ for _ in range(n_iter):
+ # Select support
+ s[:] = False
+ s[rand.permutation(x_len)[:n_nonzero]] = True # Random support selection
+
+ # Optimize
+ x = optimize(linop, x, y, alpha, s, n_iter, rel_tol, optimizer, f, grad_f, is_mse)
+
+ # Store best result
+ res_norm_tmp = f(x, linop)
+ res_norm.append(res_norm_tmp)
+ if res_norm_tmp < res_norm_best:
+ np.copyto(x_best, x)
+ res_norm_best = res_norm_tmp
+ return x_best, res_norm
+
+
+class SEA(RegressorMixin, LinearModel):
+ """
+ SEA implemented with sklearn API
+ """
+
+ def __init__(self, n_nonzero=10, n_iter=100, normalize_matrix=True,
+ random_state=None, optimizer='cg'):
+ """
+ Construct SEA estimator
+
+ :param (int) n_nonzero: Desired number of non-zero entries in the solution
+ :param (int) n_iter: Desired number o iteration of SEA
+ :param (bool) normalize_matrix: Normalize the regressors X before regression by dividing by the l2-norm
+ If True, the regressors X will be normalized before regression by
+ subtracting the mean and dividing by the l2-norm.
+ :param (Union[int, np.random.RandomState, None]) random_state: Random seed for computing spectral norm of X
+ """
+ self.n_nonzero = n_nonzero
+ self.n_iter = n_iter
+ # self.normalize = normalize
+ self.normalize_matrix = normalize_matrix
+ self.random_state = random_state
+ self.optimizer = optimizer
+ # self.fit_intercept = fit_intercept
+ # self.copy_X = copy_X
+
+ def fit(self, X, y) -> 'SEA':
+ """
+ Fit the model using X, y as training data.
+
+ :param (np.ndarray) X: Training data
+ :param (np.ndarray) y: Target values. Will be cast to X's dtype if necessary.
+ """
+ X, y = check_X_y(X, y)
+ # X, y, X_offset, y_offset, X_scale = _preprocess_data(
+ # X, y, self.fit_intercept, self.normalize, self.copy_X
+ # )
+ y: np.ndarray
+ if y.dtype == object:
+ y = y.astype(X.dtype)
+ self.random_state_ = check_random_state(self.random_state)
+ self.linop_ = SparseSupportOperator(X, y, self.random_state_)
+ self.coef_, self.res_norm_, self.exploration_ = sea_fast(self.linop_, y, self.n_nonzero, n_iter=self.n_iter,
+ f=lambda x, linop: np.linalg.norm(linop @ x - y) / 2,
+ grad_f=lambda x, linop: linop.H @ (linop @ x - y),
+ optimizer=self.optimizer, return_best=True,
+ normalize=self.normalize_matrix)
+ self.n_features_in_ = X.shape[1]
+ self.intercept_ = 0.0
+ # self._set_intercept(X_offset, y_offset, X_scale)
+ return self
+
+ # def predict(self, X):
+ # # Check if fit has been called
+ # check_is_fitted(self)
+ # # Input validation
+ # X = check_array(X)
+ # return X @ self.coef_
diff --git a/code/sksea/cholesky.py b/code/sksea/cholesky.py
new file mode 100644
index 0000000000000000000000000000000000000000..9c021e483c8b9fcb7408094603f95ffee27f69b0
--- /dev/null
+++ b/code/sksea/cholesky.py
@@ -0,0 +1,134 @@
+"""
+Functions for updating Cholesky decompositions and speeding up least-square projections
+"""
+
+import numpy as np
+from scipy.linalg import cho_solve, solve_triangular
+
+
+def cho_rk1_update(L, x, inplace_L=True, inplace_x=False):
+ """ Cholesky rank one update (in-place)
+ """
+ if not inplace_L:
+ L = np.copy(L)
+ if not inplace_x:
+ x = np.copy(x)
+ n = x.size
+ for k in range(n):
+ r = np.sqrt(L[k, k]**2 + x[k]**2)
+ c = r / L[k, k]
+ s = x[k] / L[k, k]
+ L[k, k] = r
+ if k < n - 1:
+ L[k+1:n, k] = (L[k+1:n, k] + s * x[k+1:n]) / c
+ x[k+1:n] = c * x[k+1:n] - s * L[k+1:n, k]
+ if not inplace_L:
+ return L
+
+
+def cho_rk1_downdate(L, x, inplace_L=True, inplace_x=False):
+ """ Cholesky rank one downdate (in-place)
+ """
+ if not inplace_L:
+ L = np.copy(L)
+ if not inplace_x:
+ x = np.copy(x)
+ n = x.size
+ for k in range(n):
+ r = np.sqrt(L[k, k]**2 - x[k]**2)
+ c = r / L[k, k]
+ s = x[k] / L[k, k]
+ L[k, k] = r
+ if k < n - 1:
+ L[k+1:n, k] = (L[k+1:n, k] - s * x[k+1:n]) / c
+ x[k+1:n] = c * x[k+1:n] - s * L[k+1:n, k]
+ if not inplace_L:
+ return L
+
+
+def chol_1d_update_old(L, v):
+ """
+ Cholesky 1-dimension update
+
+ Update a Cholesky decomposition of the form $A = L \times L^H$ by
+ computing the Cholesky decomposition of $(A, v[:-1]; v^H)$, i.e., by adding
+ v as a new row and column to A
+ """
+ n = L.shape[0]
+ assert L.shape[1] == n
+ assert v.shape[0] == n + 1
+ assert v.ndim == 1
+ if L is None:
+ return np.array([[np.sqrt(np.real(v[0]))]])
+ L_new = np.zeros((n+1, n+1))
+ L_new[:n, :n] = L
+ for j in range(n):
+ L_new[-1, j] = (v[j] - np.vdot(L_new[j, :j], L_new[-1, :j])) / L_new[j, j]
+ L_new[-1, -1] = np.sqrt(v[-1] - np.sum(np.abs(L_new[-1, :-1])**2))
+ return L_new
+
+
+def chol_1d_update(L, v):
+ """
+ Cholesky 1-dimension update
+
+ Update a Cholesky decomposition of the form $A = L \times L^H$ by
+ computing the Cholesky decomposition of $(A, v[:-1]; v^H)$, i.e., by adding
+ v as a new row and column to A
+ """
+ n = L.shape[0]
+ assert L.shape[1] == n
+ assert v.shape[0] == n + 1
+ assert v.ndim == 1
+ if L is None:
+ return np.array([[np.sqrt(np.real(v[0]))]])
+ L_new = np.zeros((n+1, n+1))
+ L_new[:n, :n] = L
+ if n != 0:
+ L_new[-1, :-1] = solve_triangular(L, v[:-1], lower=True, check_finite=False)
+ L_new[-1, -1] = np.sqrt(v[-1] - np.sum(np.abs(L_new[-1, :-1])**2))
+ return L_new
+
+
+def chol_1d_downdate(L, i):
+ """
+ Cholesky 1-dimension downdate
+
+ Update a Cholesky decomposition of the form $A = L \times L^H$ by
+ computing the Cholesky decomposition of the matrix obtained from $A$ by
+ removing row $i$ and column $i$.
+ """
+ L_new = np.delete(np.delete(L, i, axis=0), i, axis=1)
+ cho_rk1_update(L_new[i:, i:], L[i+1:, i], inplace_L=True, inplace_x=False)
+ return L_new
+
+
+def chol_1d_downdate_inplace_version(L, atom_indice):
+ """
+ Cholesky 1-dimension downdate
+ """
+ # https://en.wikipedia.org/w/index.php?title=Cholesky_decomposition#Adding_and_removing_rows_and_columns
+ n = L.shape[0]
+ assert L.shape[1] == n
+
+ L_new = np.delete(L, atom_indice, axis=0)
+ x = L_new[:, atom_indice]
+ L_new = np.delete(L_new, atom_indice, axis=1)
+ cho_rk1_update_inplace_version(L_new, x, atom_indice)
+ return L_new
+
+
+def cho_rk1_update_inplace_version(L, x, atom_indice):
+ """
+ Cholesky rank-1 update
+ """
+ # https://en.wikipedia.org/w/index.php?title=Cholesky_decomposition#Rank-one_update
+ n = x.size
+ for k in range(atom_indice, n):
+ r = np.sqrt(L[k, k]**2 + x[k]**2)
+ c = r / L[k, k]
+ s = x[k] / L[k, k]
+ L[k, k] = r
+ if k < n - 1:
+ L[k+1:n, k] = (L[k+1:n, k] + s * x[k+1:n]) / c
+ x[k+1:n] = c * x[k+1:n] - s * L[k+1:n, k]
diff --git a/code/sksea/dataset_operator.py b/code/sksea/dataset_operator.py
new file mode 100644
index 0000000000000000000000000000000000000000..79ae9f7d92b66a1216e5dcfdb9c20c28d74e79a5
--- /dev/null
+++ b/code/sksea/dataset_operator.py
@@ -0,0 +1,187 @@
+# Python imports
+from enum import Enum
+import inspect
+import os
+from pathlib import Path
+from typing import List, Tuple
+import zipfile
+
+# Module imports
+import gdown
+import numpy as np
+import pandas as pd
+from loguru import logger
+from sklearn.preprocessing import normalize
+
+# Script imports
+from sksea.sparse_coding import SparseSupportOperator
+
+ROOT = Path(os.path.abspath(inspect.getfile(inspect.currentframe()))).parent
+RESULT_PATH = ROOT / "results/training_tasks"
+
+
+class Task(Enum):
+ """
+ Enumerator for type of tasks
+ """
+ REGRESSION = 0
+ BINARY = 1
+
+
+class DatasetOperator:
+ """
+ Class for handling datasets as linear operators
+ """
+ _DATASETS_DL_PATH = ROOT / "downloads/datasets"
+ _DATASETS_PATH = ROOT / "datasets"
+ REGRESSIONS_NAME = ('cal_housing', 'comp-activ-harder', 'slice', 'year')
+ BINARY_NAME = ('letter', 'ijcnn1', 'kddcup04_bio', 'census')
+ K_MAX = {
+ 'kddcup04_bio': 10,
+ 'cal_housing': 9,
+ 'census': 12,
+ 'comp-activ-harder': 12,
+ 'ijcnn1': 14,
+ 'letter': 16,
+ 'slice': 40,
+ 'year': 40
+ }
+ MAX_K_MAX = max(K_MAX.values())
+ ORDER = ('cal_housing', 'letter', 'comp-activ-harder', 'ijcnn1', 'kddcup04_bio', 'year', 'slice', 'census')
+
+ def __init__(self, name, no_loading=False):
+ """
+
+ :param (str) name: Name of the dataset to load
+ :param (bool) no_loading: If True, don't load the dataset.
+ A call to load method will be needed before any computation
+ """
+ self.grad_f = None
+ self.f = None
+ self.linop = None
+ self.y = None
+ namelist = self.get_dataset_names()
+ if name not in namelist:
+ raise ValueError(f'Name must be in {namelist}')
+ self.name = name
+ self.k_max = self.K_MAX[name]
+
+ self.task = Task.REGRESSION if name in self.REGRESSIONS_NAME else Task.BINARY
+ self._loaded = False
+ if not no_loading:
+ self.load()
+
+ def load(self):
+ """
+ Load dataset in memory
+ """
+ if not self._loaded:
+ data_mat, self.y = self._load_and_preprocess()
+ self.linop = SparseSupportOperator(data_mat, self.y)
+
+ if self.task == Task.REGRESSION:
+ self.f = lambda x, linop_s=self.linop: (np.linalg.norm(linop_s @ x - self.y) ** 2) / 2
+ self.grad_f = lambda x, linop_s=self.linop: linop_s.H @ (linop_s @ x - self.y)
+ else:
+ from scipy.special import expit # For parallel computation without importation error
+ self.f = lambda x, linop_s=self.linop: -self.y.T @ np.log(expit(linop_s @ x)
+ ) - (1 - self.y).T @ np.log(
+ 1 - expit(linop_s @ x
+ ))
+ self.grad_f = lambda x, linop_s=self.linop: linop_s.H @ (expit(linop_s @ x) - self.y)
+ self._loaded = True
+
+ @classmethod
+ def get_dataset_names(cls) -> List[str]:
+ """
+ Return the list of available dataset names
+ """
+ cls._check_download()
+ return [path.stem for path in cls._DATASETS_PATH.glob('*.json')]
+
+ def _load_and_preprocess(self) -> Tuple[np.ndarray, np.ndarray]:
+ """
+ Load json file and preprocess according to the paper
+
+ :return: Data matrix and labels
+ """
+ df = pd.read_json(self._DATASETS_PATH / f'{self.name}.json').fillna(0)
+ y = df.label.to_numpy()
+ if self.task == Task.BINARY: # Put labels from {-1, 1} to {0, 1}
+ y = np.fmax(0, y)
+ x_df = df.drop(columns='label')
+ x_df["intercept"] = 1
+ x = normalize(x_df.to_numpy(), axis=0)
+ return x, y
+
+ @classmethod
+ def _check_download(cls) -> None:
+ """
+ Check that the datasets are downloaded and converted
+ """
+ if not cls._DATASETS_PATH.is_dir() or len(list(cls._DATASETS_PATH.glob('*.json'))) < 8:
+ cls._download_datasets()
+ cls._convert_datasets()
+
+ @classmethod
+ def _download_datasets(cls) -> None:
+ """
+ Download datasets from Google Drive, extract them
+ """
+ archive_path = ROOT / "downloads/datasets.zip"
+ archive_path.parent.mkdir(exist_ok=True)
+ logger.info("Downloading datasets")
+ gdown.download("https://drive.google.com/uc?id=1RDu2d46qGLI77AzliBQleSsB5WwF83TF", str(archive_path))
+ if not archive_path.is_file():
+ raise FileNotFoundError("The dataset cannot be download automatically from Google Drive. "
+ "Please download it manualy from "
+ "https://drive.google.com/file/d/1RDu2d46qGLI77AzliBQleSsB5WwF83TF/view, "
+ "and place it in code/sksea/downloads/datasets.zip. "
+ "Then, launch this command again.")
+ with zipfile.ZipFile(archive_path, 'r') as zip_ref:
+ zip_ref.extractall(cls._DATASETS_DL_PATH)
+
+ @classmethod
+ def _convert_datasets(cls) -> None:
+ """
+ Convert .vm file to .json using vm input format for simple datasets.
+ Vm format: https://github.com/VowpalWabbit/vowpal_wabbit/wiki/Input-format
+ """
+ cls._DATASETS_PATH.mkdir(exist_ok=True)
+ for filepath in cls._DATASETS_DL_PATH.glob('*.vw'):
+ logger.info(f'Converting {filepath.name}')
+ out = ['[']
+
+ # Read VM file
+ with open(filepath, 'r') as file:
+ lines = file.readlines()
+
+ # Convert to JSON structure
+ for line in lines:
+ line_stripped = line.strip().replace(' |f', '').replace('+1', '1') # Remove namespace
+ line_labelled = f'label:{line_stripped}' # Add label title
+ features = line_labelled.split(' ') # Split features
+ features_corrected = []
+
+ for feat in features:
+ if ':' not in feat: # Add default value : 1
+ feat += ':1'
+ key, value = feat.split(':')
+ features_corrected.append(f'"{key}":{value}') # Add " " to keys
+ line_with_default = ', '.join(features_corrected) # Get the JSON line together
+ out.append('{' + line_with_default + '},\n')
+
+ # Remove side effects
+ out[-1] = out[-1].strip()[:-1]
+ out.append(']')
+
+ # Save to JSON file
+ json_path = cls._DATASETS_PATH / filepath.with_suffix('.json').name
+ with open(json_path, 'w') as file:
+ file.writelines(out)
+
+ def __repr__(self) -> str:
+ """
+ Add the name of the dataset to the object representation
+ """
+ return super(DatasetOperator, self).__repr__() + ' - ' + self.name
diff --git a/code/sksea/deconvolution.py b/code/sksea/deconvolution.py
new file mode 100644
index 0000000000000000000000000000000000000000..cf2231a81570f0a73e327f03811c3ef56c25299e
--- /dev/null
+++ b/code/sksea/deconvolution.py
@@ -0,0 +1,222 @@
+# -*- coding: utf-8 -*-
+# Python imports
+from typing import Tuple
+
+# Module imports
+import numpy as np
+from numpy.random import RandomState
+from scipy.signal import convolve2d
+
+# Script imports
+from sksea.utils import find_support, AbstractLinearOperator
+from sksea.algorithms import iht, ista, sea
+
+
+class ConvolutionOperator(AbstractLinearOperator):
+ """
+ Convolution operator able to be used on restricted supports
+ Here, the 1D convolution is done by using the 2D convolution of scipy
+ """
+
+ def __init__(self, filter_conv, input_len, support=None, seed=0):
+ """
+ Linear Operator for convolutions
+
+ :param (np.ndarray) filter_conv: Filter of the convolution
+ :param (int) input_len: Size of the wanted input of the filter
+ :param (np.ndarray or None) support: If provided, upscale the provided vectors to R^{input_len}
+ and restrict the output vector of the adjoint operator to R^{support_size}
+ """
+ self._filter = filter_conv[None, :]
+ # Mapping for computing the convolution only on the provided support
+ self.support = np.ones(input_len, dtype=bool) if support is None else support
+ if support is not None and support.shape[0] != input_len:
+ raise AssertionError(f"The support size and input_len must have the same value,"
+ f" {support.shape[0]} != {input_len}")
+ self.upscaled_x = np.empty_like(self.support, dtype=float)
+ self.seed = seed
+
+ super().__init__(dtype=filter_conv.dtype, shape=[input_len, self.support.sum()])
+
+ def _matvec(self, x) -> np.ndarray:
+ """
+ Compute self @ x
+
+ :param (np.ndarray) x: Input vector of support size
+ :return: Output vector of input_len (size of the whole space) size
+ """
+ # Upscale x to the full space size
+ if x.ndim == 2:
+ x = x.flatten()
+ self.upscaled_x[self.support] = x.view() # Tentative infructueuse d'optimisation de performances
+ self.upscaled_x[~self.support] = 0
+
+ # Use the upscaled version of x for all computations
+ return convolve2d(self.upscaled_x[None, :], self._filter, 'same', 'wrap')[0]
+
+ def _rmatvec(self, x) -> np.ndarray:
+ """
+ Compute self.H @ x
+
+ :param (np.ndarray) x: Input vector of input_len (size of the whole space) size
+ :return: Output vector of support size
+ """
+ out = convolve2d(x[None, :], self._filter[:, ::-1].conj(), 'same', 'wrap')[0]
+ return out[self.support] # Restrict the output of the adjoint in the support
+
+ @property
+ def filter(self) -> np.ndarray:
+ """
+
+ :return:
+ """
+ return np.copy(self._filter[0])
+
+ def get_operator_on_support(self, s) -> 'ConvolutionOperator':
+ """
+ Return the operator truncated on the provided support
+
+ :param (np.ndarray) s: Support
+ """
+ return ConvolutionOperator(self.filter, self.shape[0], s, self.seed)
+
+ def get_normalized_operator(self) -> Tuple['ConvolutionOperator', np.ndarray]:
+ """
+ Return a normalized version of the operator using its kernel
+ """
+ norms = np.linalg.norm(self.filter)
+ return ConvolutionOperator(self.filter / norms, self.shape[0], self.support, self.seed), np.ones(self.shape[0]) / norms
+
+
+def gen_u(N, n, u_type=2, rand=RandomState(), max_size=None) -> np.ndarray:
+ """
+ Generate a signal
+
+ :param (int) N: Size of the signal
+ :param (int) n: Behaviour depends of u_type.
+ 2: Sparsity of the signal
+ Else: Position of the dirac
+ :param (int) u_type: Type of the signal.
+ 1: Dirac of size 1 in position n.
+ 2: n-sparse random signal from N(0,1).
+ 3: Dirac of random size from U(0.1,10) in position n.
+ :param (RandomState) rand: Numpy random state
+ :return: Signal of size N
+ """
+ u = np.zeros(N) # par defaut des 0
+
+ if max_size is None:
+ max_size = N
+
+ if u_type == 1: # un dirac en n
+ if 0 <= n < N:
+ u[n] = 1
+ elif u_type == 2: # un signal n-sparse
+ u = rand.normal(0, 1, N)
+ x = rand.normal(0, 1, N)
+ s = find_support(x, n)
+ u = u * s
+ elif u_type == 3: # Dirac of random size in position n
+ u[n] = rand.uniform(0.1, 10)
+ elif u_type == 4: # Uniform on [1, 2]U[-2, -1] with support space constrained by max_size
+ s = rand.permutation(max_size)[:n]
+ u[s] = rand.rand(n) + 1
+ u[s] *= (-1) ** rand.randint(0, 2, n)
+ return u
+
+
+def gen_filter(N, h_type, sigma=1) -> np.ndarray:
+ """
+ Generate a convolution filter
+
+ :param (int) N: Size of the filter
+ :param (int) h_type: Type of the filter.
+ 0: Random from N(0, 1).
+ 1: One everywhere.
+ 2: Dirac of size 1 in the middle.
+ 3: Gaussian filter of variance sigma.
+ 4: Ricker filter with sigma as parameter.
+ Else: Zero filter.
+ :param (float) sigma: Parameter for Gaussian and Ricker filters
+ :return: Filter of size N
+ """
+ half_N = N // 2
+ if h_type == 0: # aleatoire
+ h = np.random.normal(0, 1, N)
+ elif h_type == 1: # 1
+ h = np.ones(N)
+ elif h_type == 2: # un dirac
+ h = gen_u(N, half_N, 1)
+ elif h_type == 3: # gaussien
+ t = np.arange(-half_N, half_N + 1, 1)
+ h = np.exp(-(t / sigma) ** 2 / 2)
+ elif h_type == 4: # Ricker
+ t = np.arange(-half_N, half_N + 1, 1)
+ h = (1 - (t / sigma) ** 2) * np.exp(-(t / sigma) ** 2 / 2)
+ else:
+ h = np.zeros(N) # defaut
+ return h
+
+
+if __name__ == '__main__':
+ import numpy as np
+ import matplotlib.pyplot as plt
+
+ from sksea.algorithms import sea, iht as palm
+
+ # palm vs algo avec x_bar
+ x_len = 64
+ h_len = 10
+ p = 0.30 # pourcentage de données manquantes
+ n_nonzero = int(np.ceil(p * x_len))
+ sigma = 0.5
+ n_iter = 1000
+
+ # generation des donnees
+ x_true = 10 * gen_u(x_len, n_nonzero, 2)
+ # h = gen_filter(Nh,2) #Pour debugger, on prend h = un dirac
+ h = gen_filter(h_len, 4) # Pour trouver des difficiles, on prend h = 1,
+ # nh=6, avec Nx = 10
+ h_op = ConvolutionOperator(filter_conv=h, input_len=x_len)
+ y = h_op @ x_true + sigma * np.random.normal(0, 1, x_len)
+ f = lambda x, linop: np.linalg.norm(linop @ x - y)
+ grad_f = lambda x, linop: linop.H @ (linop @ x - y)
+ print(h)
+
+ # reconstruction
+ x = {}
+ res_norm = {}
+ x['IHT'], res_norm['IHT'] = iht(linop=h_op, y=y, n_nonzero=n_nonzero, n_iter=n_iter, f=f, grad_f=grad_f)
+ x['sea'], res_norm['sea'] = sea(h_op, y, n_nonzero, n_iter, return_best=True, f=f, grad_f=grad_f,
+ optimize_sea="all")
+
+ print("erreur IHT = ", np.linalg.norm(x['IHT'] - x_true))
+ print("erreur sea = ", np.linalg.norm(x['sea'] - x_true))
+
+ plt.figure()
+ plt.plot(y, "r", label="y")
+ plt.plot(h_op @ x_true, "g:", label="h*x_true")
+ plt.plot(h_op @ x['IHT'], "b:o", label="h*x_iht")
+ plt.plot(h_op @ x['sea'], "r:o", label="h*x_sea")
+ plt.legend()
+ plt.savefig('figures/deconvolution_signals.pdf')
+ #######################
+
+ plt.figure()
+ plt.plot(x_true, "g-", label="x_true")
+ plt.plot(x['IHT'], "b:o", label="x_iht")
+ plt.plot(x['sea'], "r:o", label="x_sea")
+ plt.legend()
+ plt.savefig('figures/deconvolution_sparse_vectors.pdf')
+
+ plt.figure()
+ for k in res_norm:
+ if res_norm[k] is None:
+ continue
+ plt.loglog(res_norm[k], label=k)
+ plt.xlabel('Iterations')
+ plt.ylabel('Norm of residue')
+ plt.legend()
+ plt.savefig('figures/deconvolution_res_norm.pdf')
+
+ plt.show()
diff --git a/code/sksea/exp_deconv.py b/code/sksea/exp_deconv.py
new file mode 100644
index 0000000000000000000000000000000000000000..1fd97129ba9cfc0c356731960e0a4a027dd3c656
--- /dev/null
+++ b/code/sksea/exp_deconv.py
@@ -0,0 +1,487 @@
+# Python import
+import click
+from collections import defaultdict
+from itertools import combinations
+from pathlib import Path
+import pickle
+from typing import Tuple, List, Dict
+
+# Modules imports
+import numpy as np
+import plotly.express as px
+import plotly.graph_objs as go
+from plotly.subplots import make_subplots
+from tabulate import tabulate
+from tqdm import tqdm
+
+# Scripts imports
+from sksea.algorithms import ExplorationHistory
+from sksea.deconvolution import ConvolutionOperator, gen_filter, gen_u
+from sksea.exp_phase_transition_diag import NoiseType, gen_noise
+from sksea.plot_icml import iterations_dcv, iterations_sup_dcv, plot_signal_paper, plot_signal_paper_full, \
+ plot_signal_paper_light, get_best_hist
+from sksea.utils import PAPER_LAYOUT, ALGOS_PAPER_DCV_PRECISE
+from sksea.training_tasks import select_algo, ALGOS_TYPE
+
+
+def solve_deconv_problem(h_op, x_len, spike_pos, seed, n_iter, manual=True, noise_factor=None, noise_type=None
+ ) -> Tuple[
+ Dict[str, np.ndarray], Dict[str, np.ndarray], Dict[str, float], Dict[str, 'ExplorationHistory']]:
+ """
+ It generates a signal with a few spikes, convolves it with a random matrix,
+ and then tries to recover the spikes using a few algorithms
+
+ :param (ConvolutionOperator) h_op: the linear operator that maps the signal to the observation
+ :param (int) x_len: length of the signal
+ :param (Union[List[int], Tuple[int]]) spike_pos: the position of the spikes in the signal
+ :param (int) seed: the seed for the random number generator
+ :param (int) n_iter: number of iterations to run the algorithm for
+ :return: A dictionary of the reference solutions and a dictionary of the solutions found by the algorithms.
+ """
+ # Generate signal
+ rand = np.random.RandomState(seed)
+ if manual:
+ x = np.zeros(x_len)
+ for pos in spike_pos:
+ x += gen_u(x_len, pos, 3, rand)
+ else:
+ x = gen_u(x_len, len(spike_pos), 4, rand)
+
+ y = h_op(x) # Generate observation
+ if noise_factor is not None:
+ y += gen_noise(noise_type, noise_factor, y, rand)
+ # Get algorithms we want to run
+ algos_studied = [
+ "ELSFAST", "OMPRFAST", "OMPFAST",
+ "HTPFAST", "HTPFAST-els", "IHT", "IHT-els", "IHT-omp", "HTPFAST-omp",
+ "SEAFAST", "SEAFAST-els", "SEAFAST-omp"
+ ]
+ algorithms = select_algo(ALGOS_TYPE[:4],
+ algos_studied,
+ sea_params=dict(return_both=True))
+ solutions = {}
+ refs = {f"x_{spike_pos}": x, f"y_{spike_pos}": y}
+ best_res = {}
+ res_norms = {}
+ histories = {}
+
+ # Solve problem with algorithms
+ for name, algorithm in algorithms.items():
+ label = f"{name}_{spike_pos}"
+ out = algorithm(linop=h_op, y=y, n_nonzero=len(spike_pos), n_iter=n_iter,
+ f=lambda x, linop: np.linalg.norm(linop @ x - y),
+ grad_f=lambda x, linop: linop.H @ (linop @ x - y))
+
+ if "SEA" in name or "HTP" in name:
+ (_, res_norm, hist), (x_est, res_norm_best, _) = out
+ histories[label] = hist
+ best_res[label] = res_norm_best[-1] / np.linalg.norm(y)
+ else:
+ if "FAST".lower() in name.lower():
+ x_est, res_norm, hist = out
+ histories[label] = hist
+ else:
+ x_est, res_norm = out
+ best_res[label] = res_norm[-1] / np.linalg.norm(y)
+
+ solutions[label] = x_est
+ res_norms[label] = res_norm / np.linalg.norm(y)
+ return refs, solutions, best_res, res_norms, histories
+
+
+def plot_results(refs, solutions, best_res, h, n_solutions, pos_list, out_file) -> None:
+ """
+ Plots the results of the deconvolution experiments
+
+ :param (Dict[str, np.ndarray]) solutions: a dictionary of the form {name_pos: x_sol} where name_pos is the name of
+ the algorithm and the position of the spikes, and x_sol is the solution of the algorithm
+ :param (Dict[str, np.ndarray]) refs: a dictionary of reference signals (the true signals)
+ :param (np.ndarray) h: the filter
+ :param (int) n_solutions: number of algorithm used
+ :param (List[Union[List[int], Tuple[int]]]) pos_list: list of spike positions
+ :param (Union[str, Path]) out_file: the path to the output file
+ """
+ # Color scheme for curves
+ colors = px.colors.qualitative.Plotly
+ colors1 = colors[:2]
+ colors2 = colors[2:]
+
+ fig = make_subplots(specs=[[{"secondary_y": True}]])
+
+ # Plot solutions of each algo
+ for idx, (name_pos, x_sol) in enumerate(solutions.items()):
+ name = name_pos.split('_')[0]
+ fig.add_trace(go.Scatter(y=x_sol, line=dict(color=colors2[(idx % n_solutions) % len(colors2)]),
+ name=name, legendgroup=name, visible=True),
+ secondary_y=False)
+
+ # Plot signal and observation
+ for idx, (name_pos, x_sol) in enumerate(refs.items()):
+ name = name_pos.split('_')[0]
+ fig.add_trace(go.Scatter(y=x_sol, line=dict(color=colors1[idx % len(colors1)], dash='dash'),
+ name=name, visible=True, legendgroup=name),
+ secondary_y=False)
+ fig.add_trace(go.Scatter(y=h, line=dict(color=colors[0]), name="filter"), secondary_y=True)
+
+ # Slider (start) -------------------------- https://plotly.com/python/sliders/
+ steps = []
+ n_data = len(fig.data)
+ n = (n_data - 1 - len(pos_list) * len(colors1)) // (len(pos_list))
+ n_steps = len(pos_list)
+ for idx, pos in enumerate(pos_list):
+ performances = [(name_pos.split('_')[0], res)
+ for i, (name_pos, res) in enumerate(best_res.items())
+ ] # if idx * n <= i < (idx + 1) * n]
+ performances.sort(key=lambda couple: couple[1])
+ annotation = tabulate(performances, headers=["Algorithm", "Best relative loss"],
+ tablefmt="github").replace("\n", "<br>")
+ step = dict(
+ # Update method allows us to update both trace and layout properties
+ method='update',
+ args=[
+ # Make the traces from `pos` visible
+ {'visible': [idx * n <= i < (idx + 1) * n for i in range(n_steps * n) # Get solution (x) for each algo
+ ] + [idx * len(colors1) <= i - n_steps * n < (idx + 1) * len(colors1)
+ for i in range(n_steps * n, n_data - 1) # Get y_true and x_true for each algo
+ ] + [True]}, # Keep the filter displayed
+
+ # Set the title for the these traces
+ {'title.text': f"Deconvolution experiments for spikes at {pos}",
+ # https://community.plotly.com/t/plotly-py-graph-object-updating-annotation-with-slider/52639
+ 'annotations': [
+ go.layout.Annotation(
+ align='left', # Align text to the left
+ yanchor='top', # Align text box's top edge
+ text=annotation, # Set text with '<br>' strings as newlines
+ showarrow=False, # Hide arrow head
+ width=400, # Wrap text at around 300 pixels
+ xref='paper', # Place relative to figure, not axes
+ yref='paper',
+ font={'family': 'Courier'}, # Use monospace font to keep nice indentation
+ x=1, # Place on right edge
+ y=1 # Place on the top edge
+ )
+ ]}],
+ label=str(pos)
+ )
+ steps.append(step)
+ sliders = [go.layout.Slider(
+ active=1,
+ # len=0.5,
+ currentvalue={"prefix": "Spike positions: "},
+ steps=steps
+ )]
+ # Slider (stop) --------------------------
+
+ fig.update_layout(
+ title="Deconvolution experiment. Use the slider to display the figures.",
+ legend_title="Algorithms",
+ sliders=sliders,
+ )
+ # Set y-axes titles
+ fig.update_yaxes(title_text="Signals", secondary_y=False)
+ fig.update_yaxes(title_text="Filter", secondary_y=True)
+
+ out_file = Path(out_file)
+ out_file.parent.mkdir(exist_ok=True, parents=True)
+ fig.write_html(out_file)
+
+
+def pad_signal(signal):
+ y_pad = []
+ x_pad = []
+ y_scat = []
+ x_scat = []
+ for idx, val in enumerate(signal):
+ if val == 0:
+ y_pad.append(0)
+ x_pad.append(idx)
+ else:
+ y_pad.extend((0, val, 0))
+ x_pad.extend((idx, idx, idx))
+ y_scat.append(val)
+ x_scat.append(idx)
+ return dict(y=y_pad, x=x_pad), dict(y=y_scat, x=x_scat)
+
+
+def plot_results_paper(refs, solutions, out_file, nips=False):
+ fig = go.Figure()
+
+ mapping_nips = {
+ "IHT": "$\\text{IHT}$",
+ "HTP": "$\\text{HTP}$",
+ "ELS": "$\\text{ELS, OMP, OMPR}, \\text{IHT}_{\\text{OMP}}, \\text{HTP}_{\\text{ELS}}$",
+ "SEAFAST-els": "$\\text{SEA}_0, \\text{SEA}_{\\text{OMP}}, \\text{SEA}_{\\text{ELS}}$"
+ }
+
+ size = 15
+ size_x = 17
+ width = 1
+ marker_line_width = 1
+
+ grey_level = 245
+ fig.add_trace(go.Scatter(y=[-7, 7, 7, -7, -7], x=[378, 378, 439, 439, 378],
+ fill='toself', fillcolor=f'rgb({grey_level},{grey_level},{grey_level})',
+ line_color='rgba(0,0,0,0)', showlegend=False))
+
+ legendrank = defaultdict(lambda: 1000)
+ legendrank.update(
+ {
+ "ELS": 1001,
+ "SEAFAST-els": 1002,
+ "IHT": 1000,
+ "HTP": 1000,
+ })
+ # Plot signal and observation
+ for idx, (name_pos, x_sol) in enumerate(refs.items()):
+ name = name_pos.split('_')[0]
+ if name_pos.startswith('x'):
+ continue
+ else:
+ fig.add_trace(go.Scatter(**dict(y=x_sol), line=dict(color='#24405e', width=2), # , dash='dash'),
+ name=f"${name}$"))
+
+ # Plot solutions of each algo
+ for idx, (name_pos, x_sol) in enumerate(list(solutions.items())[::-1]):
+ name = name_pos.split('_')[0]
+ try:
+ line, points = pad_signal(x_sol)
+ info = ALGOS_PAPER_DCV_PRECISE[name]
+ display_name = mapping_nips[name] if nips else info["name"]
+ marker_info = info["marker"]
+ marker_info.update(dict(size=size, line_width=marker_line_width))
+ line_info = info["line"]
+ line_info.update(dict(width=width))
+ fig.add_trace(go.Scatter(**line, line=line_info, showlegend=False,
+ name=display_name, legendgroup=info["name"], legendrank=legendrank[name]))
+ fig.add_trace(go.Scatter(**points, line=line_info, mode='markers', marker=marker_info,
+ name=display_name, legendgroup=info["name"], legendrank=legendrank[name]))
+ except Exception as e:
+ print(f"Do not plot {name} in the paper version of the figure")
+
+ # Plot signal and observation
+ for idx, (name_pos, x_sol) in enumerate(refs.items()):
+ name = name_pos.split('_')[0]
+ if name_pos.startswith('x'):
+ line, points = pad_signal(x_sol)
+ fig.add_trace(go.Scatter(**line, line=dict(color='#FF6692', dash='dash', width=width),
+ name=name, showlegend=False, legendgroup=name))
+ fig.add_trace(go.Scatter(**points, line=dict(color='#FF6692', dash='dash'),
+ mode='markers', marker=dict(symbol=100, size=size_x, line_width=marker_line_width),
+ name=r"$x^*$", legendgroup=name, legendrank=999))
+ else:
+ continue
+
+ fig.update_layout(
+ **PAPER_LAYOUT
+ )
+ tick_array = np.arange(19, 500, 20)
+ legend_position1 = dict(
+ y=1.05,
+ x=0,
+ xanchor="left",
+ yanchor="top",
+ bgcolor='rgba(0,0,0,0)'
+ )
+ fig.update_layout(
+ xaxis=dict(
+ showgrid=False,
+ showline=False,
+ position=0.515,
+ anchor="free",
+ ticks="outside",
+ tickvals=tick_array,
+ tickmode="array",
+ ticktext=[f"{i + 1 if (i + 1) % 100 == 0 and (i + 1) != 300 else ''}" for i in tick_array],
+ nticks=50,
+ tickwidth=2,
+ ticklen=10,
+ showticklabels=True,
+ ),
+ margin=dict(
+ autoexpand=False,
+ r=105,
+ l=30
+ ),
+ legend=dict(
+ title=None,
+ orientation='h',
+ **legend_position1,
+ entrywidth=0
+ ),
+ showlegend=True,
+ )
+
+ out_file = Path(out_file)
+ out_file.parent.mkdir(exist_ok=True, parents=True)
+ fig.write_html(out_file, include_mathjax='cdn')
+ fig.update_layout(showlegend=False)
+ fig.write_html(out_file.parent / (out_file.stem + "_no_legend.html"), include_mathjax='cdn')
+
+
+def plot_sea_loss(res_norms, out_dir, loss_fig):
+ # algo_order = ["SEAFAST-els", "SEAFAST", "HTP", "IHT",]
+ # legend_ranks = ["HTP", "IHT", "SEAFAST", "SEAFAST-els"]
+ algo_order = ["SEAFAST-els", "SEAFAST", "HTPFAST", "ELSFAST", ]
+ legend_ranks = ["HTPFAST", "ELSFAST", "SEAFAST", "SEAFAST-els"]
+ colors = ["#EF553B", "#19D3F3", "#FFA15A", "#636EFA", ]
+ keys_list = list(res_norms.keys())
+ keys_list.sort(key=lambda key: algo_order.index(key.split('_')[0]))
+ for idx_algo, name_pos in enumerate(keys_list):
+ loss = res_norms[name_pos]
+ fig = go.Figure()
+ name = name_pos.split('_')[0]
+ info = ALGOS_PAPER_DCV_PRECISE[name]
+ for idx, current_fig in enumerate((fig, loss_fig)):
+ if hasattr(loss, "shape") and loss.shape[0] < 1000:
+ loss = np.pad(loss, (0, 1000 - loss.shape[0]), 'edge')
+ current_fig.add_trace(go.Scatter(y=loss,
+ name=r"$x^t$" if idx == 0 else r"$x^t_{" + info["name"][1:-1] + "}$",
+ legendrank=legend_ranks.index(name),
+ line={"width": 1.5, "color": colors[idx_algo]}))
+ if "SEA" in name or "ELS" in name:
+ color = "indigo" if "els" in name_pos else "blue" if "ELS" in name else "green"
+ current_fig.add_trace(go.Scatter(y=get_best_hist(loss),
+ line={"dash": "dash", "width": 1.5, "color": color},
+ name=r"$x^{t_{\text{BEST}}}_{" + info["name"][1:-1] + "}$",
+ legendrank=legend_ranks.index(name)))
+ fig.update_layout(
+ **PAPER_LAYOUT,
+ yaxis_title=r"$\ell_{2, \text{rel}}$",
+ xaxis_title="Explored supports" if str(out_dir).endswith("hist") else "Iterations",
+ legend=dict(
+ y=0.95,
+ x=0.95,
+ xanchor="right",
+ )
+ )
+ fig.update_layout(
+ legend_title=r"",
+ )
+ out_file = out_dir / f"{info['disp_name']}_{'hist' if str(out_dir).endswith('hist') else 'losses'}.html"
+ out_file.parent.mkdir(exist_ok=True, parents=True)
+ fig.write_html(out_file, include_mathjax='cdn')
+
+
+def run_and_plot_experiment(x_len, h_len, sigma, n_spikes, seed, n_iter, noise_factor, noise_type, expe_name):
+ # Generate filter
+ h_len = x_len if h_len is None else h_len
+ h = gen_filter(h_len, 3, sigma)
+ h_op = ConvolutionOperator(h, x_len)
+
+ # Generate spike position list
+ manual = True
+ # if n_spikes == 2:
+ # begin, end = (25, 39)
+ # pos_list = [(begin, end - k) for k in range(end - begin)]
+ # elif n_spikes == 3:
+ # begin, end = (20, 30)
+ # pos_list = [(begin, *others) for others in combinations(range(begin + 1, end + 1), n_spikes - 1)]
+ if n_spikes >= 1:
+ manual = False
+ begin, end = (None, None)
+ pos_list = [[seed] * n_spikes]
+ else:
+ raise Exception
+
+ root = Path(f"figures/exp_deconv/{expe_name}")
+ root.mkdir(parents=True, exist_ok=True)
+ out_file = root / f"devonc_{seed}_{n_spikes}_{begin}-{end}.html"
+
+ # Solve problems
+ refs = {}
+ solutions = {}
+ best_res = {}
+ res_norms = {}
+ histories = {}
+ for spike_pos in tqdm(pos_list, "Solving multiple combinations of spikes"):
+ refs_pos, solutions_pos, best_res_pos, res_norms_pos, history_pos = solve_deconv_problem(h_op, x_len, spike_pos,
+ seed, n_iter, manual,
+ noise_factor,
+ noise_type)
+ refs.update(refs_pos)
+ solutions.update(solutions_pos)
+ best_res.update(best_res_pos)
+ res_norms.update(res_norms_pos)
+ histories.update(history_pos)
+ n_solutions = len(solutions_pos)
+ # solutions_y = {label: h_op(solution) for label, solution in
+ # solutions.items()} # Get y predicted for each algorithm
+ with open(root / "refs.pkl", "wb") as f:
+ pickle.dump(refs, f)
+ with open(root / "solutions.pkl", "wb") as f:
+ pickle.dump(solutions, f)
+ with open(root / "histories.pkl", "wb") as f:
+ pickle.dump(histories, f)
+ with open(root / "res_norms.pkl", "wb") as f:
+ pickle.dump(res_norms, f)
+ # # Create and save figures
+ # plot_results(refs, solutions, best_res, h, n_solutions, pos_list, out_file)
+ # # plot_results(refs, solutions_y, best_res, h, n_solutions, pos_list,
+ # # out_file.parent / (out_file.stem + "_y" + out_file.suffix))
+ #
+ # # Figure for paper
+ # # sol1, sol2 = {}, {}
+ # sol_nips = {}
+ # res_norms_sea = {}
+ # hist_sea = {}
+ # for alg_name, val in solutions.items():
+ # # print(alg_name)
+ # # if "IHT" in alg_name or "SEAFAST" in alg_name:
+ # # sol1[alg_name] = val
+ # # else:
+ # # sol2[alg_name] = val
+ # if ("SEA" in alg_name or "IHT" == alg_name.split('_')[0] or "HTPFAST" == alg_name.split('_')[0]
+ # or "ELSFAST" == alg_name.split('_')[0]) and 'omp' not in alg_name:
+ # res_norms_sea[alg_name] = res_norms[alg_name]
+ # hist = histories.get(alg_name, None)
+ # if hist is not None:
+ # hist_sea[alg_name] = hist.get_loss_by_explored_support()
+ # if alg_name.split('_')[0] not in ("SEAFAST", "SEAFAST-omp"):
+ # sol_nips[alg_name] = val
+ #
+ # # plot_results_paper(refs, sol1, root / "prec1.html")
+ # # plot_results_paper(refs, sol2, root / "prec2.html")
+ # plot_results_paper(refs, sol_nips, root / "paper_signals.html", nips=True)
+ # # plot_results_paper(refs, {}, root / "prec_pb.html")
+ # loss_fig = go.Figure()
+ # hist_fig = go.Figure()
+ # plot_sea_loss(res_norms_sea, root, loss_fig)
+ # plot_sea_loss(hist_sea, root / "hist", hist_fig)
+ # for fig, fig_name in zip((loss_fig, hist_fig), ("losses", "hist")):
+ # fig.update_layout(
+ # **PAPER_LAYOUT,
+ # yaxis_title=r"$\ell_{2, \text{rel}}$",
+ # xaxis_title="Explored supports" if fig_name == "hist" else "Iterations",
+ # )
+ # fig.update_layout(
+ # legend=dict(title="", y=0.95, x=0.95, xanchor="right", entrywidth=50, orientation='h')
+ # )
+ # fig.write_html(root / f"paper_{fig_name}.html", include_mathjax='cdn')
+
+
+@click.command(context_settings={'show_default': True, 'help_option_names': ['-h', '--help']})
+@click.option('--x_len', '-xl', default=500, help='Size of the signal')
+@click.option('--h_len', '-hl', default=None, type=int, help='Size of the gaussian filter')
+@click.option('--sigma', '-s', default=3, type=float, help='Variance of the gaussian filter')
+@click.option('--n_spikes', '-ns', default=20, type=int, help='Number of spike of the signal')
+@click.option('--seed', '-sd', default=118, type=int, help='Random seed of the experiment')
+@click.option('--n_iter', '-i', default=1000, type=int, help='Number max of iteration that the algorithm can do')
+@click.option('--noise-factor', '-nf', default=None, type=float, help="Variance of the gaussian noise")
+@click.option('--noise_type', '-nt', default=None,
+ type=click.Choice(dir(NoiseType)[:len(NoiseType)], case_sensitive=False), help="How compute the noise")
+@click.option('--expe_name', '-en', default=None)
+def main(x_len, h_len, sigma, n_spikes, seed, n_iter, noise_factor, noise_type, expe_name):
+ # Run and plot results of the experiment
+ print(locals())
+ run_and_plot_experiment(x_len, h_len, sigma, n_spikes, seed, n_iter, noise_factor, noise_type, expe_name)
+ plot_signal_paper(expe_name=expe_name)
+ plot_signal_paper_full(expe_name=expe_name)
+ iterations_dcv(expe_name=expe_name)
+ iterations_sup_dcv(expe_name=expe_name)
+ plot_signal_paper_light(expe_name=expe_name)
+
+
+# Needed for click CLI
+if __name__ == '__main__':
+ main()
diff --git a/code/sksea/exp_phase_transition_diag.py b/code/sksea/exp_phase_transition_diag.py
new file mode 100644
index 0000000000000000000000000000000000000000..8a133e6732c169ac208e034cb11e0ec9828b0268
--- /dev/null
+++ b/code/sksea/exp_phase_transition_diag.py
@@ -0,0 +1,855 @@
+# -*- coding: utf-8 -*-
+# Basic python imports
+import math
+from enum import Enum
+from pathlib import Path
+from urllib import request
+from time import time
+from typing import Tuple, Dict, Callable, Iterable, Union
+
+# Installed module imports
+from loguru import logger
+import matplotlib.pyplot as plt
+from matplotlib.colors import LogNorm
+import numpy as np
+# from mat4py import loadmat
+from scipy.spatial.distance import hamming
+from scipy.interpolate import interp1d
+import plotly.express as px
+import plotly.graph_objs as go
+from plotly.subplots import make_subplots
+
+
+# Sksea imports
+from sksea.sparse_coding import MatrixOperator, SparseSupportOperator
+from sksea.deconvolution import gen_u, gen_filter, ConvolutionOperator
+from sksea.utils import algos_paper_dt_from_factor, get_color, PAPER_LAYOUT, AbstractLinearOperator
+
+CURVE_FOLDER = "DT_curve"
+HM_CURVE_FOLDER = "HM_curve"
+NM_CURVE_FOLDER = "NM_curve"
+ALL_HM_CURVE_FOLDER = "ALL_HM_curve"
+THRESH_FMT_LABEL = [(0.95, ':r', "95%"), (0.96, ':g', "96%"), (0.97, ':b', "97%"),
+ (0.98, ':m', "98%"), (0.99, ':y', "99%")]
+# THRESH_FMT_LABEL = [(0.9, ':g', "90%"), (0.91, ':b', "91%"), (0.92, ':m', "92%"), (0.93, ':c', "93%"),
+# (0.94, ':k', "94%"), (0.95, ':r', "95%"), (0.96, ':g', "96%"), (0.97, ':b', "97%"),
+# (0.98, ':m', "98%"), (0.99, ':y', "99%")]
+
+
+
+class NoiseType(Enum):
+ NAIVE = 1
+ SNR = 2
+ NOISE_LEVEL = 3
+
+
+def gen_noise(noise_type, noise_factor, y, rand):
+ if isinstance(noise_type, str):
+ noise_type = NoiseType[noise_type]
+ if isinstance(noise_type, int):
+ noise_type = NoiseType(noise_type)
+
+ noise = rand.randn(y.shape[0])
+ if noise_type == NoiseType.SNR:
+ noise *= 10 ** (-noise_factor / 20) * np.linalg.norm(y) / np.linalg.norm(noise)
+ elif noise_type == NoiseType.NOISE_LEVEL:
+ noise *= noise_factor * np.linalg.norm(y) / np.linalg.norm(noise)
+ elif noise_type == NoiseType.NAIVE:
+ noise = noise_factor * rand.rand(y.shape[0])
+ else:
+ noise = np.zeros_like(y)
+ return noise
+
+
+def generate_problem(n_samples=None, rho=1, delta=1, distribution='gaussian', snr=None, seed=None, n_atoms=None,
+ deconv=False, noise_factor=None, noise_type=None, use_sparse_operator=False
+ ) -> Tuple[Dict[str, Union[np.ndarray, int, 'AbstractLinearOperator']], Dict[str, np.ndarray]]:
+ """
+ Generate random d, x and y matrix for problems like 'd * x = y'
+
+ :param (int or None) n_samples: Size of y, number of lines of d
+ :param (float) rho: Sparsity, number of non_zeros coefficients in x divided by n_samples.
+ Must be between 0 and 1
+ :param (float) delta: Under-sampling factor, number of lines of d divided by its number of columns.
+ Must be between 0 and 1
+ :param (str) distribution: Probability distribution to use for generating d coefficients.
+ Must be 'gaussian' or '[1, 2]' if deconv is False. Else must be between 0 and 4.
+ :param (float or None) snr: Optional, Signal-to-Noise ratio.
+ :param (int) seed: Random seed to use for d and x generation
+ :param (int or None) n_atoms: Size of x, number of columns of d. If specified, n_samples must be None.
+ Must be specified if n_samples is None.
+ :param (bool) deconv: If True, generate a deconvolution problem
+ :return:
+ | problem_data:
+ 'obs_vect': y,
+ 'dict_mat': Linear Operator,
+ 'n_nonzero': number of non_zero coefficient in x
+ | solution_data:
+ 'sp_vec': x
+ """
+ if (n_atoms is None and n_samples is None) or (n_atoms is not None and n_samples is not None):
+ print(n_atoms, n_samples)
+ raise ValueError("n_atoms or n_samples must be specified, the other must be None")
+ elif n_samples is not None:
+ n_atoms = int(np.round(n_samples / delta))
+ elif n_atoms is not None:
+ n_samples = int(np.round(n_atoms * delta))
+ rand = np.random.RandomState(seed=seed)
+
+ if deconv: # TODO: To remove
+ n_nonzero = int(np.round(rho * n_atoms))
+ try:
+ distribution = int(distribution)
+ except Exception:
+ raise TypeError("Can't convert distribution to int")
+ if distribution not in range(5):
+ raise ValueError("distribution must be between 0 and 4")
+ x_ref = gen_u(n_atoms, n_nonzero, rand=rand)
+ d = ConvolutionOperator(gen_filter(n_samples, distribution), n_atoms)
+ y = d @ x_ref
+
+ else:
+ n_nonzero = int(np.round(rho * n_samples))
+ rand_mat = rand.randn(n_samples, n_atoms)
+ x_ref = np.zeros(n_atoms)
+ s_ref = rand.permutation(n_atoms)[:n_nonzero]
+ if distribution == 'unif':
+ x_ref[s_ref] = rand.rand(n_nonzero) + 1
+ x_ref[s_ref] *= (-1) ** rand.randint(0, 2, n_nonzero)
+ elif distribution == 'gaussian':
+ x_ref[s_ref] = rand.randn(n_nonzero)
+ else:
+ raise ValueError("distribution must be 'gaussian' or 'unif'")
+ y = rand_mat @ x_ref
+ d = SparseSupportOperator(rand_mat, y, seed) if use_sparse_operator else MatrixOperator(rand_mat, seed)
+
+ if noise_factor is not None:
+ y += gen_noise(noise_type, noise_factor, y, rand)
+ problem_data = {'obs_vec': y, 'dict_mat': d, 'n_nonzero': n_nonzero}
+ solution_data = {'sp_vec': x_ref}
+ return problem_data, solution_data
+
+
+def plot_theorical_phase_diagram_curve(curve_type="crosspolytope") -> Callable[[float], float]:
+ """
+ Add the theoretical recovery DT phase diagram curves to the current figure
+ For more details see https://people.maths.ox.ac.uk/tanner/polytopes.shtml
+
+ :param (str) curve_type: Type of theoretical curves.
+ Select 'crosspolytope' for classic l1 regularisation.
+ Select 'simplex' for l1 regularisation with non-negativity.
+ Select 'orthant' for uniqueness of non-negative vectors from mean-zero measurement matrices
+ :return:
+ A interpolated function of the weak curve.
+ """
+ # Retieve curves
+ filepath = Path("downloads/polytope.mat")
+ if not filepath.is_file():
+ filepath.parent.mkdir(exist_ok=True)
+ request.urlretrieve("https://people.maths.ox.ac.uk/tanner/data/polytope.mat", filepath)
+ #matlab_data = loadmat(str(filepath))
+ # Plot curves in the current figure
+ #p1s = np.array(matlab_data[f"rhoS_{curve_type}"])
+ #p1w = np.array(matlab_data[f"rhoW_{curve_type}"])
+ #plt.plot(p1s[:, 0], p1s[:, 1], 'r', label="strong")
+ #plt.plot(p1w[:, 0], p1w[:, 1], 'b', label="weak")
+ #return interp1d(p1w[:, 0], p1w[:, 1], fill_value=(0, 1), bounds_error=False)
+ return None
+
+
+def plot_empirical_phase_diagram_curve(dt_diag, axis_range_rho, axis_range_delta, threshold, fmt, label, name,
+ fig_nums=None, axis_range_rho_sparse=None, filepath=None) -> np.ndarray:
+ """
+ Plot empirical recovery DT phase diagram curve of the provided diagram for a given threshold
+
+ :param (np.ndarray) dt_diag: DT diagram
+ :param (np.ndarray) axis_range_rho: Array of rho values for the rho axis of the DT diagram
+ :param (np.ndarray) axis_range_delta: Array of delta values for the delta axis of the DT diagram
+ :param (float) threshold: All area under the curve have a recovery percentage above this threshold
+ :param (str) fmt: Format argument of matplotlib.pyplot.plot
+ See https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.plot.html for more details
+ :param (str) label: Label to use for the curve
+ :param (str) name: Name of the algorithm used in the experiment
+ :param (List[str or int]) fig_nums: Identifier (label or num) of the pyplot figures above which
+ we want to plot the curves. If not specified, the current pyplot figure is used.
+ :param (str or Path) filepath: If specified, save the numpy array containing the curve at this location
+ :return: Empirical DT curve
+ """
+ curve = np.ones((axis_range_delta.shape[0], 2))
+ curve[:, 0] = axis_range_delta
+ if fig_nums is None:
+ fig_nums = [plt.gcf().number]
+ # Compute the recovery curve
+ for i_delta, delta in enumerate(axis_range_delta):
+ if axis_range_rho_sparse is None:
+ iterator = axis_range_rho
+ else:
+ iterator = axis_range_rho_sparse[i_delta]
+ for i_rho in range(iterator.shape[0]):
+ # Search for the first rho with a recovery percentage under the threshold for each delta
+ if dt_diag[i_rho, i_delta] < threshold:
+ curve[i_delta] = (delta, iterator[i_rho - 1])
+ break
+
+ if filepath is not None:
+ filepath.parent.mkdir(exist_ok=True, parents=True)
+ np.save(filepath, curve)
+
+ # Plot curve in the selected figures
+ for num in fig_nums:
+ plt.figure(num=num)
+ if num == label: # For the plot with all algorithms
+ if name in ['SEA_BEST', 'OMP']:
+ name_fmt = ':'
+ else:
+ name_fmt = ''
+ plt.plot(curve[:, 0], curve[:, 1], name_fmt, label=name)
+ else: # For the DT diagram of each algorithm
+ plt.plot(curve[:, 0], curve[:, 1], fmt, label=label)
+ return curve
+
+
+def compute_phase_transition_diagram(n_samples, n_steps, n_runs, n_iter,
+ algo, distribution, name, rel_tol=-np.inf, **kwargs) -> None:
+ """
+ Solve ||D * x - y||_2^2 with the provided algorithm for various configurations.
+ Results are saved in the 'data_result' folder
+
+ :param (int) n_samples: Size of y, number of lines of D
+ :param (int or Tuple[int, int]) n_steps: Discretization of rho and delta.
+ If an integer is provided, the same discretization is used for both rho and delta.
+ Else, the first element is the discretization for rho, and the second the one for delta
+ :param (int) n_runs: Number of times each configuration is tested
+ :param (int) n_iter: Number max of iteration that the algorithm can do
+ :param (Callable) algo: Solveur to use
+ :param (str) distribution: Probability distribution to use for generating D coefficients.
+ Must be 'gaussian' or '[1, 2]'
+ :param (str) name: Name of the algorithm
+ :param (float) rel_tol: The algorithm stops when the iterations relative difference is lower than rel_tol
+ """
+ # Creating output directory
+ data_dir = Path('data_results')
+ data_dir.mkdir(exist_ok=True)
+
+ if isinstance(n_steps, int):
+ n_steps_rho, n_steps_delta = n_steps, n_steps
+ elif isinstance(n_steps, tuple):
+ n_steps_rho, n_steps_delta = n_steps
+ else:
+ raise TypeError("n_step must be an int or a tuple")
+
+ # Initialize arrays
+ axis_range_rho = (np.arange(n_steps_rho) + 1) / n_steps_rho
+ axis_range_delta = (np.arange(n_steps_delta) + 1) / n_steps_delta
+ hamming_dist = np.empty((n_steps_rho, n_steps_delta, n_runs))
+ recovery_results = np.empty((n_steps_rho, n_steps_delta, n_runs), dtype=bool)
+ eucl_dist_rel = np.empty((n_steps_rho, n_steps_delta, n_runs))
+
+ print(f'Algo {name}')
+ start_time = time()
+ for i_run in range(n_runs):
+ print(f'Run {i_run}/{n_runs}')
+ for i_rho, rho in enumerate(axis_range_rho):
+ for i_delta, delta in enumerate(axis_range_delta):
+ # print(f'(rho, delta)=({rho}, {delta})')
+ problem_data, solution_data = \
+ generate_problem(n_samples=n_samples, rho=rho, delta=delta, distribution=distribution, snr=None,
+ seed=i_run)
+ y = problem_data['obs_vec']
+ D = problem_data['dict_mat']
+ n_nonzero = problem_data['n_nonzero']
+ x_est, _ = algo(linop=MatrixOperator(D), y=y, n_nonzero=n_nonzero,
+ n_iter=n_iter, rel_tol=rel_tol)
+ x_ref = solution_data['sp_vec']
+ supp_ref = x_ref != 0
+ supp_est = x_est != 0
+ hamming_dist[i_rho, i_delta, i_run] = \
+ hamming(supp_est, supp_ref)
+ recovery_results[i_rho, i_delta, i_run] = \
+ np.all(supp_est == supp_ref)
+ # Compute euclidian distance for DT phase diagram
+ dist_rel = np.linalg.norm(x_ref - x_est) / np.linalg.norm(x_ref)
+ eucl_dist_rel[i_rho, i_delta, i_run] = dist_rel
+
+ out_file = f'{name}_{n_samples}_{n_steps}_{n_runs}_{n_iter}_{rel_tol}.npz'
+ np.savez(data_dir / out_file,
+ hamming_dist=hamming_dist,
+ recovery_results=recovery_results,
+ axis_range_rho=axis_range_rho,
+ axis_range_delta=axis_range_delta,
+ eucl_dist_rel=eucl_dist_rel,
+ name=name,
+ time_spent=time() - start_time)
+
+
+def extend_data_on_y_axes(data:np.ndarray, y_axes:np.ndarray):
+ new_data = np.ones_like(data) * np.nan
+ ref_axis = y_axes[-1]
+ for j, y_axis in enumerate(y_axes):
+ for i, y_ref_value in enumerate(ref_axis):
+ y_value = (np.abs(y_axis[y_axis != 0] - y_ref_value)).argmin()
+ new_data[i, j] = data[y_value, j]
+ return new_data
+
+
+def create_and_save_diagram(data, title, extent, path=None, cmap=None, clim=True, norm=None) -> None:
+ """
+ Create a 2D figure with the specified parameters using imshow.
+ Documentation: https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.imshow.html
+
+ :param (np.ndarray) data: 2D array to plot
+ :param (str) title: Title of the plot
+ :param (Optional[Tuple[float, float, Union[float, np.ndarray], float]]) extent: Bounding box of data coordinates.
+ See imshow doc for more details.
+ :param (str or Path) path: Path of the output file
+ :param (str or None) cmap: Colormap to use. See imshow doc for more details.
+ :param (bool) clim: If true, set the colorbar between 0 and 1.
+ :param (matplotlib.colors.Normalize or None) norm: Optional. Normalisation of the colorbar
+
+ """
+ if extent is not None:
+ if extent[-1] is None:
+ y_axes = extent[-2]
+ data_to_show = extend_data_on_y_axes(data, y_axes)
+ extent_to_show = (extent[0], extent[1], y_axes[-1, 0], y_axes[-1, -1])
+ else:
+ data_to_show = data
+ extent_to_show = extent
+
+ plt.figure()
+ plt.imshow(data_to_show, cmap=cmap, origin='lower', extent=extent_to_show, norm=norm)
+ if clim:
+ if norm is None:
+ plt.clim(0, 1)
+ """ else:
+ plt.clim(1e-5, 1)"""
+ plt.colorbar()
+ plt.xlabel(r'$\delta$')
+ plt.ylabel(r'$\rho$')
+ plt.title(title)
+ if path is not None:
+ plt.savefig(path)
+ plt.close()
+
+
+def sea_vs_sea_best(sea_data, sea_best_data, fig_dir, suffix, extent):
+ """
+ Plot figures for SEA VS SEA_BEST comparison
+
+ :param (Dict[str, np.ndarray]) sea_data: Data from sea experiment
+ :param (Dict[str, np.ndarray]) sea_best_data: Data from sea_best experiment
+ :param (Path) fig_dir: Path of the folder containing the figures
+ :param (str) suffix: Suffix of the output file name
+ :param (Tuple[float, float, float, float] or None) extent: Bounding box of data coordinates.
+ See imshow doc for more details.
+ """
+ # Load data
+ sea_iter = sea_data.get("iterations")
+ sea_best_iter = sea_best_data.get("iterations")
+ sea_last_res = sea_data.get("last_res")
+ sea_best_last_res = sea_best_data.get("last_res")
+ sea_eucl = sea_data.get('eucl_dist_rel')
+ sea_best_eucl = sea_best_data.get("eucl_dist_rel")
+
+ # Iterations comparison
+ create_and_save_diagram(np.mean(sea_iter - sea_best_iter, axis=2), "SEA/SEA_BEST iterations difference",
+ extent, fig_dir / f'iterations_SEA_diff{suffix}.pdf', clim=False)
+ create_and_save_diagram(np.mean((sea_iter - sea_best_iter) / sea_iter, axis=2),
+ "SEA/SEA_BEST iterations relative difference", extent,
+ fig_dir / f'iterations_SEA_rel_diff{suffix}.pdf')
+
+ # Last residual norm comparison
+ denom = (sea_last_res == 0) + sea_last_res # Remove division by 0
+ create_and_save_diagram(np.mean((sea_last_res - sea_best_last_res) / denom, axis=2),
+ "SEA/SEA_BEST residual norm relative difference", extent,
+ fig_dir / f'res_norm_SEA_rel_diff{suffix}.pdf')
+
+ # Relative distance comparison
+ denom = (sea_eucl == 0) + sea_eucl # Remove division by 0
+ create_and_save_diagram(np.mean((sea_eucl - sea_best_eucl) / denom, axis=2),
+ "SEA/SEA_BEST relative difference of relative euclidian distance wrt x", extent,
+ fig_dir / f'eucl_SEA_rel_diff{suffix}.pdf')
+ create_and_save_diagram(np.mean(np.abs(sea_eucl - sea_best_eucl), axis=2),
+ "SEA/SEA_BEST difference of relative euclidian distance wrt x", extent,
+ fig_dir / f'eucl_SEA_diff{suffix}.pdf', clim=True, norm=LogNorm())
+ create_and_save_diagram(np.mean(sea_eucl, axis=2),
+ "SEA relative euclidian distance wrt x", extent,
+ fig_dir / f'eucl_SEA_{suffix}.pdf', clim=False)
+ create_and_save_diagram(np.mean(sea_best_eucl, axis=2),
+ "SEA_BEST relative euclidian distance wrt x", extent,
+ fig_dir / f'eucl_SEA_BEST_{suffix}.pdf', clim=False)
+
+ # DT comparison by threshold value
+ thresholds = [f"{i}e-{p}" for p in range(1, 8) for i in range(9, 0, -1)]
+ maxs = np.empty(len(thresholds))
+ mins = np.empty(len(thresholds))
+ means = np.empty(len(thresholds))
+ plt.figure()
+ for idx, epsilon in enumerate(thresholds):
+ eps = float(epsilon)
+ diff = np.mean(sea_best_eucl < eps, axis=2) - np.mean(sea_eucl < eps, axis=2)
+ maxs[idx] = diff.max()
+ mins[idx] = diff.min()
+ means[idx] = diff.mean()
+ x = [float(e) for e in thresholds]
+ plt.semilogx(x, mins, label='min')
+ plt.semilogx(x, means, label='mean')
+ plt.semilogx(x, maxs, label='max')
+ plt.title("Difference between DT diagrams by threshold value for SEA VS SEA_BEST")
+ plt.legend()
+ plt.savefig(fig_dir / 'epsilon_selection.pdf')
+ plt.close()
+
+
+def plot_all_hm_curves(diag, axis_range_rho, axis_range_delta, axis_range_rho_sparse, filepath, name, success, n_samples):
+ """
+ Plot all recovery curves in the same figure
+
+ :param (np.ndarray) diag: DT diagram
+ :param (np.ndarray) axis_range_rho: Array of rho values for the rho axis of the DT diagram
+ :param (np.ndarray) axis_range_delta: Array of delta values for the delta axis of the DT diagram
+ :param (np.ndarray) axis_range_rho_sparse: Array of rho values for the rho axis of the DT diagram by delta
+ :param (str or Path) filepath: Save the numpy array containing the curve at this location
+ :param (str) name: Name of the algorithm used in the experiment
+ """
+ curve = np.ones((axis_range_delta.shape[0], 101))
+ curve[:, 0] = axis_range_delta
+
+ # Compute all recovery curve
+ for i_delta, delta in enumerate(axis_range_delta):
+ if axis_range_rho_sparse is None:
+ iterator = axis_range_rho
+ else:
+ iterator = axis_range_rho_sparse[i_delta]
+ last = 100
+ for i_rho in range(iterator.shape[0]):
+ current = int(diag[i_rho, i_delta] * 100)
+ if current < last:
+ for j in range(current + 1, last+1):
+ if j != 0:
+ curve[i_delta, j] = iterator[i_rho - 1]
+ last = current
+ if current == 0:
+ break
+
+ # Display all recovery curve
+ cmin = int(success * 100)
+ cmax = 99
+ fig = go.Figure()
+ colorscale = 'Rainbow'
+ # add invisible markers to display the colorbar without displaying the markers
+ # fig.add_trace(go.Scatter(
+ # x=[0.5,],
+ # y=[0,],
+ # mode='markers',
+ # marker=dict(
+ # size=0,
+ # color="rgba(0,0,0,0)",
+ # colorscale=colorscale,
+ # cmin=cmin,
+ # cmax=cmax,
+ # colorbar=dict(thickness=40, orientation="h")
+ # ),
+ # showlegend=False
+ # ))
+ for i in range(cmin, cmax + 1):
+ fig.add_trace(go.Scatter(x=curve[:, 0], y=curve[:, i], mode='lines', name=f"{i}%",
+ # fill='tonexty',
+ line=dict(width=1, color=get_color(colorscale, (i - cmin) / (cmax - cmin))),
+ # showlegend=False
+ ))
+ # fig.add_trace(go.Scatter(x=curve[:, 0], y=np.zeros(curve.shape[0]), mode='lines', fill='tonexty', showlegend=False,
+ # line=dict(width=0.1, color=get_color(colorscale, 1)), name="bottom"))
+ fig.update_layout(title=f'All HM curves for {name}', xaxis_title=r'$\delta$', yaxis_title=r'$\rho$', plot_bgcolor='white',)
+ fig.write_html(filepath, include_mathjax='cdn')
+
+ # Display heatmap
+ if n_samples is not None:
+ fig = go.Figure()
+ fig.add_trace(go.Heatmap(z=diag, x=axis_range_delta, y=axis_range_rho_sparse[0], zmin=success, zmax=1))
+
+ fig.update_layout(title=f'Heatmap for {name}', xaxis_title=r'$\delta$', yaxis_title=r'$\rho$', plot_bgcolor='white',)
+ fig.write_html(filepath.parent / f"heatmap_{name}.html", include_mathjax='cdn')
+
+
+def one_column_analysis(hamming_dist, axis_range_rho, axis_range_delta, axis_range_rho_sparse, name):
+ """
+ Function for debugging ELS
+ Plot hamming distance for a given delta and rho
+
+ :param (np.ndarray) hamming_dist: Hamming distance results
+ :param (np.ndarray) axis_range_rho: Array of rho values for the rho axis of the DT diagram
+ :param (np.ndarray) axis_range_delta: Array of delta values for the delta axis of the DT diagram
+ :param (np.ndarray) axis_range_rho_sparse: Array of rho values for the rho axis of the DT diagram by delta
+ :param (str) name: Name of the algorithm used in the experiment
+ """
+ delta = 0.7
+ rho = 0.1457143
+ i_delta = np.argmin(np.abs(axis_range_delta - delta))
+ rho_axis = axis_range_rho_sparse[i_delta]
+ i_rho = np.argmin(np.abs(rho_axis - rho)) + 1
+ fig = px.line(y=hamming_dist[i_rho, i_delta, :], title=f"Hamming distance for {name}")
+ fig.show()
+
+
+def plot_results(n_samples, n_steps, n_runs, n_iter, epsilon, rel_tol=-np.inf, n_atoms=None, epsilon_x=1e-4,
+ expe_name='', success=0.01) -> None:
+ """
+ Plot results stored in the 'data_results' folder.
+ See compute_phase_transition_diagram documentation for details
+
+ :param (int or None) n_samples: Size of y, number of lines of D
+ :param (int or Tuple[int, int]) n_steps: Discretization of rho and delta.
+ If an integer is provided, the same discretization is used for both rho and delta.
+ Else, the first element is the discretization for rho, and the second the one for delta
+ :param (int) n_runs: Number of times each configuration is tested
+ :param (int) n_iter: Number max of iteration that the algorithm has done
+ :param (float) epsilon: Relative error criteria for DT phase diagram
+ :param (float) rel_tol: The algorithm stopped when the iterations relative difference was lower than rel_tol
+ :param (int or None) n_atoms: Size of x, number of columns of D. If specified, n_samples must be None.
+ :param (float) epsilon_x: Threshold on the relative distance between x_approx and x_true
+ for the definition of a successful recovery
+ :param (str) expe_name: Name of the current experiment
+ :param (float) success: Minimal success rate used to continue DT computation
+ """
+ suffix = f'_{n_samples}_{n_steps}_{n_runs}_{n_iter}_{rel_tol}_{epsilon}_{n_atoms}_{expe_name}'
+ plot_suffix = suffix + f"_{epsilon_x}"
+ fig_dir = Path('figures') / plot_suffix[1:]
+ (fig_dir / CURVE_FOLDER).mkdir(exist_ok=True, parents=True)
+ (fig_dir / HM_CURVE_FOLDER).mkdir(exist_ok=True, parents=True)
+ data_dir = Path('data_results')
+ # Variable instantiation
+ sea_data, sea_best_data, hamming_dist_iht, sea_els_best_data, els_data = None, None, None, None, None
+ for f in Path(data_dir).glob(f'*{suffix}_*.npz'):
+ # Loading results
+ data = np.load(f)
+ hamming_dist = data['hamming_dist'] # [:, :, :200]
+ recovery_results = data.get('recovery_results')
+ eucl_dist_rel = data.get('eucl_dist_rel')
+ axis_range_delta = data.get('axis_range_delta')
+ axis_range_rho = data.get('axis_range_rho')
+ axis_range_rho_sparse = data.get('axis_range_rho_per_sparcity')
+ supp_est_in_sup_ref = data.get('supp_est_in_sup_ref')
+ eucl_dist_rel_y = data.get('eucl_dist_rel_y')
+
+ stem, index = f.stem.rsplit('_', 1)
+ algo_name = stem.split('_')[0]
+ name = algo_name if index == '0' else algo_name + '_BEST'
+ time_algo = data.get('time')
+ iterations = data.get('iterations')
+ stem = stem.replace(algo_name, name)
+
+ if axis_range_rho is None and axis_range_delta is None:
+ extent = None
+ elif axis_range_rho is None:
+ extent = (axis_range_delta[0], axis_range_delta[-1], axis_range_rho_sparse, None)
+ else:
+ extent = (axis_range_delta[0], axis_range_delta[-1], axis_range_rho[0], axis_range_rho[-1])
+
+ # Recovery diagram
+ if recovery_results is not None: # For retro-compatibility
+ create_and_save_diagram(np.mean(recovery_results, axis=2), name, extent, fig_dir / f'phase_diag_{stem}.pdf')
+
+ # Euclidian distance diagram
+ create_and_save_diagram(np.mean(eucl_dist_rel, axis=2), name, extent, fig_dir / f'eucl_dist_{stem}.pdf')
+
+ # DT phase diagram
+ if eucl_dist_rel is not None: # For retro-compatibility
+ create_and_save_diagram(np.mean(eucl_dist_rel < epsilon_x, axis=2), name, extent, cmap='Greys_r')
+ num = plt.gcf().number # Store principal (active) figure number
+
+ # Add DT phase diagram curves
+ # plot_theorical_phase_diagram_curve()
+ dt_diag = np.mean(eucl_dist_rel < epsilon_x, axis=2)
+ for threshold, fmt, label in THRESH_FMT_LABEL:
+ plot_empirical_phase_diagram_curve(dt_diag, axis_range_rho, axis_range_delta,
+ threshold, fmt, label, name, (num, label), axis_range_rho_sparse,
+ filepath=fig_dir / CURVE_FOLDER / f"{name}_{threshold}.npy")
+ plt.figure(num=num) # Restore principal figure
+ plt.legend()
+
+ plt.savefig(fig_dir / f'DT_phase_diag_{stem}.pdf')
+ plt.close()
+
+ # Time spent diagram
+ if time_algo is not None: # For retro-compatibility
+ create_and_save_diagram(np.mean(time_algo, axis=2), name, extent,
+ fig_dir / f'time_{stem}.pdf', clim=False)
+
+ # Hamming distance diagram
+ create_and_save_diagram(1 - np.mean(hamming_dist, axis=2), name, extent, fig_dir / f'hamming_dist_{stem}.pdf')
+
+ # Hamming phase diagram
+ # if "els" in name.lower() and "sea" not in name.lower():
+ # one_column_analysis(hamming_dist, axis_range_rho, axis_range_delta, axis_range_rho_sparse, name)
+ hm_diag = np.mean(hamming_dist == 0, axis=2)
+ diags = [(hm_diag, "HM", HM_CURVE_FOLDER)]
+ if supp_est_in_sup_ref is not None and eucl_dist_rel_y is not None:
+ nm_diag = np.mean((hamming_dist == 0) | (supp_est_in_sup_ref & (eucl_dist_rel_y < 1e-2)), axis=2)
+ diags.append((nm_diag, "NM", NM_CURVE_FOLDER))
+ for diag, diag_name, curve_folder in diags:
+ create_and_save_diagram(hm_diag, name, extent, cmap='Greys_r')
+ num = plt.gcf().number # Store principal (active) figure number
+ for threshold, fmt, label in THRESH_FMT_LABEL:
+ plot_empirical_phase_diagram_curve(diag, axis_range_rho, axis_range_delta,
+ threshold, fmt, label, name, (num, label),
+ axis_range_rho_sparse,
+ filepath=fig_dir / curve_folder / f"{name}_{threshold}.npy")
+ plt.figure(num=num) # Restore principal figure
+ plt.legend()
+ plt.savefig(fig_dir / f'{diag_name}_phase_diag_{stem}.pdf')
+ plt.close()
+
+ plot_all_hm_curves(hm_diag, axis_range_rho, axis_range_delta,
+ axis_range_rho_sparse, fig_dir / f'ALL_HM_phase_diag_{stem}.html', name, success, n_samples)
+
+ # Iterations diagram
+ create_and_save_diagram(np.mean(iterations, axis=2), name, extent,
+ fig_dir / f'iterations_{stem}.pdf', clim=False)
+
+ logger.debug(f"Plotting {f.stem} done")
+
+ if f.stem.startswith('ELS') and f.stem.endswith('0'):
+ els_data = data
+ elif f.stem.startswith('SEAFAST-els') and f.stem.endswith('1'):
+ sea_els_best_data = data
+ # elif f.stem.startswith('IHT'):
+ # hamming_dist_iht = 1 - np.mean(hamming_dist, axis=2)
+ # recovery_results_iht = np.mean(recovery_results, axis=2)
+ # elif f.stem.startswith('SEA') and f.stem.endswith('1'):
+ # sea_best_data = data
+ # elif f.stem.startswith('SEA') and f.stem.endswith('0'):
+ # sea_data = data
+
+ # Save DT comparison
+ for _, _, label in THRESH_FMT_LABEL:
+ plt.figure(num=label)
+ plt.title(label)
+ plt.xlim([0, 1])
+ plt.ylim([0, 1])
+ plt.legend()
+ plt.savefig(fig_dir / f'DT_{label}{suffix}.pdf')
+
+ # For debugging SEA_ELS
+ if sea_els_best_data is not None and els_data is not None:
+ debug_sea_els(sea_els_best_data, els_data, fig_dir, suffix)
+
+ # Save sea/sea_best comparison
+ # if sea_data is not None and sea_best_data is not None:
+ # sea_vs_sea_best(sea_data, sea_best_data, fig_dir, suffix, extent)
+
+ # Comparison of all algorithm with IHT
+ # if hamming_dist_iht is not None:
+ # for f in Path(data_dir).glob(f'*{suffix}.npz'):
+ # if f.stem.startswith('IHT'):
+ # continue
+ # data = np.load(f)
+ # hamming_dist = data['hamming_dist']
+ # recovery_results = data['recovery_results']
+ # axis_range = data.get('axis_range') # For retro-compatibility
+ # if axis_range is None:
+ # axis_range_delta = data.get('axis_range_delta')
+ # axis_range_rho = data.get('axis_range_rho')
+ # else: # For retro-compatibility
+ # axis_range_delta, axis_range_rho = axis_range, axis_range
+ # stem, index = f.stem.rsplit('_', 1)
+ # name = str(data['name']) if index == 0 else str(data['name']) + '_BEST'
+ #
+ # extent = (axis_range_delta[0], axis_range_delta[-1], axis_range_rho[0], axis_range_rho[-1])
+ #
+ # # Hamming distance difference
+ # d_results = 1 - np.mean(hamming_dist, axis=2) - hamming_dist_iht
+ # create_and_save_diagram(d_results, f'{name} - IHT $\\approx{np.mean(d_results[~np.isnan(d_results)]):.3g}$',
+ # extent, fig_dir / f'diff_hamming_dist_{stem}.pdf', clim=False)
+ #
+ # # Recovery result difference
+ # d_results = np.mean(recovery_results, axis=2) - recovery_results_iht
+ # create_and_save_diagram(d_results, f'{name} - IHT $\\approx{np.mean(d_results[~np.isnan(d_results)]):.3g}$',
+ # extent, fig_dir / f'diff_phase_diag_{stem}.pdf', clim=False)
+
+
+def plot_threshold_curves_comparison(expe_names, filename, epsilon_x=1e-4, nm=False) -> None:
+ """
+ Create a figures for using all algorithms available for each threshold value
+
+ :param Iterable[str] expe_names: Name of the experiments to compile together
+ :param (str) filename: Name of the html file for saving the plot
+ :return:
+ """
+ fig_dir = Path("figures")
+ plot_dir = fig_dir / "comparison"
+ plot_dir.mkdir(exist_ok=True)
+ colors = ['#a6cee3', '#1f78b4', '#b2df8a', '#33a02c', '#fb9a99', '#e31a1c',
+ '#fdbf6f', '#ff7f00', '#cab2d6', '#6a3d9a', '#ffff99', '#b15928']
+
+ if nm:
+ curve_folder = NM_CURVE_FOLDER
+ file_suffix = "nm"
+ else:
+ curve_folder = HM_CURVE_FOLDER
+ file_suffix = "hm"
+
+ # Get curves
+ paths = []
+ for suff in expe_names:
+ paths.extend(list(fig_dir.glob(f'*_{suff}_*{epsilon_x}')))
+
+ # Plot curves
+ fig = go.Figure()
+ for threshold, _, label in THRESH_FMT_LABEL:
+ idx = 0
+ for path in paths:
+ npy_files = list(Path(path / curve_folder).glob(f"*_{threshold}.npy"))
+ npy_files.sort() # Keep the color order
+ for curve_path in npy_files:
+ curve = np.load(str(curve_path))
+ name = curve_path.name.rsplit('_', 1)[0]
+ for suff in expe_names: # Add experiement name
+ if f'_{suff}_' in str(path) and suff != '':
+ name += '_' + suff
+ fig.add_trace(go.Scatter(x=curve[:, 0], y=curve[:, 1], line=dict(color=colors[idx % len(colors)]),
+ name=name, visible=False))
+ idx += 1
+
+ # Slider (start) --------------------------
+ steps = []
+ n_data = len(fig.data)
+ n = n_data / len(THRESH_FMT_LABEL)
+ for idx, (threshold, _, label) in enumerate(THRESH_FMT_LABEL):
+ step = dict(
+ # Update method allows us to update both trace and layout properties
+ method='update',
+ args=[
+ # Make the ith trace visible
+ {'visible': [idx * n <= i < (idx + 1) * n for i in range(n_data)]},
+
+ # Set the title for the ith trace
+ {'title.text': f"{file_suffix.upper()} threshold comparison at {label}"}],
+ )
+ steps.append(step)
+ sliders = [go.layout.Slider(
+ active=1,
+ len=0.5,
+ currentvalue={"prefix": "Threshold: "},
+ steps=steps
+ )]
+ # Slider (stop) --------------------------
+
+ fig.update_layout(
+ xaxis_title=r"$\delta$",
+ yaxis_title=r"$\rho$",
+ legend_title="Algorithms",
+ sliders=sliders,
+ xaxis_range=[0, 1],
+ yaxis_range=[0, 1],
+ xaxis_autorange=False,
+ yaxis_autorange=False,
+ )
+ filename = f'{filename}_{epsilon_x}.html' if filename.endswith('.html') else f'{filename}_{epsilon_x}'
+ fig.write_html(plot_dir / f'{filename}_{file_suffix}.html', include_mathjax='cdn')
+
+
+def plot_paper_curves(expe_name, epsilon_x=1e-4, factors=(256,), dt=False, threshold_position=-5) -> None:
+ threshold, _, label = THRESH_FMT_LABEL[threshold_position]
+ fig_dir = Path("figures")
+ plot_dir = fig_dir / "comparison"
+ plot_dir.mkdir(exist_ok=True)
+
+ if dt:
+ curve_folder = CURVE_FOLDER
+ file_suffix = "dt"
+ else:
+ curve_folder = HM_CURVE_FOLDER
+ file_suffix = "hm"
+
+ algos_paper_dt = algos_paper_dt_from_factor(factors[0])
+
+ # Get curves
+ path = list(fig_dir.glob(f'*_{expe_name}_*{epsilon_x}'))[0]
+ npy_files = {np_path.name.rsplit('_', 1)[0]: np_path
+ for np_path in Path(path / curve_folder).glob(f"*_{threshold}.npy")
+ if np_path.name.rsplit('_', 1)[0] in algos_paper_dt.keys()}
+
+ legend_ranks = ["SEA_ELS", "SEA_OMP", "ELS", "OMPR", "IHT_OMP", "HTP_OMP", "SEA_0", "OMP", "IHT", "HTP"]
+
+ fig = go.Figure()
+ for algo, info in algos_paper_dt.items():
+ if algo in npy_files.keys() and npy_files[algo].is_file:
+ curve = np.load(str(npy_files[algo]))
+ if info["disp_name"] == "ELS":
+ display_name = "$\\text{ELS}, \\text{IHT}_{\\text{ELS}}, \\text{HTP}_{\\text{ELS}}$"
+ else:
+ display_name = info["name"]
+ fig.add_trace(go.Scatter(x=curve[:, 0], y=curve[:, 1], line=info["line"],
+ name=display_name, legendrank=legend_ranks.index(info["disp_name"])))
+
+ fig.update_layout(
+ xaxis_title=r"$\zeta = m/n$",
+ yaxis_title=r"$\rho = k/m$",
+ **PAPER_LAYOUT,
+ legend=dict(
+ orientation='h',
+ y=1,
+ x=0, #0.05,
+ xanchor="left",
+ yanchor="top",
+ bgcolor= 'rgba(0,0,0,0)',
+ entrywidth=160
+ )
+ )
+ fig.update_layout(legend=dict(title=None), font_size=17)
+ fig.write_html(plot_dir / f"paper_{expe_name}_{file_suffix}_{threshold}.html", include_mathjax='cdn')
+ # fig.update_layout(showlegend=False)
+ # fig.write_html(plot_dir / f"paper_{expe_name}_{file_suffix}_{threshold}_no_legend.html", include_mathjax='cdn')
+
+
+def debug_sea_els(sea_els_best_data, els_data, fig_dir, suffix):
+ """
+ Plot figures for debugging SEA_ELS
+
+ :param (Dict[str, np.ndarray]) sea_els_best_data: Data from sea_els_best
+ :param (Dict[str, np.ndarray]) els_data: Data from els
+ :param (Path) fig_dir: Path of the folder containing the figures
+ :param (str) suffix: Suffix of the output file name
+ """
+ # Load data
+
+ axis_range_delta = sea_els_best_data.get('axis_range_delta')
+ axis_range_rho_sparse = sea_els_best_data.get('axis_range_rho_per_sparcity')
+
+ sea_els_best_iter = sea_els_best_data.get("iterations")
+ sea_els_best_last_res = sea_els_best_data.get("last_res")
+ sea_els_best_eucl = sea_els_best_data.get("eucl_dist_rel")
+ sea_els_best_time = sea_els_best_data.get("time_spent")
+ sea_els_best_recovery_results = sea_els_best_data.get("recovery_results")
+ sea_els_best_hamming_dist = sea_els_best_data.get("hamming_dist")
+ sea_els_best_supp = sea_els_best_data.get("supp_est_in_sup_ref")
+
+
+ els_iter = els_data.get("iterations")
+ els_last_res = els_data.get("last_res")
+ els_eucl = els_data.get("eucl_dist_rel")
+ els_eucl_y = els_data.get("eucl_dist_rel_y")
+ els_time = els_data.get("time_spent")
+ els_recovery_results = els_data.get("recovery_results")
+ els_hamming_dist = els_data.get("hamming_dist")
+ els_supp = els_data.get("supp_est_in_sup_ref")
+
+ # Hamming distance comparison
+ hm_diff = els_hamming_dist - sea_els_best_hamming_dist
+ # logger.debug(f"Hamming distance difference negative positions: {list(zip(*np.where(hm_diff < 0)))}")
+
+ # metric comparison
+ hm = np.mean(els_hamming_dist == 0, axis=-1)
+ y_err_max = np.max(np.abs(els_eucl_y), axis=-1)
+ fig = make_subplots(specs=[[{"secondary_y": True}]])
+ for i, delta in enumerate(axis_range_delta):
+ fig.add_trace(go.Scatter(x=axis_range_rho_sparse[i], y=hm[:,i], name=f"Hamming distance for delta={delta}",
+ ), secondary_y=False)
+ fig.add_trace(go.Scatter(x=axis_range_rho_sparse[i], y=y_err_max[:,i], name=f"Euclidian distance for delta={delta}",
+ line=dict(dash='dash')), secondary_y=True)
+ fig.update_layout(title="Hamming distance and Euclidian distance for SEA_ELS")
+ fig.update_xaxes(title_text=r"$\rho$")
+ fig.update_yaxes(title_text="Hamming distance", secondary_y=False)
+ fig.update_yaxes(title_text="Euclidian distance", secondary_y=True)
+ fig.write_html(fig_dir / f"ELS.html", include_mathjax='cdn')
diff --git a/code/sksea/plot_icml.py b/code/sksea/plot_icml.py
new file mode 100644
index 0000000000000000000000000000000000000000..8870acfdce70a082adbd5be024cc0c1c0e5c2922
--- /dev/null
+++ b/code/sksea/plot_icml.py
@@ -0,0 +1,1131 @@
+# Python imports
+import pickle
+from copy import deepcopy
+from itertools import chain
+from pathlib import Path
+
+# Module imports
+from loguru import logger
+import matplotlib.pyplot as plt
+from matplotlib.markers import MarkerStyle
+from matplotlib.transforms import Affine2D
+import numpy as np
+from typing import Dict
+
+# Script imports
+from sksea.dataset_operator import DatasetOperator, RESULT_PATH
+from sksea.deconvolution import ConvolutionOperator
+from sksea.exp_phase_transition_diag import HM_CURVE_FOLDER
+from sksea.utils import compute_metrcs_from_file, DATA_FILENAME, RESULT_FOLDER
+
+# https://matplotlib.org/stable/gallery/text_labels_and_annotations/tex_demo.html
+els_color = "#e41a1c"
+omp_color = "#377eb8"
+zero_color = "#4daf4a"
+ompr_color = "#37b8a9"
+base_linestyle = (0, (1, 2, 0, 0)) # (0, (0, 0, 1, 4)
+sea_linestyle = "solid"
+htp_linestyle = (0, (1, 1, 0, 0))
+iht_linestyle = (0, (3, 3, 0, 0))
+alpha_dcv_iter = 0.6
+algos_base = {
+ "IHT": {"disp_name": "IHT",
+ 'plot': {"label": r"IHT", "linestyle": None},
+ 'marker': {"marker": r"$\unboldmath\bowtie$", "transform": Affine2D().rotate_deg(90), },
+ 'scatter': {"color": "#AB63FA"},
+ 'marker_label': r"IHT",
+ 'plot_dcv_iter': {},
+ },
+ "HTP": {"disp_name": "HTP",
+ 'plot': {"label": r"HTP", "linestyle": None},
+ 'marker': {"marker": "d", },
+ 'scatter': {"color": "#F39221"},
+ 'marker_label': r"HTP",
+ 'plot_dcv_iter': {},
+ },
+ "HTP-omp": {"disp_name": "HTP-omp",
+ 'plot': {"label": r"HTP$_{{OMP}}$", "linestyle": None}},
+ "IHT-omp": {"disp_name": "IHT-omp",
+ 'plot': {"label": r"IHT$_{{OMP}}$", "linestyle": None}},
+ "OMP": {"disp_name": "OMP",
+ 'plot': {"label": r"OMP", "linestyle": None}},
+ "OMPR": {"disp_name": "OMPR",
+ 'plot': {"label": r"OMPR", "linestyle": None}},
+ "ELS": {"disp_name": "ELS",
+ 'plot': {"label": r"ELS", "linestyle": None},
+ 'marker': {"marker": "d", "transform": Affine2D().rotate_deg(90), },
+ 'scatter': {"color": "#005FFA"},
+ 'marker_label': r"ELS, OMP, OMPR, IHT$_{{OMP}}$, HTP$_{{OMP}}$, IHT$_{{ELS}}$, HTP$_{{ELS}}$",
+ 'marker_label_light': r"ELS, OMP, OMPR",
+ 'plot_dcv_iter': {}, },
+ "IHT-els": {"disp_name": "IHT-els",
+ 'plot': {"label": r"IHT$_{{ELS}}$", "linestyle": None}},
+ "HTP-els": {"disp_name": "HTP-els",
+ 'plot': {"label": r"HTP$_{{ELS}}$", "linestyle": None}},
+ "SEA-0": {"disp_name": "SEA-0",
+ 'plot': {"label": r"SEA$_0$", "linestyle": None},
+ 'marker': {"marker": "s", },
+ 'scatter': {"color": "#e41a1c"},
+ 'marker_label': r"SEA$_0$, SEA$_{{OMP}}$, SEA$_{{ELS}}$",
+ 'marker_noisy_label': r"SEA$_0$",
+ 'marker_label_light': r"SEA$_0$, SEA$_{{OMP}}$, SEA$_{{ELS}}$",
+ 'plot_dcv_iter': {"alpha": alpha_dcv_iter},
+ 'plot_dcv_iter_best': {"color": "darkred"}},
+ "SEA-els": {"disp_name": "SEA-els",
+ 'plot': {"label": r"SEA$_{{ELS}}$", "linestyle": None},
+ 'marker_noisy_label': r"SEA$_{{OMP}}$, SEA$_{{ELS}}$",
+ 'scatter': {"color": "black"},
+ 'marker': {"marker": "s", "transform": Affine2D().rotate_deg(45)},
+ 'plot_dcv_iter': {"alpha": alpha_dcv_iter},
+ 'plot_dcv_iter_best': {"color": "grey"}},
+ "SEA-omp": {"disp_name": "SEA-omp",
+ 'plot': {"label": r"SEA$_{{OMP}}$", "linestyle": None},
+ 'plot_dcv_iter': {"color": "#19D3F3", "alpha": alpha_dcv_iter},
+ 'plot_dcv_iter_best': {"color": "indigo"}},
+ "remove": {"disp_name": "remove",
+ 'plot': {"label": r"remove", "linestyle": None}},
+ "SOTA": {"disp_name": "SOTA", 'plot': {"label": r"SOTA", "linestyle": None}},
+ "x": {"disp_name": "x",
+ 'plot': {"label": r"x", "linestyle": None},
+ 'marker': {"marker": "o", },
+ 'scatter': {"color": "#4daf4a"},
+ 'marker_label': r"$x^*$",
+ 'marker_label_light': r"$x^*$"
+ },
+ "y": {"disp_name": "y",
+ 'plot': {"label": r"$y$", "linestyle": None}, }
+}
+
+legend_order = ["remove", "SOTA", "x", "y",
+ "SEA-els", "SEA-omp", "SEA-0", "IHT-els", "IHT-omp", "IHT", "HTP-els", "HTP-omp", "HTP", "ELS", "OMPR",
+ "OMP", ]
+ZOOM_X_LIM = (383, 465)
+ZOOM_Y_ABS = 3.15
+
+
+def get_legend_order(label, markers=False, algo_dict=algos_base):
+ for idx, algo in enumerate(legend_order):
+ if algo in algo_dict:
+ if label == algo_dict[algo]["plot"]["label"] or markers and (
+ label == algo_dict[algo].get("marker_label_light")
+ or label == algo_dict[algo].get("marker_label")
+ or label == algo_dict[algo].get("marker_noisy_label")):
+ return idx
+
+
+def algos_from_factor(factor):
+ map = {
+ f"IHTx{factor}": "IHT",
+ f"IHT-ompx{factor}": "IHT-omp",
+ f"IHT-elsx{factor}": "IHT-els",
+ f"HTPFASTx{factor}_BEST": "HTP",
+ f"HTPFAST-ompx{factor}_BEST": "HTP-omp",
+ f"HTPFAST-elsx{factor}_BEST": "HTP-els",
+ f"OMPx{factor}": "OMP",
+ f"OMPRx{factor}": "OMPR",
+ f"ELSx{factor}": "ELS",
+ f"SEAFASTx{factor}_BEST": "SEA-0",
+ f"SEAFAST-ompx{factor}_BEST": "SEA-omp",
+ f"SEAFAST-elsx{factor}_BEST": "SEA-els",
+
+ }
+ return {algo_surname: deepcopy(algos_base[algo_name]) for algo_surname, algo_name in map.items()}
+
+
+def algos_from_factor_ml(factor):
+ map = {
+ f"IHTx{factor}": "IHT",
+ f"IHT-ompx{factor}": "IHT-omp",
+ f"IHT-elsx{factor}": "IHT-els",
+ f"HTPx{factor}": "HTP",
+ f"HTP-ompx{factor}": "HTP-omp",
+ f"HTP-elsx{factor}": "HTP-els",
+ f"OMP": "OMP",
+ f"OMPR": "OMPR",
+ f"ELS": "ELS",
+ f"SEAFASTx{factor}": "SEA-0",
+ f"SEAFAST-ompx{factor}": "SEA-omp",
+ f"SEAFAST-elsx{factor}": "SEA-els",
+
+ }
+ return {algo_surname: deepcopy(algos_base[algo_name]) for algo_surname, algo_name in map.items()}
+
+
+def algos_dcv():
+ map = {
+ f"IHT": "IHT",
+ f"IHT-omp": "IHT-omp",
+ f"IHT-els": "IHT-els",
+ f"HTPFAST": "HTP",
+ f"HTPFAST-omp": "HTP-omp",
+ f"HTPFAST-omp_fast": "HTP-omp",
+ f"HTPFAST-els": "HTP-els",
+ f"HTPFAST-els_fast": "HTP-els",
+ f"OMPFAST": "OMP",
+ f"OMPRFAST": "OMPR",
+ f"ELSFAST": "ELS",
+ f"SEAFAST": "SEA-0",
+ f"SEAFAST-omp": "SEA-omp",
+ f"SEAFAST-omp_fast": "SEA-omp",
+ f"SEAFAST-els": "SEA-els",
+ f"SEAFAST-els_fast": "SEA-els",
+
+ }
+ return {algo_surname: deepcopy(algos_base[algo_name]) for algo_surname, algo_name in map.items()}
+
+
+def algos_dcv_signal():
+ map = {
+ f"IHT": "IHT",
+ f"IHT-omp": "IHT-omp",
+ f"IHT-els": "IHT-els",
+ f"HTPFAST": "HTP",
+ f"HTPFAST-omp": "HTP-omp",
+ f"HTPFAST-els": "HTP-els",
+ f"OMPFAST": "OMP",
+ f"OMPRFAST": "OMPR",
+ f"ELSFAST": "ELS",
+ f"SEAFAST": "SEA-0",
+ f"SEAFAST-omp": "SEA-omp",
+ f"SEAFAST-els": "SEA-els",
+ "x": "x",
+ }
+ return {algo_surname: deepcopy(algos_base[algo_name]) for algo_surname, algo_name in map.items()}
+
+
+def get_best_hist(loss_array):
+ best_loss = np.empty_like(loss_array)
+ best = np.inf
+ for current_idx, current in enumerate(loss_array):
+ best = min(best, current)
+ best_loss[current_idx] = best
+ return best_loss
+
+
+# Matplotlib
+# Plot https://randalolson.com/2014/06/28/how-to-make-beautiful-data-visualizations-in-python-with-matplotlib/
+plt.rcParams.update(
+ {
+ 'axes.edgecolor': '#cccccc',
+ 'axes.linewidth': 1,
+ 'axes.spines.top': False, # Remove the plot frame lines
+ 'axes.spines.right': False, # Remove the plot frame lines
+ 'font.size': 25,
+ 'figure.dpi': 600,
+ 'figure.figsize': (10, 6),
+ 'legend.frameon': False,
+ 'legend.borderaxespad': 0,
+ 'legend.borderpad': 0,
+ 'legend.columnspacing': 1,
+ 'legend.labelspacing': 0.5,
+ 'legend.handlelength': 1.25,
+ 'legend.handletextpad': 0.7,
+ 'lines.linewidth': 3.2,
+ 'lines.markersize': 25,
+ 'mathtext.fontset': 'cm',
+ 'text.usetex': True,
+ 'text.latex.preamble': r'\usepackage[cm]{sfmath}',
+ 'xtick.bottom': False, # Remove the tick marks
+ 'ytick.left': False, # Remove the tick marks
+
+ }
+)
+
+# Add style
+for algo_selected, algo_infos in algos_base.items():
+ algo_infos["legend_order"] = legend_order.index(algo_selected) # Add curve order
+ if "OMPR" in algo_selected:
+ algo_infos["plot"]["color"] = ompr_color
+ elif "OMP".lower() in algo_selected.lower():
+ algo_infos["plot"]["color"] = omp_color
+ elif "ELS".lower() in algo_selected.lower():
+ algo_infos["plot"]["color"] = els_color
+ else:
+ algo_infos["plot"]["color"] = zero_color
+ if "SEA" in algo_selected:
+ algo_infos["plot"]["linestyle"] = sea_linestyle
+ elif "HTP" in algo_selected:
+ algo_infos["plot"]["linestyle"] = htp_linestyle
+ elif "IHT" in algo_selected:
+ algo_infos["plot"]["linestyle"] = iht_linestyle
+ else:
+ algo_infos["plot"]["linestyle"] = base_linestyle
+ algo_infos["plot"]["linewidth"] = plt.rcParams['lines.linewidth'] * 2
+ if algo_infos.get("marker") is not None:
+ algo_infos["marker"]["fillstyle"] = "none"
+ algo_infos["vlines"] = {"color": "#cccccc", "linestyle": "solid"}
+
+
+def plot_dt_paper(expe_name="unifr1000", threshold=0.95, factors=(256,)):
+ fig_dir = Path("figures")
+ plot_dir = fig_dir / "icml"
+ plot_dir.mkdir(exist_ok=True)
+
+ curve_folder = HM_CURVE_FOLDER
+
+ algos = algos_from_factor(factors[0])
+
+ # Get curves
+ path = list(fig_dir.glob(f'*_{expe_name}_0.0*'))[0]
+ npy_files = {np_path.name.rsplit('_', 1)[0]: np_path
+ for np_path in Path(path / curve_folder).glob(f"*_{threshold}.npy")
+ if np_path.name.rsplit('_', 1)[0] in algos.keys()}
+
+ # Add curve order
+ curve_order = ["remove",
+ "SEAFAST-omp", "SEAFAST-els", "SEAFAST", "IHT-els", "IHT", "IHT-omp", "HTP", "HTP-els",
+ "HTP-omp", "OMPR", "ELS", "OMP", ]
+
+ for algo, infos in algos.items():
+ algo_name = algo.split(f"x{factors[0]}")[0]
+ if algo_name in curve_order:
+ infos["plot"]["zorder"] = 100 * curve_order.index(algo_name)
+
+ fig = plt.figure()
+ # Put data in the plot
+ for algo, np_path in npy_files.items():
+ curve = np.load(np_path)
+ plt.plot(curve[:, 0], curve[:, 1], **algos[algo]["plot"])
+ if "SEA" in algo and "omp" in algo:
+ style = dict(**algos[algo]["plot"])
+ style["linewidth"] = plt.rcParams['lines.linewidth'] * 2
+ style["label"] = None
+ nb_pts_bold = 7 if "noisy" in expe_name else 8
+ plt.plot(curve[:nb_pts_bold, 0], curve[:nb_pts_bold, 1], **style)
+
+ # Limit the range of the plot to only where the data is.
+ plt.ylim(0.01, 0.36)
+ plt.xlim(0, 0.9)
+
+ plt.xlabel(r'$\zeta = m / n$', labelpad=6)
+ plt.ylabel(r'$\rho = k / m$', labelpad=10)
+
+ # Reorder legend
+ handles, labels = plt.gca().get_legend_handles_labels()
+ order = np.argsort([get_legend_order(label) for label in labels])
+ ax = plt.gca()
+ first_legend = ax.legend([handles[idx] for idx in order[:9]], [labels[idx] for idx in order[:9]], ncol=3,
+ loc='upper left', bbox_to_anchor=(0.02, 1))
+ ax.add_artist(first_legend) # Add the legend manually to the Axes.
+ plt.legend([handles[idx] for idx in order[9:]], [labels[idx] for idx in order[9:]], ncol=1,
+ loc='upper left', bbox_to_anchor=(0.02, 0.725))
+
+ fig.tight_layout(pad=0, rect=(0.015, 0, 1, 0.995))
+ plt.savefig(plot_dir / f"{expe_name}.svg")
+ plt.close("all")
+
+
+def plot_dt_paper_zoom(expe_name="unifr1000", threshold=0.95, factors=(256,), legend_order=legend_order):
+ legend_order = legend_order.copy()
+ old_linewidth = plt.rcParams['lines.linewidth']
+ plt.rcParams['lines.linewidth'] *= 1.5
+ old_font_size = plt.rcParams['font.size']
+ plt.rcParams['font.size'] = 33
+ # legend_order.insert(0, "ELS")
+
+ fig_dir = Path("figures")
+ plot_dir = fig_dir / "icml"
+ plot_dir.mkdir(exist_ok=True)
+
+ curve_folder = HM_CURVE_FOLDER
+
+ algos = algos_from_factor(factors[0])
+
+ # Get curves
+ path = list(fig_dir.glob(f'*_{expe_name}_0.0*'))[0]
+ npy_files = {np_path.name.rsplit('_', 1)[0]: np_path
+ for np_path in Path(path / curve_folder).glob(f"*_{threshold}.npy")
+ if np_path.name.rsplit('_', 1)[0] in algos.keys()
+ and np_path.name.rsplit('_', 1)[0] in
+ (f"OMPx{factors[0]}", f"SEAFAST-ompx{factors[0]}_BEST",
+ f"SEAFAST-elsx{factors[0]}_BEST", f"ELSx{factors[0]}")}
+
+ # Add curve order
+ curve_order = ["remove", "SEAFAST-omp", "SEAFAST-els", "ELS", "OMP", ]
+
+ for algo, infos in algos.items():
+ algo_name = algo.split(f"x{factors[0]}")[0]
+ if algo_name in curve_order:
+ infos["plot"]["zorder"] = 100 * curve_order.index(algo_name)
+
+ fig = plt.figure(figsize=(7, 4.5))
+ # Put data in the plot
+ for algo, np_path in npy_files.items():
+ curve = np.load(np_path)
+ plt.plot(curve[:, 0], curve[:, 1], **algos[algo]["plot"])
+ if "SEA" in algo and "omp" in algo:
+ style = dict(**algos[algo]["plot"])
+ style["linewidth"] = plt.rcParams['lines.linewidth'] * 2
+ style["label"] = None
+ nb_pts_bold = 7 if "noisy" in expe_name else 8
+ plt.plot(curve[:nb_pts_bold, 0], curve[:nb_pts_bold, 1], **style)
+
+ # Limit the range of the plot to only where the data is.
+ plt.ylim(0.075, 0.22)
+ plt.xlim(0.02, 0.4)
+
+ plt.xlabel(r'$\zeta=m/n$', labelpad=6)
+ plt.ylabel(r'$\rho=k/m$', labelpad=10)
+
+ # Reorder legend
+ handles, labels = plt.gca().get_legend_handles_labels()
+ order = np.argsort([get_legend_order(label) for label in labels])
+ plt.legend([handles[idx] for idx in order], [labels[idx] for idx in order], ncol=4,
+ loc='lower right', bbox_to_anchor=(1.05, 1.05),
+ **{'labelspacing': 0.2, 'handletextpad': 0.2, 'columnspacing': 0.5, 'borderpad': 0.2,
+ 'handlelength': 0.58
+ })
+
+ fig.tight_layout(pad=0, rect=(0, 0, 1, 1))
+ plt.savefig(plot_dir / f"{expe_name}_zoom.svg")
+ plt.rcParams['lines.linewidth'] = old_linewidth
+ plt.rcParams['font.size'] = old_font_size
+ legend_order: list
+ legend_order.remove("ELS")
+ plt.close("all")
+
+
+def plot_dcv_paper(expe_name, spars_max):
+ result_folder = RESULT_FOLDER / expe_name
+ plot_dir = result_folder / "icml"
+ plot_dir.mkdir(exist_ok=True)
+
+ solution = np.load(result_folder / "solution.npy")[:, :spars_max, :]
+ sparsity = np.arange(1, spars_max + 1)
+ with np.load(result_folder / DATA_FILENAME, allow_pickle=True) as data:
+ linop = ConvolutionOperator(data["h"], data["x_len"])
+
+ algos = algos_dcv()
+ npy_files = {np_path.stem: np_path
+ for np_path in result_folder.glob("*.npy")
+ if np_path.stem in algos.keys()}
+
+ sota_linewidth = plt.rcParams['lines.linewidth'] * 1.5
+ sota_font = 33
+ plots = {
+ "sup_dist": {
+ "ylabel": r"dist$_{supp}$", "ylim": (0, 0.75),
+ "legend1": {"ncol": 3, "loc": "lower right", "bbox_to_anchor": (1, 0.01)},
+ "rect": (0.015, 0, 1, 0.995)},
+ "ws": {
+ "ylabel": r"Wasserstein distance", "ylim": (0, 7.5e-4),
+ "legend1": {"ncol": 3, "loc": "lower right", "bbox_to_anchor": (1, 0.01)},
+ "rect": (0.015, 0, 1, 0.995)},
+ # "ws_bin": {
+ # "ylabel": r"Unnorm Support Wasserstein distance", "ylim": (None, ),
+ # "legend1": {"ncol": 3, "loc": "lower right", "bbox_to_anchor": (1, 0.01)},
+ # "rect": (0.015, 0, 1, 0.995)},
+ # "ws_bin_norm": {
+ # "ylabel": r"Support Wasserstein distance", "ylim": (None, ),
+ # "legend1": {"ncol": 3, "loc": "lower right", "bbox_to_anchor": (1, 0.01)},
+ # "rect": (0.015, 0, 1, 0.995)},
+ # "n_supports": {
+ # "ylabel": r"Number of supports explored", "ylim": (1, 4e3),
+ # "legend1": {"ncol": 1, "loc": "upper left", "bbox_to_anchor": (1.04, 1)},
+ # "rect": (0, 0, 0.8, 1)},
+ # "n_supports_new": {
+ # "ylabel": r"Number of NEW supports explored", "ylim": (1, 4e3),
+ # "legend1": {"ncol": 1, "loc": "upper left", "bbox_to_anchor": (1.04, 1)},
+ # "rect": (0, 0, 0.8, 1)},
+ # "n_supports_from_start": {
+ # "ylabel": r"Number of supports explored from start", "ylim": (1, 4e3),
+ # "legend1": {"ncol": 1, "loc": "upper left", "bbox_to_anchor": (1.04, 1)},
+ # "rect": (0, 0, 0.8, 1)},
+ # "mse": {
+ # "ylabel": r"MSE", "ylim": (None,),
+ # "legend1": {"ncol": 3, "loc": "lower right", "bbox_to_anchor": (1, 0.01)},
+ # "rect": (0.015, 0, 1, 0.995)},
+ "f_mse_y": {
+ "ylabel": r"Mean of $\ell_{2, rel\_loss}$", "ylim": (0, 0.125),
+ "legend1": {"ncol": 3, "loc": "lower right", "bbox_to_anchor": (1, 0.01)},
+ "rect": (0.015, 0, 1, 0.995)},
+ "sota": {
+ "ylabel": r"Support distance", "ylim": (0, 0.66),
+ "legend1": {
+ "ncol": 4, "loc": "lower right", "bbox_to_anchor": (1.05, 1.05), "fontsize": sota_font,
+ 'labelspacing': 0.2, 'handletextpad': 0.2, 'columnspacing': 0.5, 'borderpad': 0.2, 'handlelength': 0.6
+ },
+ "rect": (0, 0, 1, 0.85)}, # (left, bottom, right, top)
+ }
+ algos_sup_dist = []
+ # For each algo
+ for algo, file in npy_files.items():
+ metrics_file = file.parent / "temp_plot_data" / (file.stem + ".npz")
+
+ # Load or compute metrics
+ if metrics_file.is_file():
+ logger.debug(f"Loading {metrics_file}")
+ metrics = np.load(metrics_file)
+ else:
+ logger.debug(f"Computing {metrics_file}")
+ compute_metrcs_from_file(file, spars_max, linop, solution, metrics_file)
+ metrics = np.load(metrics_file)
+
+ # Plot metrics
+ for plot_name in plots.keys():
+ if "sota" == plot_name or "supports" in plot_name and "IHT" in algo:
+ continue
+ plt.figure(plot_name)
+ style = dict(**algos[algo]["plot"])
+ if algo == "OMP" and plot_name == "sup_dist":
+ plt.plot(0, -1, **style)
+ style["linestyle"] = (0, (0, 1, 1, 1))
+ style.pop("label")
+ if "OMPR" in algo and plot_name == "sup_dist":
+ plt.plot(0, -1, **style)
+ style["linewidth"] *= 1.2
+ style.pop("label")
+ if plot_name == "f_mse_y":
+ metric = metrics[plot_name].reshape(*solution.shape[:-1]).mean(axis=0)
+ if "OMPR" in algo:
+ plt.plot(0, -1, **style)
+ style["linewidth"] *= 1.2
+ style.pop("label")
+ else:
+ metric = metrics[plot_name].mean(axis=0)
+ plt.plot(sparsity, metric, **style)
+
+ # For plot of the introduction
+ if "SEA" not in algo:
+ algos_sup_dist.append(metrics["sup_dist"].mean(axis=0))
+ else:
+ old_font_size = plt.rcParams['font.size']
+ plt.rcParams.update({'font.size': sota_font})
+ plt.figure("sota", figsize=(7, 4.5))
+ plt.plot(sparsity, metrics["sup_dist"].mean(axis=0), **algos[algo]["plot"],
+ linewidth=sota_linewidth * 1.75 if "omp" in algo else sota_linewidth,
+ zorder=0 if "omp" in algo else 10)
+ plt.rcParams.update({'font.size': old_font_size})
+
+ plt.figure("sota")
+ plt.plot(sparsity, np.array(algos_sup_dist).min(axis=0), color="black", linewidth=sota_linewidth,
+ label="SOTA", zorder=100)
+
+ # Design
+ for plot_name, infos in plots.items():
+ fig = plt.figure(plot_name)
+ # Reorder legend
+ if infos.get("legend1") is not None:
+ handles, labels = plt.gca().get_legend_handles_labels()
+ order = np.argsort([get_legend_order(label) for label in labels])
+ ax = plt.gca()
+ first_legend = ax.legend([handles[idx] for idx in order[:9]], [labels[idx] for idx in order[:9]],
+ **infos["legend1"]
+ )
+ ax.add_artist(first_legend) # Add the legend manually to the Axes.
+ plt.legend([handles[idx] for idx in order[9:]], [labels[idx] for idx in order[9:]], ncol=1,
+ loc='lower right', bbox_to_anchor=(0.975, 0.28))
+
+ plt.ylim(*infos["ylim"])
+ plt.xlim(1, 50)
+ plt.xlabel(r'Sparsity')
+ plt.ylabel(infos["ylabel"], loc="top" if plot_name == "sota" else None)
+ if plot_name == "sota":
+ plt.yticks((0, 0.5))
+
+ fig.tight_layout(pad=0, rect=infos["rect"])
+ plt.savefig(plot_dir / f"{plot_name}.svg")
+ if "n_supports" in plot_name:
+ plt.yscale("log")
+ plt.savefig(plot_dir / f"{plot_name}_log.svg")
+ plt.close("all")
+
+
+def plot_dcv_n_supports(expe_name, spars_max):
+ result_folder = RESULT_FOLDER / expe_name
+ plot_dir = result_folder / "icml"
+ plot_dir.mkdir(exist_ok=True)
+
+ solution = np.load(result_folder / "solution.npy")[:, :spars_max, :]
+ sparsity = np.arange(1, spars_max + 1)
+ with np.load(result_folder / DATA_FILENAME, allow_pickle=True) as data:
+ linop = ConvolutionOperator(data["h"], data["x_len"])
+
+ algos = algos_dcv()
+ npy_files = {np_path.stem: np_path
+ for np_path in result_folder.glob("*.npy")
+ if np_path.stem in algos.keys() and "IHT" not in np_path.stem}
+
+ plots = ("n_supports_from_start", "n_supports_new",)
+ global_plot_name = "multi_n_supports"
+ fig, axs = plt.subplots(1, 2, sharey=True, sharex=True, figsize=(20, 7))
+
+ # For each algo
+ for algo, file in npy_files.items():
+ metrics_file = file.parent / "temp_plot_data" / (file.stem + ".npz")
+
+ # Load or compute metrics
+ if metrics_file.is_file():
+ logger.debug(f"Loading {metrics_file}")
+ metrics = np.load(metrics_file)
+ else:
+ logger.debug(f"Computing {metrics_file}")
+ compute_metrcs_from_file(file, spars_max, linop, solution, metrics_file)
+ metrics = np.load(metrics_file)
+
+ # Plot metrics
+ for ax, plot_name in zip(axs, plots):
+ style = dict(**algos[algo]["plot"])
+ ax.plot(sparsity, metrics[plot_name].mean(axis=0), **style, zorder=100 if "OMPR" in algo else None)
+
+ # Design
+ handles, labels = plt.gca().get_legend_handles_labels()
+ order = np.argsort([get_legend_order(label) for label in labels])
+ fig.legend([handles[idx] for idx in order], [labels[idx] for idx in order], ncol=9, loc='upper center',
+ bbox_to_anchor=(0.5, 1))
+
+ plt.ylim(1, 4e3)
+ plt.xlim(1, 50)
+ axs[0].set(ylabel="Number of explored supports")
+ axs[0].set(xlabel="Sparsity")
+ axs[1].set(xlabel="Sparsity")
+
+ fig.tight_layout(pad=1.5, rect=(-.03, -0.07, 1.015, 1.025)) # (left, bottom, right, top))
+ plt.savefig(plot_dir / f"{global_plot_name}.svg")
+
+ plt.yscale("log")
+ for ax in axs:
+ ax.grid(True, which="both", axis="y")
+ plt.savefig(plot_dir / f"{global_plot_name}_log.svg")
+ plt.close("all")
+
+
+def plot_signal_paper(expe_name: str):
+ old_rcParams = plt.rcParams.copy()
+ plt.rcParams.update({
+ 'lines.linewidth': plt.rcParams['lines.linewidth'] * 0.75,
+ 'lines.markersize': 30,
+ })
+ fig_dir = Path(f"figures/exp_deconv/{expe_name}/icml")
+ fig_dir.mkdir(exist_ok=True, parents=True)
+ refs = {}
+ solutions = {}
+ with open(fig_dir.parent / "refs.pkl", "rb") as f:
+ refs_in = pickle.load(f)
+ with open(fig_dir.parent / "solutions.pkl", "rb") as f:
+ solutions_in = pickle.load(f)
+ for dict_in, dict_new in zip((refs_in, solutions_in), (refs, solutions)):
+ for key, value in dict_in.items():
+ dict_new[key.rsplit('_', 1)[0]] = value
+
+ algos = algos_dcv_signal()
+ curve_order = ["x", "SEAFAST-els", "SEAFAST", "IHT", "ELSFAST", "HTPFAST", ]
+ for algo, infos in algos.items():
+ if infos.get("marker") is not None:
+ infos["scatter"].update({"zorder": 100 * curve_order.index(algo)})
+ if algo == "x":
+ infos["scatter"]["s"] = (plt.rcParams['lines.markersize'] * np.sqrt(2.4)) ** 2
+ infos["vlines"]["zorder"] = 10 * curve_order.index(algo)
+ infos["vlines"]["linewidth"] = plt.rcParams['lines.linewidth'] * 0.75
+
+ # Plot
+ fig = plt.figure(figsize=(20, 5))
+ plt.axhline(color=plt.rcParams['axes.edgecolor'])
+ plt.plot(refs["y"], color="black", label="$y$")
+
+ for algo, infos in algos.items():
+ if (infos.get("marker_label") is not None or
+ infos.get("marker_noisy_label") is not None and "noisy" in expe_name):
+ data = refs[algo] if algo == "x" else solutions[algo]
+ add_spikes(data, infos["vlines"])
+ add_markers(data, infos["scatter"], infos["marker"])
+ if algo == "x":
+ infos["scatter"]["s"] = plt.rcParams['lines.markersize'] ** 2
+ add_markers([100], infos["scatter"], infos["marker"],
+ legend=infos["marker_noisy_label"]
+ if infos.get("marker_noisy_label") is not None and "noisy" in expe_name
+ else infos["marker_label"])
+ rect = plt.Rectangle((ZOOM_X_LIM[0], -ZOOM_Y_ABS), ZOOM_X_LIM[1] - ZOOM_X_LIM[0], 2 * ZOOM_Y_ABS,
+ color="black", fill=False, linewidth=plt.rcParams['lines.linewidth'] * 1.2,
+ linestyle="dashed")
+ plt.xlim(*ZOOM_X_LIM)
+ yabs = ZOOM_Y_ABS
+ plt.ylim(-yabs, yabs)
+ ax = plt.gca()
+ ax.spines["bottom"].set_visible(False)
+ ax.spines['bottom'].set_position('center')
+ ax.set_xticks([400])
+ ax.xaxis.zorder = 10000
+ ax.add_patch(rect)
+
+ ax2 = ax.twiny()
+ ax2.set_xlim(*ZOOM_X_LIM)
+ ax2.plot(refs["y"], alpha=0)
+ ax2.spines["bottom"].set_visible(False)
+ ax2.spines['top'].set_position('center')
+ ax2.set_xticks([450])
+ ax2.tick_params(axis='x', labeltop=True)
+
+ handles, labels = ax.get_legend_handles_labels()
+ order = np.argsort([get_legend_order(label, markers=True) for label in labels])
+ first_legend = ax.legend([handles[idx] for idx in order[:-1]], [labels[idx] for idx in order[:-1]], ncol=6,
+ loc='upper left', bbox_to_anchor=(0.45, 0.97))
+ ax.add_artist(first_legend) # Add the legend manually to the Axes.
+ plt.legend([handles[idx] for idx in order[-1:]], [labels[idx] for idx in order[-1:]],
+ loc='upper left', bbox_to_anchor=(0.45, 0.85))
+
+ fig.tight_layout(pad=0, rect=(0, 0, 1, 1)) # (left, bottom, right, top))
+ plt.savefig(fig_dir / "zoom_signal_complete.svg")
+ plt.rcParams.update(old_rcParams)
+ plt.close("all")
+
+
+def plot_signal_paper_light(expe_name: str):
+ old_linewidth = plt.rcParams['lines.linewidth']
+ plt.rcParams['lines.linewidth'] *= 0.75
+ fig_dir = Path(f"figures/exp_deconv/{expe_name}/icml")
+ fig_dir.mkdir(exist_ok=True, parents=True)
+ refs = {}
+ solutions = {}
+ with open(fig_dir.parent / "refs.pkl", "rb") as f:
+ refs_in = pickle.load(f)
+ with open(fig_dir.parent / "solutions.pkl", "rb") as f:
+ solutions_in = pickle.load(f)
+ for dict_in, dict_new in zip((refs_in, solutions_in), (refs, solutions)):
+ for key, value in dict_in.items():
+ dict_new[key.rsplit('_', 1)[0]] = value
+
+ algos = algos_dcv()
+ algos["x"] = deepcopy(algos_base["x"])
+ curve_order = ["HTPFAST", "x", "SEAFAST", "IHT", "ELSFAST", ]
+ for algo, infos in algos.items():
+ if infos.get("marker_label_light") is not None:
+ infos["scatter"].update({"zorder": 100 * curve_order.index(algo)})
+ if algo == "x":
+ infos["scatter"]["s"] = (plt.rcParams['lines.markersize'] * np.sqrt(2.4)) ** 2
+ infos["vlines"]["zorder"] = 10 * curve_order.index(algo)
+ infos["vlines"]["linewidth"] = plt.rcParams['lines.linewidth'] * 0.75
+
+ # Plot
+ fig = plt.figure(figsize=(10, 4))
+ plt.axhline(color=plt.rcParams['axes.edgecolor'])
+ plt.plot(refs["y"], color="black", label="$y$")
+
+ for algo, infos in algos.items():
+ if infos.get("marker_label_light") is not None:
+ data = refs[algo] if algo == "x" else solutions[algo]
+ add_spikes(data, infos["vlines"])
+ add_markers(data, infos["scatter"], infos["marker"])
+ if algo == "x":
+ infos["scatter"]["s"] = plt.rcParams['lines.markersize'] ** 2
+ add_markers([100], infos["scatter"], infos["marker"], legend=infos["marker_label_light"])
+
+ plt.xlim(*ZOOM_X_LIM)
+ yabs = ZOOM_Y_ABS
+ plt.ylim(-yabs, yabs)
+ ax = plt.gca()
+ ax.spines["bottom"].set_visible(False)
+ ax.spines['bottom'].set_position('center')
+ ax.set_xticks([400])
+ ax.xaxis.zorder = 10000
+
+ ax2 = ax.twiny()
+ ax2.set_xlim(*ZOOM_X_LIM)
+ ax2.plot(refs["y"], alpha=0)
+ ax2.spines["bottom"].set_visible(False)
+ ax2.spines['top'].set_position('center')
+ ax2.set_xticks([450])
+ ax2.tick_params(axis='x', labeltop=True)
+
+ handles, labels = ax.get_legend_handles_labels()
+ order = np.argsort([get_legend_order(label, markers=True) for label in labels])
+ first_legend = ax.legend([handles[idx] for idx in order], [labels[idx] for idx in order], ncol=5,
+ loc='lower left', bbox_to_anchor=(-.05, -.15), columnspacing=0.9,
+ )
+ ax.add_artist(first_legend) # Add the legend manually to the Axes.
+
+ fig.tight_layout(pad=0, rect=(0, 0.15, 1, 1)) # (left, bottom, right, top))
+ plt.savefig(fig_dir / "zoom_signal_light.svg")
+ plt.rcParams['lines.linewidth'] = old_linewidth
+ plt.close("all")
+
+
+def plot_signal_paper_full(expe_name: str):
+ old_rcParams = plt.rcParams.copy()
+ plt.rcParams.update({
+ 'lines.linewidth': plt.rcParams['lines.linewidth'] * 0.75,
+ 'lines.markersize': 30,
+ })
+ fig_dir = Path(f"figures/exp_deconv/{expe_name}/icml")
+ fig_dir.mkdir(exist_ok=True, parents=True)
+ refs = {}
+ solutions = {}
+ with open(fig_dir.parent / "refs.pkl", "rb") as f:
+ refs_in = pickle.load(f)
+ with open(fig_dir.parent / "solutions.pkl", "rb") as f:
+ solutions_in = pickle.load(f)
+ for dict_in, dict_new in zip((refs_in, solutions_in), (refs, solutions)):
+ for key, value in dict_in.items():
+ dict_new[key.rsplit('_', 1)[0]] = value
+
+ algos = algos_dcv_signal()
+ curve_order = ["x", "SEAFAST-els", "SEAFAST", "IHT", "ELSFAST", "HTPFAST", ]
+ for algo, infos in algos.items():
+ if infos.get("marker") is not None:
+ infos["scatter"].update({"zorder": 100 * curve_order.index(algo)})
+ if algo == "x":
+ infos["scatter"]["s"] = (plt.rcParams['lines.markersize'] * np.sqrt(2.4)) ** 2
+ if algo == "IHT":
+ infos["scatter"]["s"] = 1.1 * (plt.rcParams['lines.markersize']) ** 2
+ infos["vlines"]["zorder"] = 10 * curve_order.index(algo)
+ infos["vlines"]["linewidth"] = plt.rcParams['lines.linewidth'] * 0.75
+
+ # Plot
+ fig = plt.figure(figsize=(20, 8))
+ plt.axhline(color=plt.rcParams['axes.edgecolor'])
+ plt.plot(refs["y"], color="black", label="$y$")
+
+ for algo, infos in algos.items():
+ if (infos.get("marker_label") is not None or
+ infos.get("marker_noisy_label") is not None and "noisy" in expe_name):
+ data = refs[algo] if algo == "x" else solutions[algo]
+ add_spikes(data, infos["vlines"])
+ add_markers(data, infos["scatter"], infos["marker"])
+ if algo == "x":
+ infos["scatter"]["s"] = plt.rcParams['lines.markersize'] ** 2
+ add_markers([100], infos["scatter"], infos["marker"],
+ legend=infos["marker_noisy_label"]
+ if infos.get("marker_noisy_label") is not None and "noisy" in expe_name
+ else infos["marker_label"])
+ rect = plt.Rectangle((ZOOM_X_LIM[0], -ZOOM_Y_ABS), ZOOM_X_LIM[1] - ZOOM_X_LIM[0], 2 * ZOOM_Y_ABS,
+ color="black", fill=False, linewidth=plt.rcParams['lines.linewidth'] * 1.2,
+ linestyle="dashed")
+
+ plt.xlim(-2, 500)
+ yabs = 6
+ plt.ylim(-yabs, yabs)
+ ax = plt.gca()
+ ax.add_patch(rect)
+ ax.spines["bottom"].set_visible(False)
+ ax.spines['bottom'].set_position('center')
+ ax.set_xticks([100, 200, 300, 400])
+ ax.xaxis.zorder = 10000
+ handles, labels = plt.gca().get_legend_handles_labels()
+ order = np.argsort([get_legend_order(label, markers=True) for label in labels])
+ first_legend = ax.legend([handles[idx] for idx in order[:-1]], [labels[idx] for idx in order[:-1]], ncol=6,
+ loc='upper left', bbox_to_anchor=(0.01, 1)
+ )
+ ax.add_artist(first_legend) # Add the legend manually to the Axes.
+ plt.legend([handles[idx] for idx in order[-1:]], [labels[idx] for idx in order[-1:]],
+ loc='upper left', bbox_to_anchor=(0.01, 0.925))
+
+ fig.tight_layout(pad=0, rect=(0, 0, 1, 1)) # (left, bottom, right, top))
+ plt.savefig(fig_dir / "full_signal.svg")
+ plt.rcParams.update(old_rcParams)
+ plt.close("all")
+
+
+def add_spikes(array, vlines_args: Dict):
+ for x, y in enumerate(array):
+ if y != 0:
+ plt.vlines(x, 0, y, **vlines_args)
+
+
+def add_markers(array, scatter_args: Dict, marker_args: Dict, legend=None):
+ for x, y in enumerate(array):
+ if y != 0:
+ plt.scatter(x, y, marker=MarkerStyle(**marker_args), **scatter_args, label=legend)
+
+
+def iterations_dcv(expe_name):
+ old_rcParams = plt.rcParams.copy()
+ plt.rcParams.update({
+ 'lines.linewidth': plt.rcParams['lines.linewidth'] * 0.75,
+ 'lines.markersize': 30,
+ 'legend.handlelength': 1.7,
+ })
+ fig_dir = Path(f"figures/exp_deconv/{expe_name}/icml")
+ fig_dir.mkdir(exist_ok=True, parents=True)
+ res_norms = {}
+ with open(fig_dir.parent / "res_norms.pkl", "rb") as f:
+ res_norms_in = pickle.load(f)
+ for key, value in res_norms_in.items():
+ res_norms[key.rsplit('_', 1)[0]] = value
+
+ algos = algos_dcv_signal()
+ plt.figure(figsize=(20, 8))
+ curve_order = ["SEAFAST-els", "SEAFAST", "SEAFAST-omp", "IHT", "HTPFAST", "OMPFAST", "ELSFAST", ]
+ if "noisy" not in expe_name:
+ sea_omp_infos = algos.pop("SEAFAST-omp")
+ for algo, infos in algos.items():
+ if infos.get("plot_dcv_iter") is not None:
+ if "SEAFAST-omp" in algo:
+ infos["scatter"] = {"color": infos["plot_dcv_iter"]["color"]}
+ if "SEAFAST-els" in algo and "noisy" not in expe_name:
+ infos["plot"]["label"] += "," + sea_omp_infos["plot"]["label"]
+ infos["plot_dcv_iter"].update({"label": fr'$x^t_{{{infos["plot"]["label"].replace("$", "")}}}$',
+ "zorder": 100 * curve_order.index(algo), "color": infos["scatter"]["color"],
+ "linewidth": plt.rcParams['lines.linewidth'] * 0.75})
+ if "HTPFAST" in algo:
+ infos["plot_dcv_iter"]["linewidth"] = plt.rcParams['lines.linewidth'] * 2.5
+ infos["plot_dcv_iter"]["zorder"] = 10000
+ if "ELS" in algo:
+ infos["plot_dcv_iter"]["linewidth"] = plt.rcParams['lines.linewidth'] * 3.5
+ infos["plot_dcv_iter"]["zorder"] = 10000
+ if len(res_norms[algo]) == 1:
+ add_markers([0] * 4 + [res_norms[algo]],
+ {"color": infos["scatter"]["color"], "s": 700, "zorder": 10000},
+ {"marker": "."})
+ add_markers([100], {"color": infos["scatter"]["color"], "s": 700},
+ {"marker": "."}, legend=fr'$x^t_{{{infos["plot"]["label"].replace("$", "")}}}$')
+ else:
+ plt.plot(res_norms[algo], **infos["plot_dcv_iter"])
+ if "SEAFAST" in algo:
+ infos["plot_dcv_iter_best"].update(
+ {"label": fr'$x^{{t_{{BEST}}}}_{{{infos["plot"]["label"].replace("$", "")}}}$',
+ "zorder": 1000 + 100 * curve_order.index(algo),
+ "linewidth": plt.rcParams['lines.linewidth'] * 2})
+ plt.plot(get_best_hist(res_norms[algo]), **infos["plot_dcv_iter_best"], linestyle=(0, (3, 3)))
+ plt.ylim(-0.004, 1)
+ plt.xlim(-2, 1000)
+ plt.xlabel(r'Iterations')
+ plt.ylabel(r'$\ell_{2, rel\_{loss}}$')
+ plt.legend(ncols=9, loc="upper right", bbox_to_anchor=(1, 1))
+ plt.tight_layout(pad=0)
+ plt.savefig(fig_dir / "iter.svg")
+ plt.rcParams.update(old_rcParams)
+ plt.close("all")
+
+
+def iterations_sup_dcv(expe_name):
+ old_rcParams = plt.rcParams.copy()
+ plt.rcParams.update({
+ 'lines.linewidth': plt.rcParams['lines.linewidth'] * 0.75,
+ 'lines.markersize': 30,
+ 'legend.handlelength': 1.7,
+ })
+ fig_dir = Path(f"figures/exp_deconv/{expe_name}/icml")
+ fig_dir.mkdir(exist_ok=True, parents=True)
+ refs = {}
+ histories = {}
+ with open(fig_dir.parent / "refs.pkl", "rb") as f:
+ refs_in = pickle.load(f)
+ with open(fig_dir.parent / "histories.pkl", "rb") as f:
+ histories_in = pickle.load(f)
+ for dict_in, dict_new in zip((refs_in, histories_in), (refs, histories)):
+ for key, value in dict_in.items():
+ dict_new[key.rsplit('_', 1)[0]] = value
+
+ algos = algos_dcv_signal()
+ fig = plt.figure(figsize=(20, 8))
+ curve_order = ["SEAFAST-els", "SEAFAST", "SEAFAST-omp", "IHT", "HTPFAST", "ELSFAST", ]
+ if "noisy" not in expe_name:
+ sea_omp_infos = algos.pop("SEAFAST-omp")
+ for algo, infos in algos.items():
+ if infos.get("plot_dcv_iter") is not None and histories.get(algo) is not None:
+ if "SEAFAST-omp" in algo:
+ infos["scatter"] = {"color": infos["plot_dcv_iter"]["color"]}
+ if "SEAFAST-els" in algo and "noisy" not in expe_name:
+ infos["plot"]["label"] += "," + sea_omp_infos["plot"]["label"]
+ infos["plot_dcv_iter"].update({"label": fr'$x^{{t(s)}}_{{{infos["plot"]["label"].replace("$", "")}}}$',
+ "zorder": 100 * curve_order.index(algo), "color": infos["scatter"]["color"],
+ "linewidth": plt.rcParams['lines.linewidth'] * 0.75})
+ data = histories[algo].get_loss_by_explored_support() / np.linalg.norm(refs["y"])
+ if "ELSFAST" in algo:
+ infos["plot_dcv_iter"]["linewidth"] = plt.rcParams['lines.linewidth']
+ elif "HTPFAST" in algo:
+ infos["plot_dcv_iter"]["linewidth"] = plt.rcParams['lines.linewidth'] * 2.5
+ infos["plot_dcv_iter"]["zorder"] = 10000
+ if len(data) == 1:
+ add_markers([0] * 4 + [data], {"color": infos["scatter"]["color"], "s": 700, "zorder": 10000},
+ {"marker": "."})
+ add_markers([100], {"color": infos["scatter"]["color"], "s": 700},
+ {"marker": "."}, legend=fr'$x^t_{{{infos["plot"]["label"].replace("$", "")}}}$')
+ else:
+ plt.plot(data, **infos["plot_dcv_iter"])
+ if "SEAFAST" in algo:
+ infos["plot_dcv_iter_best"].update(
+ {"label": fr'$x^{{t_{{BEST}}}}_{{{infos["plot"]["label"].replace("$", "")}}}$',
+ "zorder": 1000 + 100 * curve_order.index(algo),
+ "linewidth": plt.rcParams['lines.linewidth'] * 2})
+ plt.plot(get_best_hist(data), **infos["plot_dcv_iter_best"], linestyle=(0, (3, 3)))
+
+ plt.ylim(-0.004, 1)
+ plt.xlim(-2, 1000)
+ plt.xlabel(r'Number $s$ of explored supports')
+ plt.ylabel(r'$\ell_{2, rel\_{loss}}$')
+ plt.legend(ncols=9, loc="upper right", bbox_to_anchor=(1, 1))
+ plt.tight_layout(pad=0)
+ plt.savefig(fig_dir / "hist.svg")
+ plt.rcParams.update(old_rcParams)
+ plt.close("all")
+
+
+def plot_ml_paper(datasets=("cal_housing", "comp-activ-harder", "letter", "ijcnn1", "year", "slice"), factors=(256,)):
+ global legend_order
+ fig_dir = Path("results/training_tasks_plot_icml")
+ fig_dir.mkdir(exist_ok=True, parents=True)
+ datasets_sorted = list(datasets)
+ datasets_sorted.sort(key=lambda elt: DatasetOperator.ORDER.index(elt))
+ algos_raw = algos_from_factor_ml(factors[0])
+ algos = {}
+ # Add curve order
+ curve_order = ["remove",
+ "OMPR", "ELS", "OMP", "SEAFAST-omp", "SEAFAST-els", "SEAFAST", "IHT-els", "IHT", "IHT-omp", "HTP",
+ "HTP-els", "HTP-omp", ]
+
+ data_data = {
+ "cal_housing": {"ylim": (199, 306),
+ "legend": {"ncol": 1, "loc": "upper right", "bbox_to_anchor": (1, 1)},
+ "labels": {
+ "ELS": ["HTP-els", "IHT-els", "OMPR"],
+ "OMP": ["HTP-omp", "IHT-omp", ],
+ "HTP": [],
+ "IHT": [],
+ "SEA-els": ["SEA-omp"],
+ "SEA-0": []
+ }
+ },
+ "comp-activ-harder": {"ylim": (39, 141),
+ "legend": {"ncol": 1, "loc": "upper right", "bbox_to_anchor": (1, 1)},
+ "labels": {
+ "IHT": [],
+ "OMPR": [],
+ "HTP": [],
+ "OMP": ["HTP-omp", "IHT-omp", ],
+ "ELS": ["SEA-0", "SEA-omp", "SEA-els", "IHT-els", "HTP-els", ],
+ }
+ },
+ "letter": {"ylim": (9400, 10670),
+ "legend": {"ncol": 1, "loc": "upper right", "bbox_to_anchor": (1, 1)},
+ "labels": {
+ "ELS": ["SEA-els", "HTP-els", "IHT-els", ],
+ "OMP": ["HTP-omp", "IHT-omp", ],
+ "OMPR": [],
+ "HTP": [],
+ "IHT": [],
+ "SEA-omp": [],
+ "SEA-0": [],
+ }
+ },
+ "ijcnn1": {"ylim": (4750, 8050),
+ "legend": {"ncol": 1, "loc": "upper left", "bbox_to_anchor": (0.35, 1)},
+ "labels": {
+ "ELS": ["HTP-els", "IHT-els", "OMPR"],
+ "OMP": ["HTP-omp", "IHT-omp", ],
+ "HTP": [],
+ "IHT": [],
+ "SEA-0": ["SEA-omp", "SEA-els"],
+ }
+ },
+ "year": {"ylim": (220, 13400),
+ # "legend": {"ncol": 4, "loc": "upper left", "bbox_to_anchor": (0.5, 1)}},
+ "legend": {"ncol": 1, "loc": "upper right", "bbox_to_anchor": (1, 1)},
+ "labels": {
+ "ELS": ["HTP-els", "IHT-els", ],
+ "OMP": ["HTP-omp", "IHT-omp", ],
+ "OMPR": ["SEA-omp"],
+ "HTP": [],
+ "IHT": [],
+ "SEA-0": [],
+ "SEA-els": [],
+ }
+ },
+ "slice": {"ylim": (228, 1020),
+ # "legend": {"ncol": 4, "loc": "upper left", "bbox_to_anchor": (0.5, 1)}},
+ "legend": {"ncol": 1, "loc": "upper right", "bbox_to_anchor": (1, 1)},
+ "labels": {
+ "ELS": ["SEA-els", "HTP-els", "IHT-els", ],
+ "OMP": ["HTP-omp", "IHT-omp", ],
+ "OMPR": [],
+ "HTP": [],
+ "IHT": [],
+ "SEA-omp": [],
+ "SEA-0": [],
+ }
+ },
+ }
+
+ for algo, infos in algos_raw.items():
+ algo_name = algo.replace(f"x{factors[0]}", "").replace("FAST", "")
+ if algo_name == "SEA":
+ algo_name = "SEA-0"
+ if algo_name in curve_order:
+ infos["plot"]["zorder"] = 100 * curve_order.index(algo_name) # Add curve order
+ infos["old_name"] = algo
+ algos[algo_name] = infos
+
+ algos_backup = deepcopy(algos)
+ for data_name in datasets_sorted:
+ legend_backup = legend_order.copy()
+ if data_name in ("cal_housing",):
+ legend_order.insert(legend_order.index("HTP"), legend_order.pop(legend_order.index("SEA-els")))
+ if data_name in ("ijcnn1",):
+ legend_order.insert(legend_order.index("HTP"), legend_order.pop(legend_order.index("SEA-0")))
+ legend_order.append(legend_order.pop(legend_order.index("ELS")))
+ algos = deepcopy(algos_backup)
+ logger.info(f"Plotting for {data_name}")
+ dataset = DatasetOperator(data_name)
+ plt.figure(figsize=(20, 8))
+ for algo, infos in algos.items():
+ if data_data.get(data_name, {}).get("labels").get(algo) is None:
+ continue
+ res_by_k = []
+ for k in range(1, dataset.k_max + 1):
+ if (RESULT_PATH / f"{infos['old_name']}_1_{dataset.name}_{k}.npz").is_file():
+ path = RESULT_PATH / f"{infos['old_name']}_1_{dataset.name}_{k}.npz"
+ else:
+ path = RESULT_PATH / f"{infos['old_name']}_0_{dataset.name}_{k}.npz"
+ with np.load(path) as file:
+ res_by_k.append(dataset.f(file["x"]))
+ if algo in data_data[data_name]["labels"]:
+ infos["plot"]["label"] = ", ".join([algos[algo_key]["plot"]["label"] for algo_key in
+ chain((algo,), data_data[data_name]["labels"][algo])])
+ label = infos["plot"].get("label")
+ if "OMP" == algo:
+ plt.plot(0, 1, **infos["plot"])
+ infos["plot"]["linestyle"] = (0, (0, 2, 1, 0))
+ infos["plot"].pop("label")
+ if "ELS" in algo and data_name != "cal_housing":
+ plt.plot(0, 1, **infos["plot"])
+ infos["plot"]["linestyle"] = (0, (0, 2, 1, 0))
+ infos["plot"].pop("label")
+ plt.plot(range(1, dataset.k_max + 1), res_by_k, **infos["plot"])
+ infos["plot"]["label"] = label
+ handles, labels = plt.gca().get_legend_handles_labels()
+ order = np.argsort([get_legend_order(label, markers=True, algo_dict=algos) for label in labels])
+ plt.legend([handles[idx] for idx in order], [labels[idx] for idx in order],
+ **data_data.get(data_name, {}).get("legend", {}))
+ plt.ylim(*data_data.get(data_name, {}).get("ylim", (None, None)))
+ plt.xlim(1, dataset.k_max)
+ plt.xlabel("Sparsity")
+ plt.ylabel(r"$\ell_{2}$\_loss")
+ plt.tight_layout(pad=0)
+ plt.savefig(fig_dir / f"{data_name}.svg")
+ legend_order = legend_backup.copy()
+ plt.close("all")
+
+
+if __name__ == '__main__':
+ plot_ml_paper(('cal_housing', 'comp-activ-harder', 'ijcnn1', 'letter', 'slice', 'year',), (256,))
+
+ plot_dt_paper(expe_name="unifr1000", threshold=0.95, factors=(256, ))
+ plot_dt_paper_zoom(expe_name="unifr1000", threshold=0.95, factors=(256, ))
+ plot_dt_paper(expe_name="unifr1000_noisy_n1e2", threshold=0.95, factors=(256,))
+ plot_dt_paper_zoom(expe_name="unifr1000_noisy_n1e2", threshold=0.95, factors=(256,))
+
+ dcv_expe_name = "dcv_hist_1e5"
+ plot_dcv_paper(expe_name=dcv_expe_name, spars_max=50)
+ plot_dcv_n_supports(expe_name=dcv_expe_name, spars_max=50)
+
+ dcv_expe_name_noisy = "dcv_hist_noisy_1e5"
+ plot_dcv_paper(expe_name=dcv_expe_name_noisy, spars_max=50)
+ plot_dcv_n_supports(expe_name=dcv_expe_name_noisy, spars_max=50)
+
+ signal_expe_name = "test_noiseless"
+ signal_expe_name_noisy = "noisy"
+ for expe_name in (signal_expe_name, signal_expe_name_noisy):
+ plot_signal_paper(expe_name=expe_name)
+ plot_signal_paper_full(expe_name=expe_name)
+ iterations_dcv(expe_name=expe_name)
+ iterations_sup_dcv(expe_name=expe_name)
+ plot_signal_paper_light(expe_name=expe_name)
+
+ plt.close("all")
+ dcv_expe_name_noisy = "dcv_hist_noisy_1e5"
+ signal_expe_name_noisy = "noisy"
+ for file in chain((RESULT_FOLDER / dcv_expe_name_noisy / "icml").glob("*"),
+ Path(f"figures/exp_deconv/{signal_expe_name_noisy}/icml").glob("*")):
+ if not file.stem.endswith("noisy"):
+ # rename file
+ file.rename(file.parent / f"{file.stem}_noisy{file.suffix}")
diff --git a/code/sksea/run_exp_deconv.py b/code/sksea/run_exp_deconv.py
new file mode 100644
index 0000000000000000000000000000000000000000..511a14d92a53ec0dc6cdadf19c0adfb349cd188f
--- /dev/null
+++ b/code/sksea/run_exp_deconv.py
@@ -0,0 +1,480 @@
+# Python imports
+import pickle
+from pathlib import Path
+from shutil import rmtree
+import socket
+
+# Module imports
+import click
+from loguru import logger
+import numpy as np
+from numpy.random import RandomState
+from plotly import graph_objects as go, express as px
+import ray
+from scipy.stats import wasserstein_distance
+from tqdm import tqdm
+
+from sksea.algorithms import ExplorationHistory
+# Script imports
+from sksea.deconvolution import ConvolutionOperator, gen_u, gen_filter
+from sksea.exp_phase_transition_diag import NoiseType, gen_noise
+from sksea.plot_icml import plot_dcv_n_supports, plot_dcv_paper
+from sksea.training_tasks import ALGOS_TYPE, select_algo, ALL, get_algo_dict
+from sksea.utils import ALGOS_PAPER_DCV, compute_metrcs_from_file, PAPER_LAYOUT, RESULT_FOLDER, DATA_FILENAME
+
+MODULE_PATH = Path(__file__).parent
+ROOT = MODULE_PATH / "temp" / "deconv"
+
+
+def solve_problem(algorithm, linop, n_nonzero, seed, range_max, temp_folder, noise_factor, noise_type, keep_temp=False,
+ n_iter=1000):
+ save_file = temp_folder / (str(seed) + ".npy")
+ save_file_supp = temp_folder / (str(seed) + ".pkl")
+ if keep_temp and save_file.is_file() and (algorithm is None or save_file_supp.is_file()):
+ # logger.debug(f"{save_file} already solved")
+ pass
+ else:
+ rand = RandomState(seed=seed)
+ x = gen_u(linop.shape[0], n_nonzero, 4, rand, range_max)
+ y = linop(x)
+ if noise_factor is not None:
+ y += gen_noise(noise_type, noise_factor, y, rand)
+
+ if algorithm is None:
+ x_algo = x
+ else:
+ x_algo, *other_out = algorithm(linop=linop, y=y, n_nonzero=n_nonzero, n_iter=n_iter,
+ f=lambda x, linop: np.linalg.norm(linop @ x - y),
+ grad_f=lambda x, linop: linop.H @ (linop @ x - y))
+ with open(save_file_supp, "wb") as f:
+ pickle.dump(other_out, f)
+ np.save(save_file, x_algo, False)
+
+
+def run_experiment(sigma, sigma_factor, n_runs, h_len, x_len, expe_name, noise_factor, algos, noise_type,
+ spars_max=None, sparsities=None, keep_temp=False, store_solution=True, reverse_seed=False,
+ seeds=tuple(), n_iter=1000):
+ # Generate filter
+ h = gen_filter(h_len, 3, sigma)
+ linop = ConvolutionOperator(h, x_len)
+
+ # Get algorithms we want to run
+ algorithms_solve = select_algo(ALGOS_TYPE[:4], algos,
+ sea_params=dict(return_best=True))
+ algorithms = dict(**algorithms_solve)
+ if store_solution:
+ algorithms['solution'] = None
+ range_max = 2 * sigma * sigma_factor if sigma_factor != 0 else x_len
+ spars_max = min(range_max, x_len // 2 + 1) if spars_max is None else spars_max + 1
+ temp_root = ROOT / expe_name
+ temp_root.mkdir(parents=True, exist_ok=True)
+ data = dict(h=h, x_len=x_len, sigma=sigma, sigma_factor=sigma_factor, n_runs=n_runs, range_max=range_max,
+ h_len=h_len, noise_factor=noise_factor, spars_max=spars_max, sparsities=sparsities,
+ noise_type=noise_type, n_iter=n_iter)
+ data_path = temp_root / DATA_FILENAME
+ if not data_path.is_file() or not keep_temp:
+ np.savez(data_path, **data)
+ for name, algorithm in algorithms.items():
+ logger.info(name)
+ if not keep_temp:
+ rmtree(temp_root / name, ignore_errors=True)
+ iterator = sparsities if len(sparsities) != 0 else range(1, spars_max)
+ for n_nonzero in iterator:
+ temp_folder = temp_root / name / str(n_nonzero)
+ logger.debug(f"Solving k={n_nonzero} with {name}")
+ temp_folder.mkdir(parents=True, exist_ok=True)
+ if keep_temp and len(tuple(temp_folder.glob("*.npy"))) < n_runs or len(seeds) == 0:
+ if len(seeds) == 0:
+ seed_iterator = range(n_runs, -1, -1) if reverse_seed else range(n_runs)
+ else:
+ seed_iterator = seeds
+ for seed in tqdm(seed_iterator, desc=f"{name} , k={n_nonzero}"):
+ solve_problem(algorithm, linop, n_nonzero, seed, range_max, temp_folder, noise_factor, noise_type,
+ keep_temp, n_iter=n_iter)
+ else:
+ logger.debug(f"k={n_nonzero} Already done")
+
+
+def get_n_supports_temp_from_start(history, best=None) -> int:
+ """
+ Return the number of supports visited by the algorithm including the ones visited by the previous algorithms
+ """
+ supports = set()
+ for previous_it in history.old_it:
+ supports.update(previous_it.keys())
+ if best and history.best_it is None:
+ raise ValueError("No best support found")
+ elif best or (best is None and history.best_it is not None):
+ for buffer, iterations in history.it.items():
+ if iterations[0] <= history.best_it:
+ supports.add(buffer)
+ else:
+ supports.update(history.it.keys())
+ return len(supports)
+
+
+def get_n_supports_temp_new(history, best=None) -> int:
+ """
+ Return the number of supports visited by the algorithm
+ ONLY including the ones NOT visited by the previous algorithms
+ """
+ supports = set()
+ new_supports = set()
+ for previous_it in history.old_it:
+ supports.update(previous_it.keys())
+ if best and history.best_it is None:
+ raise ValueError("No best support found")
+ elif best or (best is None and history.best_it is not None):
+ for buffer, iterations in history.it.items():
+ if iterations[0] <= history.best_it and buffer not in supports:
+ new_supports.add(buffer)
+ else:
+ new_supports.update(set(history.it.keys()) - supports)
+ return len(new_supports)
+
+
+def get_n_supports_temp(history, best=None) -> int:
+ """
+ Return the number of supports visited by the algorithm
+ """
+ n_supports = 0
+ if best and history.best_it is None:
+ raise ValueError("No best support found")
+ elif best or (best is None and history.best_it is not None):
+ for buffer, iterations in history.it.items():
+ if iterations[0] <= history.best_it:
+ n_supports += 1
+ else:
+ n_supports += len(history.it)
+ return n_supports
+
+
+def compile_results(expe_name, keep_temp=False, n_runs=None, spars_max=None, algos_filter=(ALL,), store_solution=True):
+ temp_root = ROOT / expe_name
+ result_folder = RESULT_FOLDER / expe_name
+ result_folder.mkdir(parents=True, exist_ok=True)
+ with np.load(temp_root / DATA_FILENAME, allow_pickle=True) as data: # With statement is needed because of .npz file
+ n_runs = data["n_runs"] if n_runs is None else n_runs
+ if spars_max is None:
+ if "spars_max" in data:
+ spars_max = data["spars_max"]
+ else:
+ spars_max = data["range_max"] - 1
+ logger.info("Compiling....")
+ algorithms_name = [f.name for f in temp_root.glob("*")] if ALL in algos_filter else list(algos_filter)
+ if store_solution:
+ algorithms_name.append("solution")
+ for algo_name in algorithms_name:
+ algo_folder = temp_root / algo_name
+ results = np.zeros((n_runs, spars_max, data["x_len"]))
+ n_supports = np.zeros((n_runs, spars_max))
+ n_supports_new = np.zeros((n_runs, spars_max))
+ n_supports_from_start = np.zeros((n_runs, spars_max))
+ if algo_folder.is_dir():
+ for n_nonzero in range(1, spars_max + 1):
+ temp_folder = algo_folder / str(n_nonzero)
+ if temp_folder.is_dir():
+ logger.debug(f"Compiling {temp_folder}")
+ # for file in temp_folder.glob("*"):
+ for run_id in range(n_runs):
+ file = temp_folder / f"{run_id}.npy"
+ try:
+ results[int(file.stem), n_nonzero - 1] = np.load(file)
+ except FileNotFoundError:
+ logger.error(f"Can't open {file}")
+ if algo_folder.name != "solution":
+ try:
+ pkl_file = temp_folder / f"{run_id}.pkl"
+ with open(pkl_file, 'rb') as f:
+ history: ExplorationHistory = pickle.load(f)[1]
+ # TODO: Get back to history.get_n_supports() when fixed
+ n_supports[int(file.stem), n_nonzero - 1] = get_n_supports_temp(
+ history,
+ best=(algo_folder.name.startswith("SEAFAST") or
+ algo_folder.name.startswith("HTPFAST"))
+ )
+ n_supports_new[int(file.stem), n_nonzero - 1] = get_n_supports_temp_new(
+ history,
+ best=(algo_folder.name.startswith("SEAFAST") or
+ algo_folder.name.startswith("HTPFAST"))
+ )
+ n_supports_from_start[
+ int(file.stem), n_nonzero - 1] = get_n_supports_temp_from_start(
+ history,
+ best=(algo_folder.name.startswith("SEAFAST") or
+ algo_folder.name.startswith("HTPFAST"))
+ )
+ except FileNotFoundError:
+ logger.error(f"Can't open {pkl_file}")
+ np.save(result_folder / algo_folder.name, results)
+ np.savez(result_folder / algo_folder.name, n_supports=n_supports,
+ n_supports_new=n_supports_new, n_supports_from_start=n_supports_from_start)
+ if not keep_temp:
+ rmtree(algo_folder)
+ np.savez(result_folder / DATA_FILENAME, **data)
+ if not any(result_folder.iterdir()):
+ rmtree(temp_folder)
+
+
+def add_curve(fig, sparcity, metric, color, name, paper=False, fig_type=None):
+ mean = metric.mean(axis=0)
+ legend_ranks = ["HTP", "IHT", "OMP", "OMPR", "ELS", "SEAFAST-els", "SEAFAST-omp", "SEAFAST", ]
+
+ if paper:
+ if name == "OMP":
+ if fig_type in ("sup", "ws"):
+ display_name = "$\\text{OMP}, \\text{IHT}_{\\text{OMP}}, \\text{HTP}_{\\text{OMP}}$"
+ else:
+ display_name = "$\\text{OMP}, \\text{IHT}_{\\text{OMP}}, \\text{HTP}_{\\text{OMP}}$"
+ elif name == "ELS":
+ display_name = "$\\text{ELS}, \\text{IHT}_{\\text{ELS}}, \\text{HTP}_{\\text{ELS}}$"
+ else:
+ display_name = ALGOS_PAPER_DCV[name]["name"]
+
+ fig.add_trace(
+ go.Scatter(x=sparcity, y=mean, name=display_name, line=ALGOS_PAPER_DCV[name]["line"],
+ legendrank=legend_ranks.index(name)))
+ else:
+ fig.add_trace(
+ go.Scatter(x=sparcity, y=mean, name=name, line=dict(color=color, width=4)))
+
+
+def plot_metrics_from_file(file, file_id, spars_max, linop, solution, paper, fig_mse, fig_sup, fig_y, fig_ws, fig_n_sup,
+ fig_n_sup_new, fig_n_sup_from_start, force_recompute=False):
+ if file.is_file():
+ sparcity = np.arange(1, spars_max + 1)
+ colors = px.colors.qualitative.Plotly
+ color = colors[file_id % len(colors)]
+ temp_plot_file = file.parent / "temp_plot_data" / (file.stem + ".npz")
+ *other_dim, last = solution.shape
+ if temp_plot_file.is_file() and not force_recompute:
+ logger.debug(f"Loading {temp_plot_file}")
+ else:
+ logger.debug(f"Computing {temp_plot_file}")
+ compute_metrcs_from_file(file, spars_max, linop, solution, temp_plot_file)
+ plot_metrics_from_file(file, file_id, spars_max, linop, solution, paper, fig_mse, fig_sup, fig_y, fig_ws,
+ fig_n_sup, fig_n_sup_new, fig_n_sup_from_start, force_recompute=False)
+ return None
+ temp_plot_data = np.load(temp_plot_file)
+ mse = temp_plot_data["mse"]
+ sup_dist = temp_plot_data["sup_dist"]
+ f_mse_y = temp_plot_data["f_mse_y"]
+ ws = temp_plot_data.get("ws")
+ n_supports = temp_plot_data.get("n_supports")
+ n_supports_new = temp_plot_data.get("n_supports_new")
+ n_supports_from_start = temp_plot_data.get("n_supports_from_start")
+
+ if 0 in n_supports:
+ logger.error(f"{file.stem} not fully computed")
+ add_curve(fig_ws, sparcity, ws, color, file.stem, paper=paper, fig_type="ws")
+ add_curve(fig_n_sup, sparcity, n_supports, color, file.stem, paper=paper, fig_type="n_supports")
+ add_curve(fig_n_sup_new, sparcity, n_supports_new, color, file.stem, paper=paper, fig_type="n_supports")
+ add_curve(fig_n_sup_from_start, sparcity, n_supports_from_start, color, file.stem, paper=paper,
+ fig_type="n_supports")
+ add_curve(fig_mse, sparcity, mse, color, file.stem, paper=paper)
+ add_curve(fig_sup, sparcity, sup_dist, color, file.stem, paper=paper, fig_type="sup")
+ add_curve(fig_y, sparcity, f_mse_y.reshape(other_dim), color, file.stem, paper=paper, fig_type="mse_y")
+
+
+def plot_figures(expe_name, paper=False, spars_max=None, force_recompute=False):
+ result_folder = RESULT_FOLDER / expe_name
+ with np.load(result_folder / DATA_FILENAME, allow_pickle=True) as data:
+ fig_mse = go.Figure()
+ fig_sup = go.Figure()
+ fig_y = go.Figure()
+ fig_ws = go.Figure()
+ fig_n_sup = go.Figure()
+ fig_n_sup_new = go.Figure()
+ fig_n_sup_from_start = go.Figure()
+ spars_max = min(data["range_max"], 32) if spars_max is None else spars_max
+ solution = np.load(result_folder / "solution.npy")[:, :spars_max, :]
+ linop = ConvolutionOperator(data["h"], data["x_len"])
+ params = dict(**data)
+ params.pop("h")
+ subtitle = f"<br><sup>{params}</sup>"
+ layout = dict(xaxis_title="Sparsity")
+ if paper:
+ layout.update(PAPER_LAYOUT)
+ layout.update(
+ legend_title=None,
+ legend_orientation='h',
+ legend_bgcolor='rgba(0,0,0,0)'
+ )
+ else:
+ layout.update(dict(legend_title="Algorithms", ))
+
+ if paper:
+ paths = [result_folder / f"{name}.npy" for name in ALGOS_PAPER_DCV.keys()]
+ else:
+ paths = [path for path in result_folder.glob("*.npy") if path.name != "solution.npy"]
+ for idx, file in enumerate(paths):
+ plot_metrics_from_file(file, idx, spars_max, linop, solution, paper, fig_mse, fig_sup, fig_y, fig_ws, fig_n_sup,
+ fig_n_sup_new, fig_n_sup_from_start, force_recompute)
+ if paper:
+ fig_mse.update_layout(yaxis_title=r"$\text{Mean of } \ell_{2, \text{rel}}$", **layout,
+ legend=dict(y=1, x=0.05, xanchor="left", yanchor="top"))
+ fig_ws.update_layout(yaxis_title=r"$\text{Average Wasserstein distance}$", **layout,
+ legend=dict(y=0, x=0.45, xanchor="left", yanchor="bottom", entrywidth=200))
+ fig_ws.update_layout(margin=dict(l=80, ), yaxis=dict(showexponent='all', exponentformat='e'))
+ fig_sup.update_layout(yaxis_title=r"$\text{Mean of } \text{supp}_{\text{dist}}$", **layout,
+ legend=dict(y=0, x=1.01, xanchor="right", yanchor="bottom",
+ entrywidth=255)) # , font_size=25) )
+ fig_y.update_layout(yaxis_title=r"$\text{Mean of } \ell_{2, \text{rel}\_\text{loss}}$", **layout,
+ legend=dict(y=0, x=1.01, xanchor="right", yanchor="bottom", entrywidth=255))
+ fig_y.update_layout(margin=dict(l=70, b=55, ))
+
+ else:
+ fig_mse.update_layout(yaxis_title="MSE mean over x",
+ title=f"MSE mean and std over x by sparsity {subtitle}", **layout)
+ fig_sup.update_layout(yaxis_title="Support distance mean",
+ title=f"Support distance mean by sparsity {subtitle}", **layout)
+ fig_ws.update_layout(yaxis_title="Wasserstein distance mean over x",
+ title=f"Wasserstein distance mean by sparsity {subtitle}", **layout)
+ fig_y.update_layout(yaxis_title="MSE mean and std over y",
+ title=f"MSE mean and std over y by sparsity {subtitle}", **layout)
+ fig_n_sup.update_layout(yaxis_title="Number of support explored mean", yaxis_type="log",
+ title=f"Number of support explored mean by sparcity {subtitle}", **layout)
+ fig_n_sup_new.update_layout(yaxis_title="Number of NEW support explored mean", yaxis_type="log",
+ title=f"Number of support NEW explored mean by sparcity {subtitle}", **layout)
+ fig_n_sup_from_start.update_layout(yaxis_title="Number of support explored from start mean", yaxis_type="log",
+ title=f"Number of support explored from start mean by sparcity {subtitle}",
+ **layout)
+ write_folder = result_folder / "paper" if paper else result_folder / "draft"
+ write_folder.mkdir(parents=True, exist_ok=True)
+ fig_mse.write_html(write_folder / "mse.html", include_mathjax='cdn')
+ fig_ws.write_html(write_folder / "ws.html", include_mathjax='cdn')
+ fig_sup.write_html(write_folder / "sup_dist.html", include_mathjax='cdn')
+ fig_y.write_html(write_folder / "mse_y.html", include_mathjax='cdn')
+ fig_n_sup.write_html(write_folder / "n_sup.html", include_mathjax='cdn')
+ fig_n_sup_new.write_html(write_folder / "n_sup_new.html", include_mathjax='cdn')
+ fig_n_sup_from_start.write_html(write_folder / "n_sup_from_start.html", include_mathjax='cdn')
+ fig_n_sup.update_layout(yaxis_type="linear")
+ fig_n_sup_new.update_layout(yaxis_type="linear")
+ fig_n_sup_from_start.update_layout(yaxis_type="linear")
+ fig_n_sup.write_html(write_folder / "n_sup_linear.html", include_mathjax='cdn')
+ fig_n_sup_new.write_html(write_folder / "n_sup_new_linear.html", include_mathjax='cdn')
+ fig_n_sup_from_start.write_html(write_folder / "n_sup_from_start_linear.html", include_mathjax='cdn')
+
+
+def coherence(linop):
+ matrix = linop.get_normalized_operator()[0].matrix
+ return np.max(np.abs(np.conjugate(matrix.T) @ matrix) - np.eye(linop.shape[1]))
+
+
+def display_coherences(sigmas, h_len, x_len):
+ for sigma in sigmas:
+ h = gen_filter(h_len, 3, sigma)
+ linop = ConvolutionOperator(h, x_len)
+ print(sigma, coherence(linop))
+
+
+def debug_sigma(expe_names):
+ fig = go.Figure()
+ for expe_name in expe_names:
+ result_folder = RESULT_FOLDER / expe_name
+ data = np.load(result_folder / DATA_FILENAME)
+ params = dict(**data)
+ h = params.pop("h")
+ fig.add_trace(go.Scatter(y=h, name=f"sigma = {params['sigma']}"))
+ fig.write_html("debug.html")
+
+
+def plot_exploration_size(expe_name, algo_names, sparsities, seeds, n_runs, reverse_seed):
+ temp_root = ROOT / expe_name
+ for name in algo_names:
+ for n_nonzero in sparsities:
+ temp_folder = temp_root / name / str(n_nonzero)
+ if len(seeds) == 0:
+ seed_iterator = range(n_runs, -1, -1) if reverse_seed else range(n_runs)
+ else:
+ seed_iterator = seeds
+ len_data = []
+ for seed in seed_iterator:
+ with open(temp_folder / f"{seed}.pkl", 'rb') as f:
+ _, exploration_history = pickle.load(f)
+ len_data.append(exploration_history.get_n_supports())
+ if seed == 118:
+ print(exploration_history.get_n_supports())
+ fig = go.Figure()
+ fig.add_trace(go.Histogram(x=len_data, name="Number of supports"))
+ layout = dict()
+ layout.update(PAPER_LAYOUT)
+ layout.update(
+ legend_title=None,
+ legend_orientation='h',
+ legend_bgcolor='rgba(0,0,0,0)',
+ margin=dict(
+ autoexpand=False,
+ l=55,
+ r=5,
+ t=30,
+ b=70,
+ ),
+ )
+ display_name = ALGOS_PAPER_DCV[name]["name"]
+ fig.update_layout(**layout, xaxis_title=rf"$\text{{Number of explored supports by }}{display_name[1:-1]}$",
+ yaxis_title=r"$\text{Number of runs}$")
+ fig.write_html(temp_folder / "histogram.html", include_mathjax='cdn')
+
+
+@click.command(context_settings={'show_default': True, 'help_option_names': ['-h', '--help']})
+@click.option('--sigma', '-s', default=3, type=int, help='Variance of the gaussian filter')
+@click.option('--sigma_factor', '-sf', default=0, type=int,
+ help='If 0, the support of the signal can be spread over the whole signal, else it is spread over '
+ 'sigma_factor*sigma')
+@click.option('--n_runs', '-ru', default=200, type=int, help='Number of times each configuration is tested')
+@click.option('--h_len', '-hl', default=500, help='Size of the gaussian filter')
+@click.option('--x_len', '-xl', default=500, help='Size of the signal')
+@click.option('--expe-name', '-en', default='', help="Name of the experiment")
+@click.option('--noise-factor', '-nf', default=None, type=float, help="Variance of the gaussian noise")
+@click.option('--noise_type', '-nt', default=NoiseType.NOISE_LEVEL.name,
+ type=click.Choice(dir(NoiseType)[:len(NoiseType)], case_sensitive=False), help="How compute the noise")
+@click.option('--algos_filter', '-af', multiple=True,
+ default=["SEAFAST-els", "SEAFAST-omp", "SEAFAST", "ELSFAST", "OMPFAST", "IHT",
+ "HTPFAST", "IHT-omp", "HTP-omp", "IHT-els", "HTPFAST-els", "OMPRFAST"],
+ type=click.Choice(list(get_algo_dict(*[True] * len(ALGOS_TYPE[:4])).keys()) + [ALL],
+ case_sensitive=False),
+ help='Algorithms to run. If \'ALL\' is selected, run all algorithms.')
+@click.option('--plot', '-pl', is_flag=True,
+ help='If specified, plot results instead of running algorithmslike in the paper')
+@click.option('--plot_draft', '-pd', is_flag=True, help='If specified, plot results instead of running algorithms')
+@click.option('--plot_hist', '-plh', is_flag=True, help='If specified, plot histograms')
+@click.option('--spars_max', '-sm', default=50, type=int, help='Sparsity maximum of the problems')
+@click.option('--sparsities', '-sp', default=None, type=int, multiple=True, help='Sparsity to analyze')
+@click.option('--compile', '-cp/-ncp', is_flag=True, default=True, help='If specified, compile results if not plotting')
+@click.option('--run', '-r/-nr', is_flag=True, default=True, help='If specified, run experiment if not plotting')
+@click.option('--keep_temp', '-kt', is_flag=True, default=True, help='If specified, keep temporary files across runs')
+@click.option('--store_solution', '-ss/-nss', is_flag=True, default=True, help='If disabled, does not store solution')
+@click.option('--reverse_seed', '-rs/-nrs', is_flag=True, default=False,
+ help='If enabled, use seeds in decreasing order')
+@click.option('--seeds', '-sd', default=None, type=int, multiple=True, help='Seeds to use for computations')
+@click.option('--n_iter', '-ni', default=1000, type=int, help='Seeds to use for computations')
+@click.option('--force_data_plot', '-fdp/-nfdp', default=False, is_flag=True, help='If True, recompute data for plots')
+def main(sigma, sigma_factor, n_runs, h_len, x_len, expe_name, noise_factor, noise_type, algos_filter, plot, spars_max,
+ sparsities, compile, run, keep_temp, store_solution, reverse_seed, seeds, plot_hist, n_iter, force_data_plot,
+ plot_draft):
+ logger.info(f"Parameters: \n{locals()}")
+ if plot:
+ plot_dcv_paper(expe_name=expe_name, spars_max=spars_max)
+ plot_dcv_n_supports(expe_name=expe_name, spars_max=spars_max)
+ elif plot_draft:
+ if plot_hist:
+ plot_exploration_size(expe_name, algos_filter, sparsities, seeds, n_runs, reverse_seed)
+ # plot_figures(expe_name, paper=True, spars_max=spars_max) # Plot figures of the experiment like in the paper
+ plot_figures(expe_name, paper=False, spars_max=spars_max, force_recompute=force_data_plot)
+ else:
+ if run:
+ # Run the experiment with the parameters
+ run_experiment(sigma, sigma_factor, n_runs, h_len, x_len, expe_name, noise_factor, algos_filter, noise_type,
+ spars_max, sparsities, keep_temp, store_solution, reverse_seed, seeds, n_iter)
+ if compile:
+ # Compile the raw results in code/sksea/results/deconv/{expe_name}
+ compile_results(expe_name, keep_temp, n_runs, spars_max, algos_filter, store_solution)
+ logger.info("End")
+
+
+# Needed for click CLI
+if __name__ == '__main__':
+ main()
+ # main("-sm 50 -s 3 -ru 3 -hl 500 -xl 500 -en dcv_hist_test -r -cp -af SEAFAST -sp 1".split(" "))
+ # main("-sm 50 -s 3 -ru 200 -hl 500 -xl 500 -en dcv_hist -kt -r -ncp -sd 4 -af HTPFAST-els_fast -sp 17".split(" "))
diff --git a/code/sksea/run_exp_phase_transition.py b/code/sksea/run_exp_phase_transition.py
new file mode 100644
index 0000000000000000000000000000000000000000..9bf27556586b96660b5a19d6610fd99e4391768f
--- /dev/null
+++ b/code/sksea/run_exp_phase_transition.py
@@ -0,0 +1,603 @@
+"""
+File for running phase transition experiment on the cluster (via command line).
+You can display the help by using the command `python run_exp_compute_one_phase_transistion_run.py --help`
+The CLI was made with the click library.
+This file contains also a parallel version of the phase transition experiment
+"""
+import pickle
+# Basic python imports
+from itertools import chain
+from pathlib import Path
+from time import time
+from typing import Tuple, Dict, Union, List
+import shutil
+import socket
+import os
+
+# Installed module imports
+import click
+import pandas as pd
+from hyperopt import hp, fmin, tpe, Trials
+from loguru import logger
+import numpy as np
+import ray
+from scipy.spatial.distance import hamming
+
+# Sksea imports
+from sksea.exp_phase_transition_diag import generate_problem, plot_results, \
+ plot_paper_curves, NoiseType, plot_threshold_curves_comparison
+from sksea.plot_icml import plot_dt_paper, plot_dt_paper_zoom
+from sksea.training_tasks import ALL, select_algo, get_algo_dict, ALGOS_TYPE
+
+DATA_DIR = Path('data_results')
+ONE_POINT_DIR = DATA_DIR / 'one_point' / "raw_files"
+
+
+def get_filename(rho, delta, index, ext='npz') -> str:
+ """
+ Return the temporary file name for a given (rho, delta)
+
+ :param (float) rho: Current value of rho
+ :param (float) delta: Current value of delta
+ :param (int or None) index: Index of the result
+ """
+ if index is None:
+ return f"{int(rho * 100000)}_{int(delta * 100000)}.{ext}"
+ else:
+ return f"{int(rho * 100000)}_{int(delta * 100000)}_{index}.{ext}"
+
+
+# @ray.remote
+def solve_problem(run_number, n_samples, rho, delta, n_iter, algo, distribution, rel_tol, n_atoms, factor, deconv,
+ noise_factor, noise_type, **algo_params) -> List[Dict[str, Union[float, int]]]:
+ """
+ Solve a problem like 'D @ x = y' with the following parameters
+
+ :param (int) run_number: Used as a seed for the random generation of the problem
+ :param (int or None) n_samples: Size of y, number of lines of D
+ :param (float) rho: Sparsity, number of non_zeros coefficients in x divided by n_samples.
+ Must be between 0 and 1
+ :param (float) delta: Under-sampling factor, number of lines of D divided by its number of columns.
+ Must be between 0 and 1
+ :param (int or None) n_iter: Number max of iteration that the algorithm can do
+ :param (Callable) algo: Solver to use
+ :param (str) distribution: Probability distribution to use for generating D coefficients.
+ Must be 'gaussian' or '[1, 2]' if deconv is False. Else must be between 0 and 4.
+ :param (float) rel_tol: The algorithm stops when the iterations relative difference is lower than rel_tol
+ :param (int or None) n_atoms: Size of x, number of columns of D. If specified, n_samples must be None.
+ :param (int or None) factor: The number of iterations of the algorithm will be factor * sparsity
+ Used if n_iter is None.
+ TODO :return: Dictionaries with resolution metrics
+ """
+ results = []
+ problem_data, solution_data = \
+ generate_problem(n_samples=n_samples, rho=rho, delta=delta, distribution=distribution, noise_type=noise_type,
+ seed=run_number, n_atoms=n_atoms, deconv=deconv, noise_factor=noise_factor,
+ use_sparse_operator=True)
+ y = problem_data['obs_vec']
+ d = problem_data['dict_mat']
+ n_nonzero = problem_data['n_nonzero']
+ if n_nonzero > 0 and y.shape[0] > 1: # For removing zero-sparse vector and one line matrices
+ start_time = time()
+ params = dict(n_iter=factor * n_nonzero) if n_iter is None else dict(n_iter=n_iter)
+ params.update(**algo_params)
+ out = algo(linop=d, y=y, n_nonzero=n_nonzero, rel_tol=rel_tol,
+ f=lambda x, linop: np.linalg.norm(linop @ x - y),
+ grad_f=lambda x, linop: linop.H @ (linop @ x - y),
+ is_mse=True, **params)
+
+ outputs = out if isinstance(out[0], tuple) else (out,)
+ for idx, (x_est, res_norm, *sea_supp) in enumerate(outputs):
+ results.append({})
+ results[idx]["time"] = time() - start_time
+ # Results from first experiments: Hamming distance and support recovery
+ x_ref: np.ndarray = solution_data['sp_vec']
+ supp_ref: np.ndarray = x_ref != 0
+ supp_est = x_est != 0
+ results[idx]["hamming_dist"] = hamming(supp_est, supp_ref)
+ # Compute euclidian distance for DT phase diagram
+ results[idx]["eucl_dist_rel"] = np.linalg.norm(x_ref - x_est) / np.linalg.norm(x_ref)
+ # Compute euclidian distance on the observation for DT phase diagram
+ results[idx]["eucl_dist_rel_y"] = np.linalg.norm(d @ x_ref - d @ x_est) / np.linalg.norm(d @ x_ref)
+ results[idx]["supp_est_in_sup_ref"] = np.all(supp_ref[supp_est])
+ # Keep information about the optimization process
+ results[idx]["last_res"] = res_norm[-1]
+ results[idx]["iterations"] = len(res_norm)
+ # if len(sea_supp) > 0:
+ # results[idx]["support_history"] = sea_supp[0]
+ else:
+ results.append({})
+ return results
+
+
+@ray.remote
+def solve_batch_problem(run_numbers, n_samples, rho, delta, n_iter, algo, distribution, rel_tol, n_atoms, factor,
+ deconv, noise_factor, noise_type, **algo_params):
+ return [solve_problem(i_run, n_samples, rho, delta, n_iter, algo,
+ distribution, rel_tol, n_atoms, factor, deconv, noise_factor, noise_type, **algo_params)
+ for i_run in range(*run_numbers)]
+
+
+def compute_dt_point(n_runs, n_samples, rho, delta, n_iter, algo, distribution, rel_tol, temp_dir, epsilon, n_atoms,
+ factor, deconv, noise_factor, batch_size, noise_type, **algo_params
+ ) -> Tuple[float, float, float]:
+ """
+ Solve a problem like 'D * x = y' with the following parameters, n_runs times
+
+ :param (int) n_runs: Number of run to do.
+ :param (int or None) n_samples: Size of y, number of lines of D
+ :param (float) rho: Sparsity, number of non_zeros coefficients in x divided by n_samples.
+ Must be between 0 and 1
+ :param (float) delta: Under-sampling factor, number of lines of D divided by its number of columns.
+ Must be between 0 and 1
+ :param (int) n_iter: Number max of iteration that the algorithm can do
+ :param (Callable) algo: Solveur to use
+ :param (str) distribution: Probability distribution to use for generating D coefficients.
+ Must be 'gaussian' or '[1, 2]'
+ :param (float) rel_tol: The algorithm stops when the iterations relative difference is lower than rel_tol
+ :param (Path) temp_dir: Folder to use for temporary files
+ :param (float) epsilon: Threshold for euclidian distance relative difference in DT recovery diagram.
+ If less than 1% of problems are solved above this threshold twice in a row,
+ the computation stops for the current value of delta. If 0, compute the entire diagram.
+ :param (int or None) n_atoms: Size of x, number of columns of D. If specified, n_samples must be None.
+ :param (int or None) factor: The number of iterations of the algorithm will be factor * sparsity
+ Used if n_iter is None.
+ TODO :return: Mean of the relative euclidian distances between `x real` and `x predicted by the algorithm`
+ """
+ # Check if the run is not already done
+ if temp_dir is None:
+ out_file = None
+ else:
+ if (temp_dir / get_filename(rho, delta, 1)).is_file():
+ out_file = temp_dir / get_filename(rho, delta, 1)
+ else:
+ out_file = temp_dir / get_filename(rho, delta, 0)
+ if out_file is not None and out_file.is_file():
+ logger.debug(f"(rho, delta) = {(rho, delta)} is already computed")
+ res = np.load(str(out_file))
+ else:
+ # futures = [solve_problem.remote(i_run, n_samples, rho, delta, n_iter, algo,
+ # distribution, rel_tol, n_atoms, factor, deconv, noise_factor)
+ # for i_run in range(n_runs)]
+ # futures_completed = ray.get(futures)
+ # n_results = len(futures_completed[0])
+
+ futures = [solve_batch_problem.remote((batch_id * batch_size, min((batch_id + 1) * batch_size, n_runs)), # noqa
+ n_samples, rho, delta, n_iter, algo, distribution, rel_tol, # noqa
+ n_atoms, factor, deconv, noise_factor, noise_type, **algo_params) # noqa
+ for batch_id in range(n_runs // batch_size + 1)]
+ futures_batchs_completed = ray.get(futures)
+ futures_completed = list(chain(*futures_batchs_completed))
+ n_results = len(futures_completed[0])
+
+ # Initialize arrays
+ results = [
+ dict(
+ hamming_dist=np.ones(n_runs) * np.nan,
+ supp_est_in_sup_ref=np.zeros(n_runs, dtype=bool),
+ eucl_dist_rel=np.ones(n_runs) * np.nan,
+ eucl_dist_rel_y=np.ones(n_runs) * np.nan,
+ last_res=np.ones(n_runs) * np.nan,
+ iterations=np.ones(n_runs) * np.nan,
+ time=np.ones(n_runs) * np.nan,
+ # support_history=[],
+ ) for _ in range(n_results)]
+
+ for i_run, results_list in enumerate(futures_completed): # Get results for each run in parallel
+ for idx, result_dict in enumerate(results_list):
+ for key, value in result_dict.items():
+ # if key == "support_history":
+ # results[idx][key].append(value)
+ # else:
+ results[idx][key][i_run] = value
+ if out_file is not None:
+ for idx, result in enumerate(results): # Save files
+ np.savez(out_file.with_name(get_filename(rho, delta, idx)), **result)
+
+ res = results[-1]
+ if epsilon is not None:
+ return float(np.mean(res["eucl_dist_rel"] < epsilon)), float(np.mean(res["hamming_dist"] == 0)), float(
+ np.max(res["eucl_dist_rel_y"]))
+ else:
+ return float(np.mean(res["eucl_dist_rel"])), float(np.mean(res["hamming_dist"] == 0)), float(
+ np.max(res["eucl_dist_rel_y"]))
+
+
+@ray.remote
+def compute_dt_column(n_runs, n_samples, delta, n_steps_rho, n_iter, algo,
+ distribution, rel_tol, temp_dir, weak_curve, epsilon, n_atoms, factor, deconv, success,
+ noise_factor, batch_size, noise_type) -> None:
+ """
+ Solve a problem like 'D * x = y' with the following parameters, n_runs times for all possible values of rho
+
+ :param (int) n_runs: Number of run to do.
+ :param (int or None) n_samples: Size of y, number of lines of D
+ :param (float) delta: Under-sampling factor, number of lines of D divided by its number of columns.
+ Must be between 0 and 1
+ :param n_steps_rho:
+ :param (int) n_iter: Number max of iteration that the algorithm can do
+ :param (Callable) algo: Solveur to use
+ :param (str) distribution: Probability distribution to use for generating D coefficients.
+ Must be 'gaussian' or '[1, 2]'
+ :param (float) rel_tol: The algorithm stops when the iterations relative difference is lower than rel_tol
+ :param (Path) temp_dir: Folder to use for temporary files
+ :param (Callable[[float], float]) weak_curve: Function interpolating the asymptotic recovery weak curve.
+ The algorithm will not solve problems with a rho value above this curve and the 1% recovery curve
+ :param (float) epsilon: Threshold for euclidian distance relative difference in DT recovery diagram.
+ If less than 1% of problems are solved above this threshold twice in a row,
+ the computation stops for the current value of delta. If 0, compute the entire diagram.
+ :param (int or None) n_atoms: Size of x, number of columns of D. If specified, n_samples must be None.
+ :param (int or None) factor: The number of iterations of the algorithm will be factor * sparsity
+ Used if n_iter is None.
+ """
+ if n_steps_rho == 0:
+ n_samples_tmp = int(np.round(n_atoms * delta)) if n_samples is None else n_samples
+ axis_range_rho = (np.arange(n_samples_tmp) + 1) / n_samples_tmp
+ else:
+ axis_range_rho = (np.arange(n_steps_rho) + 1) / n_steps_rho
+ last_eucl_dist = 1
+ logger.debug(f"Begin delta={delta}")
+ last_non_recovery = -1
+ last_hamming = -1
+ max_y_rel_err = 1
+ rho = None
+ for idx, rho in enumerate(axis_range_rho):
+ if last_eucl_dist < success and last_hamming < success and max_y_rel_err > 0.1 and epsilon is not None: # rho > weak_curve(delta) and
+ if idx == last_non_recovery + 1:
+ break # We stop only if the recovery fail twice in a row above the asymptotic curve
+ else:
+ last_non_recovery = idx
+ last_eucl_dist, last_hamming, max_y_rel_err = compute_dt_point(
+ n_runs, n_samples, rho, delta, n_iter, algo, distribution, rel_tol, temp_dir, epsilon, n_atoms, factor,
+ deconv, noise_factor, batch_size, noise_type)
+ logger.debug(f"delta={delta:2f}, rho={rho:2f}, rel_y={max_y_rel_err:2f}, hamming={last_hamming:2f}")
+ logger.debug(f"End delta={delta}, rho={rho}")
+
+
+def compute_dt_complete(n_samples, n_steps, n_runs, n_iter, algo, distribution, name, epsilon, *args,
+ rel_tol=-np.inf, resume=False, n_atoms=None, expe_name='', factor=None, deconv=False,
+ success=0.01, noise_factor=None, batch_size=200, noise_type=None,
+ remove_first=False, remove_last=False, keep_temp=False, force_resume=False,
+ delta_values=(), compile_=None, **kwargs) -> None:
+ """
+ Solve ||D * x - y||_2^2 with the provided algorithm for various configurations. Runs are parallelized.
+ Results are saved in the 'data_result' folder
+
+ :param (int or None) n_samples: Size of y, number of lines of D
+ :param (int or Tuple[int, int]) n_steps: Discretization of rho and delta.
+ If an integer is provided, the same discretization is used for both rho and delta.
+ Else, the first element is the discretization for rho, and the second the one for delta
+ :param (int) n_runs: Number of times each configuration is tested
+ :param (int) n_iter: Number max of iteration that the algorithm can do
+ :param (Callable) algo: Solver to use
+ :param (str) distribution: Probability distribution to use for generating D coefficients.
+ Must be 'gaussian' or '[1, 2]'
+ :param (str) name: Name of the algorithm
+ :param (float) rel_tol: The algorithm stops when the iterations relative difference is lower than rel_tol
+ :param (bool) resume: If True, don't redo already done runs
+ :param (float) epsilon: Threshold for euclidian distance relative difference in DT recovery diagram.
+ If less than 1% of problems are solved above this threshold twice in a row,
+ the computation stops for the current value of delta. If 0, compute the entire diagram.
+ :param (int or None) n_atoms: Size of x, number of columns of D. If specified, n_samples must be None.
+ :param (str) expe_name: Name of the current experiment
+ :param (int or None) factor: The number of iterations of the algorithm will be factor * sparsity
+ Used if n_iter is None.
+ :param (Optional[bool]) compile_: If True, does only the compilation of the results.
+ If False, does only the computations of DT
+ """
+ # Creating directory
+ first_part = f'{name}' if factor is None else f'{name}x{factor}'
+ prefix = f'{first_part}_{n_samples}_{n_steps}_{n_runs}_{n_iter}_{rel_tol}_{epsilon}_{n_atoms}_{expe_name}'
+
+ if isinstance(n_steps, int):
+ n_steps_rho, n_steps_delta = n_steps, n_steps
+ elif isinstance(n_steps, tuple):
+ n_steps_rho, n_steps_delta = n_steps
+ else:
+ raise TypeError("n_steps must be an int or a tuple")
+
+ axis_range_delta = (np.arange(n_steps_delta) + 1) / n_steps_delta
+ if remove_first:
+ axis_range_delta = axis_range_delta[1:]
+ if remove_last:
+ axis_range_delta = axis_range_delta[:-1]
+ if len(delta_values) != 0:
+ axis_range_delta = np.concatenate((axis_range_delta, delta_values))
+ axis_range_delta.sort()
+
+ temp_dir = Path('temp') / prefix
+
+ if compile_ is None or not compile_:
+ temp_dir.mkdir(exist_ok=True, parents=True)
+
+ logger.info(f'Algo {first_part}')
+
+ if not resume: # Clean old experiments if needed
+ shutil.rmtree(temp_dir)
+ temp_dir.mkdir(exist_ok=True)
+ elif not force_resume:
+ logger.info(f"Resuming...")
+ if (DATA_DIR / f"{prefix}_0.npz").is_file():
+ logger.info(f"Results for {first_part} have already been computed")
+ return None
+
+ weak_curve = None # plot_theorical_phase_diagram_curve()
+ futures = [compute_dt_column.remote(n_runs, n_samples, delta, n_steps_rho, n_iter, algo, # noqa
+ distribution, rel_tol, temp_dir, weak_curve, epsilon, n_atoms, factor,
+ deconv, # noqa
+ success, noise_factor, batch_size, noise_type) # noqa
+ for delta in axis_range_delta]
+ ray.get(futures)
+
+ if compile_ is None or compile_:
+ logger.info("Compiling results")
+ compile_results(n_steps_rho, axis_range_delta, n_runs, temp_dir, name, n_atoms, n_samples)
+ if not keep_temp:
+ shutil.rmtree(temp_dir)
+
+
+def compile_results(n_steps_rho, axis_range_delta, n_runs, temp_dir, name, n_atoms, n_samples) -> None:
+ """
+ Compile the results stored in the temporary folder, and store them.
+
+ :param (int) n_steps_rho: Discretization of rho
+ :param (np.ndarray) axis_range_delta: Array of delta
+ :param (int) n_runs: Number of times each configuration is tested
+ :param (Path) temp_dir: Folder to use for temporary files
+ :param (str) name: Name of the algorithm
+ """
+ # Creating directory
+ DATA_DIR.mkdir(exist_ok=True)
+
+ n_steps_delta = axis_range_delta.shape[0]
+ idx = 0
+ while len(list(temp_dir.glob(f'*_{idx}.npz'))) > 0:
+ logger.debug(f"Compiling {len(list(temp_dir.glob(f'*_{idx}.npz')))} files for idx={idx}")
+ results = {"axis_range_delta": axis_range_delta}
+ if n_steps_rho == 0:
+ n_samples_tmp = int(
+ np.round(n_atoms * np.max(results["axis_range_delta"]))) if n_samples is None else n_samples
+ results["axis_range_rho_per_sparcity"] = np.zeros((n_steps_delta, n_samples_tmp))
+ for i_delta, delta in enumerate(results["axis_range_delta"]):
+ n_samples_delta = int(np.round(n_atoms * delta)) if n_samples is None else n_samples
+ results["axis_range_rho_per_sparcity"][i_delta] = np.pad(
+ (np.arange(n_samples_delta) + 1) / n_samples_delta, (0, n_samples_tmp - n_samples_delta))
+ n_rho = n_samples_tmp
+ else:
+ results["axis_range_rho"] = (np.arange(n_steps_rho) + 1) / n_steps_rho
+ n_rho = n_steps_rho
+
+ # Initialize arrays
+ results.update(dict(
+ hamming_dist=np.ones((n_rho, n_steps_delta, n_runs)) * np.nan,
+ eucl_dist_rel=np.ones((n_rho, n_steps_delta, n_runs)) * np.nan,
+ eucl_dist_rel_y=np.ones((n_rho, n_steps_delta, n_runs)) * np.nan,
+ supp_est_in_sup_ref=np.zeros((n_rho, n_steps_delta, n_runs), dtype=bool),
+ last_res=np.ones((n_rho, n_steps_delta, n_runs)) * np.nan,
+ iterations=np.ones((n_rho, n_steps_delta, n_runs)) * np.nan,
+ time=np.ones((n_rho, n_steps_delta, n_runs)) * np.nan,
+ # support_history={},
+ ))
+
+ # Retrieve results from parallel computation
+ for i_delta, delta in enumerate(results["axis_range_delta"]):
+ delta: float
+ if n_steps_rho == 0:
+ iterator = results["axis_range_rho_per_sparcity"][i_delta]
+ else:
+ iterator = results["axis_range_rho"]
+ for i_rho, rho in enumerate(iterator):
+ filepath = temp_dir / get_filename(rho, delta, idx)
+ if filepath.is_file():
+ result_dict = np.load(str(filepath))
+ for key, value in result_dict.items():
+ results[key][i_rho, i_delta] = value
+
+ # Store results
+ out_file = f'{temp_dir.name}_{idx}.npz'
+ print(out_file)
+ np.savez(DATA_DIR / out_file, name=name, **results)
+ idx += 1
+
+
+def compute_one_point_from_dt(n_samples, n_runs, n_iter, algo, distribution, name, epsilon, *args, rel_tol=-np.inf,
+ n_atoms=None, expe_name='', factor=None, deconv=False, noise_factor=None, batch_size=200,
+ noise_type=None, rho=0, delta=0, seed=0, max_evals=1, lip_fact=0, **kwargs):
+ if rho == 0 or delta == 0:
+ raise ValueError("rho and delta must be specified")
+ first_part = f'{name}' if factor is None else f'{name}x{factor}'
+ logger.info(f"Computing point {n_samples}x{n_atoms} (rho={rho}, delta={delta}) for {first_part} and lf={lip_fact}")
+
+ space = {'lip_fact': hp.loguniform('lip_fact', np.log(2e-3), np.log(2000))}
+ fn = lambda space_in, temp_dir=None: -compute_dt_point(n_runs, n_samples, rho, delta, n_iter, algo, distribution,
+ rel_tol, temp_dir, epsilon, n_atoms, factor, deconv,
+ noise_factor, batch_size, noise_type,
+ lip_fact=space_in['lip_fact'])[1]
+
+ out_dir = ONE_POINT_DIR / expe_name / f'{first_part}_{n_samples}x{n_atoms}_{lip_fact}'
+ out_dir.mkdir(parents=True, exist_ok=True)
+
+ if lip_fact == 0:
+ trial = Trials()
+ best = fmin(fn=fn, space=space, algo=tpe.suggest, max_evals=max_evals, rstate=np.random.default_rng(seed),
+ loss_threshold=-(1 - np.finfo(float).eps), trials=trial,
+ trials_save_file=out_dir / get_filename(rho, delta, None, ext='pkl'))
+ print(fn(best, out_dir))
+ logger.debug(f"Results: {best}")
+ else:
+ fn({'lip_fact': lip_fact}, out_dir)
+
+
+def display_one_point_from_dt(n_samples, *args, n_atoms=None, expe_name='', rho=0, delta=0, lip_fact=0., **kwargs):
+ df = pd.DataFrame(columns=['algorithm', 'best_lip_fact', 'success_rate_mean', 'success_rate_std', 'time_mean',
+ 'time_std'])
+ for temp_dir in (ONE_POINT_DIR / expe_name).glob(f"*_{n_samples}x{n_atoms}_{lip_fact}"):
+ if (temp_dir / get_filename(rho, delta, 1)).is_file():
+ out_file = temp_dir / get_filename(rho, delta, 1)
+ else:
+ out_file = temp_dir / get_filename(rho, delta, 0)
+ if out_file.is_file():
+ if lip_fact == 0:
+ trials: Trials = pickle.load(open(temp_dir / get_filename(rho, delta, None, ext='pkl'), 'rb'))
+ best_trial = trials.best_trial
+ lip_fact_iter = best_trial['misc']['vals']['lip_fact'][0]
+ else:
+ lip_fact_iter = lip_fact
+ first_part, *_ = temp_dir.name.split('_')
+ name = first_part.split('x')[0]
+ res = np.load(str(out_file))
+ df.loc[len(df.index)] = [name, lip_fact_iter,
+ float(np.mean(res["hamming_dist"] == 0)),
+ float(np.std(res["hamming_dist"] == 0)), float(np.mean(res["time"])),
+ float(np.std(res["time"]))]
+ # print(df)
+ return df
+
+
+def fuse_one_point_from_dt(n_samples, *args, n_atoms=None, expe_name='', rho=0, delta=0, **kwargs):
+ df = display_one_point_from_dt(n_samples, n_atoms=n_atoms, expe_name=expe_name, rho=rho, delta=delta,
+ lip_fact=0.)
+ temp_dirs = (ONE_POINT_DIR / expe_name).glob(f"*_{n_samples}x{n_atoms}_*")
+ lip_factors = set(float(temp_dir.name.split('_')[-1]) for temp_dir in temp_dirs)
+ for lip_fact in lip_factors:
+ print(lip_fact)
+ if lip_fact != 0.:
+ df_lip = display_one_point_from_dt(n_samples, n_atoms=n_atoms, expe_name=expe_name, rho=rho, delta=delta,
+ lip_fact=lip_fact)
+ df = df.merge(df_lip, how='outer', on=['algorithm'], suffixes=(None, str(lip_fact)))
+ df.round(4).to_csv(ONE_POINT_DIR / expe_name / f'one_point_{n_samples}x{n_atoms}.csv', index=False)
+
+
+@click.command(context_settings={'show_default': True, 'help_option_names': ['-h', '--help']})
+@click.option('--n_samples', '-ns', default=None, type=int, help='Size of y, number of lines of D')
+@click.option('--n_atoms', '-na', default=None, type=int,
+ help='Size of x, number of columns of D. '
+ 'If specified, n_samples must be None. Must be specified if n_samples is None.')
+@click.option('--n_steps', '-s', default=(0, 1), nargs=2, help='Discretization of rho and delta')
+@click.option('--n_runs', '-ru', default=1000, help='Number of times each configuration is tested')
+@click.option('--n_iter', '-i', default=None, type=int, help='Number max of iteration that the algorithm can do')
+@click.option('--distribution', '-d', default='unif',
+ help="Probability distribution to use for generating x* coefficients. Must be 'gaussian' or 'unif'")
+@click.option('--rel_tol', '-re', default=-np.inf,
+ help='Algorithms will stop if the relative difference between '
+ 'two following residuals are below this threshold')
+@click.option('--normalize/--no-normalize', '-n/-nn', default=True,
+ help='If True, normalize linear operators')
+@click.option('--plot', '-pl', is_flag=True, help='If specified, plot results instead of running algorithms')
+@click.option('--algos_type', '-at', multiple=True, default=[ALL],
+ type=click.Choice([*ALGOS_TYPE, ALL], case_sensitive=False),
+ help='Algorithms to run. If \'ALL\' is selected, run all algorithms.')
+@click.option('--algos_filter', '-af', multiple=True,
+ default=["SEAFAST-els", "SEAFAST-omp", "SEAFAST", "ELS", "OMP", "IHT",
+ "HTP", "IHT-omp", "HTP-omp", "IHT-els", "HTP-els", "OMPR"],
+ type=click.Choice(list(get_algo_dict(*[True] * len(ALGOS_TYPE)).keys()) + [ALL], case_sensitive=False),
+ help='Algorithms to run. If \'ALL\' is selected, run all algorithms.')
+@click.option('--resume', '-r', multiple=True, default=[ALL],
+ type=click.Choice(list(get_algo_dict(*[True] * len(ALGOS_TYPE)).keys()) + ["None", ALL],
+ case_sensitive=False),
+ help='Algorithms to resume instead of running them again from scratch. '
+ 'They must also be specified in algorithm filter. Use None compute everything from scratch '
+ 'Only work with the parallel version of the algorithm')
+@click.option('--factors', '-f', multiple=True, default=[256],
+ help='Sparsity factors to use for the iteration\'s number of SEA and IHT')
+@click.option('--epsilon', '-e', default=1e-4,
+ help='Threshold for euclidian distance relative difference in DT recovery diagram. '
+ 'If less than 1% of problems are solved under this threshold twice in a row, '
+ 'the computation stops for the current value of delta. If 0, compute the entire diagram')
+@click.option('--epsilon_x', '-e_x', default=1e-4,
+ help='Threshold on the relative distance between x_approx and x_true '
+ 'for the definition of a successful recovery')
+@click.option('--expe-name', '-en', default='', help="Name of the experiment")
+@click.option('--num-cpus', '-cpu', default=None, type=int, help="Limit the number of CPU if needed")
+@click.option('--deconv', '-dcv', is_flag=True, help="If specified, solve deconvolution problem")
+@click.option('--success', '-su', default=0.95, help="Minimal success rate to continue DT computation")
+@click.option('--noise-factor', '-nf', default=None, type=float, help="Variance of the gaussian noise")
+@click.option('--noise_type', '-nt', default=NoiseType.NOISE_LEVEL.name,
+ type=click.Choice(dir(NoiseType)[:len(NoiseType)], case_sensitive=False), help="How compute the noise")
+@click.option('--batch-size', '-bs', default=200, type=int,
+ help="Size of a batch of problems to send for parallelization")
+@click.option('--remove_first', '-rmf', is_flag=True, default=False,
+ help="If True, doesn't compute the first column of the phase diagram")
+@click.option('--remove_last', '-rml', is_flag=True, default=True,
+ help="If True, doesn't compute the last column of the phase diagram")
+@click.option('--keep_temp', '-kt', is_flag=True, default=True,
+ help="If True, keep temporary fil instead of deleting them")
+@click.option('--force_resume', '-fr', is_flag=True, default=True,
+ help="Set to True for continuing computations made with the `keep_temp` flag")
+@click.option('--delta_values', '-dv', multiple=True,
+ default=[round(0.025*k, 4) for k in range(1, 17)] + [round(0.1*k, 4) for k in range(5, 10)],
+ type=click.FloatRange(0, 1),
+ help='Columns to add to the diagram based on its delta value (between 0 and 1)')
+@click.option('--rho', '-rho', default=0., help="Rho value for DT with only one point")
+@click.option('--delta', '-delta', default=0., help="Delta value for DT with only one point")
+@click.option('--seed', '-sd', default=0,
+ help="Seed value for the bayesian search of DT with only one point")
+@click.option('--max_evals', '-me', default=100,
+ help="Seed value for the bayesian search of DT with only one point")
+@click.option('--lip_fact', '-lf', default=0.,
+ help="Factor of the Lipschitz constant of the gradient of the loss function")
+@click.option('--compile_/--no-compile_', '-cp/-ncp', default=None, type=bool,
+ help="Factor of the Lipschitz constant of the gradient of the loss function")
+def run_experiment(n_samples, n_steps, n_runs, n_iter, distribution, rel_tol, plot, resume, factors,
+ normalize, epsilon, n_atoms, epsilon_x, expe_name, algos_type, algos_filter, num_cpus, deconv,
+ success, noise_factor, noise_type, batch_size, remove_first, remove_last, keep_temp, force_resume,
+ delta_values, rho, delta, seed, max_evals, lip_fact, compile_
+ ) -> None:
+ """
+ Solve ||D * x - y||_2^2 with the provided algorithm for various configurations.
+ Results are saved in the 'data_result' folder
+ """
+ logger.info(f"Parameters: \n{locals()}")
+ if n_steps[0] >= 100000 or n_steps[1] >= 100000:
+ raise ValueError('n_steps values can\'t be above 100000')
+ if (n_atoms is None and n_samples is None) or (n_atoms is not None and n_samples is not None):
+ raise ValueError("n_atoms or n_samples must be specified, the other must be None")
+ if plot:
+ if rho != 0:
+ display_one_point_from_dt(n_samples, n_atoms=n_atoms, expe_name=expe_name, rho=rho, delta=delta,
+ lip_fact=lip_fact)
+ fuse_one_point_from_dt(n_samples, n_atoms=n_atoms, expe_name=expe_name, rho=rho, delta=delta,
+ lip_fact=lip_fact)
+ else:
+ plot_results(n_samples=n_samples, n_steps=n_steps, n_runs=n_runs, n_iter=n_iter, success=success,
+ rel_tol=rel_tol, epsilon=epsilon, n_atoms=n_atoms, epsilon_x=epsilon_x, expe_name=expe_name)
+ for nm in (False, ): # True):
+ plot_threshold_curves_comparison([expe_name], expe_name, epsilon_x, nm=nm)
+ # for threshold_position in (-1, -2):
+ # plot_paper_curves(expe_name, epsilon_x, factors, False, -1)
+ plot_dt_paper(expe_name=expe_name, threshold=success, factors=factors)
+ plot_dt_paper_zoom(expe_name=expe_name, threshold=success, factors=factors)
+ else:
+ # For multiprocessing
+ if compile_ is None or not compile_:
+ if os.getcwd().startswith('/baie'): # On sms cluster
+ logger.info("Running on sms")
+ ray.init(_temp_dir="/scratch/enroot-mimoun.mohamed")
+ logger.info("Ray initialized")
+ else: # On local or core5
+ logger.info(f"Running on {socket.gethostname()}")
+ ray.init(num_cpus=num_cpus)
+ algos_selected = select_algo(algos_type, algos_filter, normalize=normalize, optimizer='cg')
+ algo_to_resume = select_algo(algos_type, resume).keys()
+ if rho != 0:
+ compute_func = compute_one_point_from_dt
+ else:
+ compute_func = compute_dt_complete
+ algo_to_run = algos_selected.keys()
+ logger.info(f"Running {tuple(algo_to_run)}")
+ for algo_name in algo_to_run:
+ for factor in factors:
+ compute_func(n_samples=n_samples, n_steps=n_steps, n_runs=n_runs, n_iter=n_iter,
+ algo=algos_selected[algo_name], distribution=distribution, name=algo_name,
+ rel_tol=rel_tol, resume=algo_name in algo_to_resume, epsilon=epsilon, n_atoms=n_atoms,
+ expe_name=expe_name, factor=factor, deconv=deconv, success=success,
+ noise_factor=noise_factor, batch_size=batch_size, noise_type=NoiseType[noise_type],
+ remove_first=remove_first, remove_last=remove_last, keep_temp=keep_temp,
+ force_resume=force_resume, delta_values=delta_values, rho=rho, delta=delta, seed=seed,
+ max_evals=max_evals, lip_fact=lip_fact, compile_=compile_)
+ logger.info("End")
+
+
+# Needed for click CLI
+if __name__ == '__main__':
+ run_experiment()
diff --git a/code/sksea/sparse_coding.py b/code/sksea/sparse_coding.py
new file mode 100644
index 0000000000000000000000000000000000000000..458454fd8017cde88beaa206affc759bf54fee37
--- /dev/null
+++ b/code/sksea/sparse_coding.py
@@ -0,0 +1,164 @@
+# -*- coding: utf-8 -*-
+from typing import Tuple
+import numpy as np
+from scipy.sparse.linalg.interface import MatrixLinearOperator
+from scipy.linalg import cho_solve
+from sklearn.preprocessing import normalize
+
+from sksea.utils import AbstractLinearOperator
+from sksea.cholesky import chol_1d_update, chol_1d_downdate
+
+
+class MatrixOperator(MatrixLinearOperator, AbstractLinearOperator):
+ def __init__(self, data_mat, seed=0):
+ MatrixLinearOperator.__init__(self, A=data_mat)
+ self.seed = seed
+
+ def get_normalized_operator(self) -> Tuple['MatrixOperator', np.ndarray]:
+ """
+ Return a normalized version of the operator using normalize function of sklearn on the operator matrix
+ """
+ a, w_diag = normalize(self.A, axis=0, return_norm=True)
+ return type(self)(a, self.seed), 1 / w_diag
+
+ def get_operator_on_support(self, s) -> 'MatrixOperator':
+ """
+ Return the operator truncated on the provided support
+
+ :param (np.ndarray) s: Support
+ """
+ return type(self)(self.A[:, s], self.seed)
+
+
+class SparseSupportOperator(MatrixOperator):
+ def __init__(self, data_mat, y, seed=0):
+ MatrixOperator.__init__(self=self, data_mat=data_mat)
+ self._support = [] # list of selected atom indices
+ self._L = np.empty((0, 0)) # Cholesky
+ self._y = y
+ self._AT_y = np.empty((0,)) # Cholesky
+ self.seed = seed
+
+ def add_atom(self, i_atom):
+ if i_atom not in self._support:
+ self._support.append(i_atom)
+ a_i = self.A[:, i_atom]
+ v = np.array([np.vdot(a_i, self.A[:, j]) for j in self._support])
+ self._L = chol_1d_update(self._L, v)
+ AT_y = np.empty(self._AT_y.size+1)
+ AT_y[:-1] = self._AT_y
+ AT_y[-1] = np.vdot(a_i, self._y)
+ self._AT_y = AT_y
+
+ def remove_atom(self, i_atom):
+ atom_indice = self._support.index(i_atom)
+ self._L = chol_1d_downdate(self._L, atom_indice)
+ self._AT_y = np.delete(self._AT_y, atom_indice)
+ self._support.pop(atom_indice)
+
+ def solve(self):
+ return cho_solve((self._L, True), self._AT_y)
+
+ def change_support(self, s):
+ for i_atom in self._support[::-1]:
+ if not s[i_atom]:
+ self.remove_atom(i_atom)
+ for (i_atom, ) in np.argwhere(s):
+ if i_atom not in self._support:
+ self.add_atom(i_atom)
+
+ def get_normalized_operator(self) -> Tuple['SparseSupportOperator', np.ndarray]:
+ """
+ Return a normalized version of the operator using normalize function of sklearn on the operator matrix
+ """
+ a, w_diag = normalize(self.A, axis=0, return_norm=True)
+ return type(self)(a, self._y, self.seed), 1 / w_diag
+
+ @property
+ def support(self):
+ return self._support
+
+ def reset(self):
+ self._support = [] # list of selected atom indices
+ self._L = np.empty((0, 0)) # Cholesky
+ self._AT_y = np.empty((0,)) # Cholesky
+
+
+if __name__ == '__main__':
+ from scipy.spatial.distance import hamming
+ from sklearn.linear_model import lasso_path
+
+ from sksea.algorithms import iht, ista, sea, amp
+
+ n_samples = 64
+ n_atoms = 2 * n_samples
+ n_nonzero = n_samples // 4
+ D_mat = np.random.randn(n_samples, n_atoms)
+ D_op = MatrixOperator(D_mat)
+ x_ref = np.zeros(n_atoms)
+ s_ref = np.random.permutation(n_atoms)[:n_nonzero]
+ # x_ref[s_ref] = np.random.randn(n_nonzero)
+ x_ref[s_ref] = np.random.rand(n_nonzero) + 1
+ x_ref[s_ref] *= (-1) ** np.random.randint(0, 2, n_nonzero)
+ y = D_op @ x_ref
+
+ alphas, coefs, _ = lasso_path(X=D_mat, y=y)
+ coefs_l0 = np.count_nonzero(coefs != 0, axis=0)
+ i_alpha = np.nonzero(coefs_l0 >= n_nonzero)[0][0]
+ alpha = alphas[i_alpha] * n_samples
+
+ n_iter = 10 ** 4
+ x_est = {}
+ res_norm = {}
+ f = lambda x, linop: np.linalg.norm(linop @ x - y)
+ grad_f = lambda x, linop: linop.H @ (linop @ x - y)
+ x_est['Lasso'], res_norm['Lasso'] = coefs[:, i_alpha], None
+ x_est['ISTA'], res_norm['ISTA'] = \
+ ista(linop=D_op, y=y, alpha=alpha, n_iter=n_iter)
+ x_est['AMP'], res_norm['AMP'] = \
+ amp(linop=D_op, y=y, alpha=1, n_iter=n_iter, normalize=False)
+ x_est['IHT'], res_norm['IHT'] = \
+ iht(linop=D_op, y=y, n_nonzero=n_nonzero, n_iter=n_iter, f=f, grad_f=grad_f, normalize=False)
+ x_est['SEA'], res_norm['SEA'] = \
+ sea(linop=D_op, y=y, n_nonzero=n_nonzero, n_iter=n_iter,
+ keep_nonzero_x=True, f=f, grad_f=grad_f, normalize=False)
+ x_est['SEA best'], res_norm['SEA best'] = \
+ sea(linop=D_op, y=y, n_nonzero=n_nonzero, n_iter=n_iter, return_best=True,
+ keep_nonzero_x=True, f=f, grad_f=grad_f, normalize=False)
+ x_est['SEA-0'], res_norm['SEA-0'] = \
+ sea(linop=D_op, y=y, n_nonzero=n_nonzero, n_iter=n_iter,
+ keep_nonzero_x=False, f=f, grad_f=grad_f, normalize=False)
+ x_est['SEA-0 best'], res_norm['SEA-0 best'] = \
+ sea(linop=D_op, y=y, n_nonzero=n_nonzero, n_iter=n_iter, return_best=True,
+ keep_nonzero_x=False, f=f, grad_f=grad_f, normalize=False)
+
+ print('Sparsity (ref):', np.count_nonzero(x_ref))
+ for k, x in x_est.items():
+ x_l0 = np.count_nonzero(x != 0)
+ rel_error = np.linalg.norm(x_ref - x)/np.linalg.norm(x_ref)
+ hamming_dist = hamming(x != 0, x_ref != 0)
+ if res_norm[k] is not None:
+ print(f'{k}: '
+ f'sparsity {x_l0}, '
+ f'rel error {rel_error:.3f}, '
+ f'Hamming dist {hamming_dist:.3f}, '
+ f'last res_norm {res_norm[k][-1]:.3f},'
+ f'min res_norm {np.min(res_norm[k]):.3f}')
+ else:
+ print(f'{k}:'
+ f'sparsity {x_l0}, '
+ f'rel error {rel_error:.3f}, '
+ f'Hamming dist {hamming_dist:.3f}')
+
+ import matplotlib.pyplot as plt
+
+ plt.figure()
+ for k in res_norm:
+ if res_norm[k] is None:
+ continue
+ plt.loglog(res_norm[k], label=k)
+ plt.xlabel('Iterations')
+ plt.ylabel('Norm of residue')
+ plt.legend()
+ plt.savefig('figures/sparse_coding_res_norm.pdf')
+ plt.show()
diff --git a/code/sksea/tests/test_algorithms.py b/code/sksea/tests/test_algorithms.py
new file mode 100644
index 0000000000000000000000000000000000000000..f22f2e219e275338ed1017b757851e9ba8170a6b
--- /dev/null
+++ b/code/sksea/tests/test_algorithms.py
@@ -0,0 +1,394 @@
+# -*- coding: utf-8 -*-
+from unittest import TestCase
+import math
+import numpy as np
+import scipy
+import itertools
+from sklearn.utils.estimator_checks import check_estimator
+
+from sksea.sparse_coding import SparseSupportOperator
+
+from sksea.algorithms import els_fast, iht, ista, omp_fast, ompr_fast, sea, omp, ompr, htp_fast, htp, els, sea_fast, SEA
+
+
+# class TestAux(TestCase):
+#
+#
+class TestAlgorithms(TestCase):
+ # TODO add exact recovery in simple cases
+
+ def test_ista(self):
+ n_runs = 100
+ for i_run in range(n_runs):
+ with self.subTest(i_run=i_run):
+ n_iter = 100
+ D, y = get_random_problem()
+ n_rows, n_cols = D.shape
+ for n_nonzero in (1, n_cols, n_rows // 4):
+ with self.subTest(n_nonzero=n_nonzero):
+ x, res_norm = ista(linop=D, y=y,
+ alpha=10 ** -6, n_iter=n_iter)
+ err_msg = f'Run {i_run}, n_nonzero={n_nonzero}'
+ self.assertEqual(1, x.ndim, msg=err_msg)
+ self.assertEqual(n_cols, x.size, msg=err_msg)
+ eps = 10 ** -6
+ np.testing.assert_array_less(np.diff(res_norm), eps,
+ err_msg=err_msg)
+
+ def test_iht(self):
+ n_runs = 100
+ for i_run in range(n_runs):
+ with self.subTest(i_run=i_run):
+ n_iter = 100
+ D, y = get_random_problem(i_run)
+ n_rows, n_cols = D.shape
+ for n_nonzero in (1, n_cols, n_rows // 4):
+ with self.subTest(n_nonzero=n_nonzero):
+ x, res_norm = iht(linop=D, y=y,
+ n_nonzero=n_nonzero, n_iter=n_iter,
+ f=lambda x_iter, linop: np.linalg.norm(linop @ x_iter - y),
+ grad_f=lambda x_iter, linop: linop.H @ (linop @ x_iter - y))
+ x2, res_norm2 = iht(linop=D, y=y,
+ n_nonzero=n_nonzero, n_iter=n_iter,
+ f=lambda x_iter, linop: np.linalg.norm(linop @ x_iter - y),
+ grad_f=lambda x_iter, linop: linop.H @ (linop @ x_iter - y))
+ err_msg = f'Run {i_run}, n_nonzero={n_nonzero}'
+ self.assertTrue((x - x2 == 0).all(), msg=err_msg)
+ self.assertEqual(1, x.ndim, msg=err_msg)
+ self.assertEqual(n_cols, x.size, msg=err_msg)
+ self.assertLessEqual(np.count_nonzero(x), n_nonzero,
+ msg=err_msg)
+ # eps = 10 ** -6
+ # np.testing.assert_array_less(np.diff(res_norm), eps,
+ # err_msg=err_msg)
+
+ def test_sea(self):
+ n_runs = 50
+ sea_sklearn = SEA()
+ check_estimator(sea_sklearn)
+ for i_run in range(n_runs):
+ with self.subTest(i_run=i_run):
+ n_iter = 100
+ D, y = get_random_problem(i_run)
+ n_rows, n_cols = D.shape
+ for n_nonzero in (1, n_cols, n_rows // 4):
+ with self.subTest(n_nonzero=n_nonzero):
+ x_old, res_norm_old = sea(linop=D, y=y,
+ n_nonzero=n_nonzero, n_iter=n_iter,
+ return_best=False,
+ f=lambda x_iter, linop: np.linalg.norm(linop @ x_iter - y),
+ grad_f=lambda x_iter, linop: linop.H @ (linop @ x_iter - y),
+ optimize_sea="all", optimizer='cg'
+ )
+
+ sea_params = dict(linop=D, y=y,
+ n_nonzero=n_nonzero,
+ n_iter=n_iter,
+ f=lambda x_iter, linop: np.linalg.norm(linop @ x_iter - y),
+ grad_f=lambda x_iter, linop: linop.H @ (linop @ x_iter - y),
+ return_history=False, optimizer='cg')
+
+ x_fast, res_norm_fast = sea_fast(**sea_params, return_best=False)
+ x_best, res_norm_best = sea_fast(**sea_params, return_best=True)
+
+ sea_params["return_history"] = True
+ sea_params["n_iter"] = None
+ x_n_nonzero, _, hist_n_nonzero = sea_fast(**sea_params, return_best=False)
+
+ sea_sklearn = SEA(n_nonzero, n_iter, random_state=i_run)
+ sea_sklearn.fit(D.matrix, y)
+ x_sklearn = sea_sklearn.coef_
+ res_norm_sklearn = sea_sklearn.res_norm_
+
+ err_msg = f'Run {i_run}, n_nonzero={n_nonzero}'
+ for x, res_norm in zip((x_best, x_old, x_fast, x_sklearn),
+ (res_norm_best, res_norm_old, res_norm_fast, res_norm_sklearn)):
+ self.assertEqual(1, x.ndim, msg=err_msg)
+ self.assertEqual(n_cols, x.size, msg=err_msg)
+ self.assertLessEqual(np.count_nonzero(x), n_nonzero,
+ msg=err_msg)
+
+ # Compare residues, support and sparse iterate between implementations
+ np.testing.assert_array_almost_equal(res_norm_old[:-1], res_norm_fast[:-1], err_msg=err_msg)
+ if res_norm_old[-1] > 1e-4 or res_norm_fast[-1] > 1e-4:
+ # Tolerance of conjugate gradient descent is 1e-5 on normalized operator.
+ self.assertAlmostEqual(res_norm_old[-1], res_norm_fast[-1], msg=err_msg)
+ self.assertCountEqual(x_old.nonzero()[0], x_fast.nonzero()[0], msg=err_msg)
+ np.testing.assert_array_almost_equal(x_old, x_fast, decimal=4)
+
+ # Check if best version is better than regular version
+ np.testing.assert_array_almost_equal(res_norm_fast[:len(res_norm_best)], res_norm_best)
+ # np.testing.assert_array_almost_equal(res_norm_sklearn, res_norm_best)
+ self.assertLessEqual(res_norm_best[-1], min(res_norm_fast), msg=err_msg)
+ self.assertLessEqual(np.linalg.norm(D(x_best) - y), np.linalg.norm(D(x_fast) - y), msg=err_msg)
+ np.testing.assert_array_almost_equal(x_sklearn, x_best)
+
+ # Check if n_iter_is_n_support works
+ if hist_n_nonzero.get_n_supports(best=False) != n_nonzero + 1:
+ self.assertEqual(hist_n_nonzero.get_n_supports(), math.comb(n_cols, n_nonzero), msg=err_msg)
+
+ def test_omp(self):
+ n_runs = 100
+ for i_run in range(n_runs):
+ with self.subTest(i_run=i_run):
+ # Test copy-pasted from ista
+ n_iter = 100
+ D, y = get_random_problem(i_run)
+ n_rows, n_cols = D.shape
+ for n_nonzero in (1, n_cols, n_rows // 4):
+ with self.subTest(n_nonzero=n_nonzero):
+ x, res_norm = omp(linop=D, y=y, n_nonzero=n_nonzero, n_iter=n_iter,
+ f=lambda x_iter, linop: np.linalg.norm(linop @ x_iter - y),
+ grad_f=lambda x_iter, linop: linop.H @ (linop @ x_iter - y),
+ optimizer='cg')
+ x2, res_norm2 = omp(linop=D, y=y, n_nonzero=n_nonzero, n_iter=n_iter,
+ f=lambda x_iter, linop: np.linalg.norm(linop @ x_iter - y),
+ grad_f=lambda x_iter, linop: linop.H @ (linop @ x_iter - y),
+ optimizer='cg')
+ err_msg = f'Run {i_run}, n_nonzero={n_nonzero}'
+ self.assertTrue((x - x2 == 0).all())
+ self.assertEqual(1, x.ndim, msg=err_msg)
+ self.assertEqual(n_cols, x.size, msg=err_msg)
+ eps = 10 ** -6
+ np.testing.assert_array_less(np.diff(res_norm), eps,
+ err_msg=err_msg)
+
+ def test_omp_fast(self):
+ n_runs = 100
+ for i_run in range(n_runs):
+ with self.subTest(i_run=i_run):
+ # Test copy-pasted from ista
+ n_iter = 100
+ D, y = get_random_problem(i_run)
+ n_rows, n_cols = D.shape
+ for n_nonzero in (1, n_cols, n_rows // 4):
+ with self.subTest(n_nonzero=n_nonzero):
+ x, res_norm, history = omp_fast(linop=D, y=y, n_nonzero=n_nonzero, n_iter=n_iter,
+ f=lambda x_iter, linop: np.linalg.norm(linop @ x_iter - y),
+ grad_f=lambda x_iter, linop: linop.H @ (linop @ x_iter - y),
+ optimizer='cg')
+ x2, res_norm2 = omp(linop=D, y=y, n_nonzero=n_nonzero, n_iter=n_iter,
+ f=lambda x_iter, linop: np.linalg.norm(linop @ x_iter - y),
+ grad_f=lambda x_iter, linop: linop.H @ (linop @ x_iter - y),
+ optimizer='cg')
+ err_msg = f'Run {i_run}, n_nonzero={n_nonzero}'
+ self.assertTrue((x - x2 == 0).all())
+ self.assertEqual(1, x.ndim, msg=err_msg)
+ self.assertEqual(n_cols, x.size, msg=err_msg)
+ eps = 10 ** -6
+ np.testing.assert_array_less(np.diff(res_norm), eps,
+ err_msg=err_msg)
+
+ def test_ompr(self):
+ n_runs = 100
+ for i_run in range(n_runs):
+ with self.subTest(i_run=i_run):
+ # Test copy-pasted from ista
+ n_iter = 100
+ D, y = get_random_problem()
+ n_rows, n_cols = D.shape
+ for n_nonzero in (1, n_cols, n_rows // 4):
+ with self.subTest(n_nonzero=n_nonzero):
+ x, res_norm = ompr(linop=D, y=y, n_nonzero=n_nonzero, n_iter=n_iter,
+ f=lambda x_iter, linop: np.linalg.norm(linop @ x_iter - y),
+ grad_f=lambda x_iter, linop: linop.H @ (linop @ x_iter - y),
+ )
+ x2, res_norm2 = ompr(linop=D, y=y, n_nonzero=n_nonzero, n_iter=n_iter,
+ f=lambda x_iter, linop: np.linalg.norm(linop @ x_iter - y),
+ grad_f=lambda x_iter, linop: linop.H @ (linop @ x_iter - y),
+ )
+ err_msg = f'Run {i_run}, n_nonzero={n_nonzero}'
+ self.assertTrue((x - x2 == 0).all())
+ self.assertEqual(1, x.ndim, msg=err_msg)
+ self.assertEqual(n_cols, x.size, msg=err_msg)
+ eps = 10 ** -6
+ np.testing.assert_array_less(np.diff(res_norm), eps,
+ err_msg=err_msg)
+
+ def test_els(self):
+ n_runs = 100
+ for i_run in range(n_runs):
+ with self.subTest(i_run=i_run):
+ # Test copy-pasted from ista
+ n_iter = 100
+ D, y = get_random_problem()
+ n_rows, n_cols = D.shape
+ for n_nonzero in (1, n_cols, n_rows // 4):
+ with self.subTest(n_nonzero=n_nonzero):
+ x, res_norm = els(linop=D, y=y, n_nonzero=n_nonzero, n_iter=n_iter,
+ f=lambda x_iter, linop: np.linalg.norm(linop @ x_iter - y),
+ grad_f=lambda x_iter, linop: linop.H @ (linop @ x_iter - y),
+ )
+ x2, res_norm2 = els(linop=D, y=y, n_nonzero=n_nonzero, n_iter=n_iter,
+ f=lambda x_iter, linop: np.linalg.norm(linop @ x_iter - y),
+ grad_f=lambda x_iter, linop: linop.H @ (linop @ x_iter - y),
+ )
+ err_msg = f'Run {i_run}, n_nonzero={n_nonzero}'
+ self.assertTrue((x - x2 == 0).all())
+ self.assertEqual(1, x.ndim, msg=err_msg)
+ self.assertEqual(n_cols, x.size, msg=err_msg)
+ eps = 10 ** -6
+ np.testing.assert_array_less(np.diff(res_norm), eps,
+ err_msg=err_msg)
+
+ def test_els_fast(self):
+ n_runs = 100
+ for i_run in range(n_runs):
+ with self.subTest(i_run=i_run):
+ # Test copy-pasted from ista
+ n_iter = 100
+ D, y = get_random_problem(seed=i_run)
+ n_rows, n_cols = D.shape
+ for n_nonzero in (1, n_cols, n_rows // 4):
+ with self.subTest(n_nonzero=n_nonzero):
+ x, res_norm, history = els_fast(linop=D, y=y, n_nonzero=n_nonzero, n_iter=n_iter,
+ f=lambda x_iter, linop: np.linalg.norm(linop @ x_iter - y),
+ grad_f=lambda x_iter, linop: linop.H @ (linop @ x_iter - y),
+ )
+ x2, res_norm2 = els(linop=D, y=y, n_nonzero=n_nonzero, n_iter=n_iter,
+ f=lambda x_iter, linop: np.linalg.norm(linop @ x_iter - y),
+ grad_f=lambda x_iter, linop: linop.H @ (linop @ x_iter - y),
+ )
+ err_msg = f'Run {i_run}, n_nonzero={n_nonzero}'
+ self.assertTrue((x - x2 == 0).all())
+ np.testing.assert_allclose(x, x2, err_msg=err_msg)
+ self.assertEqual(1, x.ndim, msg=err_msg)
+ self.assertEqual(n_cols, x.size, msg=err_msg)
+ eps = 10 ** -6
+ np.testing.assert_array_less(np.diff(res_norm), eps,
+ err_msg=err_msg)
+
+ def test_ompr_fast(self):
+ n_runs = 100
+ for i_run in range(n_runs):
+ with self.subTest(i_run=i_run):
+ # Test copy-pasted from ista
+ n_iter = 100
+ D, y = get_random_problem(seed=i_run)
+ n_rows, n_cols = D.shape
+ for n_nonzero in (1, n_cols, n_rows // 4):
+ with self.subTest(n_nonzero=n_nonzero):
+ x, res_norm, history = ompr_fast(linop=D, y=y, n_nonzero=n_nonzero, n_iter=n_iter,
+ f=lambda x_iter, linop: np.linalg.norm(linop @ x_iter - y),
+ grad_f=lambda x_iter, linop: linop.H @ (linop @ x_iter - y),
+ )
+ x2, res_norm2 = ompr(linop=D, y=y, n_nonzero=n_nonzero, n_iter=n_iter,
+ f=lambda x_iter, linop: np.linalg.norm(linop @ x_iter - y),
+ grad_f=lambda x_iter, linop: linop.H @ (linop @ x_iter - y),
+ )
+ err_msg = f'Run {i_run}, n_nonzero={n_nonzero}'
+ self.assertTrue((x - x2 == 0).all())
+ np.testing.assert_allclose(x, x2, err_msg=err_msg)
+ self.assertEqual(1, x.ndim, msg=err_msg)
+ self.assertEqual(n_cols, x.size, msg=err_msg)
+ eps = 10 ** -6
+ np.testing.assert_array_less(np.diff(res_norm), eps,
+ err_msg=err_msg)
+
+ def test_fast_succession(self):
+ n_runs = 100
+ for i_run in range(n_runs):
+ with self.subTest(i_run=i_run):
+ # Test copy-pasted from ista
+ n_iter = 100
+ D, y = get_random_problem(seed=i_run)
+ n_rows, n_cols = D.shape
+ algos_init = [(omp_fast, omp), (els_fast, els)]
+ algos = [(htp_fast, htp), (sea_fast, sea)]
+ for algo_init, algo in itertools.product(algos_init, algos):
+ with self.subTest(algo_init=algo_init, algo=algo):
+ x, res_norm, history = algo[0](linop=D, y=y, n_nonzero=n_cols, n_iter=n_iter,
+ f=lambda x_iter, linop: np.linalg.norm(linop @ x_iter - y),
+ grad_f=lambda x_iter, linop: linop.H @ (linop @ x_iter - y),
+ algo_init=algo_init[0],
+ )
+ x2, res_norm2 = algo[1](linop=D, y=y, n_nonzero=n_cols, n_iter=n_iter,
+ f=lambda x_iter, linop: np.linalg.norm(linop @ x_iter - y),
+ grad_f=lambda x_iter, linop: linop.H @ (linop @ x_iter - y),
+ algo_init=algo_init[1],
+ )
+ err_msg = f'Run {i_run}, algo_init={algo_init}, algo={algo}'
+ # self.assertTrue((x - x2 == 0).all())
+ np.testing.assert_allclose(x, x2, err_msg=err_msg, atol=1e-3)
+ self.assertEqual(1, x.ndim, msg=err_msg)
+ self.assertEqual(n_cols, x.size, msg=err_msg)
+ eps = 10 ** -6
+ np.testing.assert_array_less(np.diff(res_norm), eps,
+ err_msg=err_msg)
+
+ def test_htp_fast(self):
+ n_runs = 100
+ for i_run in range(n_runs):
+ with self.subTest(i_run=i_run):
+ # Test copy-pasted from ista
+ n_iter = 100
+ D, y = get_random_problem(i_run)
+ n_rows, n_cols = D.shape
+ f = lambda x_iter, linop: np.linalg.norm(linop @ x_iter - y)
+ grad_f = lambda x_iter, linop: linop.H @ (linop @ x_iter - y)
+ for n_nonzero in (1, n_cols, n_rows // 4):
+ with self.subTest(n_nonzero=n_nonzero):
+ fast_params = dict(linop=D, y=y, n_nonzero=n_nonzero, n_iter=n_iter,
+ f=f, grad_f=grad_f, return_history=False, return_best=True,
+ optimizer="cg")
+ x_best, res_norm_best = htp_fast(**fast_params)
+
+ fast_params["return_best"] = False
+ x_fast, res_norm_fast = htp_fast(**fast_params)
+
+ x_old, res_norm_old = htp(linop=D, y=y, n_nonzero=n_nonzero, n_iter=n_iter,
+ f=f, grad_f=grad_f, optimizer="cg")
+
+ err_msg = f'Run {i_run}, n_nonzero={n_nonzero}'
+
+ eps = 10 ** -6
+ for x, res_norm in zip((x_best, x_old, x_fast), (res_norm_best, res_norm_old, res_norm_fast)):
+ self.assertEqual(1, x.ndim, msg=err_msg)
+ self.assertEqual(n_cols, x.size, msg=err_msg)
+ np.testing.assert_array_less(np.diff(res_norm), eps,
+ err_msg=err_msg)
+
+ # Compare residues, support and sparse iterate between implementations
+ np.testing.assert_array_almost_equal(res_norm_old[:-1], res_norm_fast[:-1], err_msg=err_msg)
+ if res_norm_old[-1] > 1e-4 or res_norm_fast[-1] > 1e-4:
+ # Tolerance of conjugate gradient descent is 1e-5 on normalized operator.
+ self.assertAlmostEqual(res_norm_old[-1], res_norm_fast[-1], 3, msg=err_msg)
+ self.assertCountEqual(x_old.nonzero()[0], x_fast.nonzero()[0], msg=err_msg)
+ np.testing.assert_array_almost_equal(x_old, x_fast, decimal=4)
+
+ # Check if best version is better than regular version
+ np.testing.assert_array_almost_equal(res_norm_fast[:len(res_norm_best)], res_norm_best)
+ self.assertLessEqual(res_norm_best[-1], min(res_norm_fast), msg=err_msg)
+ self.assertLessEqual(np.linalg.norm(D(x_best) - y), np.linalg.norm(D(x_fast) - y), msg=err_msg)
+
+ def test_cg(self):
+ """
+ Test conjugate gradient stability
+ """
+ n_runs = 100
+ for i_run in range(n_runs):
+ with self.subTest(i_run=i_run):
+ D, y = get_random_problem(i_run)
+ n_rows, n_cols = D.shape
+ n_repet = 10
+ all_out = np.empty((n_repet, n_cols))
+ for repet in range(n_repet):
+ all_out[repet], _ = scipy.sparse.linalg.cg(D.T @ D, D.T @ y,
+ all_out[repet - 1] if repet != 0 else None, atol=0)
+ for out in all_out:
+ np.testing.assert_array_equal(out, all_out[0])
+
+
+def get_random_problem(seed=None):
+ rand = np.random.RandomState(seed)
+ n_rows = rand.randint(5, 20)
+ n_cols = rand.randint(n_rows, 30)
+ n_nonzero = rand.randint(1, n_cols + 1)
+ D = rand.randn(n_rows, n_cols)
+ rand_mat = rand.randn(n_rows, n_cols)
+ x_ref = np.zeros(n_cols)
+ s_ref = rand.permutation(n_cols)[:n_nonzero]
+ x_ref[s_ref] = rand.rand(n_nonzero) + 1
+ x_ref[s_ref] *= (-1) ** rand.randint(0, 2, n_nonzero)
+ y = rand_mat @ x_ref
+ return SparseSupportOperator(D, y, seed), y
diff --git a/code/sksea/training_tasks.py b/code/sksea/training_tasks.py
new file mode 100644
index 0000000000000000000000000000000000000000..b2222b1f7a8c2fa29a1730e112f0c8cf6e4ce7a7
--- /dev/null
+++ b/code/sksea/training_tasks.py
@@ -0,0 +1,433 @@
+"""
+Functions for reproducing experiments in Sparse Convex Optimization via Adaptively Regularized Hard Thresholding paper
+https://arxiv.org/pdf/2006.14571.pdf
+"""
+# Python imports
+from collections import defaultdict
+from copy import deepcopy
+import gc
+import inspect
+import os
+from pathlib import Path
+import socket
+from time import time
+from typing import List, Dict, Callable, Iterable
+
+# Modules imports
+import click
+from loguru import logger
+import numpy as np
+import plotly.express as px
+import plotly.graph_objects as go
+import ray
+
+# Script imports
+from sksea.algorithms import els_fast, omp_fast, ompr_fast, sea, omp, ompr, els, amp, iht, es, PAS, sea_fast, rea, htp, \
+ htp_fast
+from sksea.dataset_operator import DatasetOperator, Task, RESULT_PATH
+from sksea.plot_icml import plot_ml_paper
+from sksea.utils import ALGOS_PAPER_TT, PAPER_LAYOUT
+
+ROOT = Path(os.path.abspath(inspect.getfile(inspect.currentframe()))).parent
+PLOT_PATH = ROOT / "results/training_tasks_plot"
+PLOT_PATH_PAPER = ROOT / "results/training_tasks_plot_paper"
+ALL = 'ALL'
+ALGOS_TYPE = ('seafast', 'sea', 'iht', 'omp', 'es', 'amp')
+
+
+def run_experiments(algorithms, datasets=DatasetOperator.REGRESSIONS_NAME, factors=(1, 2, 4, 8, 16, 100), resume=True,
+ reverse=False, sparsities=(0,)) -> None:
+ """
+ Run all simulations on regression datasets with the provided algorithms
+
+ :param (Dict[str, Callable]) algorithms: Dictionary linking each algorithm to use to its name
+ :param (Iterable[DatasetOperator) datasets:
+ :param (Iterable[int]) factors:
+ :param (bool) resume: If True, don't overwrite previous results
+ """
+ RESULT_PATH.mkdir(exist_ok=True, parents=True)
+ for name in datasets:
+ dataop = DatasetOperator(name, no_loading=True)
+ for algo_name, algo in algorithms.items():
+ if 0 in sparsities:
+ if reverse:
+ iterator = range(dataop.k_max, 0, -1)
+ else:
+ iterator = range(1, dataop.k_max + 1)
+ else:
+ iterator = sparsities
+ for n_nonzero in iterator:
+ if 'SEA' in algo_name or 'IHT' in algo_name or 'HTP' in algo_name or 'REA' in algo_name: # For SEA and IHT we use various iteration number
+ for factor in factors:
+ sea_name = f'{algo_name}x{factor}'
+ n_iter = n_nonzero * factor
+ solve_problem(dataop, lambda *args, **kwargs: algo(*args, **kwargs, n_iter=n_iter), n_nonzero,
+ sea_name, resume)
+ else:
+ solve_problem(dataop, algo, n_nonzero, algo_name, resume)
+ gc.collect() # Force linop_s (from optimize function of algorithm file) to be removed from the memory
+
+
+def solve_problem(dataop, algo, n_nonzero, algo_name, resume) -> None:
+ """
+ Solve the problem stored in linop with the provided algorithm
+
+ :param (DatasetOperator) dataop: Linear operator representing the problem matrix and the labels.
+ Everything must be already preprocessed
+ :param (Callable) algo: Algorithm to run in order to solve min_x 1/2 ||D * x - y||_2^2 w.r.t ||x||_0 = n_nonzero
+ D is linop and y is linop.y
+ :param (int) n_nonzero: Size of the wanted support
+ :param (str) algo_name: Name of the provided algorithm
+ :param (bool) resume: If True, don't overwrite previous results
+ """
+ idx = 0
+ out_file = RESULT_PATH / f"{algo_name}_{idx}_{dataop.name}_{n_nonzero}.npz"
+ if out_file.is_file() and resume:
+ logger.info(f"{algo_name} for solving {dataop.name} with k={n_nonzero} was already done")
+ else:
+ logger.info(f"Using {algo_name} for solving {dataop.name} with k={n_nonzero}")
+ dataop.load()
+ start_time = time()
+ out = algo(linop=dataop.linop, y=dataop.y, n_nonzero=n_nonzero, f=dataop.f,
+ grad_f=dataop.grad_f, is_mse=dataop.task == Task.REGRESSION,
+ optimizer="cg")
+ time_spent = time() - start_time
+ outputs = out if isinstance(out[0], tuple) else (out,)
+ for idx, (x, res_norm, *args) in enumerate(outputs):
+ np.savez(RESULT_PATH / f"{algo_name}_{idx}_{dataop.name}_{n_nonzero}.npz",
+ x=x, time_spent=np.array([time_spent]),
+ algo_name=algo_name, dataset=dataop.name, res_norm=np.array(res_norm),
+ history=args)
+
+
+def get_algo_dict(use_seafast=False, use_sea=False, use_omp=False, use_iht=False, use_amp=False, use_es=False,
+ n_iter=None, sea_params=None, **params) -> Dict[str, Callable]:
+ """
+ Get a dictionary with all available algorithms and all SEA variants.
+
+ :param (bool) use_sea: If True, add SEA and its variants to the algorithms' dictionary
+ :param (bool) use_omp: If True, add OMP and its variants to the algorithms' dictionary
+ :param (bool) use_iht: If True, add IHT to the algorithms' dictionary
+ :param (bool) use_amp: If True, add AMP to the algorithms' dictionary
+ :param (bool) use_es: If True, add ES to the algorithms' dictionary
+ :param (Optional[int]) n_iter: Maximal number of iteration the algorithm can do
+ :return: A dictionary linking each algorithm to its name
+ """
+ if sea_params is None:
+ sea_params = dict(return_both=True)
+ iter_params = dict(n_iter=n_iter) if n_iter is not None else dict() # For OMP-like algorithms
+ algos_available = {}
+ algo_inits = (None, omp, ompr, els, els_fast, omp_fast)
+ if use_iht:
+ for init in algo_inits:
+ for algo, algo_name in ((iht, 'IHT'), (htp, 'HTP')):
+ current_name = algo_name
+ if init is not None:
+ current_name += f'-{init.__name__}'
+ algos_available[current_name] = lambda *args, algo_init=init, algo_to_run=algo, **kwargs: algo_to_run(
+ *args, algo_init=algo_init, **kwargs, **params)
+ if use_omp:
+ algos_available.update({
+ 'OMP': lambda *args, **kwargs: omp(*args, **kwargs, **params, **iter_params),
+ 'OMPR': lambda *args, **kwargs: ompr(*args, **kwargs, **params, **iter_params),
+ 'ELS': lambda *args, **kwargs: els(*args, **kwargs, **params, **iter_params),
+ 'ELSFAST': lambda *args, **kwargs: els_fast(*args, return_history=True, **kwargs, **params,
+ **iter_params),
+ 'OMPFAST': lambda *args, **kwargs: omp_fast(*args, return_history=True, **kwargs, **params,
+ **iter_params),
+ 'OMPRFAST': lambda *args, **kwargs: ompr_fast(*args, return_history=True, **kwargs, **params,
+ **iter_params),
+ })
+ if use_amp:
+ algos_available['AMP'] = lambda *args, **kwargs: amp(*args, alpha=1, return_both=True,
+ **kwargs, **params, **iter_params)
+ if use_sea: # Get all SEA versions
+ for opti in ['all', None, 'last']: # , 'half', -1, -2]:
+ for init in algo_inits:
+ for pas in PAS.keys():
+ for full_explo in (True, False):
+ current_name = 'SEA'
+ if full_explo:
+ current_name += '-full'
+ if init is not None:
+ current_name += f'-{init.__name__}'
+ if opti is not None:
+ current_name += f'-{opti}'
+ if pas is not None:
+ current_name += f'-{pas}'
+ algos_available[current_name] = \
+ (lambda *args, algo_init=init, optimize_sea=opti, pas_sea=pas, full_explo_sea=full_explo,
+ **kwargs: sea(*args, algo_init=algo_init, optimize_sea=optimize_sea, pas=pas_sea,
+ full_explo_sea=full_explo, **kwargs, **sea_params, **params))
+ if use_seafast:
+ for init in algo_inits:
+ for algo, algo_name in ((sea_fast, 'SEAFAST'), (htp_fast, 'HTPFAST')):
+ current_name = algo_name
+ if init is not None:
+ current_name += f'-{init.__name__}'
+ algos_available[current_name] = lambda *args, algo_init=init, algo_to_run=algo, **kwargs: algo_to_run(
+ *args, algo_init=algo_init, return_history=True,
+ **kwargs, **sea_params, **params)
+ if use_es:
+ algos_available['ES'] = lambda *args, **kwargs: es(*args, **kwargs, **params)
+ algos_available['REA'] = lambda *args, **kwargs: rea(*args, **kwargs, **params)
+ return algos_available
+
+
+def plot_results(datasets=DatasetOperator.BINARY_NAME + DatasetOperator.REGRESSIONS_NAME, presentation=False):
+ """
+ Plot all results by dataset like in the paper using the generated results by `solve_problem` function.
+
+ :param (Iterable[str]) datasets: Datasets with results to plot
+ :param (bool) presentation: If True, plot for presentation
+ """
+ PLOT_PATH.mkdir(exist_ok=True)
+ datasets_sorted = list(datasets)
+ datasets_sorted.sort(key=lambda elt: DatasetOperator.ORDER.index(elt))
+ for data_name in datasets_sorted:
+ logger.info(f"Plotting for {data_name}")
+ dataset = DatasetOperator(data_name)
+ n = dataset.K_MAX[dataset.name] + 1 # dataset.linop.A.shape[1] + 1
+ res = defaultdict(lambda: np.zeros(n))
+ res_best = defaultdict(lambda: np.zeros(n))
+ time_spent = defaultdict(lambda: np.zeros(n))
+ iterations = defaultdict(lambda: np.zeros(n))
+ paths = list(RESULT_PATH.glob(f"*{data_name}*"))
+ paths.sort()
+ legend = dict(
+ legend_title="Algorithms",
+ xaxis_title="Sparsity",
+ xaxis_range=(1, n - 1),
+ title=data_name,
+ )
+ for path in paths:
+ try:
+ with np.load(str(path), allow_pickle=True) as file:
+ stem = path.stem
+ n_nonzeros = int(stem.rsplit('_', 1)[1])
+ data_idx = stem.find(data_name)
+ algo_name, str_idx = stem[:data_idx - 1].rsplit('_', 1)
+ idx = int(str_idx)
+ algo_name = algo_name + "_BEST" if idx == 1 else algo_name
+ res[algo_name][n_nonzeros] = dataset.f(file["x"])
+ iterations[algo_name][n_nonzeros] = file["res_norm"].shape[
+ 0] # 1 if len(file["res_norm"].shape) == 0 else
+ if presentation:
+ if algo_name not in ("ELS", "OMP", "OMPR", "IHTx1000", "SEA-init-els-opti-allx1000_BEST",
+ "SEA-opti-allx1000_BEST"):
+ continue
+ algo_name = algo_name.split('x')[0]
+ if "SEA" in algo_name:
+ algo_name = algo_name[:-len("-opti-all")]
+ if algo_name == "SEA":
+ algo_name = "SEA-init-zero"
+ if (idx == 0 and 'SEA' not in algo_name) or idx == 1:
+ time_spent[algo_name][n_nonzeros] = file["time_spent"][0]
+ res_best[algo_name][n_nonzeros] = dataset.f(file["x"])
+ if 'SEAFAST' in algo_name:
+ pass
+ # file["history"][0].get_top()
+ except Exception as e:
+ logger.error(f"Couldn't open {path}")
+
+ # fig = px.line(res)
+ # fig.update_layout(yaxis_title="Loss", **legend)
+ # fig.write_html(PLOT_PATH / f"loss_{data_name}.html")
+ fig = px.line(res_best)
+ fig.update_layout(yaxis_title="Loss", **legend)
+ fig.write_html(PLOT_PATH / f"best_loss_{data_name}.html")
+ fig = px.line(time_spent)
+ fig.update_layout(yaxis_title="Time spent", **legend)
+ fig.write_html(PLOT_PATH / f"time_spent_{data_name}.html")
+ fig = px.line(iterations)
+ fig.update_layout(yaxis_title="Iterations", **legend)
+ fig.write_html(PLOT_PATH / f"iterations_{data_name}.html")
+
+
+def plot_results_paper(datasets=DatasetOperator.BINARY_NAME + DatasetOperator.REGRESSIONS_NAME):
+ PLOT_PATH_PAPER.mkdir(exist_ok=True)
+ datasets_sorted = list(datasets)
+ datasets_sorted.sort(key=lambda elt: DatasetOperator.ORDER.index(elt))
+ for data_name in datasets_sorted:
+ logger.info(f"Plotting for {data_name}")
+ dataset = DatasetOperator(data_name)
+ n = dataset.K_MAX[dataset.name] + 1 # dataset.linop.A.shape[1] + 1
+ res = defaultdict(lambda: np.zeros(n))
+ paths = list(RESULT_PATH.glob(f"*{data_name}*"))
+ paths.sort()
+ for path in paths:
+ with np.load(str(path)) as file:
+ stem = path.stem
+ n_nonzeros = int(stem.rsplit('_', 1)[1])
+ data_idx = stem.find(data_name)
+ algo_name, str_idx = stem[:data_idx - 1].rsplit('_', 1)
+ idx = int(str_idx)
+ algo_name = algo_name + "_BEST" if idx == 1 else algo_name
+ res[algo_name][n_nonzeros] = dataset.f(file["x"])
+ fig = go.Figure()
+ for algo, info in ALGOS_PAPER_TT.items():
+ if algo in res.keys():
+ if dataset.name == "cal_housing":
+ legends_rank = ["IHT", "HTP", "OMP", "OMPR", "SEA_ELS", "SEA_0"]
+ if info["disp_name"] == "OMP":
+ display_name = "$\\text{OMP}, \\text{IHT}_{\\text{OMP}}, \\text{HTP}_{\\text{OMP}}$"
+ elif info["disp_name"] == "OMPR":
+ display_name = "$\\text{OMPR}, \\text{ELS}, \\text{IHT}_{\\text{ELS}}, \\text{HTP}_{\\text{ELS}}$"
+ elif info["disp_name"] == "SEA_ELS":
+ display_name = "$\\text{SEA}_{\\text{ELS}}, \\text{SEA}_\\text{OMP}$"
+ else:
+ display_name = info["name"]
+ elif dataset.name == "slice" or dataset.name == "letter":
+ if dataset.name == "slice":
+ legends_rank = ["HTP", "IHT", "SEA_0", "OMP", "OMPR", "SEA_OMP", "ELS", "SEA_ELS", ]
+ elif dataset.name == "letter":
+ legends_rank = ["HTP", "IHT", "OMP", "OMPR", "SEA_OMP", "ELS", "SEA_0", "SEA_ELS", ]
+ if info["disp_name"] == "OMP":
+ display_name = "$\\text{OMP}, \\text{IHT}_{\\text{OMP}}, \\text{HTP}_{\\text{OMP}}$"
+ elif info["disp_name"] == "ELS":
+ display_name = "$\\text{ELS}, \\text{IHT}_{\\text{ELS}}, \\text{HTP}_{\\text{ELS}}$"
+ else:
+ display_name = info["name"]
+ elif dataset.name == "ijcnn1":
+ legends_rank = ["HTP", "IHT", "OMP", "OMPR", "SEA_ELS", ]
+ if info["disp_name"] == "OMP":
+ display_name = "$\\text{OMP}, \\text{IHT}_{\\text{OMP}}, \\text{HTP}_{\\text{OMP}}$"
+ elif info["disp_name"] == "OMPR":
+ display_name = "$\\text{OMPR}, \\text{ELS}, \\text{IHT}_{\\text{ELS}}, \\text{HTP}_{\\text{ELS}}$"
+ elif info["disp_name"] == "SEA_ELS":
+ display_name = "$\\text{SEA}_{\\text{0}}, \\text{SEA}_{\\text{OMP}}, \\text{SEA}_{\\text{ELS}}$"
+ else:
+ display_name = info["name"]
+ curve = res[algo]
+
+ if dataset.name == 'letter' and (info["disp_name"] == "SEA_0" or info["disp_name"] == "ELS"):
+ info_line = deepcopy(info["line"])
+ info_line["width"] = info_line["width"] * 1.5
+ else:
+ info_line = info["line"]
+
+ try:
+ fig.add_trace(go.Scatter(y=curve, line=info_line, name=display_name, mode='lines',
+ legendrank=legends_rank.index(info["disp_name"])))
+ except Exception as e:
+ print(f"{algo} is not plotted in the paper version")
+ fig.update_layout(
+ xaxis_title="Sparsity",
+ yaxis_title=r"$\text{log}\_\text{loss}$" if dataset.task == Task.BINARY else r"$\ell_2\_\text{loss}$",
+ xaxis_range=(1, n - 1),
+ **PAPER_LAYOUT,
+ )
+ fig.update_layout(showlegend=True, legend=dict(
+ orientation="h",
+ title=None,
+ y=1,
+ x=0.61,
+ xanchor="left",
+ yanchor="top",
+ entrywidth=500,
+ bgcolor='rgba(0,0,0,0)'
+ ), margin=dict(l=70, b=60))
+ if dataset.task == Task.BINARY:
+ fig.update_layout(
+ margin=dict(autoexpand=False, l=80, ))
+ if dataset.name == "ijcnn1":
+ fig.update_layout(legend=dict(x=0.3, xanchor="left", ))
+ fig.write_html(PLOT_PATH_PAPER / f"paper_{data_name}.html", include_mathjax='cdn')
+
+
+def select_algo(algos_type, algos_filter, sea_params=None, **params) -> Dict[str, Callable]:
+ """
+ Select algorithms to run in experiments
+
+ :param (List[str]) algos_type: List of type of algorithm to look for. Must be a value of ALGOS_TYPE
+ :param (List[str]) algos_filter: List of algorithm to run among those selected by algos_type parameter
+ :param (Any) params: Parameters to pass to all algorithms
+ :return: Dictionary with algorithms associated to their names
+ """
+ if ALL in algos_type:
+ algos_type = ALGOS_TYPE
+ algo_dict = get_algo_dict(**{f'use_{algo_name}': True for algo_name in algos_type}, sea_params=sea_params, **params)
+ if ALL in algos_filter:
+ algo_selected = algo_dict
+ else:
+ algo_selected = {name: algo for name, algo in algo_dict.items() if name in algos_filter}
+ return algo_selected
+
+
+@ray.remote
+def run_main(cmd) -> None:
+ """
+ Run a command with the Click CLI
+
+ :param (str) cmd: Command to send
+ """
+ main(cmd.split(" "), standalone_mode=False)
+
+
+@click.command(context_settings={'show_default': True, 'help_option_names': ['-h', '--help']})
+@click.option('--algos_type', '-at', multiple=True, default=[ALL],
+ type=click.Choice([*ALGOS_TYPE, ALL], case_sensitive=False),
+ help='Algorithms to run. If \'ALL\' is selected, run all algorithms.')
+@click.option('--factors', '-f', multiple=True, default=[256],
+ help='Sparsity factors to use for the iteration\'s number of SEA and IHT')
+@click.option('--datasets', '-d', multiple=True,
+ default=list(DatasetOperator.REGRESSIONS_NAME) + list(DatasetOperator.BINARY_NAME[:2]),
+ type=click.Choice(list(DatasetOperator.REGRESSIONS_NAME) + list(DatasetOperator.BINARY_NAME) + [ALL],
+ case_sensitive=False),
+ help='Datasets to evaluate. If \'ALL\' is selected, evaluate all datasets.')
+@click.option('--algos_filter', '-af', multiple=True,
+ default=["SEAFAST-els", "SEAFAST-omp", "SEAFAST", "ELS", "OMP", "IHT",
+ "HTP", "IHT-omp", "HTP-omp", "IHT-els", "HTP-els", "OMPR"],
+ type=click.Choice(list(get_algo_dict(*[True] * len(ALGOS_TYPE)).keys()) + [ALL], case_sensitive=False),
+ help='Algorithms to run. If \'ALL\' is selected, run all algorithms.')
+@click.option('--resume/--no-resume', '-r/-nr', default=True, help='If don\'t overwrite previous results')
+@click.option('--plot/--no-plot', '-pd/-npd', default=False,
+ help='If True, only plot results for all selected datasets')
+@click.option('--plot-paper/--no-plot-paper', '-pl/-npl', default=False,
+ help='If True, only plot results for all selected datasets for the paper')
+@click.option('--param_file', '-pf', default=None, help='If specified, run in parallel multiple instance'
+ ' of the function with the specified parameters')
+@click.option('--num-cpus', '-cpu', default=None, type=int, help="Limit the number of CPU if needed")
+@click.option('--reverse/--no-reverse', '-rev/-nrev', default=False, help='If True, compute by descending sparsity')
+@click.option('--sparsities', '-s', multiple=True, default=[0], type=click.IntRange(0, DatasetOperator.MAX_K_MAX + 1),
+ help='Sparsity values to use when solving problems')
+def main(algos_type, datasets, factors, resume, algos_filter, plot, plot_paper, param_file, num_cpus, reverse,
+ sparsities) -> None:
+ """
+ Solve problem from the selected datasets with the specified algorithms
+ """
+ logger.info(f"Parameters: \n{locals()}")
+ if plot or plot_paper:
+ if plot_paper:
+ if ALL in datasets:
+ plot_ml_paper(('cal_housing', 'comp-activ-harder', 'ijcnn1', 'letter', 'slice', 'year',),
+ factors)
+ else:
+ plot_ml_paper(datasets, factors)
+ elif plot:
+ if ALL in datasets:
+ plot_results()
+ else:
+ plot_results(datasets)
+ elif param_file is None:
+ # Dict are already normalized and, we need to specify iteration limit for OMP-like algorithms
+ algo_selected = select_algo(algos_type, algos_filter, normalize=False, n_iter=1000)
+ datasets = DatasetOperator.REGRESSIONS_NAME + DatasetOperator.BINARY_NAME if ALL in datasets else datasets
+ with np.errstate(divide='ignore', invalid='ignore'):
+ run_experiments(algo_selected, datasets, factors, resume, reverse, sparsities)
+ logger.info(f'End of Training for : \n{locals()}')
+ else:
+ # For multiprocessing
+ if os.getcwd().startswith('/baie'): # On sms cluster
+ logger.info("Running on sms")
+ ray.init(_temp_dir="/scratch/ray")
+ else: # On local or core5
+ logger.info(f"Running on {socket.gethostname()}")
+ ray.init(num_cpus=num_cpus)
+ with open(param_file, 'r') as file:
+ futures = [run_main.remote(line.strip()) for line in file.readlines()]
+ ray.get(futures)
+
+
+if __name__ == '__main__':
+ main()
diff --git a/code/sksea/utils.py b/code/sksea/utils.py
new file mode 100644
index 0000000000000000000000000000000000000000..76106e10379991790b54488dc91f05908c31e9ea
--- /dev/null
+++ b/code/sksea/utils.py
@@ -0,0 +1,379 @@
+# -*- coding: utf-8 -*-
+from abc import ABC, abstractmethod
+from pathlib import Path
+from typing import Tuple
+
+import numpy as np
+from loguru import logger
+from PIL import ImageColor
+import plotly.colors
+from scipy.sparse.linalg import svds, LinearOperator
+from scipy.stats import wasserstein_distance
+
+
+def find_support(x, n_nonzero) -> np.ndarray:
+ """
+ Return an array with True value on the indexes of the n_nonzeros highest coefficients of |x|
+
+ :param (np.ndarray) x: Input vector
+ :param (int) n_nonzero: Size of the wanted support
+ :return: The support space of the wanted size
+ """
+ x_abs = np.abs(x)
+ sorted_idx = np.argsort(x_abs)
+ s = np.zeros_like(x, dtype=bool)
+ s[sorted_idx[-n_nonzero:]] = True
+ return s
+
+
+def soft_thresholding(x, threshold):
+ return np.sign(x) * np.maximum(np.abs(x) - threshold, 0)
+
+
+def soft_thresholding_der(x, threshold):
+ return np.abs(x) > threshold
+
+
+def hard_thresholding(x, n_nonzero):
+ return x * find_support(x=x, n_nonzero=n_nonzero)
+
+
+class AbstractLinearOperator(ABC, LinearOperator):
+ """
+ Class with elements to add to the LinearOperator class of scipy.
+ The abstracts methods need a specific implementation inside any subclass operator for limiting performances issues
+ """
+
+ def __init__(self, dtype, shape):
+ super().__init__(dtype, shape)
+ self.seed = None
+
+ def compute_lipschitz(self) -> float:
+ """
+ Compute lipschitz constant of the operator using svds
+ """
+ if self.shape[0] == 1:
+ raise ValueError("n_samples=1")
+ if self.shape[1] == 1:
+ raise ValueError("n_features=1")
+ return svds(A=self, k=1, return_singular_vectors=False, random_state=self.seed)[0] ** 2
+
+ @property
+ def matrix(self) -> np.ndarray:
+ """
+ Return the matrix representation of the operator
+ """
+ return self @ np.eye(self.shape[1])
+
+ @abstractmethod
+ def get_operator_on_support(self, s) -> 'AbstractLinearOperator':
+ """
+ Return the operator truncated on the provided support
+
+ :param (np.ndarray) s: Support
+ """
+ pass
+
+ @abstractmethod
+ def get_normalized_operator(self) -> Tuple['AbstractLinearOperator', np.ndarray]:
+ """
+ Return a normalized version of the operator using its kernel and the un-normalization matrix
+ """
+ pass
+
+
+def support_distance(x1, x2):
+ """
+
+ :param (np.ndarray) x1:
+ :param (np.ndarray) x2:
+ :return:
+ """
+ s1 = set(x1.nonzero()[0])
+ s2 = set(x2.nonzero()[0])
+ m = max(len(s1), len(s2))
+ return (m - len(s1.intersection(s2))) / m
+
+
+VECTOR_AXIS = -1
+MODULE_PATH = Path(__file__).parent
+RESULT_FOLDER = MODULE_PATH / "results" / "deconv"
+DATA_FILENAME = "data.npz"
+
+def compute_support_distance(a1, a2):
+ *other, last = a1.shape
+ r1 = a1.reshape(np.prod(other), last)
+ r2 = a2.reshape(np.prod(other), last)
+ n = r1.shape[0]
+ result = np.zeros(n)
+ for idx in range(n):
+ try:
+ result[idx] = support_distance(r1[idx], r2[idx])
+ except Exception as e:
+ logger.error(f"Support distance error")
+ result[idx] = np.nan
+ return result.reshape(other)
+
+
+def compute_metrcs_from_file(file, spars_max, linop, solution, temp_plot_file):
+ *other_dim, last = solution.shape
+ results = np.load(file)[:, :spars_max, :]
+
+ # Number of supports explored
+ support_file = file.parent / (file.stem + ".npz")
+ if support_file.is_file():
+ other_results = np.load(support_file, allow_pickle=True)
+ n_supports = other_results.get("n_supports")
+ n_supports_new = other_results.get("n_supports_new")
+ n_supports_from_start = other_results.get("n_supports_from_start")
+ else:
+ n_supports = np.zeros_like(results[:, :, 0])
+ n_supports_new = np.zeros_like(results[:, :, 0])
+ n_supports_from_start = np.zeros_like(results[:, :, 0])
+
+ # MSE over x signal plot
+ mse: np.ndarray = np.linalg.norm(solution - results, axis=VECTOR_AXIS
+ ) / np.linalg.norm(solution, axis=VECTOR_AXIS)
+
+ # Support error plot
+ sup_dist = compute_support_distance(solution, results)
+
+ # MSE over y plot
+ f_sol = solution.reshape(np.prod(other_dim), last)
+ f_res = results.reshape(np.prod(other_dim), last)
+ f_mse_y: np.ndarray = np.linalg.norm((linop @ f_sol.T - linop @ f_res.T).T, axis=VECTOR_AXIS
+ ) / np.linalg.norm((linop @ f_res.T).T, axis=VECTOR_AXIS)
+
+ # Wasserstein distance plot
+ ws = np.zeros_like(mse)
+ ws_bin = np.zeros_like(mse)
+ ws_bin_norm = np.zeros_like(mse)
+ n_runs, n_sparcities, _ = results.shape
+ for sparcity_id in range(n_sparcities):
+ for run_id in range(n_runs):
+ ws[run_id, sparcity_id] = wasserstein_distance(solution[run_id, sparcity_id, :] /
+ np.linalg.norm(solution[run_id, sparcity_id, :],
+ ord=1),
+ results[run_id, sparcity_id, :] /
+ np.linalg.norm(results[run_id, sparcity_id, :],
+ ord=1))
+ sol = (solution[run_id, sparcity_id, :] != 0).astype(float)
+ res = (results[run_id, sparcity_id, :] != 0).astype(float)
+ ws_bin[run_id, sparcity_id] = wasserstein_distance(sol, res)
+ ws_bin_norm[run_id, sparcity_id] = wasserstein_distance(sol / np.sum(sol), res / np.sum(res))
+ temp_plot_file.parent.mkdir(exist_ok=True)
+ np.savez(temp_plot_file, sup_dist=sup_dist, mse=mse, f_mse_y=f_mse_y, ws=ws, n_supports=n_supports,
+ n_supports_new=n_supports_new, n_supports_from_start=n_supports_from_start, ws_bin=ws_bin,
+ ws_bin_norm=ws_bin_norm)
+# https://plotly.com/python/marker-style/#custom-marker-symbols
+algos_base = {
+ "IHT": {"disp_name": "IHT", "name": "$\\text{IHT}$", "line": {}, "marker": dict(symbol=134)},
+ "HTP": {"disp_name": "HTP", "name": "$\\text{HTP}$", "line": {"dash": "dash"}, "marker": dict(symbol=124)},
+ "HTP_OMP": {"disp_name": "HTP_OMP", "name": "$\\text{HTP}_{\\text{OMP}}$", "line": {"dash": "dashdot"}, "marker": dict(symbol=106)},
+ "IHT_OMP": {"disp_name": "IHT_OMP", "name": "$\\text{IHT}_{\\text{OMP}}$", "line": {}, "marker": dict(symbol=106)},
+ "OMP": {"disp_name": "OMP", "name": "$\\text{OMP}$", "line": {"dash": "dash"}, "marker": dict(symbol=105)},
+ "OMPR": {"disp_name": "OMPR", "name": "$\\text{OMPR}$", "line": {}, "marker": dict(symbol=107)},
+ "ELS": {"disp_name": "ELS", "name": "$\\text{ELS}$", "line": {"dash": "dash"}, "marker": dict(symbol=133)},
+ "IHT_ELS": {"disp_name": "IHT_ELS", "name": "$\\text{IHT}_{\\text{ELS}}$", "line": {}, "marker": dict(symbol=106)},
+ "HTP_ELS": {"disp_name": "HTP_ELS", "name": "$\\text{HTP}_{\\text{ELS}}$", "line": {"dash": "dash"}, "marker": dict(symbol=106)},
+ "SEA_0": {"disp_name": "SEA_0", "name": "$\\text{SEA}_0$", "line": {"dash": "dot"}, "marker": dict(symbol=107)},
+ "SEA_ELS": {"disp_name": "SEA_ELS", "name": "$\\text{SEA}_{\\text{ELS}}$", "line": {"dash": "dot"},
+ "marker": dict(symbol=101)},
+ "SEA_OMP": {"disp_name": "SEA_OMP", "name": "$\\text{SEA}_{\\text{OMP}}$", "line": {"dash": "dashdot"},
+ "marker": dict(symbol=108)},
+
+}
+
+map_dt = {
+ "IHTx256": "IHT",
+ "OMPRx100": "OMPR",
+ "OMPx100": "OMP",
+ "ELSx100": "ELS",
+ "SEA-opti-allx256": "SEA_0",
+ "SEA-init-els-opti-allx256": "SEA_ELS"
+}
+
+map_tt = {
+ "IHTx256": "IHT",
+ "OMPR": "OMPR",
+ "HTPx256": "HTP",
+ "OMP": "OMP",
+ "ELS": "ELS",
+ "SEA-opti-allx256_BEST": "SEA_0",
+ "SEAFASTx256_BEST": "SEA_0",
+ #"SEA-all-Lstepx256_BEST": "SEA_0",
+ #"SEA-els-all-Lstepx256_BEST": "SEA_ELS",
+ "SEAFAST-ompx256_BEST": "SEA_OMP",
+ "SEA-init-els-opti-allx256_BEST": "SEA_ELS",
+ "SEAFAST-elsx256_BEST": "SEA_ELS",
+
+}
+
+map_dcv = {
+ "IHT": "IHT",
+ "OMPR": "OMPR",
+ "HTP": "HTP",
+ "OMP": "OMP",
+ "ELS": "ELS",
+ "SEA-all-Lstep": "SEA_0",
+ "SEAFAST": "SEA_0",
+ "SEAFAST-omp": "SEA_OMP",
+ "SEA-els-all-Lstep": "SEA_ELS",
+ "SEAFAST-els": "SEA_ELS",
+}
+
+map_dcv_precise = {
+ "HTP": "HTP",
+ "HTPFAST": "HTP",
+ "IHT-omp": "IHT_OMP",
+ "HTP-omp": "HTP_OMP",
+ "HTPFAST-omp": "HTP_OMP",
+ "IHT-els": "IHT_ELS",
+ "HTP-els": "HTP_ELS",
+ "HTPFAST-els": "HTP_ELS",
+ "OMPR": "OMPR",
+ "OMPRFAST": "OMPR",
+ "OMPFAST": "OMP",
+ "OMP": "OMP",
+ "SEA-all-Lstep": "SEA_0",
+ "SEAFAST": "SEA_0",
+ "SEA-els-all-Lstep": "SEA_ELS",
+ "SEAFAST-omp": "SEA_OMP",
+ "SEAFAST-els": "SEA_ELS",
+ "ELS": "ELS",
+ "ELSFAST": "ELS",
+ "IHT": "IHT",
+
+}
+
+colors = ["#636EFA", "#FFA15A", "#8B4513", "#FECB52", "#FF6692", "#00CC96", "#AB63FA", "#AB63FA", "#AB63FA", "#B6E880", "#EF553B", "#19D3F3"]
+for idx, info in enumerate(algos_base.values()):
+ info["line"]["color"] = colors[idx]
+ info["line"]["width"] = 3
+ info["marker"]["size"] = 20
+
+
+def algos_paper_dt_from_factor(factor):
+ map_dt2 = {
+ f"IHTx{factor}": "IHT",
+ f"IHT-ompx{factor}": "IHT_OMP",
+ "OMPR": "OMPR",
+ f"OMPRx{factor}": "OMPR",
+ f"HTPx{factor}": "HTP",
+ f"HTP-ompx{factor}": "HTP_OMP",
+ f"OMPx{factor}": "OMP",
+ f"ELSx{factor}": "ELS",
+ "OMP": "OMP",
+ "ELS": "ELS",
+ f"SEA-all-Lstepx{factor}_BEST": "SEA_0",
+ f"SEAFASTx{factor}_BEST": "SEA_0",
+ f"SEAFAST-ompx{factor}_BEST": "SEA_OMP",
+ f"SEA-els-all-Lstepx{factor}_BEST": "SEA_ELS",
+ f"SEAFAST-elsx{factor}_BEST": "SEA_ELS",
+
+ }
+ return {algo_surname: algos_base[algo_name] for algo_surname, algo_name in map_dt2.items()}
+
+
+ALGOS_PAPER_TT = {algo_surname: algos_base[algo_name] for algo_surname, algo_name in map_tt.items()}
+ALGOS_PAPER_DCV = {algo_surname: algos_base[algo_name] for algo_surname, algo_name in map_dcv.items()}
+ALGOS_PAPER_DCV_PRECISE = {algo_surname: algos_base[algo_name] for algo_surname, algo_name in map_dcv_precise.items()}
+
+PAPER_LAYOUT = dict(
+ legend_title=r"$\text{Algorithms}$",
+ margin=dict(
+ autoexpand=False,
+ l=55,
+ r=10,
+ t=30,
+ b=50,
+ ),
+ plot_bgcolor='white',
+ xaxis=dict(
+ showline=True,
+ showgrid=True,
+ showticklabels=True,
+ linecolor='rgb(204, 204, 204)',
+ ),
+ yaxis=dict(
+ showgrid=True,
+ showline=True,
+ showticklabels=True,
+ linecolor='rgb(204, 204, 204)',
+ ),
+ font=dict(
+ size=20,
+ ),
+)
+
+# https://stackoverflow.com/questions/69699744/plotly-express-line-with-continuous-color-scale
+
+# This function allows you to retrieve colors from a continuous color scale
+# by providing the name of the color scale, and the normalized location between 0 and 1
+# Reference: https://stackoverflow.com/questions/62710057/access-color-from-plotly-color-scale
+
+def get_color(colorscale_name, loc):
+ from _plotly_utils.basevalidators import ColorscaleValidator
+ # first parameter: Name of the property being validated
+ # second parameter: a string, doesn't really matter in our use case
+ cv = ColorscaleValidator("colorscale", "")
+ # colorscale will be a list of lists: [[loc1, "rgb1"], [loc2, "rgb2"], ...]
+ colorscale = cv.validate_coerce(colorscale_name)
+
+ if hasattr(loc, "__iter__"):
+ return [get_continuous_color(colorscale, x) for x in loc]
+ return get_continuous_color(colorscale, loc)
+
+
+def get_continuous_color(colorscale, intermed):
+ """
+ Plotly continuous colorscales assign colors to the range [0, 1]. This function computes the intermediate
+ color for any value in that range.
+
+ Plotly doesn't make the colorscales directly accessible in a common format.
+ Some are ready to use:
+
+ colorscale = plotly.colors.PLOTLY_SCALES["Greens"]
+
+ Others are just swatches that need to be constructed into a colorscale:
+
+ viridis_colors, scale = plotly.colors.convert_colors_to_same_type(plotly.colors.sequential.Viridis)
+ colorscale = plotly.colors.make_colorscale(viridis_colors, scale=scale)
+
+ :param colorscale: A plotly continuous colorscale defined with RGB string colors.
+ :param intermed: value in the range [0, 1]
+ :return: color in rgb string format
+ :rtype: str
+ """
+ if len(colorscale) < 1:
+ raise ValueError("colorscale must have at least one color")
+
+ hex_to_rgb = lambda c: "rgb" + str(ImageColor.getcolor(c, "RGB"))
+
+ if intermed <= 0 or len(colorscale) == 1:
+ c = colorscale[0][1]
+ return c if c[0] != "#" else hex_to_rgb(c)
+ if intermed >= 1:
+ c = colorscale[-1][1]
+ return c if c[0] != "#" else hex_to_rgb(c)
+
+ for cutoff, color in colorscale:
+ if intermed > cutoff:
+ low_cutoff, low_color = cutoff, color
+ else:
+ high_cutoff, high_color = cutoff, color
+ break
+
+ if (low_color[0] == "#") or (high_color[0] == "#"):
+ # some color scale names (such as cividis) returns:
+ # [[loc1, "hex1"], [loc2, "hex2"], ...]
+ low_color = hex_to_rgb(low_color)
+ high_color = hex_to_rgb(high_color)
+
+ return plotly.colors.find_intermediate_color(
+ lowcolor=low_color,
+ highcolor=high_color,
+ intermed=((intermed - low_cutoff) / (high_cutoff - low_cutoff)),
+ colortype="rgb",
+ )
+
diff --git a/minimal_example.py b/minimal_example.py
new file mode 100644
index 0000000000000000000000000000000000000000..2745730acd7ca7d0dc285f74af3da179601a58af
--- /dev/null
+++ b/minimal_example.py
@@ -0,0 +1,47 @@
+"""
+Minimal working example of SEA usage
+"""
+import numpy as np
+
+from sksea.algorithms import sea_fast, omp, SEA #, els, htp_fast, iht, ompr
+from sksea.sparse_coding import SparseSupportOperator
+from sksea.utils import hard_thresholding
+
+
+if __name__ == '__main__':
+ # Seed for repeatability
+ seed = 0
+ random = np.random.RandomState(seed)
+
+ # Problem instantiation
+ n_atom = 20
+ n_nonzero = 3
+ data_mat = random.random((10, n_atom))
+ x_full = random.random(n_atom)
+ x_sparse = hard_thresholding(x_full, n_nonzero) # Keep the n_nonzero highest coefficient
+ y = data_mat @ x_sparse
+
+ # SEA usage with low-level function
+ linop = SparseSupportOperator(data_mat, y, seed) # Linear Operator instantiation
+ f = lambda x, linear_operator: np.linalg.norm(linear_operator @ x - y) / 2 # Function to minimize
+ grad_f = lambda x, linear_operator: linear_operator.H @ (linear_operator @ x - y) # Gradient of the function
+
+ x_sea, res_norm_sea, exploration_history_sea = sea_fast(linop, y, n_nonzero, n_iter=20, f=f, grad_f=grad_f,
+ optimizer='chol', return_best=True)
+ print(res_norm_sea)
+
+ # SEA usage with sklearn-API
+ sea = SEA(n_nonzero, n_iter=20, random_state=seed)
+ sea.fit(data_mat, y)
+ # x_sea = sea.coef_
+ # res_norm_sea = sea.res_norm_
+ # explored_supports = sea.exploration_
+
+ # Exploration history usage
+ explored_supports = exploration_history_sea.get_supports() # Numpy array containing all explored support
+ support = explored_supports[0] # Get one support
+ x_estimated, gradient, loss_value = exploration_history_sea.get(support) # Get insight on one specific support
+
+ # SEA warm-started by omp
+ x_sea_omp, res_norm_sea_omp, exploration_history_sea_omp = sea_fast(linop, y, n_nonzero, n_iter=100, f=f, grad_f=grad_f,
+ return_best=True, algo_init=omp)