Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
B
bolsonaro
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Container Registry
Model registry
Operate
Environments
Monitor
Incidents
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
Luc Giffon
bolsonaro
Merge requests
!15
Resolve "Adding new datasets"
Code
Review changes
Check out branch
Download
Patches
Plain diff
Merged
Resolve "Adding new datasets"
17-adding-new-datasets
into
master
Overview
0
Commits
39
Pipelines
0
Changes
205
Merged
Leo Bouscarrat
requested to merge
17-adding-new-datasets
into
master
5 years ago
Overview
0
Commits
39
Pipelines
0
Changes
80
Expand
Closes
#17 (closed)
Edited
5 years ago
by
Charly Lamothe
0
0
Merge request reports
Compare
version 9
version 31
6378245a
5 years ago
version 30
2d896dd1
5 years ago
version 29
0f1d25f7
5 years ago
version 28
24d53ff8
5 years ago
version 27
006b001e
5 years ago
version 26
0edaa3f0
5 years ago
version 25
1d5b54a1
5 years ago
version 24
1b84a5c6
5 years ago
version 23
b9f6822b
5 years ago
version 22
1531c791
5 years ago
version 21
31a58a03
5 years ago
version 20
0b0af680
5 years ago
version 19
a03357d0
5 years ago
version 18
baca1281
5 years ago
version 17
264288b4
5 years ago
version 16
33453392
5 years ago
version 15
34bca5fe
5 years ago
version 14
72465f53
5 years ago
version 13
1a22e391
5 years ago
version 12
36241737
5 years ago
version 11
1bfd15fe
5 years ago
version 10
baa96e2a
5 years ago
version 9
77f3c68b
5 years ago
version 8
b2d16a25
5 years ago
version 7
7885ac63
5 years ago
version 6
effd61df
5 years ago
version 5
b5bbdccd
5 years ago
version 4
96f51d9c
5 years ago
version 3
36e191bc
5 years ago
version 2
2921c030
5 years ago
version 1
61a8e5a9
5 years ago
master (base)
and
version 27
latest version
b0e1c83e
39 commits,
5 years ago
version 31
6378245a
38 commits,
5 years ago
version 30
2d896dd1
37 commits,
5 years ago
version 29
0f1d25f7
36 commits,
5 years ago
version 28
24d53ff8
35 commits,
5 years ago
version 27
006b001e
34 commits,
5 years ago
version 26
0edaa3f0
33 commits,
5 years ago
version 25
1d5b54a1
32 commits,
5 years ago
version 24
1b84a5c6
29 commits,
5 years ago
version 23
b9f6822b
27 commits,
5 years ago
version 22
1531c791
26 commits,
5 years ago
version 21
31a58a03
25 commits,
5 years ago
version 20
0b0af680
24 commits,
5 years ago
version 19
a03357d0
22 commits,
5 years ago
version 18
baca1281
21 commits,
5 years ago
version 17
264288b4
20 commits,
5 years ago
version 16
33453392
18 commits,
5 years ago
version 15
34bca5fe
17 commits,
5 years ago
version 14
72465f53
16 commits,
5 years ago
version 13
1a22e391
15 commits,
5 years ago
version 12
36241737
14 commits,
5 years ago
version 11
1bfd15fe
13 commits,
5 years ago
version 10
baa96e2a
12 commits,
5 years ago
version 9
77f3c68b
11 commits,
5 years ago
version 8
b2d16a25
10 commits,
5 years ago
version 7
7885ac63
9 commits,
5 years ago
version 6
effd61df
8 commits,
5 years ago
version 5
b5bbdccd
7 commits,
5 years ago
version 4
96f51d9c
6 commits,
5 years ago
version 3
36e191bc
4 commits,
5 years ago
version 2
2921c030
2 commits,
5 years ago
version 1
61a8e5a9
1 commit,
5 years ago
Show latest version
80 files
+
1796
−
1213
Inline
Compare changes
Side-by-side
Inline
Show whitespace changes
Show one file at a time
Files
80
Search (e.g. *.vue) (Ctrl+P)
code/bolsonaro/data/dataset_loader.py
+
25
−
4
Options
from
bolsonaro.data.dataset
import
Dataset
from
bolsonaro.data.dataset_parameters
import
DatasetParameters
from
bolsonaro.data.task
import
Task
from
bolsonaro.utils
import
change_binary_func_load
from
bolsonaro.utils
import
change_binary_func_load
,
change_binary_func_openml
from
sklearn.datasets
import
load_boston
,
load_iris
,
load_diabetes
,
\
load_digits
,
load_linnerud
,
load_wine
,
load_breast_cancer
from
sklearn.datasets
import
fetch_olivetti_faces
,
fetch_20newsgroups
,
\
fetch_20newsgroups_vectorized
,
fetch_lfw_people
,
fetch_lfw_pairs
,
\
fetch_covtype
,
fetch_rcv1
,
fetch_kddcup99
,
fetch_california_housing
fetch_covtype
,
fetch_rcv1
,
fetch_kddcup99
,
fetch_california_housing
,
\
fetch_openml
from
sklearn.model_selection
import
train_test_split
from
sklearn
import
preprocessing
import
random
@@ -30,13 +31,15 @@ class DatasetLoader(object):
dataset_names
=
[
'
boston
'
,
'
iris
'
,
'
diabetes
'
,
'
digits
'
,
'
linnerud
'
,
'
wine
'
,
'
breast_cancer
'
,
'
olivetti_faces
'
,
'
20newsgroups_vectorized
'
,
'
lfw_people
'
,
'
lfw_pairs
'
,
'
covtype
'
,
'
rcv1
'
,
'
california_housing
'
,
'
diamonds
'
]
'
lfw_pairs
'
,
'
covtype
'
,
'
rcv1
'
,
'
california_housing
'
,
'
diamonds
'
,
'
steel-plates
'
,
'
kr-vs-kp
'
,
'
kin8nm
'
,
'
spambase
'
,
'
musk
'
,
'
gamma
'
]
dataset_seed_numbers
=
{
'
boston
'
:
15
,
'
iris
'
:
15
,
'
diabetes
'
:
15
,
'
digits
'
:
5
,
'
linnerud
'
:
15
,
'
wine
'
:
15
,
'
breast_cancer
'
:
15
,
'
olivetti_faces
'
:
15
,
'
20newsgroups_vectorized
'
:
3
,
'
lfw_people
'
:
3
,
'
lfw_pairs
'
:
3
,
'
covtype
'
:
3
,
'
rcv1
'
:
3
,
'
california_housing
'
:
3
,
'
diamonds
'
:
15
}
'
diamonds
'
:
15
,
'
steel-plates
'
:
15
,
'
kr-vs-kp
'
:
15
,
'
kin8nm
'
:
15
,
'
spambase
'
:
15
,
'
musk
'
:
15
,
'
gamma
'
:
15
}
@staticmethod
def
load
(
dataset_parameters
):
@@ -103,6 +106,24 @@ class DatasetLoader(object):
df
[
'
clarity
'
]
=
label_clarity
.
fit_transform
(
df
[
'
clarity
'
])
X
,
y
=
df
.
drop
([
'
price
'
],
axis
=
1
),
df
[
'
price
'
]
task
=
Task
.
REGRESSION
elif
name
==
'
steel-plates
'
:
dataset_loading_func
=
change_binary_func_openml
(
'
steel-plates-fault
'
)
task
=
Task
.
BINARYCLASSIFICATION
elif
name
==
'
kr-vs-kp
'
:
dataset_loading_func
=
change_binary_func_openml
(
'
kr-vs-kp
'
)
task
=
Task
.
BINARYCLASSIFICATION
elif
name
==
'
kin8nm
'
:
X
,
y
=
fetch_openml
(
'
kin8nm
'
,
return_X_y
=
True
)
task
=
Task
.
REGRESSION
elif
name
==
'
spambase
'
:
dataset_loading_func
=
change_binary_func_openml
(
'
spambase
'
)
task
=
Task
.
BINARYCLASSIFICATION
elif
name
==
'
musk
'
:
dataset_loading_func
=
change_binary_func_openml
(
'
musk
'
)
task
=
Task
.
BINARYCLASSIFICATION
elif
name
==
'
gamma
'
:
dataset_loading_func
=
change_binary_func_openml
(
'
MagicTelescope
'
)
task
=
Task
.
BINARYCLASSIFICATION
else
:
raise
ValueError
(
"
Unsupported dataset
'
{}
'"
.
format
(
name
))
Loading