Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
S
sensei_pipeline
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Container Registry
Model registry
Operate
Environments
Monitor
Incidents
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
SENSEI
sensei_pipeline
Commits
ae0db8bd
Commit
ae0db8bd
authored
8 years ago
by
Benoit Favre
Browse files
Options
Downloads
Patches
Plain Diff
update code highlighting
parent
3254328d
No related branches found
No related tags found
No related merge requests found
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
README.md
+35
-13
35 additions, 13 deletions
README.md
with
35 additions
and
13 deletions
README.md
+
35
−
13
View file @
ae0db8bd
...
@@ -10,8 +10,9 @@ Install requirements
...
@@ -10,8 +10,9 @@ Install requirements
The REST services require bottle.py to be installed.
The REST services require bottle.py to be installed.
pip install --user -r requirements.txt -U
```
pip install --user -r requirements.txt -U
```
REST services
REST services
-------------
-------------
...
@@ -23,35 +24,47 @@ server of your choice (the default WSGIRefServer is quite slow, you may install
...
@@ -23,35 +24,47 @@ server of your choice (the default WSGIRefServer is quite slow, you may install
For instance, you can run the rest/test.py script which reverses character strings.
For instance, you can run the rest/test.py script which reverses character strings.
rest/test.py
```
rest/test.py
```
It replies to three URLs: / is a short textual help for using the service.
It replies to three URLs: / is a short textual help for using the service.
curl http://localhost:1234
```
curl http://localhost:1234
```
It also recognizes GET requests of the form /test/
<text>
, where
<text>
is the
It also recognizes GET requests of the form /test/
<text>
, where
<text>
is the
string you want to reverse.
string you want to reverse.
curl http://localhost:1234/test/hello
```
curl http://localhost:1234/test/hello
```
Finally, for large inputs, it recognizes POST queries where the input string is
Finally, for large inputs, it recognizes POST queries where the input string is
specified in a "text" form parameter.
specified in a "text" form parameter.
curl --form "text=hello" http://localhost:1234/test
```
curl --form "text=hello" http://localhost:1234/test
```
A generic REST service is also available. It runs a custom command and feeds it
A generic REST service is also available. It runs a custom command and feeds it
with inputs through stdin and collects output from stdout. The following
with inputs through stdin and collects output from stdout. The following
example echoes the inputs as output. The name parameter set the name of the
example echoes the inputs as output. The name parameter set the name of the
service in the url.
service in the url.
rest/generic.py --port 1234 --name "cat" --command "cat"
```
curl http://localhost:1234/cat/hello
rest/generic.py --port 1234 --name "cat" --command "cat"
curl http://localhost:1234/cat/hello
```
The command can also be made persistent (run in the background) and fed line by
The command can also be made persistent (run in the background) and fed line by
line through stdin, and its output read line by line from stdout. Note that for
line through stdin, and its output read line by line from stdout. Note that for
this to work, the command needs to flush stdout after each input.
this to work, the command needs to flush stdout after each input.
rest/generic.py --port 1234 --name --command "awk '{x+=1;print x,
\$
0;fflush()}'" --persistent True
```
rest/generic.py --port 1234 --name --command "awk '{x+=1;print x,\$0;fflush()}'" --persistent True
```
WARNING: the generic REST service does not enforce any kind of security.
WARNING: the generic REST service does not enforce any kind of security.
...
@@ -66,7 +79,9 @@ Repository-integrated services poll the repository for new documents, process
...
@@ -66,7 +79,9 @@ Repository-integrated services poll the repository for new documents, process
them and push back annotation sets. In order to get access to the repository,
them and push back annotation sets. In order to get access to the repository,
you can create a tunnel with (given a proper ssh key is setup):
you can create a tunnel with (given a proper ssh key is setup):
util/repository.py --tunnel
```
util/repository.py --tunnel
```
The util/repository.py script has a lot more commands available. It can also be
The util/repository.py script has a lot more commands available. It can also be
used as a python module as full-functional repository client.
used as a python module as full-functional repository client.
...
@@ -75,7 +90,9 @@ Once the tunnel is setup, you can try the test annotator which polls the
...
@@ -75,7 +90,9 @@ Once the tunnel is setup, you can try the test annotator which polls the
repository and adds phony annotations. That script allows to choose the host and
repository and adds phony annotations. That script allows to choose the host and
port of the repository.
port of the repository.
repo/test.py
```
repo/test.py
```
This script gets all documents which don't have the AMU_hasTest feature, puts
This script gets all documents which don't have the AMU_hasTest feature, puts
generated annotations in the AMU_Test annotation set. Then, it sets the
generated annotations in the AMU_Test annotation set. Then, it sets the
...
@@ -84,7 +101,9 @@ AMU_hasTest feature to True.
...
@@ -84,7 +101,9 @@ AMU_hasTest feature to True.
After a few documents are processed, you can kill the script and run the
After a few documents are processed, you can kill the script and run the
cleaner which deletes all annotation sets created by the previous script.
cleaner which deletes all annotation sets created by the previous script.
repo/clean.py --presence_feature AMU_hasTest --annotation_set AMU_Test
```
repo/clean.py --presence_feature AMU_hasTest --annotation_set AMU_Test
```
The first way to integrate a novel processing module to the repository is to
The first way to integrate a novel processing module to the repository is to
use the generic annotator. It grabs all documents which don't have a given
use the generic annotator. It grabs all documents which don't have a given
...
@@ -98,9 +117,11 @@ its expectations and contain the right fields. Overwise, an error is returned.
...
@@ -98,9 +117,11 @@ its expectations and contain the right fields. Overwise, an error is returned.
For example, you can write a script which computes the length of the json
For example, you can write a script which computes the length of the json
representation of a document, and creates a new "checksum" annotation with it.
representation of a document, and creates a new "checksum" annotation with it.
```
cat script.sh
cat script.sh
awk '{print "[{\"type\": \"checksum\", \"features\": {\"value\":"length()"}, \"start\": 0, \"end\": 0}]"}'
awk '{print "[{\"type\": \"checksum\", \"features\": {\"value\":"length()"}, \"start\": 0, \"end\": 0}]"}'
repo/generic.py --command ./script.sh --mark_feature "AMU_hasChecksum" --annotation "AMU_Checksum"
repo/generic.py --command ./script.sh --mark_feature "AMU_hasChecksum" --annotation "AMU_Checksum"
```
The command can be run once for each document, or just once in the background.
The command can be run once for each document, or just once in the background.
When doing so, the command is fed line by line and its output is read line by
When doing so, the command is fed line by line and its output is read line by
...
@@ -111,6 +132,7 @@ The second way to integrate a novel processing module to the repository is to
...
@@ -111,6 +132,7 @@ The second way to integrate a novel processing module to the repository is to
subclass the repository.AnnotationGenerator class in python. See the
subclass the repository.AnnotationGenerator class in python. See the
repo/test.py script for an example.
repo/test.py script for an example.
```
python
class
Annotator
(
repository
.
AnnotationGenerator
):
class
Annotator
(
repository
.
AnnotationGenerator
):
def
__init__
(
self
):
def
__init__
(
self
):
query
=
'
_MISSING_=MarkingFeature&_MAX_=20
'
query
=
'
_MISSING_=MarkingFeature&_MAX_=20
'
...
@@ -122,4 +144,4 @@ class Annotator(repository.AnnotationGenerator):
...
@@ -122,4 +144,4 @@ class Annotator(repository.AnnotationGenerator):
...
# generate annotation
...
# generate annotation
client
.
put_annotation_set
(
doc_id
,
'
AnnotatinoName
'
,
...)
client
.
put_annotation_set
(
doc_id
,
'
AnnotatinoName
'
,
...)
client
.
put_features
(
doc_id
,
{
'
MarkingFeature
'
:
True
})
client
.
put_features
(
doc_id
,
{
'
MarkingFeature
'
:
True
})
```
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment