Skip to content
Snippets Groups Projects
Commit b74a8e24 authored by Franck Dary's avatar Franck Dary
Browse files

Give more transitions to tokenizer

parent c371b423
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
Showing
with 5 additions and 0 deletions
UD_any/data/Makefile
+ 5
− 0
View file @ b74a8e24
...@@ -27,6 +27,11 @@ tokenizer.ts: all_no_test.conllu ...@@ -27,6 +27,11 @@ tokenizer.ts: all_no_test.conllu
echo "SPLIT 6" >> $@ echo "SPLIT 6" >> $@
echo "SPLIT 7" >> $@ echo "SPLIT 7" >> $@
echo "ADDCHARTOWORD 1" >> $@ echo "ADDCHARTOWORD 1" >> $@
echo "ADDCHARTOWORD 2" >> $@
echo "ADDCHARTOWORD 3" >> $@
echo "ADDCHARTOWORD 4" >> $@
echo "ADDCHARTOWORD 5" >> $@
echo "ADDCHARTOWORD 6" >> $@
echo "IGNORECHAR" >> $@ echo "IGNORECHAR" >> $@
sed -i -e 's/^/<tokenizer> /' $@ sed -i -e 's/^/<tokenizer> /' $@
sed -i -e 's/^/<tokenizer> /' splitwords.ts sed -i -e 's/^/<tokenizer> /' splitwords.ts
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment