Skip to content
Snippets Groups Projects
Commit 36d2f53b authored by Alexis Nasr's avatar Alexis Nasr Committed by Silvio Ricardo Cordeiro
Browse files

added some rules in french tokenizer

parent a905ad9f
No related branches found
No related tags found
No related merge requests found
Show whitespace changes
Inline Side-by-side
maca_tokenizer/src/fr_tok_rules.l
+ 7
− 3
View file @ 36d2f53b
...@@ -15,9 +15,13 @@ extern int defait_amalgames; ...@@ -15,9 +15,13 @@ extern int defait_amalgames;
[0-9]+,[0-9]* printf("%s", yytext); [0-9]+,[0-9]* printf("%s", yytext);
[ \t]+ printf("\n"); [ \t]+ printf("\n");
\. printf("\n."); [ ]*\. printf("\n.");
\, printf("\n,"); [ ]*\? printf("\n?");
… printf("\n…"); [ ]*\! printf("\n!");
[ ]*, printf("\n,");
[ ]*: printf("\n:");
[ ]*; printf("\n;");
[ ]*… printf("\n…");
' printf("'\n"); ' printf("'\n");
’ printf("'\n"); ’ printf("'\n");
-je printf("\n-je"); -je printf("\n-je");
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment