modified maca_tokenizer so that it can output offset and token length

build/debug.sh

build/script.sh

maca_tokenizer/CMakeLists.txt

maca_tokenizer/src/context.c

maca_tokenizer/src/context.h

maca_tokenizer/src/fr_tok_rules.l

maca_tokenizer/src/maca_tokenizer.c

maca_tokenizer/src/maca_tokenizer_functions_for_lex.c

maca_tokenizer/src/maca_tokenizer_functions_for_lex.h