This project takes the file to be tokenized as a commandline argument
ex:
python tokenizer.py filetobetokenied.in
It writes the result to standard out.
An input file can be set in the makefile and executed with make.
To run the unit tests run make test