19 Commits

Author	SHA1	Message	Date
Ben Lavender	ec3434cf1d	Don't explode on invalid shebang	2012-11-18 20:56:06 -06:00
Joshua Peek	220108857c	Skip emiting comment tokens	2012-08-20 10:34:07 -05:00
Joshua Peek	16a67cb852	Move shebang detection into classifier ... Fixes #203	2012-08-03 15:07:36 -05:00
Joshua Peek	e5d302459f	Fix tokenzing empty strings	2012-07-24 11:49:29 -05:00
Joshua Peek	53300ca581	Add brackets to tokens	2012-07-24 11:28:27 -05:00
Joshua Peek	f5705eaf38	Parse float tokens	2012-07-24 11:23:06 -05:00
Joshua Peek	7b6caa0f6c	Rename samples subdirectories	2012-07-23 15:52:49 -05:00
Joshua Peek	36da52e383	Move specific filename samples into their own dir	2012-07-20 17:20:45 -05:00
Joshua Peek	2637d8dc55	Add tokenize helper to Tokenize class	2012-07-20 15:14:58 -05:00
Joshua Peek	175d4244c2	Extract single and multi line comment parser	2012-07-20 15:06:21 -05:00
Joshua Peek	d063089430	Add coq comments	2012-07-20 14:45:19 -05:00
Joshua Peek	5521dd08a0	Move test fixtures to samples/	2012-06-22 10:09:24 -05:00
Joshua Peek	497da86262	Strip tex and matlab leading inline comments	2012-06-21 10:38:28 -05:00
Joshua Peek	5cdd5e206c	Improve operator tokenizing	2012-06-20 17:16:53 -05:00
Joshua Peek	f68e94f181	Skip number literals	2012-06-20 11:26:14 -05:00
Joshua Peek	645a87d02b	Remove dead fixture test	2012-06-19 16:34:13 -05:00
Joshua Peek	d5fa8cbcb7	Refactor tokenizer test helper	2012-06-19 13:12:17 -05:00
Joshua Peek	6f6dd8bc38	Improve tokenizing sgml tags	2012-06-08 14:46:16 -05:00
Joshua Peek	e0cbe815a3	Add basic Tokenizer	2012-06-07 14:55:11 -05:00