18 Commits

Author	SHA1	Message	Date
落	0bc28d9424	added multi line comment flag for python	2013-04-19 15:33:02 +08:00
Ben Lavender	ec3434cf1d	Don't explode on invalid shebang	2012-11-18 20:56:06 -06:00
Joshua Peek	b183fcca05	Only read up to 100KB	2012-08-27 11:30:38 -05:00
Joshua Peek	220108857c	Skip emiting comment tokens	2012-08-20 10:34:07 -05:00
Joshua Peek	16a67cb852	Move shebang detection into classifier ... Fixes #203	2012-08-03 15:07:36 -05:00
Joshua Peek	e5d302459f	Fix tokenzing empty strings	2012-07-24 11:49:29 -05:00
Joshua Peek	53300ca581	Add brackets to tokens	2012-07-24 11:28:27 -05:00
Joshua Peek	f5705eaf38	Parse float tokens	2012-07-24 11:23:06 -05:00
Joshua Peek	2637d8dc55	Add tokenize helper to Tokenize class	2012-07-20 15:14:58 -05:00
Joshua Peek	175d4244c2	Extract single and multi line comment parser	2012-07-20 15:06:21 -05:00
Joshua Peek	d063089430	Add coq comments	2012-07-20 14:45:19 -05:00
Joshua Peek	497da86262	Strip tex and matlab leading inline comments	2012-06-21 10:38:28 -05:00
Joshua Peek	5cdd5e206c	Improve operator tokenizing	2012-06-20 17:16:53 -05:00
Joshua Peek	c353d3a050	Fix indent	2012-06-20 16:58:32 -05:00
Joshua Peek	f68e94f181	Skip number literals	2012-06-20 11:26:14 -05:00
Joshua Peek	555573071e	More tokenizer docs	2012-06-19 13:09:23 -05:00
Joshua Peek	6f6dd8bc38	Improve tokenizing sgml tags	2012-06-08 14:46:16 -05:00
Joshua Peek	e0cbe815a3	Add basic Tokenizer	2012-06-07 14:55:11 -05:00