30 Commits

Author	SHA1	Message	Date
Arfon Smith	90a293727d	Merge branch 'master' into more-encompassing-number-skips	2015-07-29 13:54:51 +01:00
Stefan Johnson	885b5aab41	Changed tokenizer number literals to be more encompassing ... Number literals now skips hexadecimal, and C style literals.	2015-02-20 14:08:39 +11:00
Arfon Smith	69b68f3a44	Extracting common methods into helper.	2015-01-26 16:22:55 -06:00
Arfon Smith	8aa4dce6f4	Merge pull request #1723 from techhat/pythonmultiline ... Python also supports triple single-quotes for comments	2015-01-24 10:54:30 -06:00
Adam Roben	65296e86a3	Switch to Minitest::Test instead of Test::Unit::TestCase ... This gives us a consistent test framework across all Ruby versions which should help avoid errors that are only found when CI runs the tests on different Rubies. (And this fixes an immediate bug where there's no `skip` method in the version of test-unit we're currently using only on Ruby 2.2.)	2014-12-31 15:26:26 -05:00
Paul Chaignon	c5a654e692	Tests for Python multiline comments during tokenization	2014-11-25 20:01:24 -05:00
Brandon Keepers	aa7ab2065b	Add test helper to make test env consistent	2014-11-18 16:46:09 -05:00
Patrick Reynolds	2db2f5a46d	add erlang, more-complex shell examples ... - some Erlang and escript files - .escript extension - .erlang extension - shell script with %, ##, name tokens	2013-06-06 15:41:44 -05:00
Joshua Peek	dcc2be0781	Merge branch 'master' into dont-explode-on-invalid-shebang ... Conflicts: lib/linguist/samples.json test/test_tokenizer.rb	2012-12-10 09:02:24 -06:00
Dominik Tobschall	f84a904ad8	fixed typo	2012-12-02 14:11:04 +01:00
Dominik Tobschall	b1684037d6	added php cli samples	2012-12-02 14:05:52 +01:00
Ben Lavender	ec3434cf1d	Don't explode on invalid shebang	2012-11-18 20:56:06 -06:00
Joshua Peek	220108857c	Skip emiting comment tokens	2012-08-20 10:34:07 -05:00
Joshua Peek	16a67cb852	Move shebang detection into classifier ... Fixes #203	2012-08-03 15:07:36 -05:00
Joshua Peek	e5d302459f	Fix tokenzing empty strings	2012-07-24 11:49:29 -05:00
Joshua Peek	53300ca581	Add brackets to tokens	2012-07-24 11:28:27 -05:00
Joshua Peek	f5705eaf38	Parse float tokens	2012-07-24 11:23:06 -05:00
Joshua Peek	7b6caa0f6c	Rename samples subdirectories	2012-07-23 15:52:49 -05:00
Joshua Peek	36da52e383	Move specific filename samples into their own dir	2012-07-20 17:20:45 -05:00
Joshua Peek	2637d8dc55	Add tokenize helper to Tokenize class	2012-07-20 15:14:58 -05:00
Joshua Peek	175d4244c2	Extract single and multi line comment parser	2012-07-20 15:06:21 -05:00
Joshua Peek	d063089430	Add coq comments	2012-07-20 14:45:19 -05:00
Joshua Peek	5521dd08a0	Move test fixtures to samples/	2012-06-22 10:09:24 -05:00
Joshua Peek	497da86262	Strip tex and matlab leading inline comments	2012-06-21 10:38:28 -05:00
Joshua Peek	5cdd5e206c	Improve operator tokenizing	2012-06-20 17:16:53 -05:00
Joshua Peek	f68e94f181	Skip number literals	2012-06-20 11:26:14 -05:00
Joshua Peek	645a87d02b	Remove dead fixture test	2012-06-19 16:34:13 -05:00
Joshua Peek	d5fa8cbcb7	Refactor tokenizer test helper	2012-06-19 13:12:17 -05:00
Joshua Peek	6f6dd8bc38	Improve tokenizing sgml tags	2012-06-08 14:46:16 -05:00
Joshua Peek	e0cbe815a3	Add basic Tokenizer	2012-06-07 14:55:11 -05:00