Ashe Connor 99eaf5faf9 Replace the tokenizer with a flex-based scanner (#3846) ...

* Lex everything except SGML, multiline, SHEBANG

* Prepend SHEBANG#! to tokens

* Support SGML tag/attribute extraction

* Multiline comments

* WIP cont'd; productionifying

* Compile before test

* Add extension to gemspec

* Add flex task to build lexer

* Reentrant extra data storage

* regenerate lexer

* use prefix

* rebuild lexer on linux

* Optimise a number of operations:

* Don't read and split the entire file if we only ever use the first/last n
  lines

* Only consider the first 50KiB when using heuristics/classifying.  This can
  save a *lot* of time; running a large number of regexes over 1MiB of text
  takes a while.

* Memoize File.size/read/stat; re-reading in a 500KiB file every time `data` is
  called adds up a lot.

* Use single regex for C++

* act like #lines

* [1][-2..-1] => nil, ffs

* k may not be set

2017-10-31 11:06:56 +11:00

..

linguist

Replace the tokenizer with a flex-based scanner (#3846)

2017-10-31 11:06:56 +11:00

linguist.rb

Update the Instrumenter doc ... (#3530)

2017-03-23 06:11:45 +01:00