Commit Graph

132 Commits

Author SHA1 Message Date
Ashe Connor 0698b0f36e Revert "Replace the tokenizer with a flex-based scanner (#3846)"
This reverts commit 99eaf5faf9.
2017-11-10 10:27:50 +11:00
Ashe Connor 99eaf5faf9 Replace the tokenizer with a flex-based scanner (#3846)
* Lex everything except SGML, multiline, SHEBANG

* Prepend SHEBANG#! to tokens

* Support SGML tag/attribute extraction

* Multiline comments

* WIP cont'd; productionifying

* Compile before test

* Add extension to gemspec

* Add flex task to build lexer

* Reentrant extra data storage

* regenerate lexer

* use prefix

* rebuild lexer on linux

* Optimise a number of operations:

* Don't read and split the entire file if we only ever use the first/last n
  lines

* Only consider the first 50KiB when using heuristics/classifying.  This can
  save a *lot* of time; running a large number of regexes over 1MiB of text
  takes a while.

* Memoize File.size/read/stat; re-reading in a 500KiB file every time `data` is
  called adds up a lot.

* Use single regex for C++

* act like #lines

* [1][-2..-1] => nil, ffs

* k may not be set
2017-10-31 11:06:56 +11:00
Colin Seymour 9d0af0da40 Update to charlock_holmes 0.7.5 (#3778)
This fixes https://github.com/github/linguist/issues/3777
2017-08-16 10:08:33 +01:00
Paul Chaignon 9468ad4947 Fix grammar hashes (#3504)
* Update Licensee hashes for grammar licenses

Licensee v8.8 changed the way licenses are normalized, thus changing hashes for
some grammars

* Update Licensee

Prevent automatic updates to major releases
2017-03-09 23:57:35 -08:00
Brandon Black f7fe1fee66 Release v5.0.5 -- part Deux (#3479)
* bumping to v5.0.5

* relaxing rugged version requirement
2017-02-15 21:29:04 -08:00
Brandon Black 4e2eba4ef8 Revert "Release v5.0.5" (#3477) 2017-02-15 12:48:45 -08:00
Colin Seymour 10457b6639 Release v5.0.5 (#3473)
Release v5.0.5

* Update submodules

* Update grammars

* Bump version to 5.0.5

* Relax dependency on rugged

It's probably not wise to depend on a beta version just yet.

* revert php.tmbundle grammar update

One of the changes in https://github.com/textmate/php.tmbundle/compare/3ed4837b43d3f650ebb525b068636281942883a0...010cc1c22c89c117ad4c985997668c3903dc37f0 leads to breakage in snippet highlighting on github.com
2017-02-15 11:12:53 +00:00
Brandon Black d761658f8b Release v5.0.4 (#3445)
* bumping to v5.0.3

* updating rugged
2017-01-31 15:08:52 -08:00
Brandon Black d7814c4899 fixing incompatibility with latest rugged 2017-01-08 22:59:00 -08:00
Brandon Black 2c78dd2c66 Bumping to v4.8.18 (#3370)
* make tests great again 

* version bump

* removing empty line in gemspec
2016-12-07 11:39:49 -08:00
Paul Chaignon ebe85788ab Rely solely on Licensee to recognize licenses
Remove our own license classification code
Add hashes for any project which does not have a standard license body
Add projects for which a license was not found to the whitelist

Requires Licensee v8.6.0 to correctly recognize TextMate bundles' .mdown README
2016-09-27 10:44:25 +02:00
Paul Chaignon 44e532c9a2 Update licensee to recognize zlib license
Linguist actually uses licensed, which in turn uses licensee
Licensee recognizes the zlib license starting with v8.3.0
zlib license added to choosealicense.com: github/choosealicense.com#438
2016-07-14 12:59:09 +02:00
Arfon Smith b5091e88ad Updating licensor -> licensed 2016-03-18 12:32:52 -06:00
Arfon Smith 3cfee4f214 Bumping to licensee v6.0.0 2016-03-17 20:30:22 -06:00
Arfon Smith 988739d566 Merge branch 'master' into combine-gems 2016-03-09 06:25:35 -06:00
Brandon Keepers 8b01e3dead Upgrade to licensee 6.0.0b1 2015-10-16 15:15:34 -04:00
Brandon Keepers a354eddf4b Update github-linguist.gemspec 2015-09-22 16:33:08 -04:00
Vicent Marti 13d1f662d1 Add the git-linguist helper 2015-09-14 08:42:51 -07:00
Ben Balter 8a7ceaa845 bump licensee to support ruby 1.9.3 2015-08-19 13:22:31 -04:00
Ben Balter fd9ce2d1cf use licensee to classify submodule licenses 2015-08-19 12:54:21 -04:00
Vicent Marti 9d61487403 gemspec: Loosen up the restriction on Rugged betas 2015-06-26 03:11:31 -07:00
Garen Torikian 004f9b4710 Use new color-proximity gem for better visual matching 2015-04-03 18:25:13 -07:00
Garen Torikian 3851b7c016 Add test for detecting color proximity 2015-04-01 10:14:52 -07:00
Patrick Toomey 738be195ef Bump escape-utils to 1.1.0 2015-03-25 14:18:46 -06:00
Andy Delcambre 7df872eb7f Bump rugged to latest release 2015-03-23 14:33:03 -06:00
Adam Roben 0cfdbfb91c Merge github-linguist-grammars into github-linguist
Now that all our grammars are licensed (or grandfathered in), we can
distribute them as part of the standard github-linguist gem. This makes
it easier for projects to get up and running with Linguist.
2015-01-07 14:47:26 -05:00
Adam Roben 65296e86a3 Switch to Minitest::Test instead of Test::Unit::TestCase
This gives us a consistent test framework across all Ruby versions which
should help avoid errors that are only found when CI runs the tests on
different Rubies. (And this fixes an immediate bug where there's no
`skip` method in the version of test-unit we're currently using only on
Ruby 2.2.)
2014-12-31 15:26:26 -05:00
Vicent Marti ecaa2a41c9 Bump Rugged again 2014-11-24 13:32:37 +01:00
Vicent Marti 74d94781cb Rugged 0.22.0b3 2014-11-24 13:05:42 +01:00
Adam Roben d699ba3a98 Allow mime-types 2.x to be used with Linguist
The API is compatible for our purposes, and this allows Linguist to be
used in apps that pull in newer versions of mime-types through other
gems.
2014-11-18 10:46:04 -05:00
Adam Roben e76ebb1a74 Upgrade to rugged 0.22.0b1
0.21.2 was just released but doesn't contain the Repository::Attributes
code we depend on. 0.22.0b1 has this code.
2014-11-18 10:40:37 -05:00
Vicent Marti 4a10b27611 Remove Pygments 2014-11-14 17:37:12 +01:00
Adam Roben 046fb18980 Add github-linguist-grammars gem
The purpose of this gem is to package up the language grammars that are
used for syntax highlighting on github.com. The grammars are TextMate,
Sublime Text, or Atom language grammars, converted to JSON and given the
filename SCOPE.json, where SCOPE is the language scope that the grammar
defines.

The github-linguist-grammars gem packages up all the grammars, and also
exports a Linguist::Grammars.path method to locate the directory
containing the grammars.

To build the gem, simply run `rake build_grammars_gem`. The grammars.yml
file lists all the repositories we download grammars from, as well as
which scopes are defined by each repository. The
script/download-grammars script takes that list and downloads and
processes the grammars into the format expected by the gem.
2014-11-13 11:03:53 -05:00
Brandon Keepers 02aeb4f895 Merge remote-tracking branch 'origin/master' into just-yajl
* origin/master: (42 commits)
  its always greener
  that new green shell
  Removing stale extension
  Update README.md
  Add moon interpreter for MoonScript
  Bumping version for 3.4.1 release
  Use text.html.erb scope for HTML+ERB files
  Add sample .dyalog file for file type APL
  Added extra Papyrus sample files.
  Add sample Papyrus script
  Add Papyrus support
  Add LOLCODE support
  Add ProGuard config files to vendored files
  Recognise *.dyalog as APL sources
  Assign a bunch more TextMate scopes
  CI step for samples
  Add .command as a Shell file extension
  CI config
  Vendored gems
  Update cibuild
  ...

Conflicts:
	Rakefile
2014-10-31 18:03:03 -04:00
Arfon Smith 89477ed2fa twiddle-wakka 2014-10-17 14:20:32 -05:00
Brandon Keepers cd743332f4 Use yajl since it is already a dependency
Both JSON and Yajl were listed as dependencies. Pygments.rb already requires yajl, so let's just use that instead of using both.
2014-10-17 14:45:28 -04:00
Arfon Smith 47843e7e78 Moving Rugged dependency back to gemspec 2014-10-17 13:36:13 -05:00
Arfon Smith 4cc679c1e5 Merge branch 'master' into 1515-local 2014-09-17 09:54:45 -05:00
The rugged tests are fragile d0cf883558 Use rugged 0.21.1b0 2014-09-11 13:51:06 +02:00
Brandon Keepers 6850499056 Remove git dependency 2014-09-10 15:49:59 -05:00
Arfon Smith 5235871fd8 Pry for development. 2014-07-23 10:26:21 -05:00
Arfon Smith 7fbb9edc0f Gem deps 2014-07-23 10:26:21 -05:00
Vicent Marti 324ac83489 Use the new Rugged release 2014-06-26 14:12:00 +02:00
Andy Lindeman 878fe95ec3 Upgrades to pygments.rb 0.6.0 2014-06-10 15:54:55 -04:00
Brian Lopez e17ebec098 Bump charlock to 0.7.3
This version includes a fix for the encoding lookup table for some
encoding aliases in the ICU detection API
2014-06-07 15:25:44 -05:00
Brian Lopez 7e8be1293e Use the :ruby_encoding value from charlock 0.7.2 2014-06-04 15:51:33 -05:00
Brian Lopez c1e560b901 use charlock 0.7.1 or better 2014-05-12 16:17:57 +02:00
Brian Lopez 92bc1cdcdf bump charlock_holmes to 0.7.0 2014-05-11 23:39:46 +02:00
Arfon Smith 7e81a9e50b Updating escape_utils 2014-05-02 13:43:24 -05:00
Arfon Smith fd7db27b48 Using Linguist::VERSION in gemspec 2014-04-24 14:09:10 -05:00