Merge pull request #1757 from github/cut-release-v4.0.3

Bump version to 4.0.3

.gitignore

@@ -1,4 +1,6 @@
-Gemfile.lock
+/Gemfile.lock
 .bundle/
 benchmark/
 lib/linguist/samples.json
+/grammars
+/node_modules

CONTRIBUTING.md

@@ -0,0 +1,31 @@
+## Contributing
+
+The majority of contributions won't need to touch any Ruby code at all. The [master language list][languages] is just a YAML configuration file.
+
+Almost all bug fixes or new language additions should come with some additional code samples. Just drop them under [`samples/`][samples] in the correct subdirectory and our test suite will automatically test them. In most cases you shouldn't need to add any new assertions.
+
+### My code is detected as the wrong language
+
+This can usually be solved either by adding a new filename or file name extension to the language's entry in [`languages.yml`][languages] or adding more [samples][samples] for your language to the repository to make Linguist's classifier smarter.
+
+### Syntax highlighting looks wrong
+
+Assuming your code is being detected as the right language (see above), in most cases this is due to a bug in the language grammar rather than a bug in Linguist. [`grammars.yml`][grammars] lists all the grammars we use for syntax highlighting on github.com. Find the one corresponding to your code's programming language and submit a bug report upstream.
+
+You can also try to fix the bug yourself and submit a Pull Request. [This piece from TextMate's documentation](http://manual.macromates.com/en/language_grammars) offers a good introduction on how to work with TextMate-compatible grammars.
+
+Once the bug has been fixed upstream, please let us know and we'll pick it up for GitHub.
+
+### I want to add support for the `X` programming language
+
+Great! You'll need to:
+
+0. Add an entry for your language to [`languages.yml`][languages].
+0. Add a grammar for your language to [`grammars.yml`][grammars] by running `script/download-grammars --add URL`. Please only add grammars that have a license that permits redistribution.
+0. Add samples for your language to the [samples directory][samples].
+
+We try only to add languages once they have some usage on GitHub, so please note in-the-wild usage examples in your pull request. In most cases we prefer that languages already be in use in hundreds of repositories before supporting them in Linguist.
+
+[grammars]: /grammars.yml
+[languages]: /lib/linguist/languages.yml
+[samples]: /samples

Gemfile

@@ -1,3 +1,4 @@
 source 'https://rubygems.org'
-gemspec
+gemspec :name => "github-linguist"
+gemspec :name => "github-linguist-grammars"
 gem 'test-unit', require: false if RUBY_VERSION >= '2.2'

README.md

@@ -1,12 +1,14 @@
 # Linguist
 
-We use this library at GitHub to detect blob languages, highlight code, ignore binary files, suppress generated files in diffs, and generate language breakdown graphs.
+We use this library at GitHub to detect blob languages, ignore binary files, suppress generated files in diffs, and generate language breakdown graphs.
+
+Tips for filing issues and creating pull requests can be found in [`CONTRIBUTING.md`](/CONTRIBUTING.md).
 
 ## Features
 
 ### Language detection
 
-Linguist defines a list of all languages known to GitHub in a [yaml file](https://github.com/github/linguist/blob/master/lib/linguist/languages.yml). In order for a file to be highlighted, a language and a lexer must be defined there.
+Linguist defines a list of all languages known to GitHub in a [yaml file](https://github.com/github/linguist/blob/master/lib/linguist/languages.yml).
 
 Most languages are detected by their file extension. For disambiguating between files with common extensions, we first apply some common-sense heuristics to pick out obvious languages. After that, we use a
 [statistical
@@ -24,7 +26,9 @@ See [lib/linguist/language.rb](https://github.com/github/linguist/blob/master/li
 
 ### Syntax Highlighting
 
-The actual syntax highlighting is handled by our Pygments wrapper, [pygments.rb](https://github.com/tmm1/pygments.rb). It also provides a [Lexer abstraction](https://github.com/tmm1/pygments.rb/blob/master/lib/pygments/lexer.rb) that determines which highlighter should be used on a file.
+Syntax highlighting in GitHub is performed using TextMate-compatible grammars. These are the same grammars that TextMate, Sublime Text and Atom use.
+
+Every language in `languages.yml` is mapped to its corresponding TM `scope`. This scope will be used when picking up a grammar for highlighting. **When adding a new language to Linguist, please add its corresponding scope too (assuming there's an existing TextMate bundle, Sublime Text package, or Atom package) so syntax highlighting works for it**.
 
 ### Stats
 
@@ -143,14 +147,6 @@ To run the tests:
 
     bundle exec rake test
 
-## Contributing
-
-The majority of contributions won't need to touch any Ruby code at all. The [master language list](https://github.com/github/linguist/blob/master/lib/linguist/languages.yml) is just a YAML configuration file.
-
-We try to only add languages once they have some usage on GitHub, so please note in-the-wild usage examples in your pull request.
-
-Almost all bug fixes or new language additions should come with some additional code samples. Just drop them under [`samples/`](https://github.com/github/linguist/tree/master/samples) in the correct subdirectory and our test suite will automatically test them. In most cases you shouldn't need to add any new assertions.
-
 ### A note on language extensions
 
 Linguist has a number of methods available to it for identifying the language of a particular file. The initial lookup is based upon the extension of the file, possible file extensions are defined in an array called `extensions`. Take a look at this example for example for `Perl`:

Rakefile

@@ -31,6 +31,12 @@ task :build_gem => :samples do
   File.delete("lib/linguist/languages.json")
 end
 
+task :build_grammars_gem do
+  rm_rf "grammars"
+  sh "script/download-grammars"
+  sh "gem", "build", "github-linguist-grammars.gemspec"
+end
+
 namespace :benchmark do
   benchmark_path = "benchmark/results"
 

github-linguist-grammars.gemspec

@@ -0,0 +1,14 @@
+require File.expand_path('../lib/linguist/version', __FILE__)
+
+Gem::Specification.new do |s|
+  s.name    = 'github-linguist-grammars'
+  s.version = Linguist::VERSION
+  s.summary = "Language grammars for use with github-linguist"
+
+  s.authors  = "GitHub"
+  s.homepage = "https://github.com/github/linguist"
+
+  s.files = ['lib/linguist/grammars.rb'] + Dir['grammars/*']
+
+  s.add_development_dependency 'plist', '~>3.1'
+end

github-linguist.gemspec

@@ -10,14 +10,13 @@ Gem::Specification.new do |s|
   s.homepage = "https://github.com/github/linguist"
   s.license  = "MIT"
 
-  s.files = Dir['lib/**/*']
+  s.files = Dir['lib/**/*'] - ['lib/linguist/grammars.rb']
   s.executables << 'linguist'
 
   s.add_dependency 'charlock_holmes', '~> 0.7.3'
   s.add_dependency 'escape_utils',    '~> 1.0.1'
-  s.add_dependency 'mime-types',      '~> 1.19'
+  s.add_dependency 'mime-types',      '>= 1.19'
-  s.add_dependency 'pygments.rb',     '~> 0.6.0'
+  s.add_dependency 'rugged',          '~> 0.22.0b1'
-  s.add_dependency 'rugged',          '~> 0.21.1b2'
 
   s.add_development_dependency 'mocha'
   s.add_development_dependency 'pry'

grammars.yml

@@ -0,0 +1,410 @@
+---
+http://svn.edgewall.org/repos/genshi/contrib/textmate/Genshi.tmbundle/Syntaxes/Markup%20Template%20%28XML%29.tmLanguage:
+- text.xml.genshi
+http://svn.textmate.org/trunk/Review/Bundles/BlitzMax.tmbundle:
+- source.blitzmax
+http://svn.textmate.org/trunk/Review/Bundles/Cython.tmbundle:
+- source.cython
+http://svn.textmate.org/trunk/Review/Bundles/F%20Sharp.tmbundle:
+- source.fsharp
+http://svn.textmate.org/trunk/Review/Bundles/Forth.tmbundle:
+- source.forth
+http://svn.textmate.org/trunk/Review/Bundles/Parrot.tmbundle:
+- source.parrot.pir
+http://svn.textmate.org/trunk/Review/Bundles/Ruby%20Sass.tmbundle:
+- source.sass
+http://svn.textmate.org/trunk/Review/Bundles/SecondLife%20LSL.tmbundle:
+- source.lsl
+http://svn.textmate.org/trunk/Review/Bundles/VHDL.tmbundle:
+- source.vhdl
+http://svn.textmate.org/trunk/Review/Bundles/XQuery.tmbundle:
+- source.xquery
+https://bitbucket.org/Clams/sublimesystemverilog/get/default.tar.gz:
+- source.systemverilog
+- source.ucfconstraints
+https://bitbucket.org/bitlang/sublime_cobol/raw/b0e9c44ac5f7a2fb553421aa986b35854cbfda4a/COBOL.tmLanguage:
+- source.cobol
+https://fan.googlecode.com/hg-history/Build%201.0.55/adm/tools/textmate/Fan.tmbundle/Syntaxes/Fan.tmLanguage:
+- source.fan
+https://github.com/AlanQuatermain/go-tmbundle:
+- source.go
+https://github.com/Anomareh/PHP-Twig.tmbundle:
+- text.html.twig
+https://github.com/Cirru/sublime-cirru/raw/master/Cirru.tmLanguage:
+- source.cirru
+https://github.com/Cykey/Sublime-Logos:
+- source.logos
+https://github.com/Drako/SublimeBrainfuck/raw/master/Brainfuck.tmLanguage:
+- source.bf
+https://github.com/JohnNilsson/awk-sublime/raw/master/AWK.tmLanguage:
+- source.awk
+https://github.com/JonBons/Sublime-SQF-Language:
+- source.sqf
+https://github.com/MarioRicalde/SCSS.tmbundle:
+- source.scss
+https://github.com/Oldes/Sublime-REBOL:
+- source.rebol
+https://github.com/PogiNate/Sublime-Inform:
+- source.Inform7
+https://github.com/Red-Nova-Technologies/autoitv3-tmbundle:
+- source.autoit.3
+https://github.com/SalGnt/Sublime-VimL:
+- source.viml
+https://github.com/Shammah/boo-sublime/raw/master/Boo.tmLanguage:
+- source.boo
+https://github.com/SublimeText/ColdFusion:
+- source.cfscript
+- source.cfscript.cfc
+- text.cfml.basic
+- text.html.cfm
+https://github.com/SublimeText/NSIS:
+- source.nsis
+https://github.com/Varriount/NimLime:
+- source.nimrod
+- source.nimrod_filter
+- source.nimrodcfg
+https://github.com/angryant0007/VBDotNetSyntax:
+- source.vbnet
+https://github.com/aroben/ada.tmbundle/raw/c45eed4d5f98fe3bcbbffbb9e436601ab5bbde4b/Syntaxes/Ada.plist:
+- source.ada
+https://github.com/aroben/ruby.tmbundle@4636a3023153c3034eb6ffc613899ba9cf33b41f:
+- source.ruby
+- text.html.erb
+https://github.com/asbjornenge/Docker.tmbundle:
+- source.dockerfile
+https://github.com/atom/language-clojure:
+- source.clojure
+https://github.com/atom/language-coffee-script:
+- source.coffee
+- source.litcoffee
+https://github.com/atom/language-csharp:
+- source.cs
+- source.csx
+- source.nant-build
+https://github.com/atom/language-javascript:
+- source.js
+- source.js.regexp
+https://github.com/atom/language-python:
+- source.python
+- source.regexp.python
+- text.python.traceback
+https://github.com/atom/language-shellscript:
+- source.shell
+- text.shell-session
+https://github.com/austinwagner/sublime-sourcepawn:
+- source.sp
+https://github.com/bfad/Sublime-Lasso:
+- file.lasso
+https://github.com/bholt/chapel-tmbundle:
+- source.chapel
+https://github.com/brandonwamboldt/sublime-nginx:
+- source.nginx
+https://github.com/bro/bro-sublime:
+- source.bro
+https://github.com/carsonoid/sublime_man_page_support/raw/master/man-groff.tmLanguage:
+- text.groff
+https://github.com/ccreutzig/sublime-MuPAD:
+- source.mupad
+https://github.com/cdwilson/nesC.tmbundle:
+- source.nesc
+https://github.com/christophevg/racket-tmbundle:
+- source.racket
+https://github.com/clemos/haxe-sublime-bundle:
+- source.erazor
+- source.haxe.2
+- source.hss.1
+- source.hxml
+- source.nmml
+https://github.com/cucumber/cucumber-tmbundle:
+- source.ruby.rspec.cucumber.steps
+- text.gherkin.feature
+https://github.com/daaain/Handlebars/raw/master/Handlebars.tmLanguage:
+- text.html.handlebars
+https://github.com/davidpeckham/powershell.tmbundle:
+- source.powershell
+https://github.com/davidrios/jade-tmbundle:
+- source.jade
+- source.pyjade
+https://github.com/elixir-lang/elixir-tmbundle:
+- source.elixir
+- text.elixir
+- text.html.elixir
+https://github.com/ericzou/ebundles/raw/master/Bundles/MSDOS%20batch%20file.tmbundle/Syntaxes/MSDOS%20batch%20file.tmLanguage:
+- source.dosbatch
+https://github.com/euler0/sublime-glsl/raw/master/GLSL.tmLanguage:
+- source.glsl
+https://github.com/fancy-lang/fancy-tmbundle:
+- source.fancy
+https://github.com/gingerbeardman/monkey.tmbundle:
+- source.monkey
+https://github.com/guillermooo/dart-sublime-bundle/raw/master/Dart.tmLanguage:
+- source.dart
+https://github.com/harrism/sublimetext-cuda-cpp/raw/master/cuda-c%2B%2B.tmLanguage:
+- source.cuda-c++
+https://github.com/hww3/pike-textmate:
+- source.pike
+https://github.com/jeancharles-roger/ceylon-sublimetext/raw/master/Ceylon.tmLanguage:
+- source.ceylon
+https://github.com/jfairbank/Sublime-Text-2-OpenEdge-ABL:
+- source.abl
+https://github.com/jhasse/sublime-rust:
+- source.rust
+https://github.com/johanasplund/sublime-befunge/raw/master/Befunge-93.tmLanguage:
+- source.befunge
+https://github.com/joshaven/RDoc.tmbundle:
+- text.rdoc
+https://github.com/jpcamara/Textmate-Gosu-Bundle/raw/master/Gosu.tmbundle/Syntaxes/Gosu.tmLanguage:
+- source.gosu.2
+https://github.com/kswedberg/jquery-tmbundle:
+- source.js.jquery
+https://github.com/laughedelic/sublime-idris/raw/master/Idris.tmLanguage:
+- source.idris
+https://github.com/lavrton/sublime-better-typescript:
+- source.ts
+https://github.com/leafo/moonscript-tmbundle:
+- source.moonscript
+https://github.com/lunixbochs/x86-assembly-textmate-bundle:
+- source.asm.x86
+https://github.com/macekond/Alloy.tmbundle:
+- source.alloy
+https://github.com/mads379/opa.tmbundle:
+- source.opa
+https://github.com/mads379/scala.tmbundle:
+- source.sbt
+- source.scala
+https://github.com/marconi/mako-tmbundle:
+- text.html.mako
+https://github.com/mattfoster/gnuplot-tmbundle:
+- source.gnuplot
+https://github.com/mgalloy/idl.tmbundle:
+- source.idl
+- source.idl-dlm
+- text.idl-idldoc
+https://github.com/michaeledgar/protobuf-tmbundle:
+- source.protobuf
+https://github.com/mkolosick/Sublime-Coq/raw/master/Coq.tmLanguage:
+- source.coq
+https://github.com/mokus0/Agda.tmbundle:
+- source.agda
+https://github.com/nanoant/Julia.tmbundle:
+- source.julia
+https://github.com/nanoant/assembly.tmbundle/raw/master/Syntaxes/objdump%20C%2B%2B.tmLanguage:
+- objdump.x86asm
+https://github.com/nilium/ooc.tmbundle:
+- source.ooc
+https://github.com/paulmillr/LiveScript.tmbundle:
+- source.livescript
+https://github.com/pferruggiaro/sublime-tea:
+- source.tea
+https://github.com/puppet-textmate-bundle/puppet-textmate-bundle:
+- source.puppet
+https://github.com/pvl/abap.tmbundle:
+- source.abap
+https://github.com/scalate/Scalate.tmbundle:
+- source.scaml
+- text.html.ssp
+https://github.com/shadanan/mathematica-tmbundle:
+- source.mathematica
+https://github.com/shellderp/sublime-robot-plugin:
+- text.robot
+https://github.com/simongregory/actionscript3-tmbundle:
+- source.actionscript.3
+- text.html.asdoc
+- text.xml.flex-config
+https://github.com/skozlovf/Sublime-QML:
+- source.qml
+https://github.com/slash-lang/Slash.tmbundle:
+- text.html.slash
+https://github.com/slavapestov/factor/raw/master/misc/Factor.tmbundle/Syntaxes/Factor.tmLanguage:
+- source.factor
+https://github.com/slim-template/ruby-slim.tmbundle:
+- text.slim
+https://github.com/staltz/SublimeXtend:
+- source.xtend
+https://github.com/statatmbundle/Stata.tmbundle:
+- source.mata
+- source.stata
+https://github.com/technosophos/Vala-TMBundle:
+- source.vala
+https://github.com/textmate/ant.tmbundle:
+- text.xml.ant
+https://github.com/textmate/antlr.tmbundle:
+- source.antlr
+https://github.com/textmate/apache.tmbundle:
+- source.apache-config
+- source.apache-config.mod_perl
+https://github.com/textmate/applescript.tmbundle:
+- source.applescript
+https://github.com/textmate/asp.tmbundle:
+- source.asp
+- text.html.asp
+https://github.com/textmate/bison.tmbundle:
+- source.bison
+https://github.com/textmate/c.tmbundle:
+- source.c
+- source.c++
+- source.c.platform
+https://github.com/textmate/capnproto.tmbundle:
+- source.capnp
+https://github.com/textmate/cmake.tmbundle:
+- source.cache.cmake
+- source.cmake
+https://github.com/textmate/cpp-qt.tmbundle:
+- source.c++.qt
+- source.qmake
+https://github.com/textmate/css.tmbundle:
+- source.css
+https://github.com/textmate/d.tmbundle:
+- source.d
+https://github.com/textmate/diff.tmbundle:
+- source.diff
+https://github.com/textmate/dylan.tmbundle:
+- source.dylan
+- source.lid
+- source.makegen
+https://github.com/textmate/eiffel.tmbundle:
+- source.eiffel
+https://github.com/textmate/erlang.tmbundle:
+- source.erlang
+- text.html.erlang.yaws
+https://github.com/textmate/fortran.tmbundle:
+- source.fortran
+- source.fortran.modern
+https://github.com/textmate/gettext.tmbundle:
+- source.po
+https://github.com/textmate/graphviz.tmbundle:
+- source.dot
+https://github.com/textmate/groovy.tmbundle:
+- source.groovy
+https://github.com/textmate/haskell.tmbundle:
+- source.haskell
+- text.tex.latex.haskell
+https://github.com/textmate/html.tmbundle:
+- text.html.basic
+https://github.com/textmate/ini.tmbundle:
+- source.ini
+https://github.com/textmate/io.tmbundle:
+- source.io
+https://github.com/textmate/java.tmbundle:
+- source.java
+- source.java-properties
+- text.html.jsp
+- text.junit-test-report
+https://github.com/textmate/javadoc.tmbundle:
+- text.html.javadoc
+https://github.com/textmate/javascript-objective-j.tmbundle:
+- source.js.objj
+https://github.com/textmate/json.tmbundle:
+- source.json
+https://github.com/textmate/latex.tmbundle:
+- text.bibtex
+- text.log.latex
+- text.tex
+- text.tex.latex
+- text.tex.latex.beamer
+- text.tex.latex.memoir
+https://github.com/textmate/less.tmbundle:
+- source.css.less
+https://github.com/textmate/lilypond.tmbundle:
+- source.lilypond
+https://github.com/textmate/lisp.tmbundle:
+- source.lisp
+https://github.com/textmate/logtalk.tmbundle:
+- source.logtalk
+https://github.com/textmate/lua.tmbundle:
+- source.lua
+https://github.com/textmate/make.tmbundle:
+- source.makefile
+https://github.com/textmate/markdown.tmbundle:
+- text.html.markdown
+https://github.com/textmate/matlab.tmbundle:
+- source.matlab
+- source.octave
+https://github.com/textmate/maven.tmbundle:
+- text.xml.pom
+https://github.com/textmate/nemerle.tmbundle:
+- source.nemerle
+https://github.com/textmate/ninja.tmbundle:
+- source.ninja
+https://github.com/textmate/objective-c.tmbundle:
+- source.objc
+- source.objc++
+- source.objc.platform
+- source.strings
+https://github.com/textmate/ocaml.tmbundle:
+- source.camlp4.ocaml
+- source.ocaml
+- source.ocamllex
+- source.ocamlyacc
+https://github.com/textmate/pascal.tmbundle:
+- source.pascal
+https://github.com/textmate/perl.tmbundle:
+- source.perl
+https://github.com/textmate/php-smarty.tmbundle:
+- source.smarty
+https://github.com/textmate/php.tmbundle:
+- text.html.php
+https://github.com/textmate/postscript.tmbundle:
+- source.postscript
+https://github.com/textmate/processing.tmbundle:
+- source.processing
+https://github.com/textmate/prolog.tmbundle:
+- source.prolog
+https://github.com/textmate/python-django.tmbundle:
+- source.python.django
+- text.html.django
+https://github.com/textmate/r.tmbundle:
+- source.r
+- text.tex.latex.rd
+https://github.com/textmate/restructuredtext.tmbundle:
+- text.restructuredtext
+https://github.com/textmate/ruby-haml.tmbundle:
+- text.haml
+https://github.com/textmate/ruby-on-rails-tmbundle:
+- source.js.erb.rails
+- source.ruby.rails
+- source.ruby.rails.rjs
+- source.sql.ruby
+- text.html.erb.rails
+https://github.com/textmate/scheme.tmbundle:
+- source.scheme
+https://github.com/textmate/scilab.tmbundle:
+- source.scilab
+https://github.com/textmate/sql.tmbundle:
+- source.sql
+https://github.com/textmate/standard-ml.tmbundle:
+- source.cm
+- source.ml
+https://github.com/textmate/swift.tmbundle:
+- source.swift
+https://github.com/textmate/tcl.tmbundle:
+- source.tcl
+- text.html.tcl
+https://github.com/textmate/text.tmbundle:
+- text.plain
+https://github.com/textmate/textile.tmbundle:
+- text.html.textile
+https://github.com/textmate/textmate.tmbundle:
+- source.regexp.oniguruma
+- source.tm-properties
+https://github.com/textmate/thrift.tmbundle:
+- source.thrift
+https://github.com/textmate/toml.tmbundle:
+- source.toml
+https://github.com/textmate/verilog.tmbundle:
+- source.verilog
+https://github.com/textmate/xml.tmbundle:
+- text.xml
+- text.xml.xsl
+https://github.com/textmate/yaml.tmbundle:
+- source.yaml
+https://github.com/tomas-stefano/smalltalk-tmbundle:
+- source.smalltalk
+https://github.com/vic/ioke-outdated/raw/master/share/TextMate/Ioke.tmbundle/Syntaxes/Ioke.tmLanguage:
+- source.ioke
+https://github.com/vkostyukov/kotlin-sublime-package:
+- source.Kotlin
+https://github.com/vmg/zephir-sublime:
+- source.php.zephir
+https://github.com/whitequark/llvm.tmbundle:
+- source.llvm

lib/linguist/blob_helper.rb

@@ -2,7 +2,6 @@ require 'linguist/generated'
 require 'charlock_holmes'
 require 'escape_utils'
 require 'mime/types'
-require 'pygments'
 require 'yaml'
 
 module Linguist
@@ -147,6 +146,13 @@ module Linguist
       end
     end
 
+    # Public: Is the blob empty?
+    #
+    # Return true or false
+    def empty?
+      data.nil? || data == ""
+    end
+
     # Public: Is the blob text?
     #
     # Return true or false
@@ -193,10 +199,6 @@ module Linguist
 
     # Public: Is the blob safe to colorize?
     #
-    # We use Pygments for syntax highlighting blobs. Pygments
-    # can be too slow for very large blobs or for certain
-    # corner-case blobs.
-    #
     # Return true or false
     def safe_to_colorize?
       !large? && text? && !high_ratio_of_long_lines?
@@ -204,9 +206,6 @@ module Linguist
 
     # Internal: Does the blob have a ratio of long lines?
     #
-    # These types of files are usually going to make Pygments.rb
-    # angry if we try to colorize them.
-    #
     # Return true or false
     def high_ratio_of_long_lines?
       return false if loc == 0
@@ -314,28 +313,9 @@ module Linguist
       @language ||= Language.detect(self)
     end
 
-    # Internal: Get the lexer of the blob.
-    #
-    # Returns a Lexer.
-    def lexer
-      language ? language.lexer : Pygments::Lexer.find_by_name('Text only')
-    end
-
     # Internal: Get the TextMate compatible scope for the blob
     def tm_scope
       language && language.tm_scope
     end
-
-    # Public: Highlight syntax of blob
-    #
-    # options - A Hash of options (defaults to {})
-    #
-    # Returns html String
-    def colorize(options = {})
-      return unless safe_to_colorize?
-      options[:options] ||= {}
-      options[:options][:encoding] ||= encoding
-      lexer.highlight(data, options)
-    end
   end
 end

lib/linguist/generated.rb

@@ -51,26 +51,25 @@ module Linguist
     #
     # Return true or false
     def generated?
-      name == 'Gemfile.lock' ||
+      minified_files? ||
-        minified_files? ||
+      compiled_coffeescript? ||
-        compiled_coffeescript? ||
+      xcode_file? ||
-        xcode_file? ||
+      generated_parser? ||
-        generated_parser? ||
+      generated_net_docfile? ||
-        generated_net_docfile? ||
+      generated_net_designer_file? ||
-        generated_net_designer_file? ||
+      generated_postscript? ||
-        generated_postscript? ||
+      generated_protocol_buffer? ||
-        generated_protocol_buffer? ||
+      generated_jni_header? ||
-        generated_jni_header? ||
+      composer_lock? ||
-        composer_lock? ||
+      node_modules? ||
-        node_modules? ||
+      godeps? ||
-        godeps? ||
+      vcr_cassette? ||
-        vcr_cassette? ||
+      generated_by_zephir?
-        generated_by_zephir?
     end
 
     # Internal: Is the blob an Xcode file?
     #
-    # Generated if the file extension is an Xcode 
+    # Generated if the file extension is an Xcode
     # file extension.
     #
     # Returns true of false.
@@ -265,4 +264,3 @@ module Linguist
     end
   end
 end
-

lib/linguist/grammars.rb

@@ -0,0 +1,13 @@
+# Note: This file is included in the github-linguist-grammars gem, not the
+# github-linguist gem.
+
+module Linguist
+  module Grammars
+    # Get the path to the directory containing the language grammar JSON files.
+    #
+    # Returns a String.
+    def self.path
+      File.expand_path("../../../grammars", __FILE__)
+    end
+  end
+end

lib/linguist/heuristics.rb

@@ -13,26 +13,31 @@ module Linguist
     # Returns an array of Languages or []
     def self.find_by_heuristics(data, languages)
       if active?
+        result = []
+
         if languages.all? { |l| ["Perl", "Prolog"].include?(l) }
-          result = disambiguate_pl(data, languages)
+          result = disambiguate_pl(data)
         end
         if languages.all? { |l| ["ECL", "Prolog"].include?(l) }
-          result = disambiguate_ecl(data, languages)
+          result = disambiguate_ecl(data)
         end
         if languages.all? { |l| ["IDL", "Prolog"].include?(l) }
-          result = disambiguate_pro(data, languages)
+          result = disambiguate_pro(data)
         end
         if languages.all? { |l| ["Common Lisp", "OpenCL"].include?(l) }
-          result = disambiguate_cl(data, languages)
+          result = disambiguate_cl(data)
         end
         if languages.all? { |l| ["Hack", "PHP"].include?(l) }
-          result = disambiguate_hack(data, languages)
+          result = disambiguate_hack(data)
         end
         if languages.all? { |l| ["Scala", "SuperCollider"].include?(l) }
-          result = disambiguate_sc(data, languages)
+          result = disambiguate_sc(data)
         end
         if languages.all? { |l| ["AsciiDoc", "AGS Script"].include?(l) }
-          result = disambiguate_asc(data, languages)
+          result = disambiguate_asc(data)
+        end
+        if languages.all? { |l| ["FORTRAN", "Forth"].include?(l) }
+          result = disambiguate_f(data)
         end
         return result
       end
@@ -42,28 +47,37 @@ module Linguist
     # We want to shortcut look for Objective-C _and_ now C++ too!
     #
     # Returns an array of Languages or []
-    def self.disambiguate_c(data, languages)
+    def self.disambiguate_c(data)
       matches = []
-      matches << Language["Objective-C"] if data.include?("@interface")
+      if data.include?("@interface")
-      matches << Language["C++"] if data.include?("#include <cstdint>")
+        matches << Language["Objective-C"]
+      elsif data.include?("#include <cstdint>")
+        matches << Language["C++"]
+      end
       matches
     end
 
-    def self.disambiguate_pl(data, languages)
+    def self.disambiguate_pl(data)
       matches = []
-      matches << Language["Prolog"] if data.include?(":-")
+      if data.include?("use strict")
-      matches << Language["Perl"] if data.include?("use strict")
+        matches << Language["Perl"]
+      elsif data.include?(":-")
+        matches << Language["Prolog"]
+      end
       matches
     end
 
-    def self.disambiguate_ecl(data, languages)
+    def self.disambiguate_ecl(data)
       matches = []
-      matches << Language["Prolog"] if data.include?(":-")
+      if data.include?(":-")
-      matches << Language["ECL"] if data.include?(":=")
+        matches << Language["Prolog"]
+      elsif data.include?(":=")
+        matches << Language["ECL"]
+      end
       matches
     end
 
-    def self.disambiguate_pro(data, languages)
+    def self.disambiguate_pro(data)
       matches = []
       if (data.include?(":-"))
         matches << Language["Prolog"]
@@ -73,7 +87,7 @@ module Linguist
       matches
     end
 
-    def self.disambiguate_ts(data, languages)
+    def self.disambiguate_ts(data)
       matches = []
       if (data.include?("</translation>"))
         matches << Language["XML"]
@@ -83,21 +97,24 @@ module Linguist
       matches
     end
 
-    def self.disambiguate_cl(data, languages)
+    def self.disambiguate_cl(data)
       matches = []
-      matches << Language["Common Lisp"] if data.include?("(defun ")
+      if data.include?("(defun ")
-      matches << Language["OpenCL"] if /\/\* |\/\/ |^\}/.match(data)
+        matches << Language["Common Lisp"]
+      elsif /\/\* |\/\/ |^\}/.match(data)
+        matches << Language["OpenCL"]
+      end
       matches
     end
 
-    def self.disambiguate_r(data, languages)
+    def self.disambiguate_r(data)
       matches = []
       matches << Language["Rebol"] if /\bRebol\b/i.match(data)
       matches << Language["R"] if data.include?("<-")
       matches
     end
 
-    def self.disambiguate_hack(data, languages)
+    def self.disambiguate_hack(data)
       matches = []
       if data.include?("<?hh")
         matches << Language["Hack"]
@@ -107,7 +124,7 @@ module Linguist
       matches
     end
 
-    def self.disambiguate_sc(data, languages)
+    def self.disambiguate_sc(data)
       matches = []
       if (/\^(this|super)\./.match(data) || /^\s*(\+|\*)\s*\w+\s*{/.match(data) || /^\s*~\w+\s*=\./.match(data))
         matches << Language["SuperCollider"]
@@ -118,12 +135,22 @@ module Linguist
       matches
     end
 
-    def self.disambiguate_asc(data, languages)
+    def self.disambiguate_asc(data)
       matches = []
       matches << Language["AsciiDoc"] if /^=+(\s|\n)/.match(data)
       matches
     end
 
+    def self.disambiguate_f(data)
+      matches = []
+      if /^: /.match(data)
+        matches << Language["Forth"]
+      elsif /^([c*][^a-z]|      subroutine\s)/i.match(data)
+        matches << Language["FORTRAN"]
+      end
+      matches
+    end
+
     def self.active?
       !!ACTIVE
     end

lib/linguist/language.rb

@@ -1,5 +1,4 @@
 require 'escape_utils'
-require 'pygments'
 require 'yaml'
 begin
   require 'yajl'
@@ -101,12 +100,8 @@ module Linguist
     def self.detect(blob)
       name = blob.name.to_s
 
-      # Check if the blob is possibly binary and bail early; this is a cheap
+      # Bail early if the blob is binary or empty.
-      # test that uses the extension name to guess a binary binary mime type.
+      return nil if blob.likely_binary? || blob.binary? || blob.empty?
-      #
-      # We'll perform a more comprehensive test later which actually involves
-      # looking for binary characters in the blob
-      return nil if blob.likely_binary? || blob.binary?
 
       # A bit of an elegant hack. If the file is executable but extensionless,
       # append a "magic" extension so it can be classified with other
@@ -125,16 +120,18 @@ module Linguist
       if possible_languages.length > 1
         data = blob.data
         possible_language_names = possible_languages.map(&:name)
+        heuristic_languages = Heuristics.find_by_heuristics(data, possible_language_names)
+
+        if heuristic_languages.size > 1
+          possible_language_names = heuristic_languages.map(&:name)
+        end
 
-        # Don't bother with binary contents or an empty file
-        if data.nil? || data == ""
-          nil
         # Check if there's a shebang line and use that as authoritative
-        elsif (result = find_by_shebang(data)) && !result.empty?
+        if (result = find_by_shebang(data)) && !result.empty?
           result.first
         # No shebang. Still more work to do. Try to find it with our heuristics.
-        elsif (determined = Heuristics.find_by_heuristics(data, possible_language_names)) && !determined.empty?
+        elsif heuristic_languages.size == 1
-          determined.first
+          heuristic_languages.first
         # Lastly, fall back to the probabilistic classifier.
         elsif classified = Classifier.classify(Samples.cache, data, possible_language_names).first
           # Return the actual Language object based of the string language name (i.e., first element of `#classify`)
@@ -164,7 +161,7 @@ module Linguist
     #
     # Returns the Language or nil if none was found.
     def self.find_by_name(name)
-      @name_index[name.downcase]
+      name && @name_index[name.downcase]
     end
 
     # Public: Look up Language by one of its aliases.
@@ -178,7 +175,7 @@ module Linguist
     #
     # Returns the Lexer or nil if none was found.
     def self.find_by_alias(name)
-      @alias_index[name.downcase]
+      name && @alias_index[name.downcase]
     end
 
     # Public: Look up Languages by filename.
@@ -243,7 +240,7 @@ module Linguist
     #
     # Returns the Language or nil if none was found.
     def self.[](name)
-      @index[name.downcase]
+      name && @index[name.downcase]
     end
 
     # Public: A List of popular languages
@@ -302,10 +299,7 @@ module Linguist
       # Set aliases
       @aliases = [default_alias_name] + (attributes[:aliases] || [])
 
-      # Lookup Lexer object
+      # Load the TextMate scope name or try to guess one
-      @lexer = Pygments::Lexer.find_by_name(attributes[:lexer] || name) ||
-        raise(ArgumentError, "#{@name} is missing lexer")
-
       @tm_scope = attributes[:tm_scope] || begin
         context = case @type
                   when :data, :markup, :prose
@@ -437,11 +431,6 @@ module Linguist
     # Returns the extensions Array
     attr_reader :filenames
 
-    # Public: Return all possible extensions for language
-    def all_extensions
-      (extensions + [primary_extension]).uniq
-    end
-
     # Deprecated: Get primary extension
     #
     # Defaults to the first extension but can be overridden
@@ -599,9 +588,9 @@ module Linguist
       :ace_mode          => options['ace_mode'],
       :wrap              => options['wrap'],
       :group_name        => options['group'],
-      :searchable        => options.key?('searchable') ? options['searchable'] : true,
+      :searchable        => options.fetch('searchable', true),
       :search_term       => options['search_term'],
-      :extensions        => [options['extensions'].first] + options['extensions'][1..-1].sort,
+      :extensions        => Array(options['extensions']),
       :interpreters      => options['interpreters'].sort,
       :filenames         => options['filenames'],
       :popular           => popular.include?(name)

lib/linguist/vendor.yml

@@ -110,6 +110,12 @@
 # MathJax
 - (^|/)MathJax/
 
+# Chart.js
+- (^|/)Chart\.js$
+
+# Codemirror
+- (^|/)[Cc]ode[Mm]irror/(lib|mode|theme|addon|keymap)
+
 # SyntaxHighlighter - http://alexgorbatchev.com/
 - (^|/)shBrush([^.]*)\.js$
 - (^|/)shCore\.js$

lib/linguist/version.rb

@@ -1,3 +1,3 @@
 module Linguist
-  VERSION = "3.5.0"
+  VERSION = "4.0.3"
 end

package.json

@@ -0,0 +1,6 @@
+{
+  "repository": "https://github.com/github/linguist",
+  "dependencies": {
+    "season": "~>3.0"
+  }
+}

samples/Ant Build System/filenames/ant.xml

@@ -0,0 +1,110 @@
+<?xml version="1.0" encoding="iso-8859-1"?>
+<project name="WebBuild">
+
+    <!-- generate timestamps -->
+    <tstamp />
+
+    <!-- Debugging Macro -->
+    <import file="echopath.xml" />
+
+    <!-- JS build files macro -->
+    <import file="rhinoscript.xml" />
+
+    <!-- Component Build Files -->
+    <import file="setup.xml" />
+    <import file="clean.xml" />
+    <import file="copy.xml" />
+    <import file="file.transform.xml" />
+    <import file="external.tools.xml" />
+    <import file="rename.xml" />
+    <import file="js.xml" />
+    <import file="css.xml" />
+    <import file="img.xml" />
+    <import file="png8.xml" />
+    <import file="yui.xml" />
+    <import file="cdn.xml" />
+    <import file="datauri.xml" />
+    <import file="devlive.xml" />
+
+    <!-- This dirname is the only complete path we know for sure, everything builds off of it -->
+    <dirname property="dir.build" file="${ant.file.WebBuild}" />
+
+    <!-- get name for newly built folder -->
+    <basename property="app.name"       file="${basedir}" />
+
+    <!-- read global properties file -->
+    <property file="${dir.build}\build.properties" />
+
+    <!-- Build Directories -->
+    <property name="dir.build.js"   location="${dir.build}/js" />
+
+    <!-- App Directories -->
+    <property name="dir.app"        location="${dir.result}/${app.name}" />
+    <property name="dir.app.temp"   location="${dir.temp}/${app.name}" />
+    <property name="dir.app.files"  location="${dir.app.temp}/${dir.files}" />
+
+    <!-- Files -->
+    <property name="mapping.js"     location="${dir.app.temp}/${mapping.file.js}" />
+    <property name="mapping.css"    location="${dir.app.temp}/${mapping.file.css}" />
+    <property name="mapping.img"    location="${dir.app.temp}/${mapping.file.img}" />
+    <property name="mapping.swf"    location="${dir.app.temp}/${mapping.file.swf}" />
+    <property name="mapping.fonts"  location="${dir.app.temp}/${mapping.file.fonts}" />
+
+    <!-- Tool Directories -->
+    <property name="dir.bin"    location="${dir.build}/Bin" />
+    <property name="dir.jar"    location="${dir.bin}/jar" />
+
+    <!-- Tool Files -->
+	<property name="tools.compressor"     location="${dir.jar}/${tools.file.compressor}" />
+	<property name="tools.cssembed"       location="${dir.jar}/${tools.file.cssembed}" />
+    <property name="tools.filetransform"  location="${dir.jar}/${tools.file.filetransform}" />
+    <property name="tools.optipng"        location="${dir.bin}/${tools.file.optipng}" />
+    <property name="tools.jpegtran"       location="${dir.bin}/${tools.file.jpegtran}" />
+
+
+    <!-- BUILD TARGETS -->
+
+    <!-- low level utility build targets -->
+
+    <!-- Build the tools -->
+    <target name="-setup.build.tools"
+            depends="-define.filetransform, -define.cssembed, -define.yuicompressor, -define.jsclasspath"
+    />
+
+    <!-- set up filesystem properties -->
+    <target
+        name="-setup"
+        depends="-setup.mode, -setup.conditions, -setup.js, -setup.css, -setup.swf, -setup.img, -setup.fonts, -setup.yui"
+    />
+
+    <!-- utility-ish targets -->
+    <target name="copy"         depends="clean, tools, -copy" />
+    <target name="tools"        depends="-setup.build.tools" />
+    <target name="finalize"     depends="copy, -finalize" />
+    <target name="-prepare"     depends="copy, -setup" />
+
+    <!-- individual component build targets (empty descriptions are to make sure they show in "ant -p") -->
+    <target name="devlive"      depends="-prepare, -devlive"            description="" />
+    <target name="js"           depends="-prepare, -js"                 description="" />
+    <target name="css"          depends="-prepare, -css"                description="" />
+    <target name="rename"       depends="-prepare, -rename"             description="" />
+    <target name="yui"          depends="-prepare, rename, -yui"        description="" />
+    <target name="cdn"          depends="-prepare, -cdn"                description="" />
+
+    <!-- high level build targets (Excluding of images is on purpose here, it's slow) -->
+    <target name="core"
+            depends="devlive, js, css, cdn, rename, yui, -js.inline"
+            description="Core build work"
+    />
+
+    <target name="prod"
+            depends="core, finalize"
+            description="Full Production Build"
+    />
+
+    <!-- debug target -->
+    <target name="debug" depends="-setup">
+        <echoproperties/>
+    </target>
+
+</project>

samples/Ant Build System/filenames/build.xml

@@ -0,0 +1 @@
+ant.xml

samples/FORTRAN/sample1.f

@@ -0,0 +1,25 @@
+c comment
+* comment
+
+      program main
+
+      end
+
+      subroutine foo( i, x, b )
+      INTEGER            i
+      REAL               x
+      LOGICAL            b
+
+      if( i.ne.0 ) then
+         call bar( -i )
+      end if
+
+      return
+      end
+
+      double complex function baz()
+
+      baz = (0.0d0,0.0d0)
+
+      return 
+      end

samples/FORTRAN/sample1.for

@@ -0,0 +1,25 @@
+c comment
+* comment
+
+      program main
+
+      end
+
+      subroutine foo( i, x, b )
+      INTEGER            i
+      REAL               x
+      LOGICAL            b
+
+      if( i.ne.0 ) then
+         call bar( -i )
+      end if
+
+      return
+      end
+
+      double complex function baz()
+
+      baz = (0.0d0,0.0d0)
+
+      return 
+      end

samples/FORTRAN/sample2.f

@@ -0,0 +1,25 @@
+      PROGRAM MAIN
+
+      END
+
+C comment
+* comment
+
+      SUBROUTINE foo( i, x, b )
+      INTEGER            i
+      REAL               x
+      LOGICAL            b
+
+      IF( i.NE.0 ) THEN
+         CALL bar( -i )
+      END IF
+
+      RETURN
+      END
+
+      DOUBLE COMPLEX FUNCTION baz()
+
+      baz = (0.0d0,0.0d0)
+
+      RETURN 
+      END

samples/FORTRAN/sample3.F

@@ -0,0 +1,25 @@
+c comment
+* comment
+
+      program main
+
+      end
+
+      subroutine foo( i, x, b )
+      INTEGER            i
+      REAL               x
+      LOGICAL            b
+
+      if( i.ne.0 ) then
+         call bar( -i )
+      end if
+
+      return
+      end
+
+      double complex function baz()
+
+      baz = (0.0d0,0.0d0)
+
+      return 
+      end

samples/Forth/core.f

@@ -0,0 +1,252 @@
+: immediate   lastxt @ dup c@ negate swap c! ;
+
+: \   source nip >in ! ; immediate \ Copyright 2004, 2012 Lars Brinkhoff
+
+: char \ ( "word" -- char )
+    bl-word here 1+ c@ ;
+
+: ahead  here 0 , ;
+
+: resolve   here swap ! ;
+
+: '   bl-word here find 0branch [ ahead ] exit [ resolve ] 0 ;
+
+: postpone-nonimmediate   [ ' literal , ' compile, ] literal , ;
+
+: create   dovariable_code header, reveal ;
+
+create postponers
+    ' postpone-nonimmediate ,
+    ' abort ,
+    ' , ,
+
+: word \ ( char "<chars>string<char>" -- caddr )
+    drop bl-word here ;
+
+: postpone \ ( C: "word" -- )
+    bl word find 1+ cells  postponers + @ execute ; immediate
+
+: unresolved \ ( C: "word" -- orig )
+    postpone postpone  postpone ahead ; immediate
+
+: chars \ ( n1 -- n2 )
+    ;
+
+: else \ ( -- ) ( C: orig1 -- orig2 )
+    unresolved branch swap resolve ; immediate
+
+: if \ ( flag -- ) ( C: -- orig )
+    unresolved 0branch ; immediate
+
+: then \ ( -- ) ( C: orig -- )
+    resolve ; immediate
+
+: [char] \ ( "word" -- )
+    char  postpone literal ; immediate
+
+: (does>)   lastxt @ dodoes_code over >code ! r> swap >does ! ;
+
+: does>   postpone (does>) ; immediate
+
+: begin \ ( -- ) ( C: -- dest )
+    here ; immediate
+
+: while \ ( x -- ) ( C: dest -- orig dest )
+    unresolved 0branch swap ; immediate
+
+: repeat \ ( -- ) ( C: orig dest -- )
+    postpone branch ,  resolve ; immediate
+
+: until \ ( x -- ) ( C: dest -- )
+    postpone 0branch , ; immediate
+
+: recurse   lastxt @ compile, ; immediate
+
+: pad \ ( -- addr )
+    here 1024 + ;
+
+: parse \ ( char "string<char>" -- addr n )
+    pad >r  begin
+	source? if <source 2dup <> else 0 0 then
+    while
+	r@ c!  r> 1+ >r
+    repeat  2drop  pad r> over - ;
+
+: ( \ ( "string<paren>" -- )
+    [ char ) ] literal parse 2drop ; immediate
+    \ TODO: If necessary, refill and keep parsing.
+
+: string, ( addr n -- )
+    here over allot align  swap cmove ;
+
+: (s") ( -- addr n ) ( R: ret1 -- ret2 )
+    r> dup @ swap cell+ 2dup + aligned >r swap ;
+
+create squote   128 allot
+
+: s" ( "string<quote>" -- addr n )
+    state @ if
+	postpone (s")  [char] " parse  dup ,  string,
+    else
+	[char] " parse  >r squote r@ cmove  squote r>
+    then ; immediate
+
+: (abort") ( ... addr n -- ) ( R: ... -- )
+    cr type cr abort ;
+
+: abort" ( ... x "string<quote>" -- ) ( R: ... -- )
+    postpone if  postpone s"  postpone (abort")  postpone then ; immediate
+
+\ ----------------------------------------------------------------------
+
+( Core words. )
+
+\ TODO: #
+\ TODO: #>
+\ TODO: #s
+
+: and  ( x y -- x&y )   nand invert ;
+
+: *   1 2>r 0 swap begin r@ while
+         r> r> swap 2dup dup + 2>r and if swap over + swap then dup +
+      repeat r> r> 2drop drop ;
+
+\ TODO: */mod
+
+: +loop ( -- ) ( C: nest-sys -- )
+    postpone (+loop)  postpone 0branch  ,  postpone unloop ; immediate
+
+: space   bl emit ;
+
+: ?.-  dup 0 < if [char] - emit negate then ;
+
+: digit   [char] 0 + emit ;
+
+: (.)   base @ /mod  ?dup if recurse then  digit ;
+
+: ." ( "string<quote>" -- )   postpone s"  postpone type ; immediate
+
+: . ( x -- )   ?.- (.) space ;
+
+: postpone-number ( caddr -- )
+    0 0 rot count >number dup 0= if
+	2drop nip
+	postpone (literal)  postpone (literal)  postpone ,
+	postpone literal  postpone ,
+    else
+	." Undefined: " type cr abort
+    then ;
+
+' postpone-number  postponers cell+  !
+
+: / ( x y -- x/y )   /mod nip ;
+
+: 0< ( n -- flag )   0 < ;
+
+: 1- ( n -- n-1 )   -1 + ;
+
+: 2! ( x1 x2 addr -- )   swap over ! cell+ ! ;
+
+: 2* ( n -- 2n )   dup + ;
+
+\ Kernel: 2/
+
+: 2@ ( addr -- x1 x2 )   dup cell+ @ swap @ ;
+
+\ Kernel: 2drop
+\ Kernel: 2dup
+
+\ TODO: 2over ( x1 x2 x3 x4 -- x1 x2 x3 x4 x1 x2 )
+\           3 pick 3 pick ;
+
+\ TODO: 2swap
+
+\ TODO: <#
+
+: abs ( n -- |n| )
+    dup 0< if negate then ;
+
+\ TODO: accept
+
+: c, ( n -- )
+    here c!  1 chars allot ;
+
+: char+ ( n1 -- n2 )
+    1+ ;
+
+: constant   create , does> @ ;
+
+: decimal ( -- )
+    10 base ! ;
+
+: depth ( -- n )
+    data_stack 100 cells +  'SP @  - /cell /  2 - ;
+
+: do ( n1 n2 -- ) ( R: -- loop-sys ) ( C: -- do-sys )
+    postpone 2>r  here ; immediate
+
+\ TODO: environment?
+\ TODO: evaluate
+\ TODO: fill
+\ TODO: fm/mod )
+\ TODO: hold
+
+: j ( -- x1 ) ( R: x1 x2 x3 -- x1 x2 x3 )
+    'RP @ 3 cells + @ ;
+
+\ TODO: leave
+
+: loop ( -- ) ( C: nest-sys -- )
+    postpone 1  postpone (+loop)
+    postpone 0branch  ,
+    postpone unloop ; immediate
+
+: lshift   begin ?dup while 1- swap dup + swap repeat ;
+
+: rshift   1 begin over while dup + swap 1- swap repeat nip
+           2>r 0 1 begin r@ while
+              r> r> 2dup swap dup + 2>r and if swap over + swap then dup +
+           repeat r> r> 2drop drop ;
+
+: max ( x y -- max[x,y] )
+    2dup > if drop else nip then ;
+
+\ Kernel: min
+\ TODO:   mod
+\ TODO:   move
+
+: (quit) ( R: ... -- )
+    return_stack 100 cells + 'RP !
+    0 'source-id !  tib ''source !  #tib ''#source !
+    postpone [
+    begin
+	refill
+    while
+	interpret  state @ 0= if ." ok" cr then
+    repeat
+    bye ;
+
+' (quit)  ' quit >body cell+  !
+
+\ TODO: s>d
+\ TODO: sign
+\ TODO: sm/rem
+
+: spaces ( n -- )
+    0 do space loop ;
+
+\ TODO: u.
+
+: signbit ( -- n )   -1 1 rshift invert ;
+
+: xor ( x y -- x^y )    2dup nand >r r@ nand swap r> nand nand ;
+
+: u<  ( x y -- flag )  signbit xor swap signbit xor > ;
+
+\ TODO: um/mod
+
+: variable ( "word" -- )
+    create /cell allot ;
+
+: ['] \ ( C: "word" -- )
+    ' postpone literal ; immediate

samples/Forth/core.for

@@ -0,0 +1,252 @@
+: immediate   lastxt @ dup c@ negate swap c! ;
+
+: \   source nip >in ! ; immediate \ Copyright 2004, 2012 Lars Brinkhoff
+
+: char \ ( "word" -- char )
+    bl-word here 1+ c@ ;
+
+: ahead  here 0 , ;
+
+: resolve   here swap ! ;
+
+: '   bl-word here find 0branch [ ahead ] exit [ resolve ] 0 ;
+
+: postpone-nonimmediate   [ ' literal , ' compile, ] literal , ;
+
+: create   dovariable_code header, reveal ;
+
+create postponers
+    ' postpone-nonimmediate ,
+    ' abort ,
+    ' , ,
+
+: word \ ( char "<chars>string<char>" -- caddr )
+    drop bl-word here ;
+
+: postpone \ ( C: "word" -- )
+    bl word find 1+ cells  postponers + @ execute ; immediate
+
+: unresolved \ ( C: "word" -- orig )
+    postpone postpone  postpone ahead ; immediate
+
+: chars \ ( n1 -- n2 )
+    ;
+
+: else \ ( -- ) ( C: orig1 -- orig2 )
+    unresolved branch swap resolve ; immediate
+
+: if \ ( flag -- ) ( C: -- orig )
+    unresolved 0branch ; immediate
+
+: then \ ( -- ) ( C: orig -- )
+    resolve ; immediate
+
+: [char] \ ( "word" -- )
+    char  postpone literal ; immediate
+
+: (does>)   lastxt @ dodoes_code over >code ! r> swap >does ! ;
+
+: does>   postpone (does>) ; immediate
+
+: begin \ ( -- ) ( C: -- dest )
+    here ; immediate
+
+: while \ ( x -- ) ( C: dest -- orig dest )
+    unresolved 0branch swap ; immediate
+
+: repeat \ ( -- ) ( C: orig dest -- )
+    postpone branch ,  resolve ; immediate
+
+: until \ ( x -- ) ( C: dest -- )
+    postpone 0branch , ; immediate
+
+: recurse   lastxt @ compile, ; immediate
+
+: pad \ ( -- addr )
+    here 1024 + ;
+
+: parse \ ( char "string<char>" -- addr n )
+    pad >r  begin
+	source? if <source 2dup <> else 0 0 then
+    while
+	r@ c!  r> 1+ >r
+    repeat  2drop  pad r> over - ;
+
+: ( \ ( "string<paren>" -- )
+    [ char ) ] literal parse 2drop ; immediate
+    \ TODO: If necessary, refill and keep parsing.
+
+: string, ( addr n -- )
+    here over allot align  swap cmove ;
+
+: (s") ( -- addr n ) ( R: ret1 -- ret2 )
+    r> dup @ swap cell+ 2dup + aligned >r swap ;
+
+create squote   128 allot
+
+: s" ( "string<quote>" -- addr n )
+    state @ if
+	postpone (s")  [char] " parse  dup ,  string,
+    else
+	[char] " parse  >r squote r@ cmove  squote r>
+    then ; immediate
+
+: (abort") ( ... addr n -- ) ( R: ... -- )
+    cr type cr abort ;
+
+: abort" ( ... x "string<quote>" -- ) ( R: ... -- )
+    postpone if  postpone s"  postpone (abort")  postpone then ; immediate
+
+\ ----------------------------------------------------------------------
+
+( Core words. )
+
+\ TODO: #
+\ TODO: #>
+\ TODO: #s
+
+: and  ( x y -- x&y )   nand invert ;
+
+: *   1 2>r 0 swap begin r@ while
+         r> r> swap 2dup dup + 2>r and if swap over + swap then dup +
+      repeat r> r> 2drop drop ;
+
+\ TODO: */mod
+
+: +loop ( -- ) ( C: nest-sys -- )
+    postpone (+loop)  postpone 0branch  ,  postpone unloop ; immediate
+
+: space   bl emit ;
+
+: ?.-  dup 0 < if [char] - emit negate then ;
+
+: digit   [char] 0 + emit ;
+
+: (.)   base @ /mod  ?dup if recurse then  digit ;
+
+: ." ( "string<quote>" -- )   postpone s"  postpone type ; immediate
+
+: . ( x -- )   ?.- (.) space ;
+
+: postpone-number ( caddr -- )
+    0 0 rot count >number dup 0= if
+	2drop nip
+	postpone (literal)  postpone (literal)  postpone ,
+	postpone literal  postpone ,
+    else
+	." Undefined: " type cr abort
+    then ;
+
+' postpone-number  postponers cell+  !
+
+: / ( x y -- x/y )   /mod nip ;
+
+: 0< ( n -- flag )   0 < ;
+
+: 1- ( n -- n-1 )   -1 + ;
+
+: 2! ( x1 x2 addr -- )   swap over ! cell+ ! ;
+
+: 2* ( n -- 2n )   dup + ;
+
+\ Kernel: 2/
+
+: 2@ ( addr -- x1 x2 )   dup cell+ @ swap @ ;
+
+\ Kernel: 2drop
+\ Kernel: 2dup
+
+\ TODO: 2over ( x1 x2 x3 x4 -- x1 x2 x3 x4 x1 x2 )
+\           3 pick 3 pick ;
+
+\ TODO: 2swap
+
+\ TODO: <#
+
+: abs ( n -- |n| )
+    dup 0< if negate then ;
+
+\ TODO: accept
+
+: c, ( n -- )
+    here c!  1 chars allot ;
+
+: char+ ( n1 -- n2 )
+    1+ ;
+
+: constant   create , does> @ ;
+
+: decimal ( -- )
+    10 base ! ;
+
+: depth ( -- n )
+    data_stack 100 cells +  'SP @  - /cell /  2 - ;
+
+: do ( n1 n2 -- ) ( R: -- loop-sys ) ( C: -- do-sys )
+    postpone 2>r  here ; immediate
+
+\ TODO: environment?
+\ TODO: evaluate
+\ TODO: fill
+\ TODO: fm/mod )
+\ TODO: hold
+
+: j ( -- x1 ) ( R: x1 x2 x3 -- x1 x2 x3 )
+    'RP @ 3 cells + @ ;
+
+\ TODO: leave
+
+: loop ( -- ) ( C: nest-sys -- )
+    postpone 1  postpone (+loop)
+    postpone 0branch  ,
+    postpone unloop ; immediate
+
+: lshift   begin ?dup while 1- swap dup + swap repeat ;
+
+: rshift   1 begin over while dup + swap 1- swap repeat nip
+           2>r 0 1 begin r@ while
+              r> r> 2dup swap dup + 2>r and if swap over + swap then dup +
+           repeat r> r> 2drop drop ;
+
+: max ( x y -- max[x,y] )
+    2dup > if drop else nip then ;
+
+\ Kernel: min
+\ TODO:   mod
+\ TODO:   move
+
+: (quit) ( R: ... -- )
+    return_stack 100 cells + 'RP !
+    0 'source-id !  tib ''source !  #tib ''#source !
+    postpone [
+    begin
+	refill
+    while
+	interpret  state @ 0= if ." ok" cr then
+    repeat
+    bye ;
+
+' (quit)  ' quit >body cell+  !
+
+\ TODO: s>d
+\ TODO: sign
+\ TODO: sm/rem
+
+: spaces ( n -- )
+    0 do space loop ;
+
+\ TODO: u.
+
+: signbit ( -- n )   -1 1 rshift invert ;
+
+: xor ( x y -- x^y )    2dup nand >r r@ nand swap r> nand nand ;
+
+: u<  ( x y -- flag )  signbit xor swap signbit xor > ;
+
+\ TODO: um/mod
+
+: variable ( "word" -- )
+    create /cell allot ;
+
+: ['] \ ( C: "word" -- )
+    ' postpone literal ; immediate

samples/Forth/core1.F

@@ -0,0 +1,252 @@
+: immediate   lastxt @ dup c@ negate swap c! ;
+
+: \   source nip >in ! ; immediate \ Copyright 2004, 2012 Lars Brinkhoff
+
+: char \ ( "word" -- char )
+    bl-word here 1+ c@ ;
+
+: ahead  here 0 , ;
+
+: resolve   here swap ! ;
+
+: '   bl-word here find 0branch [ ahead ] exit [ resolve ] 0 ;
+
+: postpone-nonimmediate   [ ' literal , ' compile, ] literal , ;
+
+: create   dovariable_code header, reveal ;
+
+create postponers
+    ' postpone-nonimmediate ,
+    ' abort ,
+    ' , ,
+
+: word \ ( char "<chars>string<char>" -- caddr )
+    drop bl-word here ;
+
+: postpone \ ( C: "word" -- )
+    bl word find 1+ cells  postponers + @ execute ; immediate
+
+: unresolved \ ( C: "word" -- orig )
+    postpone postpone  postpone ahead ; immediate
+
+: chars \ ( n1 -- n2 )
+    ;
+
+: else \ ( -- ) ( C: orig1 -- orig2 )
+    unresolved branch swap resolve ; immediate
+
+: if \ ( flag -- ) ( C: -- orig )
+    unresolved 0branch ; immediate
+
+: then \ ( -- ) ( C: orig -- )
+    resolve ; immediate
+
+: [char] \ ( "word" -- )
+    char  postpone literal ; immediate
+
+: (does>)   lastxt @ dodoes_code over >code ! r> swap >does ! ;
+
+: does>   postpone (does>) ; immediate
+
+: begin \ ( -- ) ( C: -- dest )
+    here ; immediate
+
+: while \ ( x -- ) ( C: dest -- orig dest )
+    unresolved 0branch swap ; immediate
+
+: repeat \ ( -- ) ( C: orig dest -- )
+    postpone branch ,  resolve ; immediate
+
+: until \ ( x -- ) ( C: dest -- )
+    postpone 0branch , ; immediate
+
+: recurse   lastxt @ compile, ; immediate
+
+: pad \ ( -- addr )
+    here 1024 + ;
+
+: parse \ ( char "string<char>" -- addr n )
+    pad >r  begin
+	source? if <source 2dup <> else 0 0 then
+    while
+	r@ c!  r> 1+ >r
+    repeat  2drop  pad r> over - ;
+
+: ( \ ( "string<paren>" -- )
+    [ char ) ] literal parse 2drop ; immediate
+    \ TODO: If necessary, refill and keep parsing.
+
+: string, ( addr n -- )
+    here over allot align  swap cmove ;
+
+: (s") ( -- addr n ) ( R: ret1 -- ret2 )
+    r> dup @ swap cell+ 2dup + aligned >r swap ;
+
+create squote   128 allot
+
+: s" ( "string<quote>" -- addr n )
+    state @ if
+	postpone (s")  [char] " parse  dup ,  string,
+    else
+	[char] " parse  >r squote r@ cmove  squote r>
+    then ; immediate
+
+: (abort") ( ... addr n -- ) ( R: ... -- )
+    cr type cr abort ;
+
+: abort" ( ... x "string<quote>" -- ) ( R: ... -- )
+    postpone if  postpone s"  postpone (abort")  postpone then ; immediate
+
+\ ----------------------------------------------------------------------
+
+( Core words. )
+
+\ TODO: #
+\ TODO: #>
+\ TODO: #s
+
+: and  ( x y -- x&y )   nand invert ;
+
+: *   1 2>r 0 swap begin r@ while
+         r> r> swap 2dup dup + 2>r and if swap over + swap then dup +
+      repeat r> r> 2drop drop ;
+
+\ TODO: */mod
+
+: +loop ( -- ) ( C: nest-sys -- )
+    postpone (+loop)  postpone 0branch  ,  postpone unloop ; immediate
+
+: space   bl emit ;
+
+: ?.-  dup 0 < if [char] - emit negate then ;
+
+: digit   [char] 0 + emit ;
+
+: (.)   base @ /mod  ?dup if recurse then  digit ;
+
+: ." ( "string<quote>" -- )   postpone s"  postpone type ; immediate
+
+: . ( x -- )   ?.- (.) space ;
+
+: postpone-number ( caddr -- )
+    0 0 rot count >number dup 0= if
+	2drop nip
+	postpone (literal)  postpone (literal)  postpone ,
+	postpone literal  postpone ,
+    else
+	." Undefined: " type cr abort
+    then ;
+
+' postpone-number  postponers cell+  !
+
+: / ( x y -- x/y )   /mod nip ;
+
+: 0< ( n -- flag )   0 < ;
+
+: 1- ( n -- n-1 )   -1 + ;
+
+: 2! ( x1 x2 addr -- )   swap over ! cell+ ! ;
+
+: 2* ( n -- 2n )   dup + ;
+
+\ Kernel: 2/
+
+: 2@ ( addr -- x1 x2 )   dup cell+ @ swap @ ;
+
+\ Kernel: 2drop
+\ Kernel: 2dup
+
+\ TODO: 2over ( x1 x2 x3 x4 -- x1 x2 x3 x4 x1 x2 )
+\           3 pick 3 pick ;
+
+\ TODO: 2swap
+
+\ TODO: <#
+
+: abs ( n -- |n| )
+    dup 0< if negate then ;
+
+\ TODO: accept
+
+: c, ( n -- )
+    here c!  1 chars allot ;
+
+: char+ ( n1 -- n2 )
+    1+ ;
+
+: constant   create , does> @ ;
+
+: decimal ( -- )
+    10 base ! ;
+
+: depth ( -- n )
+    data_stack 100 cells +  'SP @  - /cell /  2 - ;
+
+: do ( n1 n2 -- ) ( R: -- loop-sys ) ( C: -- do-sys )
+    postpone 2>r  here ; immediate
+
+\ TODO: environment?
+\ TODO: evaluate
+\ TODO: fill
+\ TODO: fm/mod )
+\ TODO: hold
+
+: j ( -- x1 ) ( R: x1 x2 x3 -- x1 x2 x3 )
+    'RP @ 3 cells + @ ;
+
+\ TODO: leave
+
+: loop ( -- ) ( C: nest-sys -- )
+    postpone 1  postpone (+loop)
+    postpone 0branch  ,
+    postpone unloop ; immediate
+
+: lshift   begin ?dup while 1- swap dup + swap repeat ;
+
+: rshift   1 begin over while dup + swap 1- swap repeat nip
+           2>r 0 1 begin r@ while
+              r> r> 2dup swap dup + 2>r and if swap over + swap then dup +
+           repeat r> r> 2drop drop ;
+
+: max ( x y -- max[x,y] )
+    2dup > if drop else nip then ;
+
+\ Kernel: min
+\ TODO:   mod
+\ TODO:   move
+
+: (quit) ( R: ... -- )
+    return_stack 100 cells + 'RP !
+    0 'source-id !  tib ''source !  #tib ''#source !
+    postpone [
+    begin
+	refill
+    while
+	interpret  state @ 0= if ." ok" cr then
+    repeat
+    bye ;
+
+' (quit)  ' quit >body cell+  !
+
+\ TODO: s>d
+\ TODO: sign
+\ TODO: sm/rem
+
+: spaces ( n -- )
+    0 do space loop ;
+
+\ TODO: u.
+
+: signbit ( -- n )   -1 1 rshift invert ;
+
+: xor ( x y -- x^y )    2dup nand >r r@ nand swap r> nand nand ;
+
+: u<  ( x y -- flag )  signbit xor swap signbit xor > ;
+
+\ TODO: um/mod
+
+: variable ( "word" -- )
+    create /cell allot ;
+
+: ['] \ ( C: "word" -- )
+    ' postpone literal ; immediate

samples/Graphviz (DOT)/annoying.DOT

@@ -0,0 +1,50 @@
+/* 
+  Huffman Tree DOT graph.
+
+  DOT Reference :  http://www.graphviz.org/doc/info/lang.html
+                   http://en.wikipedia.org/wiki/DOT_language
+  Timestamp     :  1415989074 
+  Phrase        :  'OH GOD WHY IS LINGUIST SO ANAL ABOUT THIS STUFF'
+
+  Generated on http://huffman.ooz.ie/
+*/
+
+digraph G {
+    edge [label=0];
+    graph [ranksep=0];
+    T [shape=record, label="{{T|4}|000}"];
+    S [shape=record, label="{{S|5}|001}"];
+    SPACE [shape=record, label="{{SPACE|9}|01}"];
+    A [shape=record, label="{{A|3}|1000}"];
+    H [shape=record, label="{{H|3}|1001}"];
+    U [shape=record, label="{{U|3}|1010}"];
+    L [shape=record, label="{{L|2}|10110}"];
+    N [shape=record, label="{{N|2}|10111}"];
+    I [shape=record, label="{{I|4}|1100}"];
+    O [shape=record, label="{{O|4}|1101}"];
+    G [shape=record, label="{{G|2}|11100}"];
+    F [shape=record, label="{{F|2}|11101}"];
+    GF [label=4];
+    W [shape=record, label="{{W|1}|111100}"];
+    Y [shape=record, label="{{Y|1}|111101}"];
+    B [shape=record, label="{{B|1}|111110}"];
+    D [shape=record, label="{{D|1}|111111}"];
+    BD [label=2];
+    WYBD [label=4];
+    GFWYBD [label=8];
+    47 -> 18 -> 9 -> T;
+    29 -> 13 -> 6 -> A;
+    7 -> U;
+    4 -> L;
+    16 -> 8 -> I;
+    GFWYBD -> GF -> G;
+    WYBD -> 2 -> W;
+    BD -> B;9 -> S [label=1];
+    18 -> SPACE [label=1];
+    6 -> H [label=1];
+    13 -> 7 -> 4 -> N [label=1];
+    8 -> O [label=1];
+    GF -> F [label=1];
+    2 -> Y [label=1];
+    47 -> 29 -> 16 -> GFWYBD -> WYBD -> BD -> D [label=1];
+}

samples/Graphviz (DOT)/sample.dot

@@ -0,0 +1,74 @@
+/* 
+  Huffman Tree DOT graph.
+
+  DOT Reference :  http://www.graphviz.org/doc/info/lang.html
+                   http://en.wikipedia.org/wiki/DOT_language
+  Timestamp     :  1415988139 
+  Phrase        :  'SERIAL KILLER AND SEX OFFENDER ANGUS SINCLAIR IS JAILED FOR A MINIMUM OF 37 YEARS FOR THE 1977 WORLDS END MURDERS OF HELEN SCOTT AND CHRISTINE EADIE.'
+
+  Generated on http://huffman.ooz.ie/
+*/
+
+digraph G {
+    edge [label=0];
+    graph [ranksep=0];
+    node [shape=record];
+    U [label="{{U|3}|00000}"];
+    G [label="{{G|1}|0000100}"];
+    K [label="{{K|1}|0000101}"];
+    _3 [label="{{3|1}|0000110}"];
+    _9 [label="{{9|1}|0000111}"];
+    _39 [label=2];
+    L [label="{{L|7}|0001}"];
+    O [label="{{O|7}|0010}"];
+    Y [label="{{Y|1}|0011000}"];
+    X [label="{{X|1}|0011001}"];
+    YX [label=2];
+    J [label="{{J|1}|0011010}"];
+    W [label="{{W|1}|0011011}"];
+    JW [label=2];
+    YXJW [label=4];
+    M [label="{{M|4}|00111}"];
+    E [label="{{E|15}|010}"];
+    D [label="{{D|8}|0110}"];
+    T [label="{{T|4}|01110}"];
+    DOT [label="{{DOT|1}|0111100}"];
+    _1 [label="{{1|1}|0111101}"];
+    DOT1 [label=2];
+    _7 [label="{{7|3}|011111}"];
+    A [label="{{A|9}|1000}"];
+    N [label="{{N|9}|1001}"];
+    S [label="{{S|10}|1010}"];
+    I [label="{{I|11}|1011}"];
+    R [label="{{R|11}|1100}"];
+    C [label="{{C|3}|110100}"];
+    H [label="{{H|3}|110101}"];
+    F [label="{{F|6}|11011}"];
+    SPACE [label="{{SPACE|26}|111}"];
+    149 -> 61 -> 29 -> 14 -> 7 -> U;
+    4 -> 2 -> G;
+    _39 -> _3;
+    15 -> O;
+    8 -> YXJW -> YX -> Y;
+    JW -> J;
+    32 -> E;
+    17 -> D;
+    9 -> T;
+    5 -> DOT1 -> DOT;
+    88 -> 39 -> 18 -> A;
+    21 -> S;
+    49 -> 23 -> R;
+    12 -> 6 -> C;2 -> K [label=1];
+    7 -> 4 -> _39 -> _9 [label=1];
+    14 -> L [label=1];
+    YX -> X [label=1];
+    YXJW -> JW -> W [label=1];
+    29 -> 15 -> 8 -> M [label=1];
+    DOT1 -> _1 [label=1];
+    61 -> 32 -> 17 -> 9 -> 5 -> _7 [label=1];
+    18 -> N [label=1];
+    39 -> 21 -> I [label=1];
+    6 -> H [label=1];
+    23 -> 12 -> F [label=1];
+    149 -> 88 -> 49 -> SPACE [label=1];
+}

samples/JavaScript/empty.js

@@ -1,3 +0,0 @@
-(function() {
-
-}).call(this);

samples/Maven POM/filenames/pom.xml

@@ -0,0 +1,207 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+	<modelVersion>4.0.0</modelVersion>
+	<groupId>renpengben</groupId>
+	<artifactId>spring4mvc-jpa</artifactId>
+	<packaging>war</packaging>
+	<version>0.0.1-SNAPSHOT</version>
+	<name>spring4mvc-jpa Maven Webapp</name>
+
+	<url>https://renpengben.github.io</url>
+
+	<description>spring4mvc-jpa</description>
+
+	<properties>
+		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+		<java.version>1.7</java.version>
+		<junit.version>4.11</junit.version>
+		<slf4j.version>1.7.7</slf4j.version>
+		<log4j.version>1.2.17</log4j.version>
+
+		<spring.version>4.0.5.RELEASE</spring.version>
+		<spring.data.jpa.version>1.6.0.RELEASE</spring.data.jpa.version>
+		<cglib.version>2.1_3</cglib.version>
+
+		<mysql.version>5.1.31</mysql.version>
+		<hibernate.version>4.3.5.Final</hibernate.version>
+		<hibernate-validator.version>5.1.1.Final</hibernate-validator.version>
+		<druid-version>1.0.6</druid-version>
+
+	</properties>
+
+
+	<dependencies>
+
+		<dependency>
+			<groupId>junit</groupId>
+			<artifactId>junit</artifactId>
+			<version>${junit.version}</version>
+			<scope>test</scope>
+		</dependency>
+
+		<dependency>
+			<groupId>org.slf4j</groupId>
+			<artifactId>slf4j-api</artifactId>
+			<version>${slf4j.version}</version>
+		</dependency>
+		<dependency>
+			<groupId>org.slf4j</groupId>
+			<artifactId>slf4j-log4j12</artifactId>
+			<version>${slf4j.version}</version>
+		</dependency>
+		<dependency>
+			<groupId>log4j</groupId>
+			<artifactId>log4j</artifactId>
+			<version>${log4j.version}</version>
+		</dependency>
+
+
+		<!-- Spring -->
+		<dependency>
+			<groupId>org.springframework</groupId>
+			<artifactId>spring-core</artifactId>
+			<version>${spring.version}</version>
+			<exclusions>
+				<exclusion>
+					<groupId>commons-logging</groupId>
+					<artifactId>commons-logging</artifactId>
+				</exclusion>
+			</exclusions>
+		</dependency>
+
+		<dependency>
+			<groupId>org.springframework</groupId>
+			<artifactId>spring-beans</artifactId>
+			<version>${spring.version}</version>
+		</dependency>
+		<dependency>
+			<groupId>org.springframework</groupId>
+			<artifactId>spring-context</artifactId>
+			<version>${spring.version}</version>
+		</dependency>
+
+		<dependency>
+			<groupId>org.springframework</groupId>
+			<artifactId>spring-aop</artifactId>
+			<version>${spring.version}</version>
+		</dependency>
+
+		<dependency>
+			<groupId>org.springframework</groupId>
+			<artifactId>spring-expression</artifactId>
+			<version>${spring.version}</version>
+		</dependency>
+		<dependency>
+			<groupId>org.springframework</groupId>
+			<artifactId>spring-tx</artifactId>
+			<version>${spring.version}</version>
+		</dependency>
+
+
+		<dependency>
+			<groupId>org.springframework</groupId>
+			<artifactId>spring-aspects</artifactId>
+			<version>${spring.version}</version>
+		</dependency>
+		<dependency>
+			<groupId>org.springframework</groupId>
+			<artifactId>spring-context-support</artifactId>
+			<version>${spring.version}</version>
+		</dependency>
+		<dependency>
+			<groupId>org.springframework</groupId>
+			<artifactId>spring-jdbc</artifactId>
+			<version>${spring.version}</version>
+		</dependency>
+		<dependency>
+			<groupId>org.springframework</groupId>
+			<artifactId>spring-orm</artifactId>
+			<version>${spring.version}</version>
+		</dependency>
+
+		<dependency>
+			<groupId>org.springframework</groupId>
+			<artifactId>spring-web</artifactId>
+			<version>${spring.version}</version>
+		</dependency>
+		<dependency>
+			<groupId>org.springframework</groupId>
+			<artifactId>spring-webmvc</artifactId>
+			<version>${spring.version}</version>
+		</dependency>
+		<dependency>
+			<groupId>org.springframework</groupId>
+			<artifactId>spring-test</artifactId>
+			<version>${spring.version}</version>
+			<scope>test</scope>
+		</dependency>
+
+		<dependency>
+			<groupId>org.springframework.data</groupId>
+			<artifactId>spring-data-jpa</artifactId>
+			<version>${spring.data.jpa.version}</version>
+			<exclusions>
+				<exclusion>
+					<artifactId>junit-dep</artifactId>
+					<groupId>junit</groupId>
+				</exclusion>
+			</exclusions>
+		</dependency>
+		<dependency>
+			<groupId>cglib</groupId>
+			<artifactId>cglib-nodep</artifactId>
+			<version>${cglib.version}</version>
+		</dependency>
+
+
+
+		<!-- JPA -->
+
+		<dependency>
+			<groupId>org.hibernate</groupId>
+			<artifactId>hibernate-core</artifactId>
+			<version>${hibernate.version}</version>
+		</dependency>
+		<dependency>
+			<groupId>org.hibernate</groupId>
+			<artifactId>hibernate-entitymanager</artifactId>
+			<version>${hibernate.version}</version>
+		</dependency>
+		<dependency>
+			<groupId>org.hibernate</groupId>
+			<artifactId>hibernate-validator</artifactId>
+			<version>${hibernate-validator.version}</version>
+			<scope>compile</scope>
+
+		</dependency>
+
+
+		<dependency>
+			<groupId>mysql</groupId>
+			<artifactId>mysql-connector-java</artifactId>
+			<version>${mysql.version}</version>
+			<scope>runtime</scope>
+		</dependency>
+		<dependency>
+			<groupId>com.alibaba</groupId>
+			<artifactId>druid</artifactId>
+			<version>${druid-version}</version>
+		</dependency>
+
+
+	</dependencies>
+	<build>
+		<plugins>
+			<plugin>
+				<groupId>org.apache.maven.plugins</groupId>
+				<artifactId>maven-compiler-plugin</artifactId>
+				<version>2.0.2</version>
+				<configuration>
+					<source>1.7</source>
+					<target>1.7</target>
+				</configuration>
+			</plugin>
+		</plugins>
+	</build>
+</project>

samples/Ruby/filenames/Gemfile.lock

@@ -0,0 +1,42 @@
+PATH
+  remote: .
+  specs:
+    github-linguist (4.0.1)
+      charlock_holmes (~> 0.7.3)
+      escape_utils (~> 1.0.1)
+      mime-types (>= 1.19)
+      rugged (~> 0.22.0b1)
+    github-linguist-grammars (4.0.1)
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    charlock_holmes (0.7.3)
+    coderay (1.1.0)
+    escape_utils (1.0.1)
+    metaclass (0.0.4)
+    method_source (0.8.2)
+    mime-types (2.4.3)
+    mocha (1.1.0)
+      metaclass (~> 0.0.1)
+    plist (3.1.0)
+    pry (0.10.1)
+      coderay (~> 1.1.0)
+      method_source (~> 0.8.1)
+      slop (~> 3.4)
+    rake (10.3.2)
+    rugged (0.22.0b1)
+    slop (3.6.0)
+    yajl-ruby (1.2.1)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  github-linguist!
+  github-linguist-grammars!
+  mocha
+  plist (~> 3.1)
+  pry
+  rake
+  yajl-ruby

script/download-grammars

@@ -0,0 +1,221 @@
+#!/usr/bin/env ruby
+
+require 'json'
+require 'net/http'
+require 'plist'
+require 'set'
+require 'tmpdir'
+require 'uri'
+require 'yaml'
+
+GRAMMARS_PATH = File.expand_path("../../grammars", __FILE__)
+SOURCES_FILE = File.expand_path("../../grammars.yml", __FILE__)
+CSONC = File.expand_path("../../node_modules/.bin/csonc", __FILE__)
+
+class TarballPackage
+  def self.fetch(tmp_dir, url)
+    `curl --silent --location --max-time 10 --output "#{tmp_dir}/archive" "#{url}"`
+    raise "Failed to fetch GH package: #{url} #{$?.to_s}" unless $?.success?
+
+    output = File.join(tmp_dir, 'extracted')
+    Dir.mkdir(output)
+    `tar -C "#{output}" -xf "#{tmp_dir}/archive"`
+    raise "Failed to uncompress tarball: #{tmp_dir}/archive (from #{url}) #{$?.to_s}" unless $?.success?
+
+    Dir["#{output}/**/*"].select do |path|
+      case File.extname(path.downcase)
+      when '.plist'
+        path.split('/')[-2] == 'Syntaxes'
+      when '.tmlanguage'
+        true
+      when '.cson'
+        path.split('/')[-2] == 'grammars'
+      else
+        false
+      end
+    end
+  end
+
+  attr_reader :url
+
+  def initialize(url)
+    @url = url
+  end
+
+  def fetch(tmp_dir)
+    self.class.fetch(tmp_dir, url)
+  end
+end
+
+class SingleGrammar
+  attr_reader :url
+
+  def initialize(url)
+    @url = url
+  end
+
+  def fetch(tmp_dir)
+    filename = File.join(tmp_dir, File.basename(url))
+    `curl --silent --location --max-time 10 --output "#{filename}" "#{url}"`
+    raise "Failed to fetch grammar: #{url}: #{$?.to_s}" unless $?.success?
+    [filename]
+  end
+end
+
+class SVNPackage
+  attr_reader :url
+
+  def initialize(url)
+    @url = url
+  end
+
+  def fetch(tmp_dir)
+    `svn export -q "#{url}/Syntaxes" "#{tmp_dir}/Syntaxes"`
+    raise "Failed to export SVN repository: #{url}: #{$?.to_s}" unless $?.success?
+    Dir["#{tmp_dir}/Syntaxes/*.{plist,tmLanguage,tmlanguage}"]
+  end
+end
+
+class GitHubPackage
+  def self.parse_url(url)
+    url, ref = url.split("@", 2)
+    path = URI.parse(url).path.split('/')
+    [path[1], path[2].chomp('.git'), ref || "master"]
+  end
+
+  attr_reader :user
+  attr_reader :repo
+  attr_reader :ref
+
+  def initialize(url)
+    @user, @repo, @ref = self.class.parse_url(url)
+  end
+
+  def url
+    suffix = "@#{ref}" unless ref == "master"
+    "https://github.com/#{user}/#{repo}#{suffix}"
+  end
+
+  def fetch(tmp_dir)
+    url = "https://github.com/#{user}/#{repo}/archive/#{ref}.tar.gz"
+    TarballPackage.fetch(tmp_dir, url)
+  end
+end
+
+def load_grammar(path)
+  case File.extname(path.downcase)
+  when '.plist', '.tmlanguage'
+    Plist::parse_xml(path)
+  when '.cson'
+    cson = `"#{CSONC}" "#{path}"`
+    raise "Failed to convert CSON grammar '#{path}': #{$?.to_s}" unless $?.success?
+    JSON.parse(cson)
+  else
+    raise "Invalid document type #{path}"
+  end
+end
+
+def install_grammar(tmp_dir, source, all_scopes)
+  p = if source.end_with?('.tmLanguage', '.plist')
+        SingleGrammar.new(source)
+      elsif source.start_with?('https://github.com')
+        GitHubPackage.new(source)
+      elsif source.start_with?('http://svn.textmate.org')
+        SVNPackage.new(source)
+      elsif source.end_with?('.tar.gz')
+        TarballPackage.new(source)
+      else
+        nil
+      end
+
+  raise "Unsupported source: #{source}" unless p
+
+  installed = []
+
+  p.fetch(tmp_dir).each do |path|
+    grammar = load_grammar(path)
+    scope = grammar['scopeName']
+
+    if all_scopes.key?(scope)
+      $stderr.puts "WARN: Duplicated scope #{scope}\n" +
+        "  Current package: #{p.url}\n" +
+      "  Previous package: #{all_scopes[scope]}"
+      next
+    end
+
+    File.write(File.join(GRAMMARS_PATH, "#{scope}.json"), JSON.pretty_generate(grammar))
+    all_scopes[scope] = p.url
+    installed << scope
+  end
+
+  $stderr.puts("OK #{p.url} (#{installed.join(', ')})")
+end
+
+def run_thread(queue, all_scopes)
+  Dir.mktmpdir do |tmpdir|
+    loop do
+      source, index = begin
+        queue.pop(true)
+      rescue ThreadError
+        # The queue is empty.
+        break
+      end
+
+      dir = "#{tmpdir}/#{index}"
+      Dir.mkdir(dir)
+
+      install_grammar(dir, source, all_scopes)
+    end
+  end
+end
+
+def generate_yaml(all_scopes, base)
+  yaml = all_scopes.each_with_object(base) do |(key,value),out|
+    out[value] ||= []
+    out[value] << key
+  end
+
+  yaml = yaml.sort.to_h
+  yaml.each { |k, v| v.sort! }
+  yaml
+end
+
+def main(sources)
+  begin
+    Dir.mkdir(GRAMMARS_PATH)
+  rescue Errno::EEXIST
+  end
+
+  `npm install`
+
+  all_scopes = {}
+
+  if ARGV[0] == '--add'
+    Dir.mktmpdir do |tmpdir|
+      install_grammar(tmpdir, ARGV[1], all_scopes)
+    end
+    generate_yaml(all_scopes, sources)
+  else
+    queue = Queue.new
+
+    sources.each do |url, scopes|
+      queue.push([url, queue.length])
+    end
+
+    threads = 8.times.map do
+      Thread.new { run_thread(queue, all_scopes) }
+    end
+    threads.each(&:join)
+    generate_yaml(all_scopes, {})
+  end
+end
+
+sources = File.open(SOURCES_FILE) do |file|
+  YAML.load(file)
+end
+
+yaml = main(sources)
+
+File.write(SOURCES_FILE, YAML.dump(yaml))
+
+$stderr.puts("Done")

script/prune-grammars

@@ -0,0 +1,57 @@
+#!/usr/bin/env ruby
+
+require "json"
+require "linguist"
+require "set"
+require "yaml"
+
+def find_includes(json)
+  case json
+  when Hash
+    result = []
+    if inc = json["include"]
+      result << inc.split("#", 2).first unless inc.start_with?("#", "$")
+    end
+    result + json.values.flat_map { |v| find_includes(v) }
+  when Array
+    json.flat_map { |v| find_includes(v) }
+  else
+    []
+  end
+end
+
+def transitive_includes(scope, includes)
+  scopes = Set.new
+  queue = includes[scope] || []
+  while s = queue.shift
+    next if scopes.include?(s)
+    scopes << s
+    queue += includes[s] || []
+  end
+  scopes
+end
+
+includes = {}
+Dir["grammars/*.json"].each do |path|
+  scope = File.basename(path).sub(/\.json/, '')
+  json = JSON.load(File.read(path))
+  incs = find_includes(json)
+  next if incs.empty?
+  includes[scope] ||= []
+  includes[scope] += incs
+end
+
+yaml = YAML.load(File.read("grammars.yml"))
+language_scopes = Linguist::Language.all.map(&:tm_scope).to_set
+
+# The set of used scopes is the scopes for each language, plus all the scopes
+# they include, transitively.
+used_scopes = language_scopes + language_scopes.flat_map { |s| transitive_includes(s, includes).to_a }.to_set
+
+unused = yaml.reject { |repo, scopes| scopes.any? { |scope| used_scopes.include?(scope) } }
+
+puts "Unused grammar repos"
+puts unused.map { |repo, scopes| sprintf("%-100s %s", repo, scopes.join(", ")) }.sort.join("\n")
+
+yaml.delete_if { |k| unused.key?(k) }
+File.write("grammars.yml", YAML.dump(yaml))

test/test_blob.rb

@@ -4,13 +4,10 @@ require 'linguist/samples'
 require 'test/unit'
 require 'mocha/setup'
 require 'mime/types'
-require 'pygments'
 
 class TestBlob < Test::Unit::TestCase
   include Linguist
 
-  Lexer = Pygments::Lexer
-
   def setup
     # git blobs are normally loaded as ASCII-8BIT since they may contain data
     # with arbitrary encoding not known ahead of time
@@ -196,8 +193,8 @@ class TestBlob < Test::Unit::TestCase
     assert blob("Binary/MainMenu.nib").generated?
     assert !blob("XML/project.pbxproj").generated?
 
-    # Gemfile.locks
+    # Gemfile.lock is NOT generated
-    assert blob("Gemfile.lock").generated?
+    assert !blob("Gemfile.lock").generated?
 
     # Generated .NET Docfiles
     assert blob("XML/net_docfile.xml").generated?
@@ -229,7 +226,6 @@ class TestBlob < Test::Unit::TestCase
     assert !blob("PostScript/sierpinski.ps").generated?
 
     # These examples are too basic to tell
-    assert !blob("JavaScript/empty.js").generated?
     assert !blob("JavaScript/hello.js").generated?
 
     assert blob("JavaScript/intro-old.js").generated?
@@ -301,6 +297,13 @@ class TestBlob < Test::Unit::TestCase
     assert blob("deps/http_parser/http_parser.c").vendored?
     assert blob("deps/v8/src/v8.h").vendored?
 
+    # Chart.js
+    assert blob("some/vendored/path/Chart.js").vendored?
+    assert !blob("some/vendored/path/chart.js").vendored?
+
+    # Codemirror deps
+    assert blob("codemirror/mode/blah.js").vendored?
+
     # Debian packaging
     assert blob("debian/cron.d").vendored?
 
@@ -469,24 +472,16 @@ class TestBlob < Test::Unit::TestCase
     end
   end
 
-  def test_lexer
+  def test_minified_files_not_safe_to_highlight
-    assert_equal Lexer['Ruby'], blob("Ruby/foo.rb").lexer
+    assert !blob("JavaScript/jquery-1.6.1.min.js").safe_to_colorize?
   end
 
-  def test_colorize
+  def test_empty
-    assert_equal <<-HTML.chomp, blob("Ruby/foo.rb").colorize
+    blob = Struct.new(:data) { include Linguist::BlobHelper }
-<div class="highlight"><pre><span class="k">module</span> <span class="nn">Foo</span>
-<span class="k">end</span>
-</pre></div>
-    HTML
-  end
 
-  def test_colorize_does_skip_minified_files
+    assert blob.new("").empty?
-    assert_nil blob("JavaScript/jquery-1.6.1.min.js").colorize
+    assert blob.new(nil).empty?
-  end
+    refute blob.new(" ").empty?
-
+    refute blob.new("nope").empty?
-  # Pygments.rb was taking exceeding long on this particular file
-  def test_colorize_doesnt_blow_up_with_files_with_high_ratio_of_long_lines
-    assert_nil blob("JavaScript/steelseries-min.js").colorize
   end
 end

test/test_heuristics.rb

@@ -20,18 +20,18 @@ class TestHeuristcs < Test::Unit::TestCase
     Dir.glob("#{samples_path}/#{language_name}/#{file}")
   end
 
+  # Candidate languages = ["C++", "Objective-C"]
   def test_obj_c_by_heuristics
-    languages = ["C++", "Objective-C"]
     # Only calling out '.h' filenames as these are the ones causing issues
     all_fixtures("Objective-C", "*.h").each do |fixture|
-      results = Heuristics.disambiguate_c(fixture("Objective-C/#{File.basename(fixture)}"), languages)
+      results = Heuristics.disambiguate_c(fixture("Objective-C/#{File.basename(fixture)}"))
       assert_equal Language["Objective-C"], results.first
     end
   end
 
+  # Candidate languages = ["C++", "Objective-C"]
   def test_cpp_by_heuristics
-    languages = ["C++", "Objective-C"]
+    results = Heuristics.disambiguate_c(fixture("C++/render_adapter.cpp"))
-    results = Heuristics.disambiguate_c(fixture("C++/render_adapter.cpp"), languages)
     assert_equal Language["C++"], results.first
   end
 
@@ -41,57 +41,57 @@ class TestHeuristcs < Test::Unit::TestCase
     assert_equal Language["Objective-C"], match
   end
 
+  # Candidate languages = ["Perl", "Prolog"]
   def test_pl_prolog_by_heuristics
-    languages = ["Perl", "Prolog"]
+    results = Heuristics.disambiguate_pl(fixture("Prolog/turing.pl"))
-    results = Heuristics.disambiguate_pl(fixture("Prolog/turing.pl"), languages)
     assert_equal Language["Prolog"], results.first
   end
 
+  # Candidate languages = ["Perl", "Prolog"]
   def test_pl_perl_by_heuristics
-    languages = ["Perl", "Prolog"]
+    results = Heuristics.disambiguate_pl(fixture("Perl/perl-test.t"))
-    results = Heuristics.disambiguate_pl(fixture("Perl/perl-test.t"), languages)
     assert_equal Language["Perl"], results.first
   end
 
+  # Candidate languages = ["ECL", "Prolog"]
   def test_ecl_prolog_by_heuristics
-    languages = ["ECL", "Prolog"]
+    results = Heuristics.disambiguate_ecl(fixture("Prolog/or-constraint.ecl"))
-    results = Heuristics.disambiguate_ecl(fixture("Prolog/or-constraint.ecl"), languages)
     assert_equal Language["Prolog"], results.first
   end
 
+  # Candidate languages = ["ECL", "Prolog"]
   def test_ecl_ecl_by_heuristics
-    languages = ["ECL", "Prolog"]
+    results = Heuristics.disambiguate_ecl(fixture("ECL/sample.ecl"))
-    results = Heuristics.disambiguate_ecl(fixture("ECL/sample.ecl"), languages)
     assert_equal Language["ECL"], results.first
   end
 
+  # Candidate languages = ["IDL", "Prolog"]
   def test_pro_prolog_by_heuristics
-    languages = ["IDL", "Prolog"]
+    results = Heuristics.disambiguate_pro(fixture("Prolog/logic-problem.pro"))
-    results = Heuristics.disambiguate_pro(fixture("Prolog/logic-problem.pro"), languages)
     assert_equal Language["Prolog"], results.first
   end
 
+  # Candidate languages = ["IDL", "Prolog"]
   def test_pro_idl_by_heuristics
-    languages = ["IDL", "Prolog"]
+    results = Heuristics.disambiguate_pro(fixture("IDL/mg_acosh.pro"))
-    results = Heuristics.disambiguate_pro(fixture("IDL/mg_acosh.pro"), languages)
     assert_equal Language["IDL"], results.first
   end
 
+  # Candidate languages = ["AGS Script", "AsciiDoc"]
   def test_asc_asciidoc_by_heuristics
-    languages = ["AGS Script", "AsciiDoc"]
+    results = Heuristics.disambiguate_asc(fixture("AsciiDoc/list.asc"))
-    results = Heuristics.disambiguate_asc(fixture("AsciiDoc/list.asc"), languages)
     assert_equal Language["AsciiDoc"], results.first
   end
 
+  # Candidate languages = ["TypeScript", "XML"]
   def test_ts_typescript_by_heuristics
-    languages = ["TypeScript", "XML"]
+    results = Heuristics.disambiguate_ts(fixture("TypeScript/classes.ts"))
-    results = Heuristics.disambiguate_ts(fixture("TypeScript/classes.ts"), languages)
     assert_equal Language["TypeScript"], results.first
   end
 
+  # Candidate languages = ["TypeScript", "XML"]
   def test_ts_xml_by_heuristics
-    languages = ["TypeScript", "XML"]
+    results = Heuristics.disambiguate_ts(fixture("XML/pt_BR.xml"))
-    results = Heuristics.disambiguate_ts(fixture("XML/pt_BR.xml"), languages)
     assert_equal Language["XML"], results.first
   end
 
@@ -99,27 +99,37 @@ class TestHeuristcs < Test::Unit::TestCase
     languages = ["Common Lisp", "OpenCL"]
     languages.each do |language|
       all_fixtures(language).each do |fixture|
-        results = Heuristics.disambiguate_cl(fixture("#{language}/#{File.basename(fixture)}"), languages)
+        results = Heuristics.disambiguate_cl(fixture("#{language}/#{File.basename(fixture)}"))
         assert_equal Language[language], results.first
       end
     end
   end
 
+  def test_f_by_heuristics
+    languages = ["FORTRAN", "Forth"]
+    languages.each do |language|
+      all_fixtures(language).each do |fixture|
+        results = Heuristics.disambiguate_f(fixture("#{language}/#{File.basename(fixture)}"))
+        assert_equal Language[language], results.first
+      end
+    end
+  end
+
+  # Candidate languages = ["Hack", "PHP"]
   def test_hack_by_heuristics
-    languages = ["Hack", "PHP"]
+    results = Heuristics.disambiguate_hack(fixture("Hack/funs.php"))
-    results = Heuristics.disambiguate_hack(fixture("Hack/funs.php"), languages)
     assert_equal Language["Hack"], results.first
   end
 
+  # Candidate languages = ["Scala", "SuperCollider"]
   def test_sc_supercollider_by_heuristics
-    languages = ["Scala", "SuperCollider"]
+    results = Heuristics.disambiguate_sc(fixture("SuperCollider/WarpPreset.sc"))
-    results = Heuristics.disambiguate_sc(fixture("SuperCollider/WarpPreset.sc"), languages)
     assert_equal Language["SuperCollider"], results.first
   end
 
+  # Candidate languages = ["Scala", "SuperCollider"]
   def test_sc_scala_by_heuristics
-    languages = ["Scala", "SuperCollider"]
+    results = Heuristics.disambiguate_sc(fixture("Scala/node11.sc"))
-    results = Heuristics.disambiguate_sc(fixture("Scala/node11.sc"), languages)
     assert_equal Language["Scala"], results.first
   end
 end

test/test_language.rb

@@ -1,64 +1,10 @@
 require 'linguist/language'
-
 require 'test/unit'
-require 'pygments'
+require 'yaml'
 
 class TestLanguage < Test::Unit::TestCase
   include Linguist
 
-  Lexer = Pygments::Lexer
-
-  def test_lexer
-    assert_equal Lexer['ActionScript 3'], Language['ActionScript'].lexer
-    assert_equal Lexer['AspectJ'], Language['AspectJ'].lexer
-    assert_equal Lexer['Bash'], Language['Gentoo Ebuild'].lexer
-    assert_equal Lexer['Bash'], Language['Gentoo Eclass'].lexer
-    assert_equal Lexer['Bash'], Language['Shell'].lexer
-    assert_equal Lexer['C'], Language['OpenCL'].lexer
-    assert_equal Lexer['C'], Language['XS'].lexer
-    assert_equal Lexer['C++'], Language['C++'].lexer
-    assert_equal Lexer['Chapel'], Language['Chapel'].lexer
-    assert_equal Lexer['Coldfusion HTML'], Language['ColdFusion'].lexer
-    assert_equal Lexer['Coq'], Language['Coq'].lexer
-    assert_equal Lexer['FSharp'], Language['F#'].lexer
-    assert_equal Lexer['FSharp'], Language['F#'].lexer
-    assert_equal Lexer['Fortran'], Language['FORTRAN'].lexer
-    assert_equal Lexer['Gherkin'], Language['Cucumber'].lexer
-    assert_equal Lexer['Groovy'], Language['Groovy'].lexer
-    assert_equal Lexer['HTML'], Language['HTML'].lexer
-    assert_equal Lexer['HTML+Django/Jinja'], Language['HTML+Django'].lexer
-    assert_equal Lexer['HTML+PHP'], Language['HTML+PHP'].lexer
-    assert_equal Lexer['HTTP'], Language['HTTP'].lexer
-    assert_equal Lexer['JSON'], Language['JSON'].lexer
-    assert_equal Lexer['Java'], Language['ChucK'].lexer
-    assert_equal Lexer['Java'], Language['Java'].lexer
-    assert_equal Lexer['JavaScript'], Language['JavaScript'].lexer
-    assert_equal Lexer['LSL'], Language['LSL'].lexer
-    assert_equal Lexer['MOOCode'], Language['Moocode'].lexer
-    assert_equal Lexer['MuPAD'], Language['mupad'].lexer
-    assert_equal Lexer['NASM'], Language['Assembly'].lexer
-    assert_equal Lexer['OCaml'], Language['OCaml'].lexer
-    assert_equal Lexer['Ooc'], Language['ooc'].lexer
-    assert_equal Lexer['OpenEdge ABL'], Language['OpenEdge ABL'].lexer
-    assert_equal Lexer['REBOL'], Language['Rebol'].lexer
-    assert_equal Lexer['RHTML'], Language['HTML+ERB'].lexer
-    assert_equal Lexer['RHTML'], Language['RHTML'].lexer
-    assert_equal Lexer['Ruby'], Language['Crystal'].lexer
-    assert_equal Lexer['Ruby'], Language['Mirah'].lexer
-    assert_equal Lexer['Ruby'], Language['Ruby'].lexer
-    assert_equal Lexer['S'], Language['R'].lexer
-    assert_equal Lexer['Scheme'], Language['Nu'].lexer
-    assert_equal Lexer['Racket'], Language['Racket'].lexer
-    assert_equal Lexer['Scheme'], Language['Scheme'].lexer
-    assert_equal Lexer['Standard ML'], Language['Standard ML'].lexer
-    assert_equal Lexer['TeX'], Language['TeX'].lexer
-    assert_equal Lexer['Verilog'], Language['Verilog'].lexer
-    assert_equal Lexer['XSLT'], Language['XSLT'].lexer
-    assert_equal Lexer['aspx-vb'], Language['ASP'].lexer
-    assert_equal Lexer['haXe'], Language['Haxe'].lexer
-    assert_equal Lexer['reStructuredText'], Language['reStructuredText'].lexer
-  end
-
   def test_find_by_alias
     assert_equal Language['ASP'], Language.find_by_alias('asp')
     assert_equal Language['ASP'], Language.find_by_alias('aspx')
@@ -119,6 +65,7 @@ class TestLanguage < Test::Unit::TestCase
     assert_equal Language['VimL'], Language.find_by_alias('viml')
     assert_equal Language['reStructuredText'], Language.find_by_alias('rst')
     assert_equal Language['YAML'], Language.find_by_alias('yml')
+    assert_nil Language.find_by_alias(nil)
   end
 
   def test_groups
@@ -193,6 +140,7 @@ class TestLanguage < Test::Unit::TestCase
     assert_equal :programming, Language['Python'].type
     assert_equal :programming, Language['Ruby'].type
     assert_equal :programming, Language['TypeScript'].type
+    assert_equal :programming, Language['Makefile'].type
   end
 
   def test_markup
@@ -211,7 +159,6 @@ class TestLanguage < Test::Unit::TestCase
 
   def test_other
     assert_nil Language['Brainfuck'].type
-    assert_nil Language['Makefile'].type
   end
 
   def test_searchable
@@ -221,6 +168,7 @@ class TestLanguage < Test::Unit::TestCase
   end
 
   def test_find_by_name
+    assert_nil Language.find_by_name(nil)
     ruby = Language['Ruby']
     assert_equal ruby, Language.find_by_name('Ruby')
   end
@@ -317,6 +265,7 @@ class TestLanguage < Test::Unit::TestCase
     assert_equal 'C#', Language['c#'].name
     assert_equal 'C#', Language['csharp'].name
     assert_nil Language['defunkt']
+    assert_nil Language[nil]
   end
 
   def test_find_ignores_case
@@ -401,12 +350,6 @@ class TestLanguage < Test::Unit::TestCase
     assert_equal '.coffee', Language['CoffeeScript'].primary_extension
     assert_equal '.t', Language['Turing'].primary_extension
     assert_equal '.ts', Language['TypeScript'].primary_extension
-
-    # This is a nasty requirement, but there's some code in GitHub that
-    # expects this. Really want to drop this.
-    Language.all.each do |language|
-      assert language.primary_extension, "#{language} has no primary extension"
-    end
   end
 
   def test_eql
@@ -418,21 +361,14 @@ class TestLanguage < Test::Unit::TestCase
     assert !Language.by_type(:prose).nil?
   end
 
-  def test_colorize
+  def test_all_languages_have_grammars
-    assert_equal <<-HTML.chomp, Language['Ruby'].colorize("def foo\n  'foo'\nend\n")
+    scopes = YAML.load(File.read(File.expand_path("../../grammars.yml", __FILE__))).values.flatten
-<div class="highlight"><pre><span class="k">def</span> <span class="nf">foo</span>
+    missing = Language.all.reject { |language| language.tm_scope == "none" || scopes.include?(language.tm_scope) }
-  <span class="s1">&#39;foo&#39;</span>
+    message = "The following languages' scopes are not listed in grammars.yml. Please add grammars for all new languages.\n"
-<span class="k">end</span>
+    message << "If no grammar exists for a language, mark the language with `tm_scope: none` in lib/linguist/languages.yml.\n"
-</pre></div>
-    HTML
-  end
 
-  def test_colorize_with_options
+    width = missing.map { |language| language.name.length }.max
-    assert_equal <<-HTML.chomp, Language['Ruby'].colorize("def foo\n  'foo'\nend\n", :options => { :cssclass => "highlight highlight-ruby" })
+    message << missing.map { |language| sprintf("%-#{width}s %s", language.name, language.tm_scope) }.sort.join("\n")
-<div class="highlight highlight-ruby"><pre><span class="k">def</span> <span class="nf">foo</span>
+    assert missing.empty?, message
-  <span class="s1">&#39;foo&#39;</span>
-<span class="k">end</span>
-</pre></div>
-    HTML
   end
 end

test/test_pedantic.rb

@@ -1,57 +1,30 @@
 require 'test/unit'
+require 'yaml'
 
 class TestPedantic < Test::Unit::TestCase
-  Lib = File.expand_path("../../lib/linguist", __FILE__)
+  filename = File.expand_path("../../lib/linguist/languages.yml", __FILE__)
-
+  LANGUAGES = YAML.load(File.read(filename))
-  def file(name)
-    File.read(File.join(Lib, name))
-  end
 
   def test_language_names_are_sorted
-    languages = []
+    assert_sorted LANGUAGES.keys
-    file("languages.yml").lines.each do |line|
-      if line =~ /^(\w+):$/
-        languages << $1
-      end
-    end
-    assert_sorted languages
   end
 
   def test_extensions_are_sorted
-    extensions = nil
+    LANGUAGES.each do |name, language|
-    file("languages.yml").lines.each do |line|
+      extensions = language['extensions']
-      if line =~ /^  extensions:$/
+      assert_sorted extensions[1..-1] if extensions && extensions.size > 1
-        extensions = []
-      elsif extensions && line =~ /^  - \.([\w-]+)( *#.*)?$/
-        extensions << $1
-      else
-        assert_sorted extensions[1..-1] if extensions
-        extensions = nil
-      end
     end
   end
 
   def test_filenames_are_sorted
-    filenames = nil
+    LANGUAGES.each do |name, language|
-    file("languages.yml").lines.each do |line|
+      assert_sorted language['filenames'] if language['filenames']
-      if line =~ /^  filenames:$/
-        filenames = []
-      elsif filenames && line =~ /^  - \.(\w+)$/
-        filenames << $1
-      else
-        assert_sorted filenames if filenames
-        filenames = nil
-      end
     end
   end
 
   def assert_sorted(list)
-    previous = nil
+    list.each_cons(2) do |previous, item|
-    list.each do |item|
+      flunk "#{previous} should come after #{item}" if previous > item
-      if previous && previous > item
-        flunk "#{previous} should come after #{item}"
-      end
-      previous = item
     end
   end
 end

test/test_samples.rb

@@ -55,20 +55,28 @@ class TestSamples < Test::Unit::TestCase
   end
 
   # If a language extension isn't globally unique then make sure there are samples
-  def test_presence
+  Linguist::Language.all.each do |language|
-    Linguist::Language.all.each do |language|
+    define_method "test_#{language.name}_has_samples" do
-      language.all_extensions.each do |extension|
+      language.extensions.each do |extension|
-        language_matches = Language.find_by_filename("foo#{extension}")
+        language_matches = Language.find_by_extension(extension)
 
-        # If there is more than one language match for a given extension
+        # Check for samples if more than one language matches the given extension.
-        # then check that there are examples for that language with the extension
         if language_matches.length > 1
-          language_matches.each do |language|
+          language_matches.each do |match|
-            assert File.directory?("samples/#{language.name}"), "#{language.name} is missing a samples directory"
+            samples = "samples/#{match.name}/*#{extension}"
-            assert Dir.glob("samples/#{language.name}/*#{extension}").any?, "#{language.name} is missing samples for extension #{extension}"
+            assert Dir.glob(samples).any?, "Missing samples in #{samples.inspect}. See https://github.com/github/linguist/blob/master/CONTRIBUTING.md"
           end
         end
       end
+
+      language.filenames.each do |filename|
+        # Check for samples if more than one language matches the given filename
+        if Language.find_by_filename(filename).size > 1
+          sample = "samples/#{language.name}/filenames/#{filename}"
+          assert File.exists?(sample),
+            "Missing sample in #{sample.inspect}. See https://github.com/github/linguist/blob/master/CONTRIBUTING.md"
+        end
+      end
     end
   end
 end