Merge remote-tracking branch 'origin/master' into test-helper

* origin/master: (31 commits)
  Link to Lightshow in CONTRIBUTING.md
  Switch to a better F# grammar
  Bump Rugged again
  Checkout the master for testing
  Rugged 0.22.0b3
  Reordering
  Bump version to 4.0.3
  Add some docs for tm_scope
  Change NONE to none
  Checking other case for Chart.jS
  Test that all languages have grammars
  Fix RHTML's tm_scope
  Chart JS is vendored
  Switch to a better grammar for Bro
  reorder again…
  put cjsx at the top
  Use a SQF grammar for SQF files
  move cjsx before iced
  move cjsx before iced
  change component name
  ...

Conflicts:
	test/test_language.rb

.travis.yml

@@ -2,6 +2,7 @@ before_install:
   - git fetch origin master:master
   - git fetch origin v2.0.0:v2.0.0
   - git fetch origin test/attributes:test/attributes
+  - git fetch origin test/master:test/master
   - sudo apt-get install libicu-dev -y
 rvm:
   - 1.9.3

CONTRIBUTING.md

@@ -12,7 +12,7 @@ This can usually be solved either by adding a new filename or file name extensio
 
 Assuming your code is being detected as the right language (see above), in most cases this is due to a bug in the language grammar rather than a bug in Linguist. [`grammars.yml`][grammars] lists all the grammars we use for syntax highlighting on github.com. Find the one corresponding to your code's programming language and submit a bug report upstream.
 
-You can also try to fix the bug yourself and submit a Pull Request. [This piece from TextMate's documentation](http://manual.macromates.com/en/language_grammars) offers a good introduction on how to work with TextMate-compatible grammars.
+You can also try to fix the bug yourself and submit a Pull Request. [This piece from TextMate's documentation](http://manual.macromates.com/en/language_grammars) offers a good introduction on how to work with TextMate-compatible grammars. You can test grammars using [Lightshow](https://lightshow.githubapp.com).
 
 Once the bug has been fixed upstream, please let us know and we'll pick it up for GitHub.
 

github-linguist.gemspec

@@ -16,7 +16,7 @@ Gem::Specification.new do |s|
   s.add_dependency 'charlock_holmes', '~> 0.7.3'
   s.add_dependency 'escape_utils',    '~> 1.0.1'
   s.add_dependency 'mime-types',      '>= 1.19'
-  s.add_dependency 'rugged',          '~> 0.22.0b1'
+  s.add_dependency 'rugged',          '~> 0.22.0b4'
 
   s.add_development_dependency 'mocha'
   s.add_development_dependency 'pry'

grammars.yml

@@ -5,8 +5,6 @@ http://svn.textmate.org/trunk/Review/Bundles/BlitzMax.tmbundle:
 - source.blitzmax
 http://svn.textmate.org/trunk/Review/Bundles/Cython.tmbundle:
 - source.cython
-http://svn.textmate.org/trunk/Review/Bundles/F%20Sharp.tmbundle:
-- source.fsharp
 http://svn.textmate.org/trunk/Review/Bundles/Forth.tmbundle:
 - source.forth
 http://svn.textmate.org/trunk/Review/Bundles/Parrot.tmbundle:
@@ -38,6 +36,8 @@ https://github.com/Drako/SublimeBrainfuck/raw/master/Brainfuck.tmLanguage:
 - source.bf
 https://github.com/JohnNilsson/awk-sublime/raw/master/AWK.tmLanguage:
 - source.awk
+https://github.com/JonBons/Sublime-SQF-Language:
+- source.sqf
 https://github.com/MarioRicalde/SCSS.tmbundle:
 - source.scss
 https://github.com/Oldes/Sublime-REBOL:
@@ -97,6 +97,8 @@ https://github.com/bholt/chapel-tmbundle:
 - source.chapel
 https://github.com/brandonwamboldt/sublime-nginx:
 - source.nginx
+https://github.com/bro/bro-sublime:
+- source.bro
 https://github.com/carsonoid/sublime_man_page_support/raw/master/man-groff.tmLanguage:
 - text.groff
 https://github.com/ccreutzig/sublime-MuPAD:
@@ -131,6 +133,8 @@ https://github.com/euler0/sublime-glsl/raw/master/GLSL.tmLanguage:
 - source.glsl
 https://github.com/fancy-lang/fancy-tmbundle:
 - source.fancy
+https://github.com/fsharp/fsharpbinding:
+- source.fsharp
 https://github.com/gingerbeardman/monkey.tmbundle:
 - source.monkey
 https://github.com/guillermooo/dart-sublime-bundle/raw/master/Dart.tmLanguage:
@@ -215,9 +219,6 @@ https://github.com/slavapestov/factor/raw/master/misc/Factor.tmbundle/Syntaxes/F
 - source.factor
 https://github.com/slim-template/ruby-slim.tmbundle:
 - text.slim
-https://github.com/smiledawgg/Bro.tmbundle:
-- source.bro
-- source.bro.sig
 https://github.com/staltz/SublimeXtend:
 - source.xtend
 https://github.com/statatmbundle/Stata.tmbundle:
@@ -242,6 +243,7 @@ https://github.com/textmate/bison.tmbundle:
 https://github.com/textmate/c.tmbundle:
 - source.c
 - source.c++
+- source.c.platform
 https://github.com/textmate/capnproto.tmbundle:
 - source.capnp
 https://github.com/textmate/cmake.tmbundle:
@@ -327,6 +329,7 @@ https://github.com/textmate/ninja.tmbundle:
 https://github.com/textmate/objective-c.tmbundle:
 - source.objc
 - source.objc++
+- source.objc.platform
 - source.strings
 https://github.com/textmate/ocaml.tmbundle:
 - source.camlp4.ocaml

lib/linguist/languages.yml

@@ -12,6 +12,10 @@
 # search_term       - Deprecated: Some languages maybe indexed under a
 #                     different alias. Avoid defining new exceptions.
 # color             - CSS hex color to represent the language.
+# tm_scope          - The TextMate scope that represents this programming
+#                     language. This should match one of the scopes listed in
+#                     the grammars.yml file. Use "none" if there is no grammar
+#                     for this language.
 #
 # Any additions or modifications (even trivial) should have corresponding
 # test change in `test/test_blob.rb`.
@@ -45,6 +49,7 @@ APL:
   extensions:
   - .apl
   - .dyalog
+  tm_scope: none
 
 ASP:
   type: programming
@@ -146,6 +151,7 @@ Arc:
   color: "#ca2afe"
   extensions:
   - .arc
+  tm_scope: none
 
 Arduino:
   type: programming
@@ -162,12 +168,14 @@ AsciiDoc:
   - .asciidoc
   - .adoc
   - .asc
+  tm_scope: none
 
 AspectJ:
   type: programming
   color: "#1957b0"
   extensions:
   - .aj
+  tm_scope: none
 
 Assembly:
   type: programming
@@ -185,6 +193,7 @@ Augeas:
   type: programming
   extensions:
   - .aug
+  tm_scope: none
 
 AutoHotkey:
   type: programming
@@ -194,6 +203,7 @@ AutoHotkey:
   extensions:
   - .ahk
   - .ahkl
+  tm_scope: none
 
 AutoIt:
   type: programming
@@ -286,6 +296,7 @@ Brightscript:
   type: programming
   extensions:
   - .brs
+  tm_scope: none
 
 Bro:
   type: programming
@@ -359,6 +370,7 @@ CLIPS:
   type: programming
   extensions:
   - .clp
+  tm_scope: none
 
 CMake:
   extensions:
@@ -421,6 +433,7 @@ Clean:
   extensions:
   - .icl
   - .dcl
+  tm_scope: none
 
 Clojure:
   type: programming
@@ -449,6 +462,7 @@ CoffeeScript:
   extensions:
   - .coffee
   - ._coffee
+  - .cjsx
   - .cson
   - .iced
   filenames:
@@ -537,6 +551,7 @@ Creole:
   wrap: true
   extensions:
   - .creole
+  tm_scope: none
 
 Crystal:
   type: programming
@@ -604,6 +619,7 @@ Darcs Patch:
   extensions:
   - .darcspatch
   - .dpatch
+  tm_scope: none
 
 Dart:
   type: programming
@@ -631,6 +647,7 @@ Dogescript:
   color: "#cca760"
   extensions:
   - .djs
+  tm_scope: none
 
 Dylan:
   type: programming
@@ -646,6 +663,7 @@ E:
   color: "#ccce35"
   extensions:
   - .E
+  tm_scope: none
 
 ECL:
   type: programming
@@ -653,6 +671,7 @@ ECL:
   extensions:
   - .ecl
   - .eclxml
+  tm_scope: none
 
 Eagle:
   type: markup
@@ -736,6 +755,7 @@ FLUX:
   extensions:
   - .fx
   - .flux
+  tm_scope: none
 
 FORTRAN:
   type: programming
@@ -809,11 +829,13 @@ G-code:
   - .g
   - .gco
   - .gcode
+  tm_scope: none
 
 GAMS:
   type: programming
   extensions:
   - .gms
+  tm_scope: none
 
 GAP:
   type: programming
@@ -822,6 +844,7 @@ GAP:
   - .gap
   - .gd
   - .gi
+  tm_scope: none
 
 GAS:
   type: programming
@@ -835,6 +858,7 @@ GDScript:
   type: programming
   extensions:
   - .gd
+  tm_scope: none
 
 GLSL:
   group: C
@@ -919,6 +943,7 @@ Golo:
   color: "#f6a51f"
   extensions:
   - .golo
+  tm_scope: none
 
 Gosu:
   type: programming
@@ -934,6 +959,7 @@ Grace:
   type: programming
   extensions:
   - .grace
+  tm_scope: none
 
 Grammatical Framework:
   type: programming
@@ -950,6 +976,7 @@ Graph Modeling Language:
   type: data
   extensions:
   - .gml
+  tm_scope: none
 
 Graphviz (DOT):
   type: data
@@ -1004,6 +1031,7 @@ HTML:
   - .html
   - .htm
   - .st
+  - .xht
   - .xhtml
 
 HTML+Django:
@@ -1039,6 +1067,7 @@ HTTP:
   type: data
   extensions:
   - .http
+  tm_scope: none
 
 Hack:
   type: programming
@@ -1046,6 +1075,7 @@ Hack:
   extensions:
   - .hh
   - .php
+  tm_scope: none
 
 Haml:
   group: HTML
@@ -1068,6 +1098,7 @@ Harbour:
   color: "#0e60e3"
   extensions:
   - .hb
+  tm_scope: none
 
 Haskell:
   type: programming
@@ -1093,6 +1124,7 @@ Hy:
   - .hy
   aliases:
   - hylang
+  tm_scope: none
 
 IDL:
   type: programming
@@ -1108,6 +1140,7 @@ IGOR Pro:
   aliases:
   - igor
   - igorpro
+  tm_scope: none
 
 INI:
   type: data
@@ -1128,6 +1161,7 @@ IRC log:
   extensions:
   - .irclog
   - .weechatlog
+  tm_scope: none
 
 Idris:
   type: programming
@@ -1149,6 +1183,7 @@ Inform 7:
 Inno Setup:
   extensions:
   - .iss
+  tm_scope: none
 
 Io:
   type: programming
@@ -1167,11 +1202,13 @@ Isabelle:
   color: "#fdcd00"
   extensions:
   - .thy
+  tm_scope: none
 
 J:
   type: programming
   extensions:
   - .ijs
+  tm_scope: none
 
 JSON:
   type: data
@@ -1279,6 +1316,7 @@ KRL:
   color: "#f5c800"
   extensions:
   - .krl
+  tm_scope: none
 
 Kit:
   type: markup
@@ -1312,6 +1350,7 @@ LOLCODE:
   extensions:
   - .lol
   color: "#cc9900"
+  tm_scope: none
 
 LSL:
   type: programming
@@ -1326,6 +1365,7 @@ LabVIEW:
   type: programming
   extensions:
   - .lvproj
+  tm_scope: none
 
 Lasso:
   type: programming
@@ -1364,12 +1404,14 @@ Liquid:
   type: markup
   extensions:
   - .liquid
+  tm_scope: none
 
 Literate Agda:
   type: programming
   group: Agda
   extensions:
   - .lagda
+  tm_scope: none
 
 Literate CoffeeScript:
   type: programming
@@ -1546,6 +1588,7 @@ MediaWiki:
   wrap: true
   extensions:
   - .mediawiki
+  tm_scope: none
 
 Mercury:
   type: programming
@@ -1560,6 +1603,7 @@ MiniD: # Legacy
   searchable: false
   extensions:
   - .minid # Dummy extension
+  tm_scope: none
 
 Mirah:
   type: programming
@@ -1581,6 +1625,7 @@ Moocode:
   type: programming
   extensions:
   - .moo
+  tm_scope: none
 
 MoonScript:
   type: programming
@@ -1592,6 +1637,7 @@ MoonScript:
 Myghty:
   extensions:
   - .myt
+  tm_scope: none
 
 NSIS:
   extensions:
@@ -1637,6 +1683,7 @@ Nit:
   color: "#0d8921"
   extensions:
   - .nit
+  tm_scope: none
 
 Nix:
   type: programming
@@ -1644,6 +1691,7 @@ Nix:
   - .nix
   aliases:
   - nixos
+  tm_scope: none
 
 Nu:
   type: programming
@@ -1662,6 +1710,7 @@ NumPy:
   - .numpy
   - .numpyw
   - .numsc
+  tm_scope: none
 
 OCaml:
   type: programming
@@ -1722,6 +1771,7 @@ Omgrofl:
   extensions:
   - .omgrofl
   color: "#cabbff"
+  tm_scope: none
 
 Opa:
   type: programming
@@ -1733,6 +1783,7 @@ Opal:
   color: "#f7ede0"
   extensions:
   - .opal
+  tm_scope: none
 
 OpenCL:
   type: programming
@@ -1757,12 +1808,14 @@ OpenSCAD:
   type: programming
   extensions:
   - .scad
+  tm_scope: none
 
 Org:
   type: prose
   wrap: true
   extensions:
   - .org
+  tm_scope: none
 
 Ox:
   type: programming
@@ -1770,12 +1823,14 @@ Ox:
   - .ox
   - .oxh
   - .oxo
+  tm_scope: none
 
 Oxygene:
   type: programming
   color: "#5a63a3"
   extensions:
   - .oxygene
+  tm_scope: none
 
 PAWN:
   type: programming
@@ -1811,18 +1866,21 @@ Pan:
   color: '#cc0000'
   extensions:
   - .pan
+  tm_scope: none
 
 Papyrus:
   type: programming
   color: "#6600cc"
   extensions:
   - .psc
+  tm_scope: none
 
 Parrot:
   type: programming
   color: "#f3ca0a"
   extensions:
   - .parrot # Dummy extension
+  tm_scope: none
 
 Parrot Assembly:
   group: Parrot
@@ -1831,6 +1889,7 @@ Parrot Assembly:
   - pasm
   extensions:
   - .pasm
+  tm_scope: none
 
 Parrot Internal Representation:
   group: Parrot
@@ -1882,12 +1941,14 @@ Perl6:
   - .p6m
   - .pl6
   - .pm6
+  tm_scope: none
 
 PigLatin:
   type: programming
   color: "#fcd7de"
   extensions:
   - .pig
+  tm_scope: none
 
 Pike:
   type: programming
@@ -1902,12 +1963,14 @@ Pod:
   wrap: true
   extensions:
   - .pod
+  tm_scope: none
 
 PogoScript:
   type: programming
   color: "#d80074"
   extensions:
   - .pogo
+  tm_scope: none
 
 PostScript:
   type: markup
@@ -1948,6 +2011,7 @@ Propeller Spin:
   color: "#2b446d"
   extensions:
   - .spin
+  tm_scope: none
 
 Protocol Buffer:
   type: markup
@@ -1971,6 +2035,7 @@ Pure Data:
   color: "#91de79"
   extensions:
   - .pd
+  tm_scope: none
 
 PureScript:
   type: programming
@@ -2066,7 +2131,7 @@ RHTML:
   group: HTML
   extensions:
   - .rhtml
-  tm_scope: text.html.ruby
+  tm_scope: text.html.erb
   aliases:
   - html+ruby
 
@@ -2077,6 +2142,7 @@ RMarkdown:
   extensions:
   - .rmd
   - .Rmd
+  tm_scope: none
 
 Racket:
   type: programming
@@ -2096,6 +2162,7 @@ Ragel in Ruby Host:
   aliases:
   - ragel-rb
   - ragel-ruby
+  tm_scope: none
 
 Raw token data:
   search_term: raw
@@ -2103,6 +2170,7 @@ Raw token data:
   - raw
   extensions:
   - .raw
+  tm_scope: none
 
 Rebol:
   type: programming
@@ -2122,10 +2190,12 @@ Red:
   - .reds
   aliases:
   - red/system
+  tm_scope: none
 
 Redcode:
   extensions:
   - .cw
+  tm_scope: none
 
 RobotFramework:
   type: programming
@@ -2199,6 +2269,7 @@ SAS:
   color: "#1E90FF"
   extensions:
   - .sas
+  tm_scope: none
 
 SCSS:
   type: markup
@@ -2214,7 +2285,7 @@ SQF:
   extensions:
   - .sqf
   - .hqf
-  tm_scope: source.c++
+  tm_scope: source.sqf
 
 SQL:
   type: data
@@ -2292,6 +2363,7 @@ Self:
   color: "#0579aa"
   extensions:
   - .self
+  tm_scope: none
 
 Shell:
   type: programming
@@ -2330,6 +2402,7 @@ Shen:
   color: "#120F14"
   extensions:
   - .shen
+  tm_scope: none
 
 Slash:
   type: programming
@@ -2400,6 +2473,7 @@ Stylus:
   group: CSS
   extensions:
   - .styl
+  tm_scope: none
 
 SuperCollider:
   type: programming
@@ -2407,6 +2481,7 @@ SuperCollider:
   extensions:
   - .scd
   - .sc
+  tm_scope: none
 
 Swift:
   type: programming
@@ -2432,6 +2507,7 @@ TXL:
   type: programming
   extensions:
   - .txl
+  tm_scope: none
 
 Tcl:
   type: programming
@@ -2488,6 +2564,7 @@ Textile:
   wrap: true
   extensions:
   - .textile
+  tm_scope: none
 
 Thrift:
   type: programming
@@ -2501,6 +2578,7 @@ Turing:
   extensions:
   - .t
   - .tu
+  tm_scope: none
 
 Twig:
   type: markup
@@ -2761,6 +2839,7 @@ Zimpl:
   - .zimpl
   - .zmpl
   - .zpl
+  tm_scope: none
 
 eC:
   type: programming
@@ -2768,6 +2847,7 @@ eC:
   extensions:
   - .ec
   - .eh
+  tm_scope: none
 
 edn:
   type: data
@@ -2782,6 +2862,7 @@ fish:
   group: Shell
   extensions:
   - .fish
+  tm_scope: none
 
 mupad:
   extensions:
@@ -2822,3 +2903,4 @@ xBase:
   color: "#3a4040"
   extensions:
   - .prg
+  tm_scope: none

lib/linguist/vendor.yml

@@ -110,6 +110,12 @@
 # MathJax
 - (^|/)MathJax/
 
+# Chart.js
+- (^|/)Chart\.js$
+
+# Codemirror
+- (^|/)[Cc]ode[Mm]irror/(lib|mode|theme|addon|keymap)
+
 # SyntaxHighlighter - http://alexgorbatchev.com/
 - (^|/)shBrush([^.]*)\.js$
 - (^|/)shCore\.js$

lib/linguist/version.rb

@@ -1,3 +1,3 @@
 module Linguist
-  VERSION = "4.0.1"
+  VERSION = "4.0.3"
 end

samples/CoffeeScript/example.cjsx

@@ -0,0 +1,40 @@
+###* @cjsx React.DOM ###
+define 'myProject.ReactExampleComponent', [
+  'React'
+  'myProject.ExampleStore'
+  'myProject.ExampleActions'
+  'myProject.ReactExampleTable'
+], (React, ExampleStore, ExampleActions, ReactExampleTable ) ->
+
+  ReactExampleComponent = React.createClass
+    mixins: [ListenMixin]
+
+    getInitialState: ->
+      rows: ExampleStore.getRows()
+      meta: ExampleStore.getMeta()
+
+    componentWillMount: ->
+      @listenTo ExampleStore
+
+    componentDidMount: ->
+      ExampleActions.getExampleData()
+
+    onStoreChange: ->
+      if this.isMounted()
+        @setState
+          rows: ExampleStore.getRows()
+          meta: ExampleStore.getMeta()
+
+    componentWillUnmount: ->
+      @stopListening ExampleStore
+
+    render: ->
+      <div className="page-wrap">
+          <header>
+            <strong> {@state.title} </strong>
+          <header>
+            <ReactExampleTable
+              rows={@state.rows},
+              meta={@state.meta}
+            />
+      </div>

samples/HTML/example.xht

@@ -0,0 +1,17 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml">
+    <head>
+        <title>This is a XHTML sample file</title>
+        <style type="text/css"><![CDATA[
+            #example {
+                background-color: yellow;
+            }
+        ]]></style>
+    </head>
+    <body>
+        <div id="example">
+            Just a simple <strong>XHTML</strong> test page.
+        </div>
+    </body>
+</html>
+

test/test_blob.rb

@@ -292,6 +292,13 @@ class TestBlob < Test::Unit::TestCase
     assert blob("deps/http_parser/http_parser.c").vendored?
     assert blob("deps/v8/src/v8.h").vendored?
 
+    # Chart.js
+    assert blob("some/vendored/path/Chart.js").vendored?
+    assert !blob("some/vendored/path/chart.js").vendored?
+
+    # Codemirror deps
+    assert blob("codemirror/mode/blah.js").vendored?
+
     # Debian packaging
     assert blob("debian/cron.d").vendored?
 

test/test_language.rb

@@ -358,4 +358,15 @@ class TestLanguage < Test::Unit::TestCase
   def test_by_type
     assert !Language.by_type(:prose).nil?
   end
+
+  def test_all_languages_have_grammars
+    scopes = YAML.load(File.read(File.expand_path("../../grammars.yml", __FILE__))).values.flatten
+    missing = Language.all.reject { |language| language.tm_scope == "none" || scopes.include?(language.tm_scope) }
+    message = "The following languages' scopes are not listed in grammars.yml. Please add grammars for all new languages.\n"
+    message << "If no grammar exists for a language, mark the language with `tm_scope: none` in lib/linguist/languages.yml.\n"
+
+    width = missing.map { |language| language.name.length }.max
+    message << missing.map { |language| sprintf("%-#{width}s %s", language.name, language.tm_scope) }.sort.join("\n")
+    assert missing.empty?, message
+  end
 end