diff --git a/.gitignore b/.gitignore index 08e2205d..90d94706 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,4 @@ -Gemfile.lock +/Gemfile.lock .bundle/ benchmark/ lib/linguist/samples.json diff --git a/.travis.yml b/.travis.yml index ac6800a7..7b013349 100644 --- a/.travis.yml +++ b/.travis.yml @@ -2,6 +2,7 @@ before_install: - git fetch origin master:master - git fetch origin v2.0.0:v2.0.0 - git fetch origin test/attributes:test/attributes + - git fetch origin test/master:test/master - sudo apt-get install libicu-dev -y rvm: - 1.9.3 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 00000000..ee13ae71 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,31 @@ +## Contributing + +The majority of contributions won't need to touch any Ruby code at all. The [master language list][languages] is just a YAML configuration file. + +Almost all bug fixes or new language additions should come with some additional code samples. Just drop them under [`samples/`][samples] in the correct subdirectory and our test suite will automatically test them. In most cases you shouldn't need to add any new assertions. + +### My code is detected as the wrong language + +This can usually be solved either by adding a new filename or file name extension to the language's entry in [`languages.yml`][languages] or adding more [samples][samples] for your language to the repository to make Linguist's classifier smarter. + +### Syntax highlighting looks wrong + +Assuming your code is being detected as the right language (see above), in most cases this is due to a bug in the language grammar rather than a bug in Linguist. [`grammars.yml`][grammars] lists all the grammars we use for syntax highlighting on github.com. Find the one corresponding to your code's programming language and submit a bug report upstream. + +You can also try to fix the bug yourself and submit a Pull Request. [This piece from TextMate's documentation](http://manual.macromates.com/en/language_grammars) offers a good introduction on how to work with TextMate-compatible grammars. + +Once the bug has been fixed upstream, please let us know and we'll pick it up for GitHub. + +### I want to add support for the `X` programming language + +Great! You'll need to: + +0. Add an entry for your language to [`languages.yml`][languages]. +0. Add a grammar for your language to [`grammars.yml`][grammars] by running `script/download-grammars --add URL`. Please only add grammars that have a license that permits redistribution. +0. Add samples for your language to the [samples directory][samples]. + +We try only to add languages once they have some usage on GitHub, so please note in-the-wild usage examples in your pull request. In most cases we prefer that languages already be in use in hundreds of repositories before supporting them in Linguist. + +[grammars]: /grammars.yml +[languages]: /lib/linguist/languages.yml +[samples]: /samples diff --git a/README.md b/README.md index 36d1fcd2..5eaa2f20 100644 --- a/README.md +++ b/README.md @@ -2,6 +2,8 @@ We use this library at GitHub to detect blob languages, ignore binary files, suppress generated files in diffs, and generate language breakdown graphs. +Tips for filing issues and creating pull requests can be found in [`CONTRIBUTING.md`](/CONTRIBUTING.md). + ## Features ### Language detection @@ -26,15 +28,7 @@ See [lib/linguist/language.rb](https://github.com/github/linguist/blob/master/li Syntax highlighting in GitHub is performed using TextMate-compatible grammars. These are the same grammars that TextMate, Sublime Text and Atom use. -Every language in `languages.yml` is mapped to its corresponding TM `scope`. This scope will be used when picking up a grammar for highlighting. **When adding a new language to Linguist, please add its corrsponding scope too (assuming there's an existing TextMate bundle) so syntax highlighting works for it**. - -#### I found a bug! - -The `grammars.yml` file contains the list of all the repositories where we fetch TextMate grammars for highlighting. If you find a bug in the highlighting for any given language, please consult this Grammars list to find the source of the grammar, and submit the bug report upstream. - -You can also try to fix the bug yourself and submit a Pull Request. [This piece from TextMate's documentation](http://manual.macromates.com/en/language_grammars) offers a good introduction on how to work with TextMate-compatible grammars. - -Once the bug has been fixed upstream, please let us know and we'll pick it up for GitHub. +Every language in `languages.yml` is mapped to its corresponding TM `scope`. This scope will be used when picking up a grammar for highlighting. **When adding a new language to Linguist, please add its corresponding scope too (assuming there's an existing TextMate bundle, Sublime Text package, or Atom package) so syntax highlighting works for it**. ### Stats @@ -153,14 +147,6 @@ To run the tests: bundle exec rake test -## Contributing - -The majority of contributions won't need to touch any Ruby code at all. The [master language list](https://github.com/github/linguist/blob/master/lib/linguist/languages.yml) is just a YAML configuration file. - -We try to only add languages once they have some usage on GitHub, so please note in-the-wild usage examples in your pull request. - -Almost all bug fixes or new language additions should come with some additional code samples. Just drop them under [`samples/`](https://github.com/github/linguist/tree/master/samples) in the correct subdirectory and our test suite will automatically test them. In most cases you shouldn't need to add any new assertions. - ### A note on language extensions Linguist has a number of methods available to it for identifying the language of a particular file. The initial lookup is based upon the extension of the file, possible file extensions are defined in an array called `extensions`. Take a look at this example for example for `Perl`: diff --git a/github-linguist.gemspec b/github-linguist.gemspec index a30e0f19..578e823c 100644 --- a/github-linguist.gemspec +++ b/github-linguist.gemspec @@ -15,8 +15,8 @@ Gem::Specification.new do |s| s.add_dependency 'charlock_holmes', '~> 0.7.3' s.add_dependency 'escape_utils', '~> 1.0.1' - s.add_dependency 'mime-types', '~> 1.19' - s.add_dependency 'rugged', '~> 0.21.1b2' + s.add_dependency 'mime-types', '>= 1.19' + s.add_dependency 'rugged', '~> 0.22.0b4' s.add_development_dependency 'mocha' s.add_development_dependency 'pry' diff --git a/grammars.yml b/grammars.yml index 074bd04a..fbf087d3 100644 --- a/grammars.yml +++ b/grammars.yml @@ -1,6 +1,4 @@ --- -http://hww3.riverweb.com/dist/Pike_TextMate.tar.gz: -- source.pike http://svn.edgewall.org/repos/genshi/contrib/textmate/Genshi.tmbundle/Syntaxes/Markup%20Template%20%28XML%29.tmLanguage: - text.xml.genshi http://svn.textmate.org/trunk/Review/Bundles/BlitzMax.tmbundle: @@ -40,10 +38,14 @@ https://github.com/Drako/SublimeBrainfuck/raw/master/Brainfuck.tmLanguage: - source.bf https://github.com/JohnNilsson/awk-sublime/raw/master/AWK.tmLanguage: - source.awk +https://github.com/JonBons/Sublime-SQF-Language: +- source.sqf https://github.com/MarioRicalde/SCSS.tmbundle: - source.scss https://github.com/Oldes/Sublime-REBOL: - source.rebol +https://github.com/PogiNate/Sublime-Inform: +- source.Inform7 https://github.com/Red-Nova-Technologies/autoitv3-tmbundle: - source.autoit.3 https://github.com/SalGnt/Sublime-VimL: @@ -70,6 +72,8 @@ https://github.com/aroben/ruby.tmbundle@4636a3023153c3034eb6ffc613899ba9cf33b41f - text.html.erb https://github.com/asbjornenge/Docker.tmbundle: - source.dockerfile +https://github.com/atom/language-clojure: +- source.clojure https://github.com/atom/language-coffee-script: - source.coffee - source.litcoffee @@ -95,6 +99,8 @@ https://github.com/bholt/chapel-tmbundle: - source.chapel https://github.com/brandonwamboldt/sublime-nginx: - source.nginx +https://github.com/bro/bro-sublime: +- source.bro https://github.com/carsonoid/sublime_man_page_support/raw/master/man-groff.tmLanguage: - text.groff https://github.com/ccreutzig/sublime-MuPAD: @@ -135,6 +141,8 @@ https://github.com/guillermooo/dart-sublime-bundle/raw/master/Dart.tmLanguage: - source.dart https://github.com/harrism/sublimetext-cuda-cpp/raw/master/cuda-c%2B%2B.tmLanguage: - source.cuda-c++ +https://github.com/hww3/pike-textmate: +- source.pike https://github.com/jeancharles-roger/ceylon-sublimetext/raw/master/Ceylon.tmLanguage: - source.ceylon https://github.com/jfairbank/Sublime-Text-2-OpenEdge-ABL: @@ -211,16 +219,11 @@ https://github.com/slavapestov/factor/raw/master/misc/Factor.tmbundle/Syntaxes/F - source.factor https://github.com/slim-template/ruby-slim.tmbundle: - text.slim -https://github.com/smiledawgg/Bro.tmbundle: -- source.bro -- source.bro.sig https://github.com/staltz/SublimeXtend: - source.xtend https://github.com/statatmbundle/Stata.tmbundle: - source.mata - source.stata -https://github.com/swannodette/textmate-clojure: -- source.clojure https://github.com/technosophos/Vala-TMBundle: - source.vala https://github.com/textmate/ant.tmbundle: @@ -240,6 +243,7 @@ https://github.com/textmate/bison.tmbundle: https://github.com/textmate/c.tmbundle: - source.c - source.c++ +- source.c.platform https://github.com/textmate/capnproto.tmbundle: - source.capnp https://github.com/textmate/cmake.tmbundle: @@ -277,8 +281,6 @@ https://github.com/textmate/haskell.tmbundle: - text.tex.latex.haskell https://github.com/textmate/html.tmbundle: - text.html.basic -https://github.com/textmate/inform.tmbundle: -- source.inform https://github.com/textmate/ini.tmbundle: - source.ini https://github.com/textmate/io.tmbundle: @@ -327,6 +329,7 @@ https://github.com/textmate/ninja.tmbundle: https://github.com/textmate/objective-c.tmbundle: - source.objc - source.objc++ +- source.objc.platform - source.strings https://github.com/textmate/ocaml.tmbundle: - source.camlp4.ocaml diff --git a/lib/linguist/blob_helper.rb b/lib/linguist/blob_helper.rb index d6d3dd30..c368b4d0 100644 --- a/lib/linguist/blob_helper.rb +++ b/lib/linguist/blob_helper.rb @@ -146,6 +146,13 @@ module Linguist end end + # Public: Is the blob empty? + # + # Return true or false + def empty? + data.nil? || data == "" + end + # Public: Is the blob text? # # Return true or false diff --git a/lib/linguist/generated.rb b/lib/linguist/generated.rb index 0f911c45..291fa8bd 100644 --- a/lib/linguist/generated.rb +++ b/lib/linguist/generated.rb @@ -51,26 +51,25 @@ module Linguist # # Return true or false def generated? - name == 'Gemfile.lock' || - minified_files? || - compiled_coffeescript? || - xcode_file? || - generated_parser? || - generated_net_docfile? || - generated_net_designer_file? || - generated_postscript? || - generated_protocol_buffer? || - generated_jni_header? || - composer_lock? || - node_modules? || - godeps? || - vcr_cassette? || - generated_by_zephir? + minified_files? || + compiled_coffeescript? || + xcode_file? || + generated_parser? || + generated_net_docfile? || + generated_net_designer_file? || + generated_postscript? || + generated_protocol_buffer? || + generated_jni_header? || + composer_lock? || + node_modules? || + godeps? || + vcr_cassette? || + generated_by_zephir? end # Internal: Is the blob an Xcode file? # - # Generated if the file extension is an Xcode + # Generated if the file extension is an Xcode # file extension. # # Returns true of false. @@ -265,4 +264,3 @@ module Linguist end end end - diff --git a/lib/linguist/heuristics.rb b/lib/linguist/heuristics.rb index c7519881..13f0173f 100644 --- a/lib/linguist/heuristics.rb +++ b/lib/linguist/heuristics.rb @@ -13,26 +13,31 @@ module Linguist # Returns an array of Languages or [] def self.find_by_heuristics(data, languages) if active? + result = [] + if languages.all? { |l| ["Perl", "Prolog"].include?(l) } - result = disambiguate_pl(data, languages) + result = disambiguate_pl(data) end if languages.all? { |l| ["ECL", "Prolog"].include?(l) } - result = disambiguate_ecl(data, languages) + result = disambiguate_ecl(data) end if languages.all? { |l| ["IDL", "Prolog"].include?(l) } - result = disambiguate_pro(data, languages) + result = disambiguate_pro(data) end if languages.all? { |l| ["Common Lisp", "OpenCL"].include?(l) } - result = disambiguate_cl(data, languages) + result = disambiguate_cl(data) end if languages.all? { |l| ["Hack", "PHP"].include?(l) } - result = disambiguate_hack(data, languages) + result = disambiguate_hack(data) end if languages.all? { |l| ["Scala", "SuperCollider"].include?(l) } - result = disambiguate_sc(data, languages) + result = disambiguate_sc(data) end if languages.all? { |l| ["AsciiDoc", "AGS Script"].include?(l) } - result = disambiguate_asc(data, languages) + result = disambiguate_asc(data) + end + if languages.all? { |l| ["FORTRAN", "Forth"].include?(l) } + result = disambiguate_f(data) end return result end @@ -42,28 +47,37 @@ module Linguist # We want to shortcut look for Objective-C _and_ now C++ too! # # Returns an array of Languages or [] - def self.disambiguate_c(data, languages) + def self.disambiguate_c(data) matches = [] - matches << Language["Objective-C"] if data.include?("@interface") - matches << Language["C++"] if data.include?("#include ") + if data.include?("@interface") + matches << Language["Objective-C"] + elsif data.include?("#include ") + matches << Language["C++"] + end matches end - def self.disambiguate_pl(data, languages) + def self.disambiguate_pl(data) matches = [] - matches << Language["Prolog"] if data.include?(":-") - matches << Language["Perl"] if data.include?("use strict") + if data.include?("use strict") + matches << Language["Perl"] + elsif data.include?(":-") + matches << Language["Prolog"] + end matches end - def self.disambiguate_ecl(data, languages) + def self.disambiguate_ecl(data) matches = [] - matches << Language["Prolog"] if data.include?(":-") - matches << Language["ECL"] if data.include?(":=") + if data.include?(":-") + matches << Language["Prolog"] + elsif data.include?(":=") + matches << Language["ECL"] + end matches end - def self.disambiguate_pro(data, languages) + def self.disambiguate_pro(data) matches = [] if (data.include?(":-")) matches << Language["Prolog"] @@ -73,7 +87,7 @@ module Linguist matches end - def self.disambiguate_ts(data, languages) + def self.disambiguate_ts(data) matches = [] if (data.include?("")) matches << Language["XML"] @@ -83,21 +97,24 @@ module Linguist matches end - def self.disambiguate_cl(data, languages) + def self.disambiguate_cl(data) matches = [] - matches << Language["Common Lisp"] if data.include?("(defun ") - matches << Language["OpenCL"] if /\/\* |\/\/ |^\}/.match(data) + if data.include?("(defun ") + matches << Language["Common Lisp"] + elsif /\/\* |\/\/ |^\}/.match(data) + matches << Language["OpenCL"] + end matches end - def self.disambiguate_r(data, languages) + def self.disambiguate_r(data) matches = [] matches << Language["Rebol"] if /\bRebol\b/i.match(data) matches << Language["R"] if data.include?("<-") matches end - def self.disambiguate_hack(data, languages) + def self.disambiguate_hack(data) matches = [] if data.include?(" 1 data = blob.data possible_language_names = possible_languages.map(&:name) + heuristic_languages = Heuristics.find_by_heuristics(data, possible_language_names) + + if heuristic_languages.size > 1 + possible_language_names = heuristic_languages.map(&:name) + end - # Don't bother with binary contents or an empty file - if data.nil? || data == "" - nil # Check if there's a shebang line and use that as authoritative - elsif (result = find_by_shebang(data)) && !result.empty? + if (result = find_by_shebang(data)) && !result.empty? result.first # No shebang. Still more work to do. Try to find it with our heuristics. - elsif (determined = Heuristics.find_by_heuristics(data, possible_language_names)) && !determined.empty? - determined.first + elsif heuristic_languages.size == 1 + heuristic_languages.first # Lastly, fall back to the probabilistic classifier. elsif classified = Classifier.classify(Samples.cache, data, possible_language_names).first # Return the actual Language object based of the string language name (i.e., first element of `#classify`) @@ -433,11 +431,6 @@ module Linguist # Returns the extensions Array attr_reader :filenames - # Public: Return all possible extensions for language - def all_extensions - (extensions + [primary_extension]).uniq - end - # Deprecated: Get primary extension # # Defaults to the first extension but can be overridden @@ -595,9 +588,9 @@ module Linguist :ace_mode => options['ace_mode'], :wrap => options['wrap'], :group_name => options['group'], - :searchable => options.key?('searchable') ? options['searchable'] : true, + :searchable => options.fetch('searchable', true), :search_term => options['search_term'], - :extensions => [options['extensions'].first] + options['extensions'][1..-1].sort, + :extensions => Array(options['extensions']), :interpreters => options['interpreters'].sort, :filenames => options['filenames'], :popular => popular.include?(name) diff --git a/lib/linguist/languages.yml b/lib/linguist/languages.yml index e199d33f..8c405356 100644 --- a/lib/linguist/languages.yml +++ b/lib/linguist/languages.yml @@ -12,6 +12,10 @@ # search_term - Deprecated: Some languages maybe indexed under a # different alias. Avoid defining new exceptions. # color - CSS hex color to represent the language. +# tm_scope - The TextMate scope that represents this programming +# language. This should match one of the scopes listed in +# the grammars.yml file. Use "none" if there is no grammar +# for this language. # # Any additions or modifications (even trivial) should have corresponding # test change in `test/test_blob.rb`. @@ -33,15 +37,6 @@ AGS Script: - .ash tm_scope: source.c++ -Ant Build System: - type: data - tm_scope: text.xml.ant - extensions: - - .ant.xml - filenames: - - build.xml - - ant.xml - ANTLR: type: programming color: "#9DC3FF" @@ -54,6 +49,7 @@ APL: extensions: - .apl - .dyalog + tm_scope: none ASP: type: programming @@ -117,6 +113,13 @@ Alloy: extensions: - .als +Ant Build System: + type: data + tm_scope: text.xml.ant + filenames: + - ant.xml + - build.xml + ApacheConf: type: markup aliases: @@ -148,6 +151,7 @@ Arc: color: "#ca2afe" extensions: - .arc + tm_scope: none Arduino: type: programming @@ -164,12 +168,14 @@ AsciiDoc: - .asciidoc - .adoc - .asc + tm_scope: none AspectJ: type: programming color: "#1957b0" extensions: - .aj + tm_scope: none Assembly: type: programming @@ -187,6 +193,7 @@ Augeas: type: programming extensions: - .aug + tm_scope: none AutoHotkey: type: programming @@ -196,6 +203,7 @@ AutoHotkey: extensions: - .ahk - .ahkl + tm_scope: none AutoIt: type: programming @@ -288,6 +296,7 @@ Brightscript: type: programming extensions: - .brs + tm_scope: none Bro: type: programming @@ -361,6 +370,7 @@ CLIPS: type: programming extensions: - .clp + tm_scope: none CMake: extensions: @@ -423,6 +433,7 @@ Clean: extensions: - .icl - .dcl + tm_scope: none Clojure: type: programming @@ -451,6 +462,7 @@ CoffeeScript: extensions: - .coffee - ._coffee + - .cjsx - .cson - .iced filenames: @@ -539,6 +551,7 @@ Creole: wrap: true extensions: - .creole + tm_scope: none Crystal: type: programming @@ -606,6 +619,7 @@ Darcs Patch: extensions: - .darcspatch - .dpatch + tm_scope: none Dart: type: programming @@ -633,6 +647,7 @@ Dogescript: color: "#cca760" extensions: - .djs + tm_scope: none Dylan: type: programming @@ -648,13 +663,7 @@ E: color: "#ccce35" extensions: - .E - -Ecere Projects: - type: data - group: JavaScript - extensions: - - .epj - tm_scope: source.json + tm_scope: none ECL: type: programming @@ -662,6 +671,7 @@ ECL: extensions: - .ecl - .eclxml + tm_scope: none Eagle: type: markup @@ -671,6 +681,13 @@ Eagle: - .brd tm_scope: text.xml +Ecere Projects: + type: data + group: JavaScript + extensions: + - .epj + tm_scope: source.json + Eiffel: type: programming color: "#946d57" @@ -738,6 +755,7 @@ FLUX: extensions: - .fx - .flux + tm_scope: none FORTRAN: type: programming @@ -767,8 +785,8 @@ Factor: extensions: - .factor filenames: - - .factor-rc - .factor-boot-rc + - .factor-rc Fancy: type: programming @@ -792,6 +810,9 @@ Forth: extensions: - .fth - .4th + - .F + - .f + - .for - .forth - .frt @@ -808,18 +829,13 @@ G-code: - .g - .gco - .gcode - -Game Maker Language: - type: programming - color: "#8ad353" - extensions: - - .gml - tm_scope: source.js + tm_scope: none GAMS: type: programming extensions: - .gms + tm_scope: none GAP: type: programming @@ -828,6 +844,7 @@ GAP: - .gap - .gd - .gi + tm_scope: none GAS: type: programming @@ -841,6 +858,7 @@ GDScript: type: programming extensions: - .gd + tm_scope: none GLSL: group: C @@ -860,6 +878,13 @@ GLSL: - .vrx - .vshader +Game Maker Language: + type: programming + color: "#8ad353" + extensions: + - .gml + tm_scope: source.js + Genshi: extensions: - .kid @@ -918,6 +943,7 @@ Golo: color: "#f6a51f" extensions: - .golo + tm_scope: none Gosu: type: programming @@ -933,6 +959,7 @@ Grace: type: programming extensions: - .grace + tm_scope: none Grammatical Framework: type: programming @@ -949,6 +976,7 @@ Graph Modeling Language: type: data extensions: - .gml + tm_scope: none Graphviz (DOT): type: data @@ -1003,6 +1031,7 @@ HTML: - .html - .htm - .st + - .xht - .xhtml HTML+Django: @@ -1038,6 +1067,7 @@ HTTP: type: data extensions: - .http + tm_scope: none Hack: type: programming @@ -1045,6 +1075,7 @@ Hack: extensions: - .hh - .php + tm_scope: none Haml: group: HTML @@ -1067,6 +1098,7 @@ Harbour: color: "#0e60e3" extensions: - .hb + tm_scope: none Haskell: type: programming @@ -1092,6 +1124,7 @@ Hy: - .hy aliases: - hylang + tm_scope: none IDL: type: programming @@ -1107,6 +1140,7 @@ IGOR Pro: aliases: - igor - igorpro + tm_scope: none INI: type: data @@ -1119,9 +1153,15 @@ INI: aliases: - dosini -Inno Setup: +IRC log: + search_term: irc + aliases: + - irc + - irc logs extensions: - - .iss + - .irclog + - .weechatlog + tm_scope: none Idris: type: programming @@ -1135,7 +1175,7 @@ Inform 7: extensions: - .ni - .i7x - tm_scope: source.inform + tm_scope: source.Inform7 aliases: - i7 - inform7 @@ -1143,15 +1183,7 @@ Inform 7: Inno Setup: extensions: - .iss - -IRC log: - search_term: irc - aliases: - - irc - - irc logs - extensions: - - .irclog - - .weechatlog + tm_scope: none Io: type: programming @@ -1170,11 +1202,13 @@ Isabelle: color: "#fdcd00" extensions: - .thy + tm_scope: none J: type: programming extensions: - .ijs + tm_scope: none JSON: type: data @@ -1282,6 +1316,7 @@ KRL: color: "#f5c800" extensions: - .krl + tm_scope: none Kit: type: markup @@ -1315,6 +1350,7 @@ LOLCODE: extensions: - .lol color: "#cc9900" + tm_scope: none LSL: type: programming @@ -1329,6 +1365,7 @@ LabVIEW: type: programming extensions: - .lvproj + tm_scope: none Lasso: type: programming @@ -1367,12 +1404,14 @@ Liquid: type: markup extensions: - .liquid + tm_scope: none Literate Agda: type: programming group: Agda extensions: - .lagda + tm_scope: none Literate CoffeeScript: type: programming @@ -1462,6 +1501,7 @@ MTML: tm_scope: text.html.basic Makefile: + type: programming aliases: - bsdmake - make @@ -1470,9 +1510,9 @@ Makefile: - .mak - .mk filenames: - - makefile - - Makefile - GNUmakefile + - Makefile + - makefile interpreters: - make @@ -1525,8 +1565,6 @@ Matlab: Maven POM: type: data tm_scope: text.xml.pom - extensions: - - .pom.xml filenames: - pom.xml @@ -1550,6 +1588,7 @@ MediaWiki: wrap: true extensions: - .mediawiki + tm_scope: none Mercury: type: programming @@ -1564,6 +1603,7 @@ MiniD: # Legacy searchable: false extensions: - .minid # Dummy extension + tm_scope: none Mirah: type: programming @@ -1585,6 +1625,7 @@ Moocode: type: programming extensions: - .moo + tm_scope: none MoonScript: type: programming @@ -1596,6 +1637,7 @@ MoonScript: Myghty: extensions: - .myt + tm_scope: none NSIS: extensions: @@ -1641,6 +1683,7 @@ Nit: color: "#0d8921" extensions: - .nit + tm_scope: none Nix: type: programming @@ -1648,6 +1691,7 @@ Nix: - .nix aliases: - nixos + tm_scope: none Nu: type: programming @@ -1666,6 +1710,7 @@ NumPy: - .numpy - .numpyw - .numsc + tm_scope: none OCaml: type: programming @@ -1726,6 +1771,7 @@ Omgrofl: extensions: - .omgrofl color: "#cabbff" + tm_scope: none Opa: type: programming @@ -1737,6 +1783,7 @@ Opal: color: "#f7ede0" extensions: - .opal + tm_scope: none OpenCL: type: programming @@ -1761,12 +1808,14 @@ OpenSCAD: type: programming extensions: - .scad + tm_scope: none Org: type: prose wrap: true extensions: - .org + tm_scope: none Ox: type: programming @@ -1774,12 +1823,14 @@ Ox: - .ox - .oxh - .oxo + tm_scope: none Oxygene: type: programming color: "#5a63a3" extensions: - .oxygene + tm_scope: none PAWN: type: programming @@ -1815,18 +1866,30 @@ Pan: color: '#cc0000' extensions: - .pan + tm_scope: none Papyrus: type: programming color: "#6600cc" extensions: - .psc + tm_scope: none Parrot: type: programming color: "#f3ca0a" extensions: - .parrot # Dummy extension + tm_scope: none + +Parrot Assembly: + group: Parrot + type: programming + aliases: + - pasm + extensions: + - .pasm + tm_scope: none Parrot Internal Representation: group: Parrot @@ -1837,14 +1900,6 @@ Parrot Internal Representation: extensions: - .pir -Parrot Assembly: - group: Parrot - type: programming - aliases: - - pasm - extensions: - - .pasm - Pascal: type: programming color: "#b0ce4e" @@ -1886,12 +1941,14 @@ Perl6: - .p6m - .pl6 - .pm6 + tm_scope: none PigLatin: type: programming color: "#fcd7de" extensions: - .pig + tm_scope: none Pike: type: programming @@ -1906,12 +1963,14 @@ Pod: wrap: true extensions: - .pod + tm_scope: none PogoScript: type: programming color: "#d80074" extensions: - .pogo + tm_scope: none PostScript: type: markup @@ -1952,6 +2011,7 @@ Propeller Spin: color: "#2b446d" extensions: - .spin + tm_scope: none Protocol Buffer: type: markup @@ -1975,6 +2035,7 @@ Pure Data: color: "#91de79" extensions: - .pd + tm_scope: none PureScript: type: programming @@ -2001,10 +2062,10 @@ Python: - .wsgi - .xpy filenames: - - wscript - - SConstruct - - SConscript - BUILD + - SConscript + - SConstruct + - wscript interpreters: - python @@ -2070,7 +2131,7 @@ RHTML: group: HTML extensions: - .rhtml - tm_scope: text.html.ruby + tm_scope: text.html.erb aliases: - html+ruby @@ -2081,6 +2142,7 @@ RMarkdown: extensions: - .rmd - .Rmd + tm_scope: none Racket: type: programming @@ -2100,6 +2162,7 @@ Ragel in Ruby Host: aliases: - ragel-rb - ragel-ruby + tm_scope: none Raw token data: search_term: raw @@ -2107,6 +2170,7 @@ Raw token data: - raw extensions: - .raw + tm_scope: none Rebol: type: programming @@ -2126,10 +2190,12 @@ Red: - .reds aliases: - red/system + tm_scope: none Redcode: extensions: - .cw + tm_scope: none RobotFramework: type: programming @@ -2203,6 +2269,7 @@ SAS: color: "#1E90FF" extensions: - .sas + tm_scope: none SCSS: type: markup @@ -2218,7 +2285,7 @@ SQF: extensions: - .sqf - .hqf - tm_scope: source.c++ + tm_scope: source.sqf SQL: type: data @@ -2296,6 +2363,7 @@ Self: color: "#0579aa" extensions: - .self + tm_scope: none Shell: type: programming @@ -2334,6 +2402,7 @@ Shen: color: "#120F14" extensions: - .shen + tm_scope: none Slash: type: programming @@ -2404,6 +2473,7 @@ Stylus: group: CSS extensions: - .styl + tm_scope: none SuperCollider: type: programming @@ -2411,6 +2481,7 @@ SuperCollider: extensions: - .scd - .sc + tm_scope: none Swift: type: programming @@ -2436,6 +2507,7 @@ TXL: type: programming extensions: - .txl + tm_scope: none Tcl: type: programming @@ -2492,6 +2564,7 @@ Textile: wrap: true extensions: - .textile + tm_scope: none Thrift: type: programming @@ -2505,6 +2578,7 @@ Turing: extensions: - .t - .tu + tm_scope: none Twig: type: markup @@ -2584,8 +2658,8 @@ VimL: filenames: - .vimrc - _vimrc - - vimrc - gvimrc + - vimrc Visual Basic: type: programming @@ -2687,12 +2761,12 @@ XML: filenames: - .classpath - .project + - Web.Debug.config + - Web.Release.config + - Web.config - build.xml.dist - packages.config - phpunit.xml.dist - - Web.config - - Web.Debug.config - - Web.Release.config XProc: type: programming @@ -2765,6 +2839,7 @@ Zimpl: - .zimpl - .zmpl - .zpl + tm_scope: none eC: type: programming @@ -2772,6 +2847,7 @@ eC: extensions: - .ec - .eh + tm_scope: none edn: type: data @@ -2786,6 +2862,7 @@ fish: group: Shell extensions: - .fish + tm_scope: none mupad: extensions: @@ -2826,3 +2903,4 @@ xBase: color: "#3a4040" extensions: - .prg + tm_scope: none diff --git a/lib/linguist/vendor.yml b/lib/linguist/vendor.yml index 0bf4bb6c..31ff7ce7 100644 --- a/lib/linguist/vendor.yml +++ b/lib/linguist/vendor.yml @@ -110,6 +110,12 @@ # MathJax - (^|/)MathJax/ +# Chart.js +- (^|/)Chart\.js$ + +# Codemirror +- (^|/)[Cc]ode[Mm]irror/(lib|mode|theme|addon|keymap) + # SyntaxHighlighter - http://alexgorbatchev.com/ - (^|/)shBrush([^.]*)\.js$ - (^|/)shCore\.js$ diff --git a/lib/linguist/version.rb b/lib/linguist/version.rb index c8777321..6aa2f696 100644 --- a/lib/linguist/version.rb +++ b/lib/linguist/version.rb @@ -1,3 +1,3 @@ module Linguist - VERSION = "4.0.0" + VERSION = "4.0.3" end diff --git a/samples/Ant Build System/filenames/ant.xml b/samples/Ant Build System/filenames/ant.xml new file mode 100644 index 00000000..54e96ddc --- /dev/null +++ b/samples/Ant Build System/filenames/ant.xml @@ -0,0 +1,110 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/samples/Ant Build System/filenames/build.xml b/samples/Ant Build System/filenames/build.xml new file mode 120000 index 00000000..6c7d5cd4 --- /dev/null +++ b/samples/Ant Build System/filenames/build.xml @@ -0,0 +1 @@ +ant.xml \ No newline at end of file diff --git a/samples/CoffeeScript/empty.coffee b/samples/CoffeeScript/empty.coffee deleted file mode 100644 index e69de29b..00000000 diff --git a/samples/CoffeeScript/example.cjsx b/samples/CoffeeScript/example.cjsx new file mode 100644 index 00000000..def9cf2f --- /dev/null +++ b/samples/CoffeeScript/example.cjsx @@ -0,0 +1,40 @@ +###* @cjsx React.DOM ### +define 'myProject.ReactExampleComponent', [ + 'React' + 'myProject.ExampleStore' + 'myProject.ExampleActions' + 'myProject.ReactExampleTable' +], (React, ExampleStore, ExampleActions, ReactExampleTable ) -> + + ReactExampleComponent = React.createClass + mixins: [ListenMixin] + + getInitialState: -> + rows: ExampleStore.getRows() + meta: ExampleStore.getMeta() + + componentWillMount: -> + @listenTo ExampleStore + + componentDidMount: -> + ExampleActions.getExampleData() + + onStoreChange: -> + if this.isMounted() + @setState + rows: ExampleStore.getRows() + meta: ExampleStore.getMeta() + + componentWillUnmount: -> + @stopListening ExampleStore + + render: -> +
+
+ {@state.title} +
+ +
diff --git a/samples/FORTRAN/sample1.f b/samples/FORTRAN/sample1.f new file mode 100644 index 00000000..39ba97cb --- /dev/null +++ b/samples/FORTRAN/sample1.f @@ -0,0 +1,25 @@ +c comment +* comment + + program main + + end + + subroutine foo( i, x, b ) + INTEGER i + REAL x + LOGICAL b + + if( i.ne.0 ) then + call bar( -i ) + end if + + return + end + + double complex function baz() + + baz = (0.0d0,0.0d0) + + return + end diff --git a/samples/FORTRAN/sample1.for b/samples/FORTRAN/sample1.for new file mode 100644 index 00000000..39ba97cb --- /dev/null +++ b/samples/FORTRAN/sample1.for @@ -0,0 +1,25 @@ +c comment +* comment + + program main + + end + + subroutine foo( i, x, b ) + INTEGER i + REAL x + LOGICAL b + + if( i.ne.0 ) then + call bar( -i ) + end if + + return + end + + double complex function baz() + + baz = (0.0d0,0.0d0) + + return + end diff --git a/samples/FORTRAN/sample2.f b/samples/FORTRAN/sample2.f new file mode 100644 index 00000000..19538ac7 --- /dev/null +++ b/samples/FORTRAN/sample2.f @@ -0,0 +1,25 @@ + PROGRAM MAIN + + END + +C comment +* comment + + SUBROUTINE foo( i, x, b ) + INTEGER i + REAL x + LOGICAL b + + IF( i.NE.0 ) THEN + CALL bar( -i ) + END IF + + RETURN + END + + DOUBLE COMPLEX FUNCTION baz() + + baz = (0.0d0,0.0d0) + + RETURN + END diff --git a/samples/FORTRAN/sample3.F b/samples/FORTRAN/sample3.F new file mode 100644 index 00000000..39ba97cb --- /dev/null +++ b/samples/FORTRAN/sample3.F @@ -0,0 +1,25 @@ +c comment +* comment + + program main + + end + + subroutine foo( i, x, b ) + INTEGER i + REAL x + LOGICAL b + + if( i.ne.0 ) then + call bar( -i ) + end if + + return + end + + double complex function baz() + + baz = (0.0d0,0.0d0) + + return + end diff --git a/samples/Forth/core.f b/samples/Forth/core.f new file mode 100644 index 00000000..4a13e217 --- /dev/null +++ b/samples/Forth/core.f @@ -0,0 +1,252 @@ +: immediate lastxt @ dup c@ negate swap c! ; + +: \ source nip >in ! ; immediate \ Copyright 2004, 2012 Lars Brinkhoff + +: char \ ( "word" -- char ) + bl-word here 1+ c@ ; + +: ahead here 0 , ; + +: resolve here swap ! ; + +: ' bl-word here find 0branch [ ahead ] exit [ resolve ] 0 ; + +: postpone-nonimmediate [ ' literal , ' compile, ] literal , ; + +: create dovariable_code header, reveal ; + +create postponers + ' postpone-nonimmediate , + ' abort , + ' , , + +: word \ ( char "string" -- caddr ) + drop bl-word here ; + +: postpone \ ( C: "word" -- ) + bl word find 1+ cells postponers + @ execute ; immediate + +: unresolved \ ( C: "word" -- orig ) + postpone postpone postpone ahead ; immediate + +: chars \ ( n1 -- n2 ) + ; + +: else \ ( -- ) ( C: orig1 -- orig2 ) + unresolved branch swap resolve ; immediate + +: if \ ( flag -- ) ( C: -- orig ) + unresolved 0branch ; immediate + +: then \ ( -- ) ( C: orig -- ) + resolve ; immediate + +: [char] \ ( "word" -- ) + char postpone literal ; immediate + +: (does>) lastxt @ dodoes_code over >code ! r> swap >does ! ; + +: does> postpone (does>) ; immediate + +: begin \ ( -- ) ( C: -- dest ) + here ; immediate + +: while \ ( x -- ) ( C: dest -- orig dest ) + unresolved 0branch swap ; immediate + +: repeat \ ( -- ) ( C: orig dest -- ) + postpone branch , resolve ; immediate + +: until \ ( x -- ) ( C: dest -- ) + postpone 0branch , ; immediate + +: recurse lastxt @ compile, ; immediate + +: pad \ ( -- addr ) + here 1024 + ; + +: parse \ ( char "string" -- addr n ) + pad >r begin + source? if else 0 0 then + while + r@ c! r> 1+ >r + repeat 2drop pad r> over - ; + +: ( \ ( "string" -- ) + [ char ) ] literal parse 2drop ; immediate + \ TODO: If necessary, refill and keep parsing. + +: string, ( addr n -- ) + here over allot align swap cmove ; + +: (s") ( -- addr n ) ( R: ret1 -- ret2 ) + r> dup @ swap cell+ 2dup + aligned >r swap ; + +create squote 128 allot + +: s" ( "string" -- addr n ) + state @ if + postpone (s") [char] " parse dup , string, + else + [char] " parse >r squote r@ cmove squote r> + then ; immediate + +: (abort") ( ... addr n -- ) ( R: ... -- ) + cr type cr abort ; + +: abort" ( ... x "string" -- ) ( R: ... -- ) + postpone if postpone s" postpone (abort") postpone then ; immediate + +\ ---------------------------------------------------------------------- + +( Core words. ) + +\ TODO: # +\ TODO: #> +\ TODO: #s + +: and ( x y -- x&y ) nand invert ; + +: * 1 2>r 0 swap begin r@ while + r> r> swap 2dup dup + 2>r and if swap over + swap then dup + + repeat r> r> 2drop drop ; + +\ TODO: */mod + +: +loop ( -- ) ( C: nest-sys -- ) + postpone (+loop) postpone 0branch , postpone unloop ; immediate + +: space bl emit ; + +: ?.- dup 0 < if [char] - emit negate then ; + +: digit [char] 0 + emit ; + +: (.) base @ /mod ?dup if recurse then digit ; + +: ." ( "string" -- ) postpone s" postpone type ; immediate + +: . ( x -- ) ?.- (.) space ; + +: postpone-number ( caddr -- ) + 0 0 rot count >number dup 0= if + 2drop nip + postpone (literal) postpone (literal) postpone , + postpone literal postpone , + else + ." Undefined: " type cr abort + then ; + +' postpone-number postponers cell+ ! + +: / ( x y -- x/y ) /mod nip ; + +: 0< ( n -- flag ) 0 < ; + +: 1- ( n -- n-1 ) -1 + ; + +: 2! ( x1 x2 addr -- ) swap over ! cell+ ! ; + +: 2* ( n -- 2n ) dup + ; + +\ Kernel: 2/ + +: 2@ ( addr -- x1 x2 ) dup cell+ @ swap @ ; + +\ Kernel: 2drop +\ Kernel: 2dup + +\ TODO: 2over ( x1 x2 x3 x4 -- x1 x2 x3 x4 x1 x2 ) +\ 3 pick 3 pick ; + +\ TODO: 2swap + +\ TODO: <# + +: abs ( n -- |n| ) + dup 0< if negate then ; + +\ TODO: accept + +: c, ( n -- ) + here c! 1 chars allot ; + +: char+ ( n1 -- n2 ) + 1+ ; + +: constant create , does> @ ; + +: decimal ( -- ) + 10 base ! ; + +: depth ( -- n ) + data_stack 100 cells + 'SP @ - /cell / 2 - ; + +: do ( n1 n2 -- ) ( R: -- loop-sys ) ( C: -- do-sys ) + postpone 2>r here ; immediate + +\ TODO: environment? +\ TODO: evaluate +\ TODO: fill +\ TODO: fm/mod ) +\ TODO: hold + +: j ( -- x1 ) ( R: x1 x2 x3 -- x1 x2 x3 ) + 'RP @ 3 cells + @ ; + +\ TODO: leave + +: loop ( -- ) ( C: nest-sys -- ) + postpone 1 postpone (+loop) + postpone 0branch , + postpone unloop ; immediate + +: lshift begin ?dup while 1- swap dup + swap repeat ; + +: rshift 1 begin over while dup + swap 1- swap repeat nip + 2>r 0 1 begin r@ while + r> r> 2dup swap dup + 2>r and if swap over + swap then dup + + repeat r> r> 2drop drop ; + +: max ( x y -- max[x,y] ) + 2dup > if drop else nip then ; + +\ Kernel: min +\ TODO: mod +\ TODO: move + +: (quit) ( R: ... -- ) + return_stack 100 cells + 'RP ! + 0 'source-id ! tib ''source ! #tib ''#source ! + postpone [ + begin + refill + while + interpret state @ 0= if ." ok" cr then + repeat + bye ; + +' (quit) ' quit >body cell+ ! + +\ TODO: s>d +\ TODO: sign +\ TODO: sm/rem + +: spaces ( n -- ) + 0 do space loop ; + +\ TODO: u. + +: signbit ( -- n ) -1 1 rshift invert ; + +: xor ( x y -- x^y ) 2dup nand >r r@ nand swap r> nand nand ; + +: u< ( x y -- flag ) signbit xor swap signbit xor > ; + +\ TODO: um/mod + +: variable ( "word" -- ) + create /cell allot ; + +: ['] \ ( C: "word" -- ) + ' postpone literal ; immediate diff --git a/samples/Forth/core.for b/samples/Forth/core.for new file mode 100644 index 00000000..4a13e217 --- /dev/null +++ b/samples/Forth/core.for @@ -0,0 +1,252 @@ +: immediate lastxt @ dup c@ negate swap c! ; + +: \ source nip >in ! ; immediate \ Copyright 2004, 2012 Lars Brinkhoff + +: char \ ( "word" -- char ) + bl-word here 1+ c@ ; + +: ahead here 0 , ; + +: resolve here swap ! ; + +: ' bl-word here find 0branch [ ahead ] exit [ resolve ] 0 ; + +: postpone-nonimmediate [ ' literal , ' compile, ] literal , ; + +: create dovariable_code header, reveal ; + +create postponers + ' postpone-nonimmediate , + ' abort , + ' , , + +: word \ ( char "string" -- caddr ) + drop bl-word here ; + +: postpone \ ( C: "word" -- ) + bl word find 1+ cells postponers + @ execute ; immediate + +: unresolved \ ( C: "word" -- orig ) + postpone postpone postpone ahead ; immediate + +: chars \ ( n1 -- n2 ) + ; + +: else \ ( -- ) ( C: orig1 -- orig2 ) + unresolved branch swap resolve ; immediate + +: if \ ( flag -- ) ( C: -- orig ) + unresolved 0branch ; immediate + +: then \ ( -- ) ( C: orig -- ) + resolve ; immediate + +: [char] \ ( "word" -- ) + char postpone literal ; immediate + +: (does>) lastxt @ dodoes_code over >code ! r> swap >does ! ; + +: does> postpone (does>) ; immediate + +: begin \ ( -- ) ( C: -- dest ) + here ; immediate + +: while \ ( x -- ) ( C: dest -- orig dest ) + unresolved 0branch swap ; immediate + +: repeat \ ( -- ) ( C: orig dest -- ) + postpone branch , resolve ; immediate + +: until \ ( x -- ) ( C: dest -- ) + postpone 0branch , ; immediate + +: recurse lastxt @ compile, ; immediate + +: pad \ ( -- addr ) + here 1024 + ; + +: parse \ ( char "string" -- addr n ) + pad >r begin + source? if else 0 0 then + while + r@ c! r> 1+ >r + repeat 2drop pad r> over - ; + +: ( \ ( "string" -- ) + [ char ) ] literal parse 2drop ; immediate + \ TODO: If necessary, refill and keep parsing. + +: string, ( addr n -- ) + here over allot align swap cmove ; + +: (s") ( -- addr n ) ( R: ret1 -- ret2 ) + r> dup @ swap cell+ 2dup + aligned >r swap ; + +create squote 128 allot + +: s" ( "string" -- addr n ) + state @ if + postpone (s") [char] " parse dup , string, + else + [char] " parse >r squote r@ cmove squote r> + then ; immediate + +: (abort") ( ... addr n -- ) ( R: ... -- ) + cr type cr abort ; + +: abort" ( ... x "string" -- ) ( R: ... -- ) + postpone if postpone s" postpone (abort") postpone then ; immediate + +\ ---------------------------------------------------------------------- + +( Core words. ) + +\ TODO: # +\ TODO: #> +\ TODO: #s + +: and ( x y -- x&y ) nand invert ; + +: * 1 2>r 0 swap begin r@ while + r> r> swap 2dup dup + 2>r and if swap over + swap then dup + + repeat r> r> 2drop drop ; + +\ TODO: */mod + +: +loop ( -- ) ( C: nest-sys -- ) + postpone (+loop) postpone 0branch , postpone unloop ; immediate + +: space bl emit ; + +: ?.- dup 0 < if [char] - emit negate then ; + +: digit [char] 0 + emit ; + +: (.) base @ /mod ?dup if recurse then digit ; + +: ." ( "string" -- ) postpone s" postpone type ; immediate + +: . ( x -- ) ?.- (.) space ; + +: postpone-number ( caddr -- ) + 0 0 rot count >number dup 0= if + 2drop nip + postpone (literal) postpone (literal) postpone , + postpone literal postpone , + else + ." Undefined: " type cr abort + then ; + +' postpone-number postponers cell+ ! + +: / ( x y -- x/y ) /mod nip ; + +: 0< ( n -- flag ) 0 < ; + +: 1- ( n -- n-1 ) -1 + ; + +: 2! ( x1 x2 addr -- ) swap over ! cell+ ! ; + +: 2* ( n -- 2n ) dup + ; + +\ Kernel: 2/ + +: 2@ ( addr -- x1 x2 ) dup cell+ @ swap @ ; + +\ Kernel: 2drop +\ Kernel: 2dup + +\ TODO: 2over ( x1 x2 x3 x4 -- x1 x2 x3 x4 x1 x2 ) +\ 3 pick 3 pick ; + +\ TODO: 2swap + +\ TODO: <# + +: abs ( n -- |n| ) + dup 0< if negate then ; + +\ TODO: accept + +: c, ( n -- ) + here c! 1 chars allot ; + +: char+ ( n1 -- n2 ) + 1+ ; + +: constant create , does> @ ; + +: decimal ( -- ) + 10 base ! ; + +: depth ( -- n ) + data_stack 100 cells + 'SP @ - /cell / 2 - ; + +: do ( n1 n2 -- ) ( R: -- loop-sys ) ( C: -- do-sys ) + postpone 2>r here ; immediate + +\ TODO: environment? +\ TODO: evaluate +\ TODO: fill +\ TODO: fm/mod ) +\ TODO: hold + +: j ( -- x1 ) ( R: x1 x2 x3 -- x1 x2 x3 ) + 'RP @ 3 cells + @ ; + +\ TODO: leave + +: loop ( -- ) ( C: nest-sys -- ) + postpone 1 postpone (+loop) + postpone 0branch , + postpone unloop ; immediate + +: lshift begin ?dup while 1- swap dup + swap repeat ; + +: rshift 1 begin over while dup + swap 1- swap repeat nip + 2>r 0 1 begin r@ while + r> r> 2dup swap dup + 2>r and if swap over + swap then dup + + repeat r> r> 2drop drop ; + +: max ( x y -- max[x,y] ) + 2dup > if drop else nip then ; + +\ Kernel: min +\ TODO: mod +\ TODO: move + +: (quit) ( R: ... -- ) + return_stack 100 cells + 'RP ! + 0 'source-id ! tib ''source ! #tib ''#source ! + postpone [ + begin + refill + while + interpret state @ 0= if ." ok" cr then + repeat + bye ; + +' (quit) ' quit >body cell+ ! + +\ TODO: s>d +\ TODO: sign +\ TODO: sm/rem + +: spaces ( n -- ) + 0 do space loop ; + +\ TODO: u. + +: signbit ( -- n ) -1 1 rshift invert ; + +: xor ( x y -- x^y ) 2dup nand >r r@ nand swap r> nand nand ; + +: u< ( x y -- flag ) signbit xor swap signbit xor > ; + +\ TODO: um/mod + +: variable ( "word" -- ) + create /cell allot ; + +: ['] \ ( C: "word" -- ) + ' postpone literal ; immediate diff --git a/samples/Forth/core1.F b/samples/Forth/core1.F new file mode 100644 index 00000000..4a13e217 --- /dev/null +++ b/samples/Forth/core1.F @@ -0,0 +1,252 @@ +: immediate lastxt @ dup c@ negate swap c! ; + +: \ source nip >in ! ; immediate \ Copyright 2004, 2012 Lars Brinkhoff + +: char \ ( "word" -- char ) + bl-word here 1+ c@ ; + +: ahead here 0 , ; + +: resolve here swap ! ; + +: ' bl-word here find 0branch [ ahead ] exit [ resolve ] 0 ; + +: postpone-nonimmediate [ ' literal , ' compile, ] literal , ; + +: create dovariable_code header, reveal ; + +create postponers + ' postpone-nonimmediate , + ' abort , + ' , , + +: word \ ( char "string" -- caddr ) + drop bl-word here ; + +: postpone \ ( C: "word" -- ) + bl word find 1+ cells postponers + @ execute ; immediate + +: unresolved \ ( C: "word" -- orig ) + postpone postpone postpone ahead ; immediate + +: chars \ ( n1 -- n2 ) + ; + +: else \ ( -- ) ( C: orig1 -- orig2 ) + unresolved branch swap resolve ; immediate + +: if \ ( flag -- ) ( C: -- orig ) + unresolved 0branch ; immediate + +: then \ ( -- ) ( C: orig -- ) + resolve ; immediate + +: [char] \ ( "word" -- ) + char postpone literal ; immediate + +: (does>) lastxt @ dodoes_code over >code ! r> swap >does ! ; + +: does> postpone (does>) ; immediate + +: begin \ ( -- ) ( C: -- dest ) + here ; immediate + +: while \ ( x -- ) ( C: dest -- orig dest ) + unresolved 0branch swap ; immediate + +: repeat \ ( -- ) ( C: orig dest -- ) + postpone branch , resolve ; immediate + +: until \ ( x -- ) ( C: dest -- ) + postpone 0branch , ; immediate + +: recurse lastxt @ compile, ; immediate + +: pad \ ( -- addr ) + here 1024 + ; + +: parse \ ( char "string" -- addr n ) + pad >r begin + source? if else 0 0 then + while + r@ c! r> 1+ >r + repeat 2drop pad r> over - ; + +: ( \ ( "string" -- ) + [ char ) ] literal parse 2drop ; immediate + \ TODO: If necessary, refill and keep parsing. + +: string, ( addr n -- ) + here over allot align swap cmove ; + +: (s") ( -- addr n ) ( R: ret1 -- ret2 ) + r> dup @ swap cell+ 2dup + aligned >r swap ; + +create squote 128 allot + +: s" ( "string" -- addr n ) + state @ if + postpone (s") [char] " parse dup , string, + else + [char] " parse >r squote r@ cmove squote r> + then ; immediate + +: (abort") ( ... addr n -- ) ( R: ... -- ) + cr type cr abort ; + +: abort" ( ... x "string" -- ) ( R: ... -- ) + postpone if postpone s" postpone (abort") postpone then ; immediate + +\ ---------------------------------------------------------------------- + +( Core words. ) + +\ TODO: # +\ TODO: #> +\ TODO: #s + +: and ( x y -- x&y ) nand invert ; + +: * 1 2>r 0 swap begin r@ while + r> r> swap 2dup dup + 2>r and if swap over + swap then dup + + repeat r> r> 2drop drop ; + +\ TODO: */mod + +: +loop ( -- ) ( C: nest-sys -- ) + postpone (+loop) postpone 0branch , postpone unloop ; immediate + +: space bl emit ; + +: ?.- dup 0 < if [char] - emit negate then ; + +: digit [char] 0 + emit ; + +: (.) base @ /mod ?dup if recurse then digit ; + +: ." ( "string" -- ) postpone s" postpone type ; immediate + +: . ( x -- ) ?.- (.) space ; + +: postpone-number ( caddr -- ) + 0 0 rot count >number dup 0= if + 2drop nip + postpone (literal) postpone (literal) postpone , + postpone literal postpone , + else + ." Undefined: " type cr abort + then ; + +' postpone-number postponers cell+ ! + +: / ( x y -- x/y ) /mod nip ; + +: 0< ( n -- flag ) 0 < ; + +: 1- ( n -- n-1 ) -1 + ; + +: 2! ( x1 x2 addr -- ) swap over ! cell+ ! ; + +: 2* ( n -- 2n ) dup + ; + +\ Kernel: 2/ + +: 2@ ( addr -- x1 x2 ) dup cell+ @ swap @ ; + +\ Kernel: 2drop +\ Kernel: 2dup + +\ TODO: 2over ( x1 x2 x3 x4 -- x1 x2 x3 x4 x1 x2 ) +\ 3 pick 3 pick ; + +\ TODO: 2swap + +\ TODO: <# + +: abs ( n -- |n| ) + dup 0< if negate then ; + +\ TODO: accept + +: c, ( n -- ) + here c! 1 chars allot ; + +: char+ ( n1 -- n2 ) + 1+ ; + +: constant create , does> @ ; + +: decimal ( -- ) + 10 base ! ; + +: depth ( -- n ) + data_stack 100 cells + 'SP @ - /cell / 2 - ; + +: do ( n1 n2 -- ) ( R: -- loop-sys ) ( C: -- do-sys ) + postpone 2>r here ; immediate + +\ TODO: environment? +\ TODO: evaluate +\ TODO: fill +\ TODO: fm/mod ) +\ TODO: hold + +: j ( -- x1 ) ( R: x1 x2 x3 -- x1 x2 x3 ) + 'RP @ 3 cells + @ ; + +\ TODO: leave + +: loop ( -- ) ( C: nest-sys -- ) + postpone 1 postpone (+loop) + postpone 0branch , + postpone unloop ; immediate + +: lshift begin ?dup while 1- swap dup + swap repeat ; + +: rshift 1 begin over while dup + swap 1- swap repeat nip + 2>r 0 1 begin r@ while + r> r> 2dup swap dup + 2>r and if swap over + swap then dup + + repeat r> r> 2drop drop ; + +: max ( x y -- max[x,y] ) + 2dup > if drop else nip then ; + +\ Kernel: min +\ TODO: mod +\ TODO: move + +: (quit) ( R: ... -- ) + return_stack 100 cells + 'RP ! + 0 'source-id ! tib ''source ! #tib ''#source ! + postpone [ + begin + refill + while + interpret state @ 0= if ." ok" cr then + repeat + bye ; + +' (quit) ' quit >body cell+ ! + +\ TODO: s>d +\ TODO: sign +\ TODO: sm/rem + +: spaces ( n -- ) + 0 do space loop ; + +\ TODO: u. + +: signbit ( -- n ) -1 1 rshift invert ; + +: xor ( x y -- x^y ) 2dup nand >r r@ nand swap r> nand nand ; + +: u< ( x y -- flag ) signbit xor swap signbit xor > ; + +\ TODO: um/mod + +: variable ( "word" -- ) + create /cell allot ; + +: ['] \ ( C: "word" -- ) + ' postpone literal ; immediate diff --git a/samples/HTML/example.xht b/samples/HTML/example.xht new file mode 100644 index 00000000..a49a0a53 --- /dev/null +++ b/samples/HTML/example.xht @@ -0,0 +1,17 @@ + + + + This is a XHTML sample file + + + +
+ Just a simple XHTML test page. +
+ + + diff --git a/samples/JavaScript/empty.js b/samples/JavaScript/empty.js deleted file mode 100644 index f5e757a8..00000000 --- a/samples/JavaScript/empty.js +++ /dev/null @@ -1,3 +0,0 @@ -(function() { - -}).call(this); diff --git a/samples/Maven POM/filenames/pom.xml b/samples/Maven POM/filenames/pom.xml new file mode 100644 index 00000000..6e329fd9 --- /dev/null +++ b/samples/Maven POM/filenames/pom.xml @@ -0,0 +1,207 @@ + + + 4.0.0 + renpengben + spring4mvc-jpa + war + 0.0.1-SNAPSHOT + spring4mvc-jpa Maven Webapp + + https://renpengben.github.io + + spring4mvc-jpa + + + UTF-8 + 1.7 + 4.11 + 1.7.7 + 1.2.17 + + 4.0.5.RELEASE + 1.6.0.RELEASE + 2.1_3 + + 5.1.31 + 4.3.5.Final + 5.1.1.Final + 1.0.6 + + + + + + + + junit + junit + ${junit.version} + test + + + + org.slf4j + slf4j-api + ${slf4j.version} + + + org.slf4j + slf4j-log4j12 + ${slf4j.version} + + + log4j + log4j + ${log4j.version} + + + + + + org.springframework + spring-core + ${spring.version} + + + commons-logging + commons-logging + + + + + + org.springframework + spring-beans + ${spring.version} + + + org.springframework + spring-context + ${spring.version} + + + + org.springframework + spring-aop + ${spring.version} + + + + org.springframework + spring-expression + ${spring.version} + + + org.springframework + spring-tx + ${spring.version} + + + + + org.springframework + spring-aspects + ${spring.version} + + + org.springframework + spring-context-support + ${spring.version} + + + org.springframework + spring-jdbc + ${spring.version} + + + org.springframework + spring-orm + ${spring.version} + + + + org.springframework + spring-web + ${spring.version} + + + org.springframework + spring-webmvc + ${spring.version} + + + org.springframework + spring-test + ${spring.version} + test + + + + org.springframework.data + spring-data-jpa + ${spring.data.jpa.version} + + + junit-dep + junit + + + + + cglib + cglib-nodep + ${cglib.version} + + + + + + + + org.hibernate + hibernate-core + ${hibernate.version} + + + org.hibernate + hibernate-entitymanager + ${hibernate.version} + + + org.hibernate + hibernate-validator + ${hibernate-validator.version} + compile + + + + + + mysql + mysql-connector-java + ${mysql.version} + runtime + + + com.alibaba + druid + ${druid-version} + + + + + + + + org.apache.maven.plugins + maven-compiler-plugin + 2.0.2 + + 1.7 + 1.7 + + + + + diff --git a/samples/Ruby/filenames/Gemfile.lock b/samples/Ruby/filenames/Gemfile.lock new file mode 100644 index 00000000..22208028 --- /dev/null +++ b/samples/Ruby/filenames/Gemfile.lock @@ -0,0 +1,42 @@ +PATH + remote: . + specs: + github-linguist (4.0.1) + charlock_holmes (~> 0.7.3) + escape_utils (~> 1.0.1) + mime-types (>= 1.19) + rugged (~> 0.22.0b1) + github-linguist-grammars (4.0.1) + +GEM + remote: https://rubygems.org/ + specs: + charlock_holmes (0.7.3) + coderay (1.1.0) + escape_utils (1.0.1) + metaclass (0.0.4) + method_source (0.8.2) + mime-types (2.4.3) + mocha (1.1.0) + metaclass (~> 0.0.1) + plist (3.1.0) + pry (0.10.1) + coderay (~> 1.1.0) + method_source (~> 0.8.1) + slop (~> 3.4) + rake (10.3.2) + rugged (0.22.0b1) + slop (3.6.0) + yajl-ruby (1.2.1) + +PLATFORMS + ruby + +DEPENDENCIES + github-linguist! + github-linguist-grammars! + mocha + plist (~> 3.1) + pry + rake + yajl-ruby diff --git a/samples/TypeScript/empty.ts b/samples/TypeScript/empty.ts deleted file mode 100644 index e69de29b..00000000 diff --git a/test/test_blob.rb b/test/test_blob.rb index fe3f38ee..eb758534 100644 --- a/test/test_blob.rb +++ b/test/test_blob.rb @@ -193,8 +193,8 @@ class TestBlob < Test::Unit::TestCase assert blob("Binary/MainMenu.nib").generated? assert !blob("XML/project.pbxproj").generated? - # Gemfile.locks - assert blob("Gemfile.lock").generated? + # Gemfile.lock is NOT generated + assert !blob("Gemfile.lock").generated? # Generated .NET Docfiles assert blob("XML/net_docfile.xml").generated? @@ -226,7 +226,6 @@ class TestBlob < Test::Unit::TestCase assert !blob("PostScript/sierpinski.ps").generated? # These examples are too basic to tell - assert !blob("JavaScript/empty.js").generated? assert !blob("JavaScript/hello.js").generated? assert blob("JavaScript/intro-old.js").generated? @@ -298,6 +297,13 @@ class TestBlob < Test::Unit::TestCase assert blob("deps/http_parser/http_parser.c").vendored? assert blob("deps/v8/src/v8.h").vendored? + # Chart.js + assert blob("some/vendored/path/Chart.js").vendored? + assert !blob("some/vendored/path/chart.js").vendored? + + # Codemirror deps + assert blob("codemirror/mode/blah.js").vendored? + # Debian packaging assert blob("debian/cron.d").vendored? @@ -469,4 +475,13 @@ class TestBlob < Test::Unit::TestCase def test_minified_files_not_safe_to_highlight assert !blob("JavaScript/jquery-1.6.1.min.js").safe_to_colorize? end + + def test_empty + blob = Struct.new(:data) { include Linguist::BlobHelper } + + assert blob.new("").empty? + assert blob.new(nil).empty? + refute blob.new(" ").empty? + refute blob.new("nope").empty? + end end diff --git a/test/test_heuristics.rb b/test/test_heuristics.rb index e75fb470..db8f286e 100644 --- a/test/test_heuristics.rb +++ b/test/test_heuristics.rb @@ -20,18 +20,18 @@ class TestHeuristcs < Test::Unit::TestCase Dir.glob("#{samples_path}/#{language_name}/#{file}") end + # Candidate languages = ["C++", "Objective-C"] def test_obj_c_by_heuristics - languages = ["C++", "Objective-C"] # Only calling out '.h' filenames as these are the ones causing issues all_fixtures("Objective-C", "*.h").each do |fixture| - results = Heuristics.disambiguate_c(fixture("Objective-C/#{File.basename(fixture)}"), languages) + results = Heuristics.disambiguate_c(fixture("Objective-C/#{File.basename(fixture)}")) assert_equal Language["Objective-C"], results.first end end + # Candidate languages = ["C++", "Objective-C"] def test_cpp_by_heuristics - languages = ["C++", "Objective-C"] - results = Heuristics.disambiguate_c(fixture("C++/render_adapter.cpp"), languages) + results = Heuristics.disambiguate_c(fixture("C++/render_adapter.cpp")) assert_equal Language["C++"], results.first end @@ -41,57 +41,57 @@ class TestHeuristcs < Test::Unit::TestCase assert_equal Language["Objective-C"], match end + # Candidate languages = ["Perl", "Prolog"] def test_pl_prolog_by_heuristics - languages = ["Perl", "Prolog"] - results = Heuristics.disambiguate_pl(fixture("Prolog/turing.pl"), languages) + results = Heuristics.disambiguate_pl(fixture("Prolog/turing.pl")) assert_equal Language["Prolog"], results.first end + # Candidate languages = ["Perl", "Prolog"] def test_pl_perl_by_heuristics - languages = ["Perl", "Prolog"] - results = Heuristics.disambiguate_pl(fixture("Perl/perl-test.t"), languages) + results = Heuristics.disambiguate_pl(fixture("Perl/perl-test.t")) assert_equal Language["Perl"], results.first end + # Candidate languages = ["ECL", "Prolog"] def test_ecl_prolog_by_heuristics - languages = ["ECL", "Prolog"] - results = Heuristics.disambiguate_ecl(fixture("Prolog/or-constraint.ecl"), languages) + results = Heuristics.disambiguate_ecl(fixture("Prolog/or-constraint.ecl")) assert_equal Language["Prolog"], results.first end + # Candidate languages = ["ECL", "Prolog"] def test_ecl_ecl_by_heuristics - languages = ["ECL", "Prolog"] - results = Heuristics.disambiguate_ecl(fixture("ECL/sample.ecl"), languages) + results = Heuristics.disambiguate_ecl(fixture("ECL/sample.ecl")) assert_equal Language["ECL"], results.first end + # Candidate languages = ["IDL", "Prolog"] def test_pro_prolog_by_heuristics - languages = ["IDL", "Prolog"] - results = Heuristics.disambiguate_pro(fixture("Prolog/logic-problem.pro"), languages) + results = Heuristics.disambiguate_pro(fixture("Prolog/logic-problem.pro")) assert_equal Language["Prolog"], results.first end + # Candidate languages = ["IDL", "Prolog"] def test_pro_idl_by_heuristics - languages = ["IDL", "Prolog"] - results = Heuristics.disambiguate_pro(fixture("IDL/mg_acosh.pro"), languages) + results = Heuristics.disambiguate_pro(fixture("IDL/mg_acosh.pro")) assert_equal Language["IDL"], results.first end + # Candidate languages = ["AGS Script", "AsciiDoc"] def test_asc_asciidoc_by_heuristics - languages = ["AGS Script", "AsciiDoc"] - results = Heuristics.disambiguate_asc(fixture("AsciiDoc/list.asc"), languages) + results = Heuristics.disambiguate_asc(fixture("AsciiDoc/list.asc")) assert_equal Language["AsciiDoc"], results.first end + # Candidate languages = ["TypeScript", "XML"] def test_ts_typescript_by_heuristics - languages = ["TypeScript", "XML"] - results = Heuristics.disambiguate_ts(fixture("TypeScript/classes.ts"), languages) + results = Heuristics.disambiguate_ts(fixture("TypeScript/classes.ts")) assert_equal Language["TypeScript"], results.first end + # Candidate languages = ["TypeScript", "XML"] def test_ts_xml_by_heuristics - languages = ["TypeScript", "XML"] - results = Heuristics.disambiguate_ts(fixture("XML/pt_BR.xml"), languages) + results = Heuristics.disambiguate_ts(fixture("XML/pt_BR.xml")) assert_equal Language["XML"], results.first end @@ -99,27 +99,37 @@ class TestHeuristcs < Test::Unit::TestCase languages = ["Common Lisp", "OpenCL"] languages.each do |language| all_fixtures(language).each do |fixture| - results = Heuristics.disambiguate_cl(fixture("#{language}/#{File.basename(fixture)}"), languages) + results = Heuristics.disambiguate_cl(fixture("#{language}/#{File.basename(fixture)}")) assert_equal Language[language], results.first end end end + def test_f_by_heuristics + languages = ["FORTRAN", "Forth"] + languages.each do |language| + all_fixtures(language).each do |fixture| + results = Heuristics.disambiguate_f(fixture("#{language}/#{File.basename(fixture)}")) + assert_equal Language[language], results.first + end + end + end + + # Candidate languages = ["Hack", "PHP"] def test_hack_by_heuristics - languages = ["Hack", "PHP"] - results = Heuristics.disambiguate_hack(fixture("Hack/funs.php"), languages) + results = Heuristics.disambiguate_hack(fixture("Hack/funs.php")) assert_equal Language["Hack"], results.first end + # Candidate languages = ["Scala", "SuperCollider"] def test_sc_supercollider_by_heuristics - languages = ["Scala", "SuperCollider"] - results = Heuristics.disambiguate_sc(fixture("SuperCollider/WarpPreset.sc"), languages) + results = Heuristics.disambiguate_sc(fixture("SuperCollider/WarpPreset.sc")) assert_equal Language["SuperCollider"], results.first end + # Candidate languages = ["Scala", "SuperCollider"] def test_sc_scala_by_heuristics - languages = ["Scala", "SuperCollider"] - results = Heuristics.disambiguate_sc(fixture("Scala/node11.sc"), languages) + results = Heuristics.disambiguate_sc(fixture("Scala/node11.sc")) assert_equal Language["Scala"], results.first end end diff --git a/test/test_language.rb b/test/test_language.rb index cd48bc75..1ad47e33 100644 --- a/test/test_language.rb +++ b/test/test_language.rb @@ -1,5 +1,6 @@ require 'linguist/language' require 'test/unit' +require 'yaml' class TestLanguage < Test::Unit::TestCase include Linguist @@ -139,6 +140,7 @@ class TestLanguage < Test::Unit::TestCase assert_equal :programming, Language['Python'].type assert_equal :programming, Language['Ruby'].type assert_equal :programming, Language['TypeScript'].type + assert_equal :programming, Language['Makefile'].type end def test_markup @@ -157,7 +159,6 @@ class TestLanguage < Test::Unit::TestCase def test_other assert_nil Language['Brainfuck'].type - assert_nil Language['Makefile'].type end def test_searchable @@ -349,12 +350,6 @@ class TestLanguage < Test::Unit::TestCase assert_equal '.coffee', Language['CoffeeScript'].primary_extension assert_equal '.t', Language['Turing'].primary_extension assert_equal '.ts', Language['TypeScript'].primary_extension - - # This is a nasty requirement, but there's some code in GitHub that - # expects this. Really want to drop this. - Language.all.each do |language| - assert language.primary_extension, "#{language} has no primary extension" - end end def test_eql @@ -365,4 +360,15 @@ class TestLanguage < Test::Unit::TestCase def test_by_type assert !Language.by_type(:prose).nil? end + + def test_all_languages_have_grammars + scopes = YAML.load(File.read(File.expand_path("../../grammars.yml", __FILE__))).values.flatten + missing = Language.all.reject { |language| language.tm_scope == "none" || scopes.include?(language.tm_scope) } + message = "The following languages' scopes are not listed in grammars.yml. Please add grammars for all new languages.\n" + message << "If no grammar exists for a language, mark the language with `tm_scope: none` in lib/linguist/languages.yml.\n" + + width = missing.map { |language| language.name.length }.max + message << missing.map { |language| sprintf("%-#{width}s %s", language.name, language.tm_scope) }.sort.join("\n") + assert missing.empty?, message + end end diff --git a/test/test_pedantic.rb b/test/test_pedantic.rb index 264c54f3..c1819fb6 100644 --- a/test/test_pedantic.rb +++ b/test/test_pedantic.rb @@ -1,57 +1,30 @@ require 'test/unit' +require 'yaml' class TestPedantic < Test::Unit::TestCase - Lib = File.expand_path("../../lib/linguist", __FILE__) - - def file(name) - File.read(File.join(Lib, name)) - end + filename = File.expand_path("../../lib/linguist/languages.yml", __FILE__) + LANGUAGES = YAML.load(File.read(filename)) def test_language_names_are_sorted - languages = [] - file("languages.yml").lines.each do |line| - if line =~ /^(\w+):$/ - languages << $1 - end - end - assert_sorted languages + assert_sorted LANGUAGES.keys end def test_extensions_are_sorted - extensions = nil - file("languages.yml").lines.each do |line| - if line =~ /^ extensions:$/ - extensions = [] - elsif extensions && line =~ /^ - \.([\w\-\.]+)( *#.*)?$/ - extensions << $1 - else - assert_sorted extensions[1..-1] if extensions - extensions = nil - end + LANGUAGES.each do |name, language| + extensions = language['extensions'] + assert_sorted extensions[1..-1] if extensions && extensions.size > 1 end end def test_filenames_are_sorted - filenames = nil - file("languages.yml").lines.each do |line| - if line =~ /^ filenames:$/ - filenames = [] - elsif filenames && line =~ /^ - \.(\w+)$/ - filenames << $1 - else - assert_sorted filenames if filenames - filenames = nil - end + LANGUAGES.each do |name, language| + assert_sorted language['filenames'] if language['filenames'] end end def assert_sorted(list) - previous = nil - list.each do |item| - if previous && previous > item - flunk "#{previous} should come after #{item}" - end - previous = item + list.each_cons(2) do |previous, item| + flunk "#{previous} should come after #{item}" if previous > item end end end diff --git a/test/test_samples.rb b/test/test_samples.rb index 992618cf..929a7a00 100644 --- a/test/test_samples.rb +++ b/test/test_samples.rb @@ -55,20 +55,28 @@ class TestSamples < Test::Unit::TestCase end # If a language extension isn't globally unique then make sure there are samples - def test_presence - Linguist::Language.all.each do |language| - language.all_extensions.each do |extension| - language_matches = Language.find_by_filename("foo#{extension}") + Linguist::Language.all.each do |language| + define_method "test_#{language.name}_has_samples" do + language.extensions.each do |extension| + language_matches = Language.find_by_extension(extension) - # If there is more than one language match for a given extension - # then check that there are examples for that language with the extension + # Check for samples if more than one language matches the given extension. if language_matches.length > 1 - language_matches.each do |language| - assert File.directory?("samples/#{language.name}"), "#{language.name} is missing a samples directory" - assert Dir.glob("samples/#{language.name}/*#{extension}").any?, "#{language.name} is missing samples for extension #{extension}" + language_matches.each do |match| + samples = "samples/#{match.name}/*#{extension}" + assert Dir.glob(samples).any?, "Missing samples in #{samples.inspect}. See https://github.com/github/linguist/blob/master/CONTRIBUTING.md" end end end + + language.filenames.each do |filename| + # Check for samples if more than one language matches the given filename + if Language.find_by_filename(filename).size > 1 + sample = "samples/#{language.name}/filenames/#{filename}" + assert File.exists?(sample), + "Missing sample in #{sample.inspect}. See https://github.com/github/linguist/blob/master/CONTRIBUTING.md" + end + end end end end diff --git a/test/test_tokenizer.rb b/test/test_tokenizer.rb index 0521f4da..a460afc3 100644 --- a/test/test_tokenizer.rb +++ b/test/test_tokenizer.rb @@ -43,6 +43,8 @@ class TestTokenizer < Test::Unit::TestCase assert_equal %w(foo), tokenize("foo {- Comment -}") assert_equal %w(foo), tokenize("foo (* Comment *)") assert_equal %w(%), tokenize("2 % 10\n% Comment") + assert_equal %w(foo bar), tokenize("foo\n\"\"\"\nComment\n\"\"\"\nbar") + assert_equal %w(foo bar), tokenize("foo\n'''\nComment\n'''\nbar") end def test_sgml_tags diff --git a/vendor/cache/mime-types-1.25.1.gem b/vendor/cache/mime-types-1.25.1.gem deleted file mode 100644 index 877d8a97..00000000 Binary files a/vendor/cache/mime-types-1.25.1.gem and /dev/null differ diff --git a/vendor/cache/mime-types-2.4.3.gem b/vendor/cache/mime-types-2.4.3.gem new file mode 100644 index 00000000..670e7110 Binary files /dev/null and b/vendor/cache/mime-types-2.4.3.gem differ diff --git a/vendor/cache/rugged-0.21.1b2.gem b/vendor/cache/rugged-0.21.1b2.gem deleted file mode 100644 index 1d5a7dc8..00000000 Binary files a/vendor/cache/rugged-0.21.1b2.gem and /dev/null differ diff --git a/vendor/cache/rugged-0.22.0b4.gem b/vendor/cache/rugged-0.22.0b4.gem new file mode 100644 index 00000000..5f8a68bc Binary files /dev/null and b/vendor/cache/rugged-0.22.0b4.gem differ diff --git a/vendor/cache/yajl-ruby-1.2.1.gem b/vendor/cache/yajl-ruby-1.2.1.gem new file mode 100644 index 00000000..1d70c76b Binary files /dev/null and b/vendor/cache/yajl-ruby-1.2.1.gem differ