KevinMidboe/linguist
1
0
Fork 0
mirror of https://github.com/KevinMidboe/linguist.git synced 2025-12-08 20:38:47 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #1 from pchaigno/pythonmultiline

Browse Source 
Multiline comments for Python during tokenization
This commit is contained in:
Joseph Hall
2015-01-24 09:00:00 -07:00
parent 7702583314 c5a654e692
commit feeceefe99
40 changed files with 1651 additions and 237 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.gitignore vendored
View File

@@ -1,4 +1,4 @@
Gemfile.lock
/Gemfile.lock
.bundle/
benchmark/
lib/linguist/samples.json

1
.travis.yml
View File

@@ -2,6 +2,7 @@ before_install:
- git fetch origin master:master
- git fetch origin v2.0.0:v2.0.0
- git fetch origin test/attributes:test/attributes
- git fetch origin test/master:test/master
- sudo apt-get install libicu-dev -y
rvm:
- 1.9.3

31
CONTRIBUTING.md Normal file
View File

@@ -0,0 +1,31 @@
## Contributing
The majority of contributions won't need to touch any Ruby code at all. The [master language list][languages] is just a YAML configuration file.
Almost all bug fixes or new language additions should come with some additional code samples. Just drop them under [`samples/`][samples] in the correct subdirectory and our test suite will automatically test them. In most cases you shouldn't need to add any new assertions.
### My code is detected as the wrong language
This can usually be solved either by adding a new filename or file name extension to the language's entry in [`languages.yml`][languages] or adding more [samples][samples] for your language to the repository to make Linguist's classifier smarter.
### Syntax highlighting looks wrong
Assuming your code is being detected as the right language (see above), in most cases this is due to a bug in the language grammar rather than a bug in Linguist. [`grammars.yml`][grammars] lists all the grammars we use for syntax highlighting on github.com. Find the one corresponding to your code's programming language and submit a bug report upstream.
You can also try to fix the bug yourself and submit a Pull Request. [This piece from TextMate's documentation](http://manual.macromates.com/en/language_grammars) offers a good introduction on how to work with TextMate-compatible grammars.
Once the bug has been fixed upstream, please let us know and we'll pick it up for GitHub.
### I want to add support for the `X` programming language
Great! You'll need to:
0. Add an entry for your language to [`languages.yml`][languages].
0. Add a grammar for your language to [`grammars.yml`][grammars] by running `script/download-grammars --add URL`. Please only add grammars that have a license that permits redistribution.
0. Add samples for your language to the [samples directory][samples].
We try only to add languages once they have some usage on GitHub, so please note in-the-wild usage examples in your pull request. In most cases we prefer that languages already be in use in hundreds of repositories before supporting them in Linguist.
[grammars]: /grammars.yml
[languages]: /lib/linguist/languages.yml
[samples]: /samples

20
README.md
View File

@@ -2,6 +2,8 @@
We use this library at GitHub to detect blob languages, ignore binary files, suppress generated files in diffs, and generate language breakdown graphs.
Tips for filing issues and creating pull requests can be found in [`CONTRIBUTING.md`](/CONTRIBUTING.md).
## Features
### Language detection
@@ -26,15 +28,7 @@ See [lib/linguist/language.rb](https://github.com/github/linguist/blob/master/li
Syntax highlighting in GitHub is performed using TextMate-compatible grammars. These are the same grammars that TextMate, Sublime Text and Atom use.
Every language in `languages.yml` is mapped to its corresponding TM `scope`. This scope will be used when picking up a grammar for highlighting. **When adding a new language to Linguist, please add its corrsponding scope too (assuming there's an existing TextMate bundle) so syntax highlighting works for it**.
#### I found a bug!
The `grammars.yml` file contains the list of all the repositories where we fetch TextMate grammars for highlighting. If you find a bug in the highlighting for any given language, please consult this Grammars list to find the source of the grammar, and submit the bug report upstream.
You can also try to fix the bug yourself and submit a Pull Request. [This piece from TextMate's documentation](http://manual.macromates.com/en/language_grammars) offers a good introduction on how to work with TextMate-compatible grammars.
Once the bug has been fixed upstream, please let us know and we'll pick it up for GitHub.
Every language in `languages.yml` is mapped to its corresponding TM `scope`. This scope will be used when picking up a grammar for highlighting. **When adding a new language to Linguist, please add its corresponding scope too (assuming there's an existing TextMate bundle, Sublime Text package, or Atom package) so syntax highlighting works for it**.
### Stats
@@ -153,14 +147,6 @@ To run the tests:
bundle exec rake test
## Contributing
The majority of contributions won't need to touch any Ruby code at all. The [master language list](https://github.com/github/linguist/blob/master/lib/linguist/languages.yml) is just a YAML configuration file.
We try to only add languages once they have some usage on GitHub, so please note in-the-wild usage examples in your pull request.
Almost all bug fixes or new language additions should come with some additional code samples. Just drop them under [`samples/`](https://github.com/github/linguist/tree/master/samples) in the correct subdirectory and our test suite will automatically test them. In most cases you shouldn't need to add any new assertions.
### A note on language extensions
Linguist has a number of methods available to it for identifying the language of a particular file. The initial lookup is based upon the extension of the file, possible file extensions are defined in an array called `extensions`. Take a look at this example for example for `Perl`:

4
github-linguist.gemspec
View File

@@ -15,8 +15,8 @@ Gem::Specification.new do |s|
s.add_dependency 'charlock_holmes', '~> 0.7.3'
s.add_dependency 'escape_utils', '~> 1.0.1'
s.add_dependency 'mime-types', '~> 1.19'
s.add_dependency 'rugged', '~> 0.21.1b2'
s.add_dependency 'mime-types', '>= 1.19'
s.add_dependency 'rugged', '~> 0.22.0b4'
s.add_development_dependency 'mocha'
s.add_development_dependency 'pry'

21
grammars.yml
View File

@@ -1,6 +1,4 @@
---
http://hww3.riverweb.com/dist/Pike_TextMate.tar.gz:
- source.pike
http://svn.edgewall.org/repos/genshi/contrib/textmate/Genshi.tmbundle/Syntaxes/Markup%20Template%20%28XML%29.tmLanguage:
- text.xml.genshi
http://svn.textmate.org/trunk/Review/Bundles/BlitzMax.tmbundle:
@@ -40,10 +38,14 @@ https://github.com/Drako/SublimeBrainfuck/raw/master/Brainfuck.tmLanguage:
- source.bf
https://github.com/JohnNilsson/awk-sublime/raw/master/AWK.tmLanguage:
- source.awk
https://github.com/JonBons/Sublime-SQF-Language:
- source.sqf
https://github.com/MarioRicalde/SCSS.tmbundle:
- source.scss
https://github.com/Oldes/Sublime-REBOL:
- source.rebol
https://github.com/PogiNate/Sublime-Inform:
- source.Inform7
https://github.com/Red-Nova-Technologies/autoitv3-tmbundle:
- source.autoit.3
https://github.com/SalGnt/Sublime-VimL:
@@ -70,6 +72,8 @@ https://github.com/aroben/ruby.tmbundle@4636a3023153c3034eb6ffc613899ba9cf33b41f
- text.html.erb
https://github.com/asbjornenge/Docker.tmbundle:
- source.dockerfile
https://github.com/atom/language-clojure:
- source.clojure
https://github.com/atom/language-coffee-script:
- source.coffee
- source.litcoffee
@@ -95,6 +99,8 @@ https://github.com/bholt/chapel-tmbundle:
- source.chapel
https://github.com/brandonwamboldt/sublime-nginx:
- source.nginx
https://github.com/bro/bro-sublime:
- source.bro
https://github.com/carsonoid/sublime_man_page_support/raw/master/man-groff.tmLanguage:
- text.groff
https://github.com/ccreutzig/sublime-MuPAD:
@@ -135,6 +141,8 @@ https://github.com/guillermooo/dart-sublime-bundle/raw/master/Dart.tmLanguage:
- source.dart
https://github.com/harrism/sublimetext-cuda-cpp/raw/master/cuda-c%2B%2B.tmLanguage:
- source.cuda-c++
https://github.com/hww3/pike-textmate:
- source.pike
https://github.com/jeancharles-roger/ceylon-sublimetext/raw/master/Ceylon.tmLanguage:
- source.ceylon
https://github.com/jfairbank/Sublime-Text-2-OpenEdge-ABL:
@@ -211,16 +219,11 @@ https://github.com/slavapestov/factor/raw/master/misc/Factor.tmbundle/Syntaxes/F
- source.factor
https://github.com/slim-template/ruby-slim.tmbundle:
- text.slim
https://github.com/smiledawgg/Bro.tmbundle:
- source.bro
- source.bro.sig
https://github.com/staltz/SublimeXtend:
- source.xtend
https://github.com/statatmbundle/Stata.tmbundle:
- source.mata
- source.stata
https://github.com/swannodette/textmate-clojure:
- source.clojure
https://github.com/technosophos/Vala-TMBundle:
- source.vala
https://github.com/textmate/ant.tmbundle:
@@ -240,6 +243,7 @@ https://github.com/textmate/bison.tmbundle:
https://github.com/textmate/c.tmbundle:
- source.c
- source.c++
- source.c.platform
https://github.com/textmate/capnproto.tmbundle:
- source.capnp
https://github.com/textmate/cmake.tmbundle:
@@ -277,8 +281,6 @@ https://github.com/textmate/haskell.tmbundle:
- text.tex.latex.haskell
https://github.com/textmate/html.tmbundle:
- text.html.basic
https://github.com/textmate/inform.tmbundle:
- source.inform
https://github.com/textmate/ini.tmbundle:
- source.ini
https://github.com/textmate/io.tmbundle:
@@ -327,6 +329,7 @@ https://github.com/textmate/ninja.tmbundle:
https://github.com/textmate/objective-c.tmbundle:
- source.objc
- source.objc++
- source.objc.platform
- source.strings
https://github.com/textmate/ocaml.tmbundle:
- source.camlp4.ocaml

7
lib/linguist/blob_helper.rb
View File

@@ -146,6 +146,13 @@ module Linguist
end
end
# Public: Is the blob empty?
#
# Return true or false
def empty?
data.nil? || data == ""
end
# Public: Is the blob text?
#
# Return true or false

32
lib/linguist/generated.rb
View File

@@ -51,26 +51,25 @@ module Linguist
#
# Return true or false
def generated?
name == 'Gemfile.lock' ||
minified_files? ||
compiled_coffeescript? ||
xcode_file? ||
generated_parser? ||
generated_net_docfile? ||
generated_net_designer_file? ||
generated_postscript? ||
generated_protocol_buffer? ||
generated_jni_header? ||
composer_lock? ||
node_modules? ||
godeps? ||
vcr_cassette? ||
generated_by_zephir?
minified_files? ||
compiled_coffeescript? ||
xcode_file? ||
generated_parser? ||
generated_net_docfile? ||
generated_net_designer_file? ||
generated_postscript? ||
generated_protocol_buffer? ||
generated_jni_header? ||
composer_lock? ||
node_modules? ||
godeps? ||
vcr_cassette? ||
generated_by_zephir?
end
# Internal: Is the blob an Xcode file?
#
# Generated if the file extension is an Xcode
# Generated if the file extension is an Xcode
# file extension.
#
# Returns true of false.
@@ -265,4 +264,3 @@ module Linguist
end
end
end

77
lib/linguist/heuristics.rb
View File

@@ -13,26 +13,31 @@ module Linguist
# Returns an array of Languages or []
def self.find_by_heuristics(data, languages)
if active?
result = []
if languages.all? { |l| ["Perl", "Prolog"].include?(l) }
result = disambiguate_pl(data, languages)
result = disambiguate_pl(data)
end
if languages.all? { |l| ["ECL", "Prolog"].include?(l) }
result = disambiguate_ecl(data, languages)
result = disambiguate_ecl(data)
end
if languages.all? { |l| ["IDL", "Prolog"].include?(l) }
result = disambiguate_pro(data, languages)
result = disambiguate_pro(data)
end
if languages.all? { |l| ["Common Lisp", "OpenCL"].include?(l) }
result = disambiguate_cl(data, languages)
result = disambiguate_cl(data)
end
if languages.all? { |l| ["Hack", "PHP"].include?(l) }
result = disambiguate_hack(data, languages)
result = disambiguate_hack(data)
end
if languages.all? { |l| ["Scala", "SuperCollider"].include?(l) }
result = disambiguate_sc(data, languages)
result = disambiguate_sc(data)
end
if languages.all? { |l| ["AsciiDoc", "AGS Script"].include?(l) }
result = disambiguate_asc(data, languages)
result = disambiguate_asc(data)
end
if languages.all? { |l| ["FORTRAN", "Forth"].include?(l) }
result = disambiguate_f(data)
end
return result
end
@@ -42,28 +47,37 @@ module Linguist
# We want to shortcut look for Objective-C _and_ now C++ too!
#
# Returns an array of Languages or []
def self.disambiguate_c(data, languages)
def self.disambiguate_c(data)
matches = []
matches << Language["Objective-C"] if data.include?("@interface")
matches << Language["C++"] if data.include?("#include <cstdint>")
if data.include?("@interface")
matches << Language["Objective-C"]
elsif data.include?("#include <cstdint>")
matches << Language["C++"]
end
matches
end
def self.disambiguate_pl(data, languages)
def self.disambiguate_pl(data)
matches = []
matches << Language["Prolog"] if data.include?(":-")
matches << Language["Perl"] if data.include?("use strict")
if data.include?("use strict")
matches << Language["Perl"]
elsif data.include?(":-")
matches << Language["Prolog"]
end
matches
end
def self.disambiguate_ecl(data, languages)
def self.disambiguate_ecl(data)
matches = []
matches << Language["Prolog"] if data.include?(":-")
matches << Language["ECL"] if data.include?(":=")
if data.include?(":-")
matches << Language["Prolog"]
elsif data.include?(":=")
matches << Language["ECL"]
end
matches
end
def self.disambiguate_pro(data, languages)
def self.disambiguate_pro(data)
matches = []
if (data.include?(":-"))
matches << Language["Prolog"]
@@ -73,7 +87,7 @@ module Linguist
matches
end
def self.disambiguate_ts(data, languages)
def self.disambiguate_ts(data)
matches = []
if (data.include?("</translation>"))
matches << Language["XML"]
@@ -83,21 +97,24 @@ module Linguist
matches
end
def self.disambiguate_cl(data, languages)
def self.disambiguate_cl(data)
matches = []
matches << Language["Common Lisp"] if data.include?("(defun ")
matches << Language["OpenCL"] if /\/\* |\/\/ |^\}/.match(data)
if data.include?("(defun ")
matches << Language["Common Lisp"]
elsif /\/\* |\/\/ |^\}/.match(data)
matches << Language["OpenCL"]
end
matches
end
def self.disambiguate_r(data, languages)
def self.disambiguate_r(data)
matches = []
matches << Language["Rebol"] if /\bRebol\b/i.match(data)
matches << Language["R"] if data.include?("<-")
matches
end
def self.disambiguate_hack(data, languages)
def self.disambiguate_hack(data)
matches = []
if data.include?("<?hh")
matches << Language["Hack"]
@@ -107,7 +124,7 @@ module Linguist
matches
end
def self.disambiguate_sc(data, languages)
def self.disambiguate_sc(data)
matches = []
if (/\^(this|super)\./.match(data) || /^\s*(\+|\*)\s*\w+\s*{/.match(data) || /^\s*~\w+\s*=\./.match(data))
matches << Language["SuperCollider"]
@@ -118,12 +135,22 @@ module Linguist
matches
end
def self.disambiguate_asc(data, languages)
def self.disambiguate_asc(data)
matches = []
matches << Language["AsciiDoc"] if /^=+(\s|\n)/.match(data)
matches
end
def self.disambiguate_f(data)
matches = []
if /^: /.match(data)
matches << Language["Forth"]
elsif /^([c*][^a-z]| subroutine\s)/i.match(data)
matches << Language["FORTRAN"]
end
matches
end
def self.active?
!!ACTIVE
end

31
lib/linguist/language.rb
View File

@@ -100,12 +100,8 @@ module Linguist
def self.detect(blob)
name = blob.name.to_s
# Check if the blob is possibly binary and bail early; this is a cheap
# test that uses the extension name to guess a binary binary mime type.
#
# We'll perform a more comprehensive test later which actually involves
# looking for binary characters in the blob
return nil if blob.likely_binary? || blob.binary?
# Bail early if the blob is binary or empty.
return nil if blob.likely_binary? || blob.binary? || blob.empty?
# A bit of an elegant hack. If the file is executable but extensionless,
# append a "magic" extension so it can be classified with other
@@ -124,16 +120,18 @@ module Linguist
if possible_languages.length > 1
data = blob.data
possible_language_names = possible_languages.map(&:name)
heuristic_languages = Heuristics.find_by_heuristics(data, possible_language_names)
if heuristic_languages.size > 1
possible_language_names = heuristic_languages.map(&:name)
end
# Don't bother with binary contents or an empty file
if data.nil? || data == ""
nil
# Check if there's a shebang line and use that as authoritative
elsif (result = find_by_shebang(data)) && !result.empty?
if (result = find_by_shebang(data)) && !result.empty?
result.first
# No shebang. Still more work to do. Try to find it with our heuristics.
elsif (determined = Heuristics.find_by_heuristics(data, possible_language_names)) && !determined.empty?
determined.first
elsif heuristic_languages.size == 1
heuristic_languages.first
# Lastly, fall back to the probabilistic classifier.
elsif classified = Classifier.classify(Samples.cache, data, possible_language_names).first
# Return the actual Language object based of the string language name (i.e., first element of `#classify`)
@@ -433,11 +431,6 @@ module Linguist
# Returns the extensions Array
attr_reader :filenames
# Public: Return all possible extensions for language
def all_extensions
(extensions + [primary_extension]).uniq
end
# Deprecated: Get primary extension
#
# Defaults to the first extension but can be overridden
@@ -595,9 +588,9 @@ module Linguist
:ace_mode => options['ace_mode'],
:wrap => options['wrap'],
:group_name => options['group'],
:searchable => options.key?('searchable') ? options['searchable'] : true,
:searchable => options.fetch('searchable', true),
:search_term => options['search_term'],
:extensions => [options['extensions'].first] + options['extensions'][1..-1].sort,
:extensions => Array(options['extensions']),
:interpreters => options['interpreters'].sort,
:filenames => options['filenames'],
:popular => popular.include?(name)

192
lib/linguist/languages.yml
View File

@@ -12,6 +12,10 @@
# search_term - Deprecated: Some languages maybe indexed under a
# different alias. Avoid defining new exceptions.
# color - CSS hex color to represent the language.
# tm_scope - The TextMate scope that represents this programming
# language. This should match one of the scopes listed in
# the grammars.yml file. Use "none" if there is no grammar
# for this language.
#
# Any additions or modifications (even trivial) should have corresponding
# test change in `test/test_blob.rb`.
@@ -33,15 +37,6 @@ AGS Script:
- .ash
tm_scope: source.c++
Ant Build System:
type: data
tm_scope: text.xml.ant
extensions:
- .ant.xml
filenames:
- build.xml
- ant.xml
ANTLR:
type: programming
color: "#9DC3FF"
@@ -54,6 +49,7 @@ APL:
extensions:
- .apl
- .dyalog
tm_scope: none
ASP:
type: programming
@@ -117,6 +113,13 @@ Alloy:
extensions:
- .als
Ant Build System:
type: data
tm_scope: text.xml.ant
filenames:
- ant.xml
- build.xml
ApacheConf:
type: markup
aliases:
@@ -148,6 +151,7 @@ Arc:
color: "#ca2afe"
extensions:
- .arc
tm_scope: none
Arduino:
type: programming
@@ -164,12 +168,14 @@ AsciiDoc:
- .asciidoc
- .adoc
- .asc
tm_scope: none
AspectJ:
type: programming
color: "#1957b0"
extensions:
- .aj
tm_scope: none
Assembly:
type: programming
@@ -187,6 +193,7 @@ Augeas:
type: programming
extensions:
- .aug
tm_scope: none
AutoHotkey:
type: programming
@@ -196,6 +203,7 @@ AutoHotkey:
extensions:
- .ahk
- .ahkl
tm_scope: none
AutoIt:
type: programming
@@ -288,6 +296,7 @@ Brightscript:
type: programming
extensions:
- .brs
tm_scope: none
Bro:
type: programming
@@ -361,6 +370,7 @@ CLIPS:
type: programming
extensions:
- .clp
tm_scope: none
CMake:
extensions:
@@ -423,6 +433,7 @@ Clean:
extensions:
- .icl
- .dcl
tm_scope: none
Clojure:
type: programming
@@ -451,6 +462,7 @@ CoffeeScript:
extensions:
- .coffee
- ._coffee
- .cjsx
- .cson
- .iced
filenames:
@@ -539,6 +551,7 @@ Creole:
wrap: true
extensions:
- .creole
tm_scope: none
Crystal:
type: programming
@@ -606,6 +619,7 @@ Darcs Patch:
extensions:
- .darcspatch
- .dpatch
tm_scope: none
Dart:
type: programming
@@ -633,6 +647,7 @@ Dogescript:
color: "#cca760"
extensions:
- .djs
tm_scope: none
Dylan:
type: programming
@@ -648,13 +663,7 @@ E:
color: "#ccce35"
extensions:
- .E
Ecere Projects:
type: data
group: JavaScript
extensions:
- .epj
tm_scope: source.json
tm_scope: none
ECL:
type: programming
@@ -662,6 +671,7 @@ ECL:
extensions:
- .ecl
- .eclxml
tm_scope: none
Eagle:
type: markup
@@ -671,6 +681,13 @@ Eagle:
- .brd
tm_scope: text.xml
Ecere Projects:
type: data
group: JavaScript
extensions:
- .epj
tm_scope: source.json
Eiffel:
type: programming
color: "#946d57"
@@ -738,6 +755,7 @@ FLUX:
extensions:
- .fx
- .flux
tm_scope: none
FORTRAN:
type: programming
@@ -767,8 +785,8 @@ Factor:
extensions:
- .factor
filenames:
- .factor-rc
- .factor-boot-rc
- .factor-rc
Fancy:
type: programming
@@ -792,6 +810,9 @@ Forth:
extensions:
- .fth
- .4th
- .F
- .f
- .for
- .forth
- .frt
@@ -808,18 +829,13 @@ G-code:
- .g
- .gco
- .gcode
Game Maker Language:
type: programming
color: "#8ad353"
extensions:
- .gml
tm_scope: source.js
tm_scope: none
GAMS:
type: programming
extensions:
- .gms
tm_scope: none
GAP:
type: programming
@@ -828,6 +844,7 @@ GAP:
- .gap
- .gd
- .gi
tm_scope: none
GAS:
type: programming
@@ -841,6 +858,7 @@ GDScript:
type: programming
extensions:
- .gd
tm_scope: none
GLSL:
group: C
@@ -860,6 +878,13 @@ GLSL:
- .vrx
- .vshader
Game Maker Language:
type: programming
color: "#8ad353"
extensions:
- .gml
tm_scope: source.js
Genshi:
extensions:
- .kid
@@ -918,6 +943,7 @@ Golo:
color: "#f6a51f"
extensions:
- .golo
tm_scope: none
Gosu:
type: programming
@@ -933,6 +959,7 @@ Grace:
type: programming
extensions:
- .grace
tm_scope: none
Grammatical Framework:
type: programming
@@ -949,6 +976,7 @@ Graph Modeling Language:
type: data
extensions:
- .gml
tm_scope: none
Graphviz (DOT):
type: data
@@ -1003,6 +1031,7 @@ HTML:
- .html
- .htm
- .st
- .xht
- .xhtml
HTML+Django:
@@ -1038,6 +1067,7 @@ HTTP:
type: data
extensions:
- .http
tm_scope: none
Hack:
type: programming
@@ -1045,6 +1075,7 @@ Hack:
extensions:
- .hh
- .php
tm_scope: none
Haml:
group: HTML
@@ -1067,6 +1098,7 @@ Harbour:
color: "#0e60e3"
extensions:
- .hb
tm_scope: none
Haskell:
type: programming
@@ -1092,6 +1124,7 @@ Hy:
- .hy
aliases:
- hylang
tm_scope: none
IDL:
type: programming
@@ -1107,6 +1140,7 @@ IGOR Pro:
aliases:
- igor
- igorpro
tm_scope: none
INI:
type: data
@@ -1119,9 +1153,15 @@ INI:
aliases:
- dosini
Inno Setup:
IRC log:
search_term: irc
aliases:
- irc
- irc logs
extensions:
- .iss
- .irclog
- .weechatlog
tm_scope: none
Idris:
type: programming
@@ -1135,7 +1175,7 @@ Inform 7:
extensions:
- .ni
- .i7x
tm_scope: source.inform
tm_scope: source.Inform7
aliases:
- i7
- inform7
@@ -1143,15 +1183,7 @@ Inform 7:
Inno Setup:
extensions:
- .iss
IRC log:
search_term: irc
aliases:
- irc
- irc logs
extensions:
- .irclog
- .weechatlog
tm_scope: none
Io:
type: programming
@@ -1170,11 +1202,13 @@ Isabelle:
color: "#fdcd00"
extensions:
- .thy
tm_scope: none
J:
type: programming
extensions:
- .ijs
tm_scope: none
JSON:
type: data
@@ -1282,6 +1316,7 @@ KRL:
color: "#f5c800"
extensions:
- .krl
tm_scope: none
Kit:
type: markup
@@ -1315,6 +1350,7 @@ LOLCODE:
extensions:
- .lol
color: "#cc9900"
tm_scope: none
LSL:
type: programming
@@ -1329,6 +1365,7 @@ LabVIEW:
type: programming
extensions:
- .lvproj
tm_scope: none
Lasso:
type: programming
@@ -1367,12 +1404,14 @@ Liquid:
type: markup
extensions:
- .liquid
tm_scope: none
Literate Agda:
type: programming
group: Agda
extensions:
- .lagda
tm_scope: none
Literate CoffeeScript:
type: programming
@@ -1462,6 +1501,7 @@ MTML:
tm_scope: text.html.basic
Makefile:
type: programming
aliases:
- bsdmake
- make
@@ -1470,9 +1510,9 @@ Makefile:
- .mak
- .mk
filenames:
- makefile
- Makefile
- GNUmakefile
- Makefile
- makefile
interpreters:
- make
@@ -1525,8 +1565,6 @@ Matlab:
Maven POM:
type: data
tm_scope: text.xml.pom
extensions:
- .pom.xml
filenames:
- pom.xml
@@ -1550,6 +1588,7 @@ MediaWiki:
wrap: true
extensions:
- .mediawiki
tm_scope: none
Mercury:
type: programming
@@ -1564,6 +1603,7 @@ MiniD: # Legacy
searchable: false
extensions:
- .minid # Dummy extension
tm_scope: none
Mirah:
type: programming
@@ -1585,6 +1625,7 @@ Moocode:
type: programming
extensions:
- .moo
tm_scope: none
MoonScript:
type: programming
@@ -1596,6 +1637,7 @@ MoonScript:
Myghty:
extensions:
- .myt
tm_scope: none
NSIS:
extensions:
@@ -1641,6 +1683,7 @@ Nit:
color: "#0d8921"
extensions:
- .nit
tm_scope: none
Nix:
type: programming
@@ -1648,6 +1691,7 @@ Nix:
- .nix
aliases:
- nixos
tm_scope: none
Nu:
type: programming
@@ -1666,6 +1710,7 @@ NumPy:
- .numpy
- .numpyw
- .numsc
tm_scope: none
OCaml:
type: programming
@@ -1726,6 +1771,7 @@ Omgrofl:
extensions:
- .omgrofl
color: "#cabbff"
tm_scope: none
Opa:
type: programming
@@ -1737,6 +1783,7 @@ Opal:
color: "#f7ede0"
extensions:
- .opal
tm_scope: none
OpenCL:
type: programming
@@ -1761,12 +1808,14 @@ OpenSCAD:
type: programming
extensions:
- .scad
tm_scope: none
Org:
type: prose
wrap: true
extensions:
- .org
tm_scope: none
Ox:
type: programming
@@ -1774,12 +1823,14 @@ Ox:
- .ox
- .oxh
- .oxo
tm_scope: none
Oxygene:
type: programming
color: "#5a63a3"
extensions:
- .oxygene
tm_scope: none
PAWN:
type: programming
@@ -1815,18 +1866,30 @@ Pan:
color: '#cc0000'
extensions:
- .pan
tm_scope: none
Papyrus:
type: programming
color: "#6600cc"
extensions:
- .psc
tm_scope: none
Parrot:
type: programming
color: "#f3ca0a"
extensions:
- .parrot # Dummy extension
tm_scope: none
Parrot Assembly:
group: Parrot
type: programming
aliases:
- pasm
extensions:
- .pasm
tm_scope: none
Parrot Internal Representation:
group: Parrot
@@ -1837,14 +1900,6 @@ Parrot Internal Representation:
extensions:
- .pir
Parrot Assembly:
group: Parrot
type: programming
aliases:
- pasm
extensions:
- .pasm
Pascal:
type: programming
color: "#b0ce4e"
@@ -1886,12 +1941,14 @@ Perl6:
- .p6m
- .pl6
- .pm6
tm_scope: none
PigLatin:
type: programming
color: "#fcd7de"
extensions:
- .pig
tm_scope: none
Pike:
type: programming
@@ -1906,12 +1963,14 @@ Pod:
wrap: true
extensions:
- .pod
tm_scope: none
PogoScript:
type: programming
color: "#d80074"
extensions:
- .pogo
tm_scope: none
PostScript:
type: markup
@@ -1952,6 +2011,7 @@ Propeller Spin:
color: "#2b446d"
extensions:
- .spin
tm_scope: none
Protocol Buffer:
type: markup
@@ -1975,6 +2035,7 @@ Pure Data:
color: "#91de79"
extensions:
- .pd
tm_scope: none
PureScript:
type: programming
@@ -2001,10 +2062,10 @@ Python:
- .wsgi
- .xpy
filenames:
- wscript
- SConstruct
- SConscript
- BUILD
- SConscript
- SConstruct
- wscript
interpreters:
- python
@@ -2070,7 +2131,7 @@ RHTML:
group: HTML
extensions:
- .rhtml
tm_scope: text.html.ruby
tm_scope: text.html.erb
aliases:
- html+ruby
@@ -2081,6 +2142,7 @@ RMarkdown:
extensions:
- .rmd
- .Rmd
tm_scope: none
Racket:
type: programming
@@ -2100,6 +2162,7 @@ Ragel in Ruby Host:
aliases:
- ragel-rb
- ragel-ruby
tm_scope: none
Raw token data:
search_term: raw
@@ -2107,6 +2170,7 @@ Raw token data:
- raw
extensions:
- .raw
tm_scope: none
Rebol:
type: programming
@@ -2126,10 +2190,12 @@ Red:
- .reds
aliases:
- red/system
tm_scope: none
Redcode:
extensions:
- .cw
tm_scope: none
RobotFramework:
type: programming
@@ -2203,6 +2269,7 @@ SAS:
color: "#1E90FF"
extensions:
- .sas
tm_scope: none
SCSS:
type: markup
@@ -2218,7 +2285,7 @@ SQF:
extensions:
- .sqf
- .hqf
tm_scope: source.c++
tm_scope: source.sqf
SQL:
type: data
@@ -2296,6 +2363,7 @@ Self:
color: "#0579aa"
extensions:
- .self
tm_scope: none
Shell:
type: programming
@@ -2334,6 +2402,7 @@ Shen:
color: "#120F14"
extensions:
- .shen
tm_scope: none
Slash:
type: programming
@@ -2404,6 +2473,7 @@ Stylus:
group: CSS
extensions:
- .styl
tm_scope: none
SuperCollider:
type: programming
@@ -2411,6 +2481,7 @@ SuperCollider:
extensions:
- .scd
- .sc
tm_scope: none
Swift:
type: programming
@@ -2436,6 +2507,7 @@ TXL:
type: programming
extensions:
- .txl
tm_scope: none
Tcl:
type: programming
@@ -2492,6 +2564,7 @@ Textile:
wrap: true
extensions:
- .textile
tm_scope: none
Thrift:
type: programming
@@ -2505,6 +2578,7 @@ Turing:
extensions:
- .t
- .tu
tm_scope: none
Twig:
type: markup
@@ -2584,8 +2658,8 @@ VimL:
filenames:
- .vimrc
- _vimrc
- vimrc
- gvimrc
- vimrc
Visual Basic:
type: programming
@@ -2687,12 +2761,12 @@ XML:
filenames:
- .classpath
- .project
- Web.Debug.config
- Web.Release.config
- Web.config
- build.xml.dist
- packages.config
- phpunit.xml.dist
- Web.config
- Web.Debug.config
- Web.Release.config
XProc:
type: programming
@@ -2765,6 +2839,7 @@ Zimpl:
- .zimpl
- .zmpl
- .zpl
tm_scope: none
eC:
type: programming
@@ -2772,6 +2847,7 @@ eC:
extensions:
- .ec
- .eh
tm_scope: none
edn:
type: data
@@ -2786,6 +2862,7 @@ fish:
group: Shell
extensions:
- .fish
tm_scope: none
mupad:
extensions:
@@ -2826,3 +2903,4 @@ xBase:
color: "#3a4040"
extensions:
- .prg
tm_scope: none

6
lib/linguist/vendor.yml
View File

@@ -110,6 +110,12 @@
# MathJax
- (^|/)MathJax/
# Chart.js
- (^|/)Chart\.js$
# Codemirror
- (^|/)[Cc]ode[Mm]irror/(lib|mode|theme|addon|keymap)
# SyntaxHighlighter - http://alexgorbatchev.com/
- (^|/)shBrush([^.]*)\.js$
- (^|/)shCore\.js$

2
lib/linguist/version.rb
View File

@@ -1,3 +1,3 @@
module Linguist
VERSION = "4.0.0"
VERSION = "4.0.3"
end

110
samples/Ant Build System/filenames/ant.xml Normal file
View File

@@ -0,0 +1,110 @@
<?xml version="1.0" encoding="iso-8859-1"?>
<project name="WebBuild">
<!-- generate timestamps -->
<tstamp />
<!-- Debugging Macro -->
<import file="echopath.xml" />
<!-- JS build files macro -->
<import file="rhinoscript.xml" />
<!-- Component Build Files -->
<import file="setup.xml" />
<import file="clean.xml" />
<import file="copy.xml" />
<import file="file.transform.xml" />
<import file="external.tools.xml" />
<import file="rename.xml" />
<import file="js.xml" />
<import file="css.xml" />
<import file="img.xml" />
<import file="png8.xml" />
<import file="yui.xml" />
<import file="cdn.xml" />
<import file="datauri.xml" />
<import file="devlive.xml" />
<!-- This dirname is the only complete path we know for sure, everything builds off of it -->
<dirname property="dir.build" file="${ant.file.WebBuild}" />
<!-- get name for newly built folder -->
<basename property="app.name" file="${basedir}" />
<!-- read global properties file -->
<property file="${dir.build}\build.properties" />
<!-- Build Directories -->
<property name="dir.build.js" location="${dir.build}/js" />
<!-- App Directories -->
<property name="dir.app" location="${dir.result}/${app.name}" />
<property name="dir.app.temp" location="${dir.temp}/${app.name}" />
<property name="dir.app.files" location="${dir.app.temp}/${dir.files}" />
<!-- Files -->
<property name="mapping.js" location="${dir.app.temp}/${mapping.file.js}" />
<property name="mapping.css" location="${dir.app.temp}/${mapping.file.css}" />
<property name="mapping.img" location="${dir.app.temp}/${mapping.file.img}" />
<property name="mapping.swf" location="${dir.app.temp}/${mapping.file.swf}" />
<property name="mapping.fonts" location="${dir.app.temp}/${mapping.file.fonts}" />
<!-- Tool Directories -->
<property name="dir.bin" location="${dir.build}/Bin" />
<property name="dir.jar" location="${dir.bin}/jar" />
<!-- Tool Files -->
<property name="tools.compressor" location="${dir.jar}/${tools.file.compressor}" />
<property name="tools.cssembed" location="${dir.jar}/${tools.file.cssembed}" />
<property name="tools.filetransform" location="${dir.jar}/${tools.file.filetransform}" />
<property name="tools.optipng" location="${dir.bin}/${tools.file.optipng}" />
<property name="tools.jpegtran" location="${dir.bin}/${tools.file.jpegtran}" />
<!-- BUILD TARGETS -->
<!-- low level utility build targets -->
<!-- Build the tools -->
<target name="-setup.build.tools"
depends="-define.filetransform, -define.cssembed, -define.yuicompressor, -define.jsclasspath"
/>
<!-- set up filesystem properties -->
<target
name="-setup"
depends="-setup.mode, -setup.conditions, -setup.js, -setup.css, -setup.swf, -setup.img, -setup.fonts, -setup.yui"
/>
<!-- utility-ish targets -->
<target name="copy" depends="clean, tools, -copy" />
<target name="tools" depends="-setup.build.tools" />
<target name="finalize" depends="copy, -finalize" />
<target name="-prepare" depends="copy, -setup" />
<!-- individual component build targets (empty descriptions are to make sure they show in "ant -p") -->
<target name="devlive" depends="-prepare, -devlive" description="" />
<target name="js" depends="-prepare, -js" description="" />
<target name="css" depends="-prepare, -css" description="" />
<target name="rename" depends="-prepare, -rename" description="" />
<target name="yui" depends="-prepare, rename, -yui" description="" />
<target name="cdn" depends="-prepare, -cdn" description="" />
<!-- high level build targets (Excluding of images is on purpose here, it's slow) -->
<target name="core"
depends="devlive, js, css, cdn, rename, yui, -js.inline"
description="Core build work"
/>
<target name="prod"
depends="core, finalize"
description="Full Production Build"
/>
<!-- debug target -->
<target name="debug" depends="-setup">
<echoproperties/>
</target>
</project>

1
samples/Ant Build System/filenames/build.xml Symbolic link
View File

@@ -0,0 +1 @@
ant.xml

0
samples/CoffeeScript/empty.coffee
View File

40
samples/CoffeeScript/example.cjsx Normal file
View File

@@ -0,0 +1,40 @@
###* @cjsx React.DOM ###
define 'myProject.ReactExampleComponent', [
'React'
'myProject.ExampleStore'
'myProject.ExampleActions'
'myProject.ReactExampleTable'
], (React, ExampleStore, ExampleActions, ReactExampleTable ) ->
ReactExampleComponent = React.createClass
mixins: [ListenMixin]
getInitialState: ->
rows: ExampleStore.getRows()
meta: ExampleStore.getMeta()
componentWillMount: ->
@listenTo ExampleStore
componentDidMount: ->
ExampleActions.getExampleData()
onStoreChange: ->
if this.isMounted()
@setState
rows: ExampleStore.getRows()
meta: ExampleStore.getMeta()
componentWillUnmount: ->
@stopListening ExampleStore
render: ->
<div className="page-wrap">
<header>
<strong> {@state.title} </strong>
<header>
<ReactExampleTable
rows={@state.rows},
meta={@state.meta}
/>
</div>

25
samples/FORTRAN/sample1.f Normal file
View File

@@ -0,0 +1,25 @@
c comment
* comment
program main
end
subroutine foo( i, x, b )
INTEGER i
REAL x
LOGICAL b
if( i.ne.0 ) then
call bar( -i )
end if
return
end
double complex function baz()
baz = (0.0d0,0.0d0)
return
end

25
samples/FORTRAN/sample1.for Normal file
View File

@@ -0,0 +1,25 @@
c comment
* comment
program main
end
subroutine foo( i, x, b )
INTEGER i
REAL x
LOGICAL b
if( i.ne.0 ) then
call bar( -i )
end if
return
end
double complex function baz()
baz = (0.0d0,0.0d0)
return
end

25
samples/FORTRAN/sample2.f Normal file
View File

@@ -0,0 +1,25 @@
PROGRAM MAIN
END
C comment
* comment
SUBROUTINE foo( i, x, b )
INTEGER i
REAL x
LOGICAL b
IF( i.NE.0 ) THEN
CALL bar( -i )
END IF
RETURN
END
DOUBLE COMPLEX FUNCTION baz()
baz = (0.0d0,0.0d0)
RETURN
END

25
samples/FORTRAN/sample3.F Normal file
View File

@@ -0,0 +1,25 @@
c comment
* comment
program main
end
subroutine foo( i, x, b )
INTEGER i
REAL x
LOGICAL b
if( i.ne.0 ) then
call bar( -i )
end if
return
end
double complex function baz()
baz = (0.0d0,0.0d0)
return
end

252
samples/Forth/core.f Normal file
View File

@@ -0,0 +1,252 @@
: immediate lastxt @ dup c@ negate swap c! ;
: \ source nip >in ! ; immediate \ Copyright 2004, 2012 Lars Brinkhoff
: char \ ( "word" -- char )
bl-word here 1+ c@ ;
: ahead here 0 , ;
: resolve here swap ! ;
: ' bl-word here find 0branch [ ahead ] exit [ resolve ] 0 ;
: postpone-nonimmediate [ ' literal , ' compile, ] literal , ;
: create dovariable_code header, reveal ;
create postponers
' postpone-nonimmediate ,
' abort ,
' , ,
: word \ ( char "<chars>string<char>" -- caddr )
drop bl-word here ;
: postpone \ ( C: "word" -- )
bl word find 1+ cells postponers + @ execute ; immediate
: unresolved \ ( C: "word" -- orig )
postpone postpone postpone ahead ; immediate
: chars \ ( n1 -- n2 )
;
: else \ ( -- ) ( C: orig1 -- orig2 )
unresolved branch swap resolve ; immediate
: if \ ( flag -- ) ( C: -- orig )
unresolved 0branch ; immediate
: then \ ( -- ) ( C: orig -- )
resolve ; immediate
: [char] \ ( "word" -- )
char postpone literal ; immediate
: (does>) lastxt @ dodoes_code over >code ! r> swap >does ! ;
: does> postpone (does>) ; immediate
: begin \ ( -- ) ( C: -- dest )
here ; immediate
: while \ ( x -- ) ( C: dest -- orig dest )
unresolved 0branch swap ; immediate
: repeat \ ( -- ) ( C: orig dest -- )
postpone branch , resolve ; immediate
: until \ ( x -- ) ( C: dest -- )
postpone 0branch , ; immediate
: recurse lastxt @ compile, ; immediate
: pad \ ( -- addr )
here 1024 + ;
: parse \ ( char "string<char>" -- addr n )
pad >r begin
source? if <source 2dup <> else 0 0 then
while
r@ c! r> 1+ >r
repeat 2drop pad r> over - ;
: ( \ ( "string<paren>" -- )
[ char ) ] literal parse 2drop ; immediate
\ TODO: If necessary, refill and keep parsing.
: string, ( addr n -- )
here over allot align swap cmove ;
: (s") ( -- addr n ) ( R: ret1 -- ret2 )
r> dup @ swap cell+ 2dup + aligned >r swap ;
create squote 128 allot
: s" ( "string<quote>" -- addr n )
state @ if
postpone (s") [char] " parse dup , string,
else
[char] " parse >r squote r@ cmove squote r>
then ; immediate
: (abort") ( ... addr n -- ) ( R: ... -- )
cr type cr abort ;
: abort" ( ... x "string<quote>" -- ) ( R: ... -- )
postpone if postpone s" postpone (abort") postpone then ; immediate
\ ----------------------------------------------------------------------
( Core words. )
\ TODO: #
\ TODO: #>
\ TODO: #s
: and ( x y -- x&y ) nand invert ;
: * 1 2>r 0 swap begin r@ while
r> r> swap 2dup dup + 2>r and if swap over + swap then dup +
repeat r> r> 2drop drop ;
\ TODO: */mod
: +loop ( -- ) ( C: nest-sys -- )
postpone (+loop) postpone 0branch , postpone unloop ; immediate
: space bl emit ;
: ?.- dup 0 < if [char] - emit negate then ;
: digit [char] 0 + emit ;
: (.) base @ /mod ?dup if recurse then digit ;
: ." ( "string<quote>" -- ) postpone s" postpone type ; immediate
: . ( x -- ) ?.- (.) space ;
: postpone-number ( caddr -- )
0 0 rot count >number dup 0= if
2drop nip
postpone (literal) postpone (literal) postpone ,
postpone literal postpone ,
else
." Undefined: " type cr abort
then ;
' postpone-number postponers cell+ !
: / ( x y -- x/y ) /mod nip ;
: 0< ( n -- flag ) 0 < ;
: 1- ( n -- n-1 ) -1 + ;
: 2! ( x1 x2 addr -- ) swap over ! cell+ ! ;
: 2* ( n -- 2n ) dup + ;
\ Kernel: 2/
: 2@ ( addr -- x1 x2 ) dup cell+ @ swap @ ;
\ Kernel: 2drop
\ Kernel: 2dup
\ TODO: 2over ( x1 x2 x3 x4 -- x1 x2 x3 x4 x1 x2 )
\ 3 pick 3 pick ;
\ TODO: 2swap
\ TODO: <#
: abs ( n -- |n| )
dup 0< if negate then ;
\ TODO: accept
: c, ( n -- )
here c! 1 chars allot ;
: char+ ( n1 -- n2 )
1+ ;
: constant create , does> @ ;
: decimal ( -- )
10 base ! ;
: depth ( -- n )
data_stack 100 cells + 'SP @ - /cell / 2 - ;
: do ( n1 n2 -- ) ( R: -- loop-sys ) ( C: -- do-sys )
postpone 2>r here ; immediate
\ TODO: environment?
\ TODO: evaluate
\ TODO: fill
\ TODO: fm/mod )
\ TODO: hold
: j ( -- x1 ) ( R: x1 x2 x3 -- x1 x2 x3 )
'RP @ 3 cells + @ ;
\ TODO: leave
: loop ( -- ) ( C: nest-sys -- )
postpone 1 postpone (+loop)
postpone 0branch ,
postpone unloop ; immediate
: lshift begin ?dup while 1- swap dup + swap repeat ;
: rshift 1 begin over while dup + swap 1- swap repeat nip
2>r 0 1 begin r@ while
r> r> 2dup swap dup + 2>r and if swap over + swap then dup +
repeat r> r> 2drop drop ;
: max ( x y -- max[x,y] )
2dup > if drop else nip then ;
\ Kernel: min
\ TODO: mod
\ TODO: move
: (quit) ( R: ... -- )
return_stack 100 cells + 'RP !
0 'source-id ! tib ''source ! #tib ''#source !
postpone [
begin
refill
while
interpret state @ 0= if ." ok" cr then
repeat
bye ;
' (quit) ' quit >body cell+ !
\ TODO: s>d
\ TODO: sign
\ TODO: sm/rem
: spaces ( n -- )
0 do space loop ;
\ TODO: u.
: signbit ( -- n ) -1 1 rshift invert ;
: xor ( x y -- x^y ) 2dup nand >r r@ nand swap r> nand nand ;
: u< ( x y -- flag ) signbit xor swap signbit xor > ;
\ TODO: um/mod
: variable ( "word" -- )
create /cell allot ;
: ['] \ ( C: "word" -- )
' postpone literal ; immediate

252
samples/Forth/core.for Normal file
View File

@@ -0,0 +1,252 @@
: immediate lastxt @ dup c@ negate swap c! ;
: \ source nip >in ! ; immediate \ Copyright 2004, 2012 Lars Brinkhoff
: char \ ( "word" -- char )
bl-word here 1+ c@ ;
: ahead here 0 , ;
: resolve here swap ! ;
: ' bl-word here find 0branch [ ahead ] exit [ resolve ] 0 ;
: postpone-nonimmediate [ ' literal , ' compile, ] literal , ;
: create dovariable_code header, reveal ;
create postponers
' postpone-nonimmediate ,
' abort ,
' , ,
: word \ ( char "<chars>string<char>" -- caddr )
drop bl-word here ;
: postpone \ ( C: "word" -- )
bl word find 1+ cells postponers + @ execute ; immediate
: unresolved \ ( C: "word" -- orig )
postpone postpone postpone ahead ; immediate
: chars \ ( n1 -- n2 )
;
: else \ ( -- ) ( C: orig1 -- orig2 )
unresolved branch swap resolve ; immediate
: if \ ( flag -- ) ( C: -- orig )
unresolved 0branch ; immediate
: then \ ( -- ) ( C: orig -- )
resolve ; immediate
: [char] \ ( "word" -- )
char postpone literal ; immediate
: (does>) lastxt @ dodoes_code over >code ! r> swap >does ! ;
: does> postpone (does>) ; immediate
: begin \ ( -- ) ( C: -- dest )
here ; immediate
: while \ ( x -- ) ( C: dest -- orig dest )
unresolved 0branch swap ; immediate
: repeat \ ( -- ) ( C: orig dest -- )
postpone branch , resolve ; immediate
: until \ ( x -- ) ( C: dest -- )
postpone 0branch , ; immediate
: recurse lastxt @ compile, ; immediate
: pad \ ( -- addr )
here 1024 + ;
: parse \ ( char "string<char>" -- addr n )
pad >r begin
source? if <source 2dup <> else 0 0 then
while
r@ c! r> 1+ >r
repeat 2drop pad r> over - ;
: ( \ ( "string<paren>" -- )
[ char ) ] literal parse 2drop ; immediate
\ TODO: If necessary, refill and keep parsing.
: string, ( addr n -- )
here over allot align swap cmove ;
: (s") ( -- addr n ) ( R: ret1 -- ret2 )
r> dup @ swap cell+ 2dup + aligned >r swap ;
create squote 128 allot
: s" ( "string<quote>" -- addr n )
state @ if
postpone (s") [char] " parse dup , string,
else
[char] " parse >r squote r@ cmove squote r>
then ; immediate
: (abort") ( ... addr n -- ) ( R: ... -- )
cr type cr abort ;
: abort" ( ... x "string<quote>" -- ) ( R: ... -- )
postpone if postpone s" postpone (abort") postpone then ; immediate
\ ----------------------------------------------------------------------
( Core words. )
\ TODO: #
\ TODO: #>
\ TODO: #s
: and ( x y -- x&y ) nand invert ;
: * 1 2>r 0 swap begin r@ while
r> r> swap 2dup dup + 2>r and if swap over + swap then dup +
repeat r> r> 2drop drop ;
\ TODO: */mod
: +loop ( -- ) ( C: nest-sys -- )
postpone (+loop) postpone 0branch , postpone unloop ; immediate
: space bl emit ;
: ?.- dup 0 < if [char] - emit negate then ;
: digit [char] 0 + emit ;
: (.) base @ /mod ?dup if recurse then digit ;
: ." ( "string<quote>" -- ) postpone s" postpone type ; immediate
: . ( x -- ) ?.- (.) space ;
: postpone-number ( caddr -- )
0 0 rot count >number dup 0= if
2drop nip
postpone (literal) postpone (literal) postpone ,
postpone literal postpone ,
else
." Undefined: " type cr abort
then ;
' postpone-number postponers cell+ !
: / ( x y -- x/y ) /mod nip ;
: 0< ( n -- flag ) 0 < ;
: 1- ( n -- n-1 ) -1 + ;
: 2! ( x1 x2 addr -- ) swap over ! cell+ ! ;
: 2* ( n -- 2n ) dup + ;
\ Kernel: 2/
: 2@ ( addr -- x1 x2 ) dup cell+ @ swap @ ;
\ Kernel: 2drop
\ Kernel: 2dup
\ TODO: 2over ( x1 x2 x3 x4 -- x1 x2 x3 x4 x1 x2 )
\ 3 pick 3 pick ;
\ TODO: 2swap
\ TODO: <#
: abs ( n -- |n| )
dup 0< if negate then ;
\ TODO: accept
: c, ( n -- )
here c! 1 chars allot ;
: char+ ( n1 -- n2 )
1+ ;
: constant create , does> @ ;
: decimal ( -- )
10 base ! ;
: depth ( -- n )
data_stack 100 cells + 'SP @ - /cell / 2 - ;
: do ( n1 n2 -- ) ( R: -- loop-sys ) ( C: -- do-sys )
postpone 2>r here ; immediate
\ TODO: environment?
\ TODO: evaluate
\ TODO: fill
\ TODO: fm/mod )
\ TODO: hold
: j ( -- x1 ) ( R: x1 x2 x3 -- x1 x2 x3 )
'RP @ 3 cells + @ ;
\ TODO: leave
: loop ( -- ) ( C: nest-sys -- )
postpone 1 postpone (+loop)
postpone 0branch ,
postpone unloop ; immediate
: lshift begin ?dup while 1- swap dup + swap repeat ;
: rshift 1 begin over while dup + swap 1- swap repeat nip
2>r 0 1 begin r@ while
r> r> 2dup swap dup + 2>r and if swap over + swap then dup +
repeat r> r> 2drop drop ;
: max ( x y -- max[x,y] )
2dup > if drop else nip then ;
\ Kernel: min
\ TODO: mod
\ TODO: move
: (quit) ( R: ... -- )
return_stack 100 cells + 'RP !
0 'source-id ! tib ''source ! #tib ''#source !
postpone [
begin
refill
while
interpret state @ 0= if ." ok" cr then
repeat
bye ;
' (quit) ' quit >body cell+ !
\ TODO: s>d
\ TODO: sign
\ TODO: sm/rem
: spaces ( n -- )
0 do space loop ;
\ TODO: u.
: signbit ( -- n ) -1 1 rshift invert ;
: xor ( x y -- x^y ) 2dup nand >r r@ nand swap r> nand nand ;
: u< ( x y -- flag ) signbit xor swap signbit xor > ;
\ TODO: um/mod
: variable ( "word" -- )
create /cell allot ;
: ['] \ ( C: "word" -- )
' postpone literal ; immediate

252
samples/Forth/core1.F Normal file
View File

@@ -0,0 +1,252 @@
: immediate lastxt @ dup c@ negate swap c! ;
: \ source nip >in ! ; immediate \ Copyright 2004, 2012 Lars Brinkhoff
: char \ ( "word" -- char )
bl-word here 1+ c@ ;
: ahead here 0 , ;
: resolve here swap ! ;
: ' bl-word here find 0branch [ ahead ] exit [ resolve ] 0 ;
: postpone-nonimmediate [ ' literal , ' compile, ] literal , ;
: create dovariable_code header, reveal ;
create postponers
' postpone-nonimmediate ,
' abort ,
' , ,
: word \ ( char "<chars>string<char>" -- caddr )
drop bl-word here ;
: postpone \ ( C: "word" -- )
bl word find 1+ cells postponers + @ execute ; immediate
: unresolved \ ( C: "word" -- orig )
postpone postpone postpone ahead ; immediate
: chars \ ( n1 -- n2 )
;
: else \ ( -- ) ( C: orig1 -- orig2 )
unresolved branch swap resolve ; immediate
: if \ ( flag -- ) ( C: -- orig )
unresolved 0branch ; immediate
: then \ ( -- ) ( C: orig -- )
resolve ; immediate
: [char] \ ( "word" -- )
char postpone literal ; immediate
: (does>) lastxt @ dodoes_code over >code ! r> swap >does ! ;
: does> postpone (does>) ; immediate
: begin \ ( -- ) ( C: -- dest )
here ; immediate
: while \ ( x -- ) ( C: dest -- orig dest )
unresolved 0branch swap ; immediate
: repeat \ ( -- ) ( C: orig dest -- )
postpone branch , resolve ; immediate
: until \ ( x -- ) ( C: dest -- )
postpone 0branch , ; immediate
: recurse lastxt @ compile, ; immediate
: pad \ ( -- addr )
here 1024 + ;
: parse \ ( char "string<char>" -- addr n )
pad >r begin
source? if <source 2dup <> else 0 0 then
while
r@ c! r> 1+ >r
repeat 2drop pad r> over - ;
: ( \ ( "string<paren>" -- )
[ char ) ] literal parse 2drop ; immediate
\ TODO: If necessary, refill and keep parsing.
: string, ( addr n -- )
here over allot align swap cmove ;
: (s") ( -- addr n ) ( R: ret1 -- ret2 )
r> dup @ swap cell+ 2dup + aligned >r swap ;
create squote 128 allot
: s" ( "string<quote>" -- addr n )
state @ if
postpone (s") [char] " parse dup , string,
else
[char] " parse >r squote r@ cmove squote r>
then ; immediate
: (abort") ( ... addr n -- ) ( R: ... -- )
cr type cr abort ;
: abort" ( ... x "string<quote>" -- ) ( R: ... -- )
postpone if postpone s" postpone (abort") postpone then ; immediate
\ ----------------------------------------------------------------------
( Core words. )
\ TODO: #
\ TODO: #>
\ TODO: #s
: and ( x y -- x&y ) nand invert ;
: * 1 2>r 0 swap begin r@ while
r> r> swap 2dup dup + 2>r and if swap over + swap then dup +
repeat r> r> 2drop drop ;
\ TODO: */mod
: +loop ( -- ) ( C: nest-sys -- )
postpone (+loop) postpone 0branch , postpone unloop ; immediate
: space bl emit ;
: ?.- dup 0 < if [char] - emit negate then ;
: digit [char] 0 + emit ;
: (.) base @ /mod ?dup if recurse then digit ;
: ." ( "string<quote>" -- ) postpone s" postpone type ; immediate
: . ( x -- ) ?.- (.) space ;
: postpone-number ( caddr -- )
0 0 rot count >number dup 0= if
2drop nip
postpone (literal) postpone (literal) postpone ,
postpone literal postpone ,
else
." Undefined: " type cr abort
then ;
' postpone-number postponers cell+ !
: / ( x y -- x/y ) /mod nip ;
: 0< ( n -- flag ) 0 < ;
: 1- ( n -- n-1 ) -1 + ;
: 2! ( x1 x2 addr -- ) swap over ! cell+ ! ;
: 2* ( n -- 2n ) dup + ;
\ Kernel: 2/
: 2@ ( addr -- x1 x2 ) dup cell+ @ swap @ ;
\ Kernel: 2drop
\ Kernel: 2dup
\ TODO: 2over ( x1 x2 x3 x4 -- x1 x2 x3 x4 x1 x2 )
\ 3 pick 3 pick ;
\ TODO: 2swap
\ TODO: <#
: abs ( n -- |n| )
dup 0< if negate then ;
\ TODO: accept
: c, ( n -- )
here c! 1 chars allot ;
: char+ ( n1 -- n2 )
1+ ;
: constant create , does> @ ;
: decimal ( -- )
10 base ! ;
: depth ( -- n )
data_stack 100 cells + 'SP @ - /cell / 2 - ;
: do ( n1 n2 -- ) ( R: -- loop-sys ) ( C: -- do-sys )
postpone 2>r here ; immediate
\ TODO: environment?
\ TODO: evaluate
\ TODO: fill
\ TODO: fm/mod )
\ TODO: hold
: j ( -- x1 ) ( R: x1 x2 x3 -- x1 x2 x3 )
'RP @ 3 cells + @ ;
\ TODO: leave
: loop ( -- ) ( C: nest-sys -- )
postpone 1 postpone (+loop)
postpone 0branch ,
postpone unloop ; immediate
: lshift begin ?dup while 1- swap dup + swap repeat ;
: rshift 1 begin over while dup + swap 1- swap repeat nip
2>r 0 1 begin r@ while
r> r> 2dup swap dup + 2>r and if swap over + swap then dup +
repeat r> r> 2drop drop ;
: max ( x y -- max[x,y] )
2dup > if drop else nip then ;
\ Kernel: min
\ TODO: mod
\ TODO: move
: (quit) ( R: ... -- )
return_stack 100 cells + 'RP !
0 'source-id ! tib ''source ! #tib ''#source !
postpone [
begin
refill
while
interpret state @ 0= if ." ok" cr then
repeat
bye ;
' (quit) ' quit >body cell+ !
\ TODO: s>d
\ TODO: sign
\ TODO: sm/rem
: spaces ( n -- )
0 do space loop ;
\ TODO: u.
: signbit ( -- n ) -1 1 rshift invert ;
: xor ( x y -- x^y ) 2dup nand >r r@ nand swap r> nand nand ;
: u< ( x y -- flag ) signbit xor swap signbit xor > ;
\ TODO: um/mod
: variable ( "word" -- )
create /cell allot ;
: ['] \ ( C: "word" -- )
' postpone literal ; immediate

17
samples/HTML/example.xht Normal file
View File

@@ -0,0 +1,17 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>This is a XHTML sample file</title>
<style type="text/css"><![CDATA[
#example {
background-color: yellow;
}
]]></style>
</head>
<body>
<div id="example">
Just a simple <strong>XHTML</strong> test page.
</div>
</body>
</html>

3
samples/JavaScript/empty.js
View File

@@ -1,3 +0,0 @@
(function() {
}).call(this);

207
samples/Maven POM/filenames/pom.xml Normal file
View File

@@ -0,0 +1,207 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>renpengben</groupId>
<artifactId>spring4mvc-jpa</artifactId>
<packaging>war</packaging>
<version>0.0.1-SNAPSHOT</version>
<name>spring4mvc-jpa Maven Webapp</name>
<url>https://renpengben.github.io</url>
<description>spring4mvc-jpa</description>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<java.version>1.7</java.version>
<junit.version>4.11</junit.version>
<slf4j.version>1.7.7</slf4j.version>
<log4j.version>1.2.17</log4j.version>
<spring.version>4.0.5.RELEASE</spring.version>
<spring.data.jpa.version>1.6.0.RELEASE</spring.data.jpa.version>
<cglib.version>2.1_3</cglib.version>
<mysql.version>5.1.31</mysql.version>
<hibernate.version>4.3.5.Final</hibernate.version>
<hibernate-validator.version>5.1.1.Final</hibernate-validator.version>
<druid-version>1.0.6</druid-version>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>${junit.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>${slf4j.version}</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>${slf4j.version}</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>${log4j.version}</version>
</dependency>
<!-- Spring -->
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-core</artifactId>
<version>${spring.version}</version>
<exclusions>
<exclusion>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-beans</artifactId>
<version>${spring.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
<version>${spring.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-aop</artifactId>
<version>${spring.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-expression</artifactId>
<version>${spring.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-tx</artifactId>
<version>${spring.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-aspects</artifactId>
<version>${spring.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context-support</artifactId>
<version>${spring.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-jdbc</artifactId>
<version>${spring.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-orm</artifactId>
<version>${spring.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-web</artifactId>
<version>${spring.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-webmvc</artifactId>
<version>${spring.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-test</artifactId>
<version>${spring.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-jpa</artifactId>
<version>${spring.data.jpa.version}</version>
<exclusions>
<exclusion>
<artifactId>junit-dep</artifactId>
<groupId>junit</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>cglib</groupId>
<artifactId>cglib-nodep</artifactId>
<version>${cglib.version}</version>
</dependency>
<!-- JPA -->
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-core</artifactId>
<version>${hibernate.version}</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-entitymanager</artifactId>
<version>${hibernate.version}</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-validator</artifactId>
<version>${hibernate-validator.version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>${mysql.version}</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>druid</artifactId>
<version>${druid-version}</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.0.2</version>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>
</plugins>
</build>
</project>

42
samples/Ruby/filenames/Gemfile.lock Normal file
View File

@@ -0,0 +1,42 @@
PATH
remote: .
specs:
github-linguist (4.0.1)
charlock_holmes (~> 0.7.3)
escape_utils (~> 1.0.1)
mime-types (>= 1.19)
rugged (~> 0.22.0b1)
github-linguist-grammars (4.0.1)
GEM
remote: https://rubygems.org/
specs:
charlock_holmes (0.7.3)
coderay (1.1.0)
escape_utils (1.0.1)
metaclass (0.0.4)
method_source (0.8.2)
mime-types (2.4.3)
mocha (1.1.0)
metaclass (~> 0.0.1)
plist (3.1.0)
pry (0.10.1)
coderay (~> 1.1.0)
method_source (~> 0.8.1)
slop (~> 3.4)
rake (10.3.2)
rugged (0.22.0b1)
slop (3.6.0)
yajl-ruby (1.2.1)
PLATFORMS
ruby
DEPENDENCIES
github-linguist!
github-linguist-grammars!
mocha
plist (~> 3.1)
pry
rake
yajl-ruby

0
samples/TypeScript/empty.ts
View File

21
test/test_blob.rb
View File

@@ -193,8 +193,8 @@ class TestBlob < Test::Unit::TestCase
assert blob("Binary/MainMenu.nib").generated?
assert !blob("XML/project.pbxproj").generated?
# Gemfile.locks
assert blob("Gemfile.lock").generated?
# Gemfile.lock is NOT generated
assert !blob("Gemfile.lock").generated?
# Generated .NET Docfiles
assert blob("XML/net_docfile.xml").generated?
@@ -226,7 +226,6 @@ class TestBlob < Test::Unit::TestCase
assert !blob("PostScript/sierpinski.ps").generated?
# These examples are too basic to tell
assert !blob("JavaScript/empty.js").generated?
assert !blob("JavaScript/hello.js").generated?
assert blob("JavaScript/intro-old.js").generated?
@@ -298,6 +297,13 @@ class TestBlob < Test::Unit::TestCase
assert blob("deps/http_parser/http_parser.c").vendored?
assert blob("deps/v8/src/v8.h").vendored?
# Chart.js
assert blob("some/vendored/path/Chart.js").vendored?
assert !blob("some/vendored/path/chart.js").vendored?
# Codemirror deps
assert blob("codemirror/mode/blah.js").vendored?
# Debian packaging
assert blob("debian/cron.d").vendored?
@@ -469,4 +475,13 @@ class TestBlob < Test::Unit::TestCase
def test_minified_files_not_safe_to_highlight
assert !blob("JavaScript/jquery-1.6.1.min.js").safe_to_colorize?
end
def test_empty
blob = Struct.new(:data) { include Linguist::BlobHelper }
assert blob.new("").empty?
assert blob.new(nil).empty?
refute blob.new(" ").empty?
refute blob.new("nope").empty?
end
end

68
test/test_heuristics.rb
View File

@@ -20,18 +20,18 @@ class TestHeuristcs < Test::Unit::TestCase
Dir.glob("#{samples_path}/#{language_name}/#{file}")
end
# Candidate languages = ["C++", "Objective-C"]
def test_obj_c_by_heuristics
languages = ["C++", "Objective-C"]
# Only calling out '.h' filenames as these are the ones causing issues
all_fixtures("Objective-C", "*.h").each do |fixture|
results = Heuristics.disambiguate_c(fixture("Objective-C/#{File.basename(fixture)}"), languages)
results = Heuristics.disambiguate_c(fixture("Objective-C/#{File.basename(fixture)}"))
assert_equal Language["Objective-C"], results.first
end
end
# Candidate languages = ["C++", "Objective-C"]
def test_cpp_by_heuristics
languages = ["C++", "Objective-C"]
results = Heuristics.disambiguate_c(fixture("C++/render_adapter.cpp"), languages)
results = Heuristics.disambiguate_c(fixture("C++/render_adapter.cpp"))
assert_equal Language["C++"], results.first
end
@@ -41,57 +41,57 @@ class TestHeuristcs < Test::Unit::TestCase
assert_equal Language["Objective-C"], match
end
# Candidate languages = ["Perl", "Prolog"]
def test_pl_prolog_by_heuristics
languages = ["Perl", "Prolog"]
results = Heuristics.disambiguate_pl(fixture("Prolog/turing.pl"), languages)
results = Heuristics.disambiguate_pl(fixture("Prolog/turing.pl"))
assert_equal Language["Prolog"], results.first
end
# Candidate languages = ["Perl", "Prolog"]
def test_pl_perl_by_heuristics
languages = ["Perl", "Prolog"]
results = Heuristics.disambiguate_pl(fixture("Perl/perl-test.t"), languages)
results = Heuristics.disambiguate_pl(fixture("Perl/perl-test.t"))
assert_equal Language["Perl"], results.first
end
# Candidate languages = ["ECL", "Prolog"]
def test_ecl_prolog_by_heuristics
languages = ["ECL", "Prolog"]
results = Heuristics.disambiguate_ecl(fixture("Prolog/or-constraint.ecl"), languages)
results = Heuristics.disambiguate_ecl(fixture("Prolog/or-constraint.ecl"))
assert_equal Language["Prolog"], results.first
end
# Candidate languages = ["ECL", "Prolog"]
def test_ecl_ecl_by_heuristics
languages = ["ECL", "Prolog"]
results = Heuristics.disambiguate_ecl(fixture("ECL/sample.ecl"), languages)
results = Heuristics.disambiguate_ecl(fixture("ECL/sample.ecl"))
assert_equal Language["ECL"], results.first
end
# Candidate languages = ["IDL", "Prolog"]
def test_pro_prolog_by_heuristics
languages = ["IDL", "Prolog"]
results = Heuristics.disambiguate_pro(fixture("Prolog/logic-problem.pro"), languages)
results = Heuristics.disambiguate_pro(fixture("Prolog/logic-problem.pro"))
assert_equal Language["Prolog"], results.first
end
# Candidate languages = ["IDL", "Prolog"]
def test_pro_idl_by_heuristics
languages = ["IDL", "Prolog"]
results = Heuristics.disambiguate_pro(fixture("IDL/mg_acosh.pro"), languages)
results = Heuristics.disambiguate_pro(fixture("IDL/mg_acosh.pro"))
assert_equal Language["IDL"], results.first
end
# Candidate languages = ["AGS Script", "AsciiDoc"]
def test_asc_asciidoc_by_heuristics
languages = ["AGS Script", "AsciiDoc"]
results = Heuristics.disambiguate_asc(fixture("AsciiDoc/list.asc"), languages)
results = Heuristics.disambiguate_asc(fixture("AsciiDoc/list.asc"))
assert_equal Language["AsciiDoc"], results.first
end
# Candidate languages = ["TypeScript", "XML"]
def test_ts_typescript_by_heuristics
languages = ["TypeScript", "XML"]
results = Heuristics.disambiguate_ts(fixture("TypeScript/classes.ts"), languages)
results = Heuristics.disambiguate_ts(fixture("TypeScript/classes.ts"))
assert_equal Language["TypeScript"], results.first
end
# Candidate languages = ["TypeScript", "XML"]
def test_ts_xml_by_heuristics
languages = ["TypeScript", "XML"]
results = Heuristics.disambiguate_ts(fixture("XML/pt_BR.xml"), languages)
results = Heuristics.disambiguate_ts(fixture("XML/pt_BR.xml"))
assert_equal Language["XML"], results.first
end
@@ -99,27 +99,37 @@ class TestHeuristcs < Test::Unit::TestCase
languages = ["Common Lisp", "OpenCL"]
languages.each do |language|
all_fixtures(language).each do |fixture|
results = Heuristics.disambiguate_cl(fixture("#{language}/#{File.basename(fixture)}"), languages)
results = Heuristics.disambiguate_cl(fixture("#{language}/#{File.basename(fixture)}"))
assert_equal Language[language], results.first
end
end
end
def test_f_by_heuristics
languages = ["FORTRAN", "Forth"]
languages.each do |language|
all_fixtures(language).each do |fixture|
results = Heuristics.disambiguate_f(fixture("#{language}/#{File.basename(fixture)}"))
assert_equal Language[language], results.first
end
end
end
# Candidate languages = ["Hack", "PHP"]
def test_hack_by_heuristics
languages = ["Hack", "PHP"]
results = Heuristics.disambiguate_hack(fixture("Hack/funs.php"), languages)
results = Heuristics.disambiguate_hack(fixture("Hack/funs.php"))
assert_equal Language["Hack"], results.first
end
# Candidate languages = ["Scala", "SuperCollider"]
def test_sc_supercollider_by_heuristics
languages = ["Scala", "SuperCollider"]
results = Heuristics.disambiguate_sc(fixture("SuperCollider/WarpPreset.sc"), languages)
results = Heuristics.disambiguate_sc(fixture("SuperCollider/WarpPreset.sc"))
assert_equal Language["SuperCollider"], results.first
end
# Candidate languages = ["Scala", "SuperCollider"]
def test_sc_scala_by_heuristics
languages = ["Scala", "SuperCollider"]
results = Heuristics.disambiguate_sc(fixture("Scala/node11.sc"), languages)
results = Heuristics.disambiguate_sc(fixture("Scala/node11.sc"))
assert_equal Language["Scala"], results.first
end
end

20
test/test_language.rb
View File

@@ -1,5 +1,6 @@
require 'linguist/language'
require 'test/unit'
require 'yaml'
class TestLanguage < Test::Unit::TestCase
include Linguist
@@ -139,6 +140,7 @@ class TestLanguage < Test::Unit::TestCase
assert_equal :programming, Language['Python'].type
assert_equal :programming, Language['Ruby'].type
assert_equal :programming, Language['TypeScript'].type
assert_equal :programming, Language['Makefile'].type
end
def test_markup
@@ -157,7 +159,6 @@ class TestLanguage < Test::Unit::TestCase
def test_other
assert_nil Language['Brainfuck'].type
assert_nil Language['Makefile'].type
end
def test_searchable
@@ -349,12 +350,6 @@ class TestLanguage < Test::Unit::TestCase
assert_equal '.coffee', Language['CoffeeScript'].primary_extension
assert_equal '.t', Language['Turing'].primary_extension
assert_equal '.ts', Language['TypeScript'].primary_extension
# This is a nasty requirement, but there's some code in GitHub that
# expects this. Really want to drop this.
Language.all.each do |language|
assert language.primary_extension, "#{language} has no primary extension"
end
end
def test_eql
@@ -365,4 +360,15 @@ class TestLanguage < Test::Unit::TestCase
def test_by_type
assert !Language.by_type(:prose).nil?
end
def test_all_languages_have_grammars
scopes = YAML.load(File.read(File.expand_path("../../grammars.yml", __FILE__))).values.flatten
missing = Language.all.reject { |language| language.tm_scope == "none" || scopes.include?(language.tm_scope) }
message = "The following languages' scopes are not listed in grammars.yml. Please add grammars for all new languages.\n"
message << "If no grammar exists for a language, mark the language with `tm_scope: none` in lib/linguist/languages.yml.\n"
width = missing.map { |language| language.name.length }.max
message << missing.map { |language| sprintf("%-#{width}s %s", language.name, language.tm_scope) }.sort.join("\n")
assert missing.empty?, message
end
end

49
test/test_pedantic.rb
View File

@@ -1,57 +1,30 @@
require 'test/unit'
require 'yaml'
class TestPedantic < Test::Unit::TestCase
Lib = File.expand_path("../../lib/linguist", __FILE__)
def file(name)
File.read(File.join(Lib, name))
end
filename = File.expand_path("../../lib/linguist/languages.yml", __FILE__)
LANGUAGES = YAML.load(File.read(filename))
def test_language_names_are_sorted
languages = []
file("languages.yml").lines.each do |line|
if line =~ /^(\w+):$/
languages << $1
end
end
assert_sorted languages
assert_sorted LANGUAGES.keys
end
def test_extensions_are_sorted
extensions = nil
file("languages.yml").lines.each do |line|
if line =~ /^ extensions:$/
extensions = []
elsif extensions && line =~ /^ - \.([\w\-\.]+)( *#.*)?$/
extensions << $1
else
assert_sorted extensions[1..-1] if extensions
extensions = nil
end
LANGUAGES.each do |name, language|
extensions = language['extensions']
assert_sorted extensions[1..-1] if extensions && extensions.size > 1
end
end
def test_filenames_are_sorted
filenames = nil
file("languages.yml").lines.each do |line|
if line =~ /^ filenames:$/
filenames = []
elsif filenames && line =~ /^ - \.(\w+)$/
filenames << $1
else
assert_sorted filenames if filenames
filenames = nil
end
LANGUAGES.each do |name, language|
assert_sorted language['filenames'] if language['filenames']
end
end
def assert_sorted(list)
previous = nil
list.each do |item|
if previous && previous > item
flunk "#{previous} should come after #{item}"
end
previous = item
list.each_cons(2) do |previous, item|
flunk "#{previous} should come after #{item}" if previous > item
end
end
end

26
test/test_samples.rb
View File

@@ -55,20 +55,28 @@ class TestSamples < Test::Unit::TestCase
end
# If a language extension isn't globally unique then make sure there are samples
def test_presence
Linguist::Language.all.each do |language|
language.all_extensions.each do |extension|
language_matches = Language.find_by_filename("foo#{extension}")
Linguist::Language.all.each do |language|
define_method "test_#{language.name}_has_samples" do
language.extensions.each do |extension|
language_matches = Language.find_by_extension(extension)
# If there is more than one language match for a given extension
# then check that there are examples for that language with the extension
# Check for samples if more than one language matches the given extension.
if language_matches.length > 1
language_matches.each do |language|
assert File.directory?("samples/#{language.name}"), "#{language.name} is missing a samples directory"
assert Dir.glob("samples/#{language.name}/*#{extension}").any?, "#{language.name} is missing samples for extension #{extension}"
language_matches.each do |match|
samples = "samples/#{match.name}/*#{extension}"
assert Dir.glob(samples).any?, "Missing samples in #{samples.inspect}. See https://github.com/github/linguist/blob/master/CONTRIBUTING.md"
end
end
end
language.filenames.each do |filename|
# Check for samples if more than one language matches the given filename
if Language.find_by_filename(filename).size > 1
sample = "samples/#{language.name}/filenames/#{filename}"
assert File.exists?(sample),
"Missing sample in #{sample.inspect}. See https://github.com/github/linguist/blob/master/CONTRIBUTING.md"
end
end
end
end
end

2
test/test_tokenizer.rb
View File

@@ -43,6 +43,8 @@ class TestTokenizer < Test::Unit::TestCase
assert_equal %w(foo), tokenize("foo {- Comment -}")
assert_equal %w(foo), tokenize("foo (* Comment *)")
assert_equal %w(%), tokenize("2 % 10\n% Comment")
assert_equal %w(foo bar), tokenize("foo\n\"\"\"\nComment\n\"\"\"\nbar")
assert_equal %w(foo bar), tokenize("foo\n'''\nComment\n'''\nbar")
end
def test_sgml_tags

BIN
vendor/cache/mime-types-1.25.1.gem vendored
View File

Binary file not shown.

BIN
vendor/cache/mime-types-2.4.3.gem vendored Normal file
View File

Binary file not shown.

BIN
vendor/cache/rugged-0.21.1b2.gem vendored
View File

Binary file not shown.

BIN
vendor/cache/rugged-0.22.0b4.gem vendored Normal file
View File

Binary file not shown.

BIN
vendor/cache/yajl-ruby-1.2.1.gem vendored Normal file
View File

Binary file not shown.