Merge pull request #1747 from github/cut-release-v4.0.2

Cut release v4.0.2

.gitignore

@@ -1,4 +1,6 @@
-Gemfile.lock
+/Gemfile.lock
 .bundle/
 benchmark/
 lib/linguist/samples.json
+/grammars
+/node_modules

.travis.yml

@@ -6,6 +6,7 @@ before_install:
 rvm:
   - 1.9.3
   - 2.0.0
-  - 2.1.1
+  - 2.1
+  - 2.2
 notifications:
   disabled: true

CONTRIBUTING.md

@@ -0,0 +1,31 @@
+## Contributing
+
+The majority of contributions won't need to touch any Ruby code at all. The [master language list][languages] is just a YAML configuration file.
+
+Almost all bug fixes or new language additions should come with some additional code samples. Just drop them under [`samples/`][samples] in the correct subdirectory and our test suite will automatically test them. In most cases you shouldn't need to add any new assertions.
+
+### My code is detected as the wrong language
+
+This can usually be solved either by adding a new filename or file name extension to the language's entry in [`languages.yml`][languages] or adding more [samples][samples] for your language to the repository to make Linguist's classifier smarter.
+
+### Syntax highlighting looks wrong
+
+Assuming your code is being detected as the right language (see above), in most cases this is due to a bug in the language grammar rather than a bug in Linguist. [`grammars.yml`][grammars] lists all the grammars we use for syntax highlighting on github.com. Find the one corresponding to your code's programming language and submit a bug report upstream.
+
+You can also try to fix the bug yourself and submit a Pull Request. [This piece from TextMate's documentation](http://manual.macromates.com/en/language_grammars) offers a good introduction on how to work with TextMate-compatible grammars.
+
+Once the bug has been fixed upstream, please let us know and we'll pick it up for GitHub.
+
+### I want to add support for the `X` programming language
+
+Great! You'll need to:
+
+0. Add an entry for your language to [`languages.yml`][languages].
+0. Add a grammar for your language to [`grammars.yml`][grammars] by running `script/download-grammars --add URL`. Please only add grammars that have a license that permits redistribution.
+0. Add samples for your language to the [samples directory][samples].
+
+We try only to add languages once they have some usage on GitHub, so please note in-the-wild usage examples in your pull request. In most cases we prefer that languages already be in use in hundreds of repositories before supporting them in Linguist.
+
+[grammars]: /grammars.yml
+[languages]: /lib/linguist/languages.yml
+[samples]: /samples

Gemfile

@@ -1,2 +1,4 @@
 source 'https://rubygems.org'
-gemspec
+gemspec :name => "github-linguist"
+gemspec :name => "github-linguist-grammars"
+gem 'test-unit', require: false if RUBY_VERSION >= '2.2'

README.md

@@ -1,12 +1,14 @@
 # Linguist
 
-We use this library at GitHub to detect blob languages, highlight code, ignore binary files, suppress generated files in diffs, and generate language breakdown graphs.
+We use this library at GitHub to detect blob languages, ignore binary files, suppress generated files in diffs, and generate language breakdown graphs.
+
+Tips for filing issues and creating pull requests can be found in [`CONTRIBUTING.md`](/CONTRIBUTING.md).
 
 ## Features
 
 ### Language detection
 
-Linguist defines a list of all languages known to GitHub in a [yaml file](https://github.com/github/linguist/blob/master/lib/linguist/languages.yml). In order for a file to be highlighted, a language and a lexer must be defined there.
+Linguist defines a list of all languages known to GitHub in a [yaml file](https://github.com/github/linguist/blob/master/lib/linguist/languages.yml).
 
 Most languages are detected by their file extension. For disambiguating between files with common extensions, we first apply some common-sense heuristics to pick out obvious languages. After that, we use a
 [statistical
@@ -24,7 +26,9 @@ See [lib/linguist/language.rb](https://github.com/github/linguist/blob/master/li
 
 ### Syntax Highlighting
 
-The actual syntax highlighting is handled by our Pygments wrapper, [pygments.rb](https://github.com/tmm1/pygments.rb). It also provides a [Lexer abstraction](https://github.com/tmm1/pygments.rb/blob/master/lib/pygments/lexer.rb) that determines which highlighter should be used on a file.
+Syntax highlighting in GitHub is performed using TextMate-compatible grammars. These are the same grammars that TextMate, Sublime Text and Atom use.
+
+Every language in `languages.yml` is mapped to its corresponding TM `scope`. This scope will be used when picking up a grammar for highlighting. **When adding a new language to Linguist, please add its corresponding scope too (assuming there's an existing TextMate bundle, Sublime Text package, or Atom package) so syntax highlighting works for it**.
 
 ### Stats
 
@@ -108,6 +112,8 @@ See [Linguist::Generated#generated?](https://github.com/github/linguist/blob/mas
 
 Linguist supports custom overrides for language definitions and vendored paths. Add a `.gitattributes` file to your project using the keys `linguist-language` and `linguist-vendored` with the standard git-style path matchers for the files you want to override.
 
+Please note that the overrides currently only affect the language statistics for a repository and not the syntax-highlighting of files.
+
 ```
 $ cat .gitattributes
 *.rb linguist-language=Java
@@ -141,14 +147,6 @@ To run the tests:
 
     bundle exec rake test
 
-## Contributing
-
-The majority of contributions won't need to touch any Ruby code at all. The [master language list](https://github.com/github/linguist/blob/master/lib/linguist/languages.yml) is just a YAML configuration file.
-
-We try to only add languages once they have some usage on GitHub, so please note in-the-wild usage examples in your pull request.
-
-Almost all bug fixes or new language additions should come with some additional code samples. Just drop them under [`samples/`](https://github.com/github/linguist/tree/master/samples) in the correct subdirectory and our test suite will automatically test them. In most cases you shouldn't need to add any new assertions.
-
 ### A note on language extensions
 
 Linguist has a number of methods available to it for identifying the language of a particular file. The initial lookup is based upon the extension of the file, possible file extensions are defined in an array called `extensions`. Take a look at this example for example for `Perl`:

Rakefile

@@ -1,8 +1,8 @@
 require 'bundler/setup'
-require 'json'
 require 'rake/clean'
 require 'rake/testtask'
 require 'yaml'
+require 'yajl'
 
 task :default => :test
 
@@ -20,19 +20,23 @@ end
 
 task :samples do
   require 'linguist/samples'
-  require 'yajl'
-  data = Linguist::Samples.data
-  json = Yajl::Encoder.encode(data, :pretty => true)
-  File.open('lib/linguist/samples.json', 'w') { |io| io.write json }
+  json = Yajl.dump(Linguist::Samples.data, :pretty => true)
+  File.write 'lib/linguist/samples.json', json
 end
 
 task :build_gem => :samples do
   languages = YAML.load_file("lib/linguist/languages.yml")
-  File.write("lib/linguist/languages.json", JSON.dump(languages))
+  File.write("lib/linguist/languages.json", Yajl.dump(languages))
   `gem build github-linguist.gemspec`
   File.delete("lib/linguist/languages.json")
 end
 
+task :build_grammars_gem do
+  rm_rf "grammars"
+  sh "script/download-grammars"
+  sh "gem", "build", "github-linguist-grammars.gemspec"
+end
+
 namespace :benchmark do
   benchmark_path = "benchmark/results"
 
@@ -72,11 +76,11 @@ namespace :benchmark do
     reference_file = ENV["REFERENCE"]
     candidate_file = ENV["CANDIDATE"]
 
-    reference = JSON.parse(File.read(reference_file))
+    reference = Yajl.load(File.read(reference_file))
     reference_counts = Hash.new(0)
     reference.each { |filename, language| reference_counts[language] += 1 }
 
-    candidate = JSON.parse(File.read(candidate_file))
+    candidate = Yajl.load(File.read(candidate_file))
     candidate_counts = Hash.new(0)
     candidate.each { |filename, language| candidate_counts[language] += 1 }
 
@@ -126,14 +130,12 @@ namespace :classifier do
 
   def each_public_gist
     require 'open-uri'
-    require 'json'
-
     url = "https://api.github.com/gists/public"
 
     loop do
       resp = open(url)
       url = resp.meta['link'][/<([^>]+)>; rel="next"/, 1]
-      gists = JSON.parse(resp.read)
+      gists = Yajl.load(resp.read)
 
       for gist in gists
         for filename, attrs in gist['files']

github-linguist-grammars.gemspec

@@ -0,0 +1,14 @@
+require File.expand_path('../lib/linguist/version', __FILE__)
+
+Gem::Specification.new do |s|
+  s.name    = 'github-linguist-grammars'
+  s.version = Linguist::VERSION
+  s.summary = "Language grammars for use with github-linguist"
+
+  s.authors  = "GitHub"
+  s.homepage = "https://github.com/github/linguist"
+
+  s.files = ['lib/linguist/grammars.rb'] + Dir['grammars/*']
+
+  s.add_development_dependency 'plist', '~>3.1'
+end

github-linguist.gemspec

@@ -10,16 +10,14 @@ Gem::Specification.new do |s|
   s.homepage = "https://github.com/github/linguist"
   s.license  = "MIT"
 
-  s.files = Dir['lib/**/*']
+  s.files = Dir['lib/**/*'] - ['lib/linguist/grammars.rb']
   s.executables << 'linguist'
 
   s.add_dependency 'charlock_holmes', '~> 0.7.3'
   s.add_dependency 'escape_utils',    '~> 1.0.1'
-  s.add_dependency 'mime-types',      '~> 1.19'
-  s.add_dependency 'pygments.rb',     '~> 0.6.0'
-  s.add_dependency 'rugged',          '~> 0.21.1b2'
+  s.add_dependency 'mime-types',      '>= 1.19'
+  s.add_dependency 'rugged',          '~> 0.22.0b1'
 
-  s.add_development_dependency 'json'
   s.add_development_dependency 'mocha'
   s.add_development_dependency 'pry'
   s.add_development_dependency 'rake'

grammars.yml

@@ -0,0 +1,409 @@
+---
+http://svn.edgewall.org/repos/genshi/contrib/textmate/Genshi.tmbundle/Syntaxes/Markup%20Template%20%28XML%29.tmLanguage:
+- text.xml.genshi
+http://svn.textmate.org/trunk/Review/Bundles/BlitzMax.tmbundle:
+- source.blitzmax
+http://svn.textmate.org/trunk/Review/Bundles/Cython.tmbundle:
+- source.cython
+http://svn.textmate.org/trunk/Review/Bundles/F%20Sharp.tmbundle:
+- source.fsharp
+http://svn.textmate.org/trunk/Review/Bundles/Forth.tmbundle:
+- source.forth
+http://svn.textmate.org/trunk/Review/Bundles/Parrot.tmbundle:
+- source.parrot.pir
+http://svn.textmate.org/trunk/Review/Bundles/Ruby%20Sass.tmbundle:
+- source.sass
+http://svn.textmate.org/trunk/Review/Bundles/SecondLife%20LSL.tmbundle:
+- source.lsl
+http://svn.textmate.org/trunk/Review/Bundles/VHDL.tmbundle:
+- source.vhdl
+http://svn.textmate.org/trunk/Review/Bundles/XQuery.tmbundle:
+- source.xquery
+https://bitbucket.org/Clams/sublimesystemverilog/get/default.tar.gz:
+- source.systemverilog
+- source.ucfconstraints
+https://bitbucket.org/bitlang/sublime_cobol/raw/b0e9c44ac5f7a2fb553421aa986b35854cbfda4a/COBOL.tmLanguage:
+- source.cobol
+https://fan.googlecode.com/hg-history/Build%201.0.55/adm/tools/textmate/Fan.tmbundle/Syntaxes/Fan.tmLanguage:
+- source.fan
+https://github.com/AlanQuatermain/go-tmbundle:
+- source.go
+https://github.com/Anomareh/PHP-Twig.tmbundle:
+- text.html.twig
+https://github.com/Cirru/sublime-cirru/raw/master/Cirru.tmLanguage:
+- source.cirru
+https://github.com/Cykey/Sublime-Logos:
+- source.logos
+https://github.com/Drako/SublimeBrainfuck/raw/master/Brainfuck.tmLanguage:
+- source.bf
+https://github.com/JohnNilsson/awk-sublime/raw/master/AWK.tmLanguage:
+- source.awk
+https://github.com/MarioRicalde/SCSS.tmbundle:
+- source.scss
+https://github.com/Oldes/Sublime-REBOL:
+- source.rebol
+https://github.com/PogiNate/Sublime-Inform:
+- source.Inform7
+https://github.com/Red-Nova-Technologies/autoitv3-tmbundle:
+- source.autoit.3
+https://github.com/SalGnt/Sublime-VimL:
+- source.viml
+https://github.com/Shammah/boo-sublime/raw/master/Boo.tmLanguage:
+- source.boo
+https://github.com/SublimeText/ColdFusion:
+- source.cfscript
+- source.cfscript.cfc
+- text.cfml.basic
+- text.html.cfm
+https://github.com/SublimeText/NSIS:
+- source.nsis
+https://github.com/Varriount/NimLime:
+- source.nimrod
+- source.nimrod_filter
+- source.nimrodcfg
+https://github.com/angryant0007/VBDotNetSyntax:
+- source.vbnet
+https://github.com/aroben/ada.tmbundle/raw/c45eed4d5f98fe3bcbbffbb9e436601ab5bbde4b/Syntaxes/Ada.plist:
+- source.ada
+https://github.com/aroben/ruby.tmbundle@4636a3023153c3034eb6ffc613899ba9cf33b41f:
+- source.ruby
+- text.html.erb
+https://github.com/asbjornenge/Docker.tmbundle:
+- source.dockerfile
+https://github.com/atom/language-clojure:
+- source.clojure
+https://github.com/atom/language-coffee-script:
+- source.coffee
+- source.litcoffee
+https://github.com/atom/language-csharp:
+- source.cs
+- source.csx
+- source.nant-build
+https://github.com/atom/language-javascript:
+- source.js
+- source.js.regexp
+https://github.com/atom/language-python:
+- source.python
+- source.regexp.python
+- text.python.traceback
+https://github.com/atom/language-shellscript:
+- source.shell
+- text.shell-session
+https://github.com/austinwagner/sublime-sourcepawn:
+- source.sp
+https://github.com/bfad/Sublime-Lasso:
+- file.lasso
+https://github.com/bholt/chapel-tmbundle:
+- source.chapel
+https://github.com/brandonwamboldt/sublime-nginx:
+- source.nginx
+https://github.com/carsonoid/sublime_man_page_support/raw/master/man-groff.tmLanguage:
+- text.groff
+https://github.com/ccreutzig/sublime-MuPAD:
+- source.mupad
+https://github.com/cdwilson/nesC.tmbundle:
+- source.nesc
+https://github.com/christophevg/racket-tmbundle:
+- source.racket
+https://github.com/clemos/haxe-sublime-bundle:
+- source.erazor
+- source.haxe.2
+- source.hss.1
+- source.hxml
+- source.nmml
+https://github.com/cucumber/cucumber-tmbundle:
+- source.ruby.rspec.cucumber.steps
+- text.gherkin.feature
+https://github.com/daaain/Handlebars/raw/master/Handlebars.tmLanguage:
+- text.html.handlebars
+https://github.com/davidpeckham/powershell.tmbundle:
+- source.powershell
+https://github.com/davidrios/jade-tmbundle:
+- source.jade
+- source.pyjade
+https://github.com/elixir-lang/elixir-tmbundle:
+- source.elixir
+- text.elixir
+- text.html.elixir
+https://github.com/ericzou/ebundles/raw/master/Bundles/MSDOS%20batch%20file.tmbundle/Syntaxes/MSDOS%20batch%20file.tmLanguage:
+- source.dosbatch
+https://github.com/euler0/sublime-glsl/raw/master/GLSL.tmLanguage:
+- source.glsl
+https://github.com/fancy-lang/fancy-tmbundle:
+- source.fancy
+https://github.com/gingerbeardman/monkey.tmbundle:
+- source.monkey
+https://github.com/guillermooo/dart-sublime-bundle/raw/master/Dart.tmLanguage:
+- source.dart
+https://github.com/harrism/sublimetext-cuda-cpp/raw/master/cuda-c%2B%2B.tmLanguage:
+- source.cuda-c++
+https://github.com/hww3/pike-textmate:
+- source.pike
+https://github.com/jeancharles-roger/ceylon-sublimetext/raw/master/Ceylon.tmLanguage:
+- source.ceylon
+https://github.com/jfairbank/Sublime-Text-2-OpenEdge-ABL:
+- source.abl
+https://github.com/jhasse/sublime-rust:
+- source.rust
+https://github.com/johanasplund/sublime-befunge/raw/master/Befunge-93.tmLanguage:
+- source.befunge
+https://github.com/joshaven/RDoc.tmbundle:
+- text.rdoc
+https://github.com/jpcamara/Textmate-Gosu-Bundle/raw/master/Gosu.tmbundle/Syntaxes/Gosu.tmLanguage:
+- source.gosu.2
+https://github.com/kswedberg/jquery-tmbundle:
+- source.js.jquery
+https://github.com/laughedelic/sublime-idris/raw/master/Idris.tmLanguage:
+- source.idris
+https://github.com/lavrton/sublime-better-typescript:
+- source.ts
+https://github.com/leafo/moonscript-tmbundle:
+- source.moonscript
+https://github.com/lunixbochs/x86-assembly-textmate-bundle:
+- source.asm.x86
+https://github.com/macekond/Alloy.tmbundle:
+- source.alloy
+https://github.com/mads379/opa.tmbundle:
+- source.opa
+https://github.com/mads379/scala.tmbundle:
+- source.sbt
+- source.scala
+https://github.com/marconi/mako-tmbundle:
+- text.html.mako
+https://github.com/mattfoster/gnuplot-tmbundle:
+- source.gnuplot
+https://github.com/mgalloy/idl.tmbundle:
+- source.idl
+- source.idl-dlm
+- text.idl-idldoc
+https://github.com/michaeledgar/protobuf-tmbundle:
+- source.protobuf
+https://github.com/mkolosick/Sublime-Coq/raw/master/Coq.tmLanguage:
+- source.coq
+https://github.com/mokus0/Agda.tmbundle:
+- source.agda
+https://github.com/nanoant/Julia.tmbundle:
+- source.julia
+https://github.com/nanoant/assembly.tmbundle/raw/master/Syntaxes/objdump%20C%2B%2B.tmLanguage:
+- objdump.x86asm
+https://github.com/nilium/ooc.tmbundle:
+- source.ooc
+https://github.com/paulmillr/LiveScript.tmbundle:
+- source.livescript
+https://github.com/pferruggiaro/sublime-tea:
+- source.tea
+https://github.com/puppet-textmate-bundle/puppet-textmate-bundle:
+- source.puppet
+https://github.com/pvl/abap.tmbundle:
+- source.abap
+https://github.com/scalate/Scalate.tmbundle:
+- source.scaml
+- text.html.ssp
+https://github.com/shadanan/mathematica-tmbundle:
+- source.mathematica
+https://github.com/shellderp/sublime-robot-plugin:
+- text.robot
+https://github.com/simongregory/actionscript3-tmbundle:
+- source.actionscript.3
+- text.html.asdoc
+- text.xml.flex-config
+https://github.com/skozlovf/Sublime-QML:
+- source.qml
+https://github.com/slash-lang/Slash.tmbundle:
+- text.html.slash
+https://github.com/slavapestov/factor/raw/master/misc/Factor.tmbundle/Syntaxes/Factor.tmLanguage:
+- source.factor
+https://github.com/slim-template/ruby-slim.tmbundle:
+- text.slim
+https://github.com/smiledawgg/Bro.tmbundle:
+- source.bro
+- source.bro.sig
+https://github.com/staltz/SublimeXtend:
+- source.xtend
+https://github.com/statatmbundle/Stata.tmbundle:
+- source.mata
+- source.stata
+https://github.com/technosophos/Vala-TMBundle:
+- source.vala
+https://github.com/textmate/ant.tmbundle:
+- text.xml.ant
+https://github.com/textmate/antlr.tmbundle:
+- source.antlr
+https://github.com/textmate/apache.tmbundle:
+- source.apache-config
+- source.apache-config.mod_perl
+https://github.com/textmate/applescript.tmbundle:
+- source.applescript
+https://github.com/textmate/asp.tmbundle:
+- source.asp
+- text.html.asp
+https://github.com/textmate/bison.tmbundle:
+- source.bison
+https://github.com/textmate/c.tmbundle:
+- source.c
+- source.c++
+- source.c.platform
+https://github.com/textmate/capnproto.tmbundle:
+- source.capnp
+https://github.com/textmate/cmake.tmbundle:
+- source.cache.cmake
+- source.cmake
+https://github.com/textmate/cpp-qt.tmbundle:
+- source.c++.qt
+- source.qmake
+https://github.com/textmate/css.tmbundle:
+- source.css
+https://github.com/textmate/d.tmbundle:
+- source.d
+https://github.com/textmate/diff.tmbundle:
+- source.diff
+https://github.com/textmate/dylan.tmbundle:
+- source.dylan
+- source.lid
+- source.makegen
+https://github.com/textmate/eiffel.tmbundle:
+- source.eiffel
+https://github.com/textmate/erlang.tmbundle:
+- source.erlang
+- text.html.erlang.yaws
+https://github.com/textmate/fortran.tmbundle:
+- source.fortran
+- source.fortran.modern
+https://github.com/textmate/gettext.tmbundle:
+- source.po
+https://github.com/textmate/graphviz.tmbundle:
+- source.dot
+https://github.com/textmate/groovy.tmbundle:
+- source.groovy
+https://github.com/textmate/haskell.tmbundle:
+- source.haskell
+- text.tex.latex.haskell
+https://github.com/textmate/html.tmbundle:
+- text.html.basic
+https://github.com/textmate/ini.tmbundle:
+- source.ini
+https://github.com/textmate/io.tmbundle:
+- source.io
+https://github.com/textmate/java.tmbundle:
+- source.java
+- source.java-properties
+- text.html.jsp
+- text.junit-test-report
+https://github.com/textmate/javadoc.tmbundle:
+- text.html.javadoc
+https://github.com/textmate/javascript-objective-j.tmbundle:
+- source.js.objj
+https://github.com/textmate/json.tmbundle:
+- source.json
+https://github.com/textmate/latex.tmbundle:
+- text.bibtex
+- text.log.latex
+- text.tex
+- text.tex.latex
+- text.tex.latex.beamer
+- text.tex.latex.memoir
+https://github.com/textmate/less.tmbundle:
+- source.css.less
+https://github.com/textmate/lilypond.tmbundle:
+- source.lilypond
+https://github.com/textmate/lisp.tmbundle:
+- source.lisp
+https://github.com/textmate/logtalk.tmbundle:
+- source.logtalk
+https://github.com/textmate/lua.tmbundle:
+- source.lua
+https://github.com/textmate/make.tmbundle:
+- source.makefile
+https://github.com/textmate/markdown.tmbundle:
+- text.html.markdown
+https://github.com/textmate/matlab.tmbundle:
+- source.matlab
+- source.octave
+https://github.com/textmate/maven.tmbundle:
+- text.xml.pom
+https://github.com/textmate/nemerle.tmbundle:
+- source.nemerle
+https://github.com/textmate/ninja.tmbundle:
+- source.ninja
+https://github.com/textmate/objective-c.tmbundle:
+- source.objc
+- source.objc++
+- source.objc.platform
+- source.strings
+https://github.com/textmate/ocaml.tmbundle:
+- source.camlp4.ocaml
+- source.ocaml
+- source.ocamllex
+- source.ocamlyacc
+https://github.com/textmate/pascal.tmbundle:
+- source.pascal
+https://github.com/textmate/perl.tmbundle:
+- source.perl
+https://github.com/textmate/php-smarty.tmbundle:
+- source.smarty
+https://github.com/textmate/php.tmbundle:
+- text.html.php
+https://github.com/textmate/postscript.tmbundle:
+- source.postscript
+https://github.com/textmate/processing.tmbundle:
+- source.processing
+https://github.com/textmate/prolog.tmbundle:
+- source.prolog
+https://github.com/textmate/python-django.tmbundle:
+- source.python.django
+- text.html.django
+https://github.com/textmate/r.tmbundle:
+- source.r
+- text.tex.latex.rd
+https://github.com/textmate/restructuredtext.tmbundle:
+- text.restructuredtext
+https://github.com/textmate/ruby-haml.tmbundle:
+- text.haml
+https://github.com/textmate/ruby-on-rails-tmbundle:
+- source.js.erb.rails
+- source.ruby.rails
+- source.ruby.rails.rjs
+- source.sql.ruby
+- text.html.erb.rails
+https://github.com/textmate/scheme.tmbundle:
+- source.scheme
+https://github.com/textmate/scilab.tmbundle:
+- source.scilab
+https://github.com/textmate/sql.tmbundle:
+- source.sql
+https://github.com/textmate/standard-ml.tmbundle:
+- source.cm
+- source.ml
+https://github.com/textmate/swift.tmbundle:
+- source.swift
+https://github.com/textmate/tcl.tmbundle:
+- source.tcl
+- text.html.tcl
+https://github.com/textmate/text.tmbundle:
+- text.plain
+https://github.com/textmate/textile.tmbundle:
+- text.html.textile
+https://github.com/textmate/textmate.tmbundle:
+- source.regexp.oniguruma
+- source.tm-properties
+https://github.com/textmate/thrift.tmbundle:
+- source.thrift
+https://github.com/textmate/toml.tmbundle:
+- source.toml
+https://github.com/textmate/verilog.tmbundle:
+- source.verilog
+https://github.com/textmate/xml.tmbundle:
+- text.xml
+- text.xml.xsl
+https://github.com/textmate/yaml.tmbundle:
+- source.yaml
+https://github.com/tomas-stefano/smalltalk-tmbundle:
+- source.smalltalk
+https://github.com/vic/ioke-outdated/raw/master/share/TextMate/Ioke.tmbundle/Syntaxes/Ioke.tmLanguage:
+- source.ioke
+https://github.com/vkostyukov/kotlin-sublime-package:
+- source.Kotlin
+https://github.com/vmg/zephir-sublime:
+- source.php.zephir
+https://github.com/whitequark/llvm.tmbundle:
+- source.llvm

lib/linguist/blob_helper.rb

@@ -2,7 +2,6 @@ require 'linguist/generated'
 require 'charlock_holmes'
 require 'escape_utils'
 require 'mime/types'
-require 'pygments'
 require 'yaml'
 
 module Linguist
@@ -147,6 +146,13 @@ module Linguist
       end
     end
 
+    # Public: Is the blob empty?
+    #
+    # Return true or false
+    def empty?
+      data.nil? || data == ""
+    end
+
     # Public: Is the blob text?
     #
     # Return true or false
@@ -193,10 +199,6 @@ module Linguist
 
     # Public: Is the blob safe to colorize?
     #
-    # We use Pygments for syntax highlighting blobs. Pygments
-    # can be too slow for very large blobs or for certain
-    # corner-case blobs.
-    #
     # Return true or false
     def safe_to_colorize?
       !large? && text? && !high_ratio_of_long_lines?
@@ -204,9 +206,6 @@ module Linguist
 
     # Internal: Does the blob have a ratio of long lines?
     #
-    # These types of files are usually going to make Pygments.rb
-    # angry if we try to colorize them.
-    #
     # Return true or false
     def high_ratio_of_long_lines?
       return false if loc == 0
@@ -314,28 +313,9 @@ module Linguist
       @language ||= Language.detect(self)
     end
 
-    # Internal: Get the lexer of the blob.
-    #
-    # Returns a Lexer.
-    def lexer
-      language ? language.lexer : Pygments::Lexer.find_by_name('Text only')
-    end
-
     # Internal: Get the TextMate compatible scope for the blob
     def tm_scope
       language && language.tm_scope
     end
-
-    # Public: Highlight syntax of blob
-    #
-    # options - A Hash of options (defaults to {})
-    #
-    # Returns html String
-    def colorize(options = {})
-      return unless safe_to_colorize?
-      options[:options] ||= {}
-      options[:options][:encoding] ||= encoding
-      lexer.highlight(data, options)
-    end
   end
 end

lib/linguist/generated.rb

@@ -51,26 +51,25 @@ module Linguist
     #
     # Return true or false
     def generated?
-      name == 'Gemfile.lock' ||
-        minified_files? ||
-        compiled_coffeescript? ||
-        xcode_file? ||
-        generated_parser? ||
-        generated_net_docfile? ||
-        generated_net_designer_file? ||
-        generated_postscript? ||
-        generated_protocol_buffer? ||
-        generated_jni_header? ||
-        composer_lock? ||
-        node_modules? ||
-        godeps? ||
-        vcr_cassette? ||
-        generated_by_zephir?
+      minified_files? ||
+      compiled_coffeescript? ||
+      xcode_file? ||
+      generated_parser? ||
+      generated_net_docfile? ||
+      generated_net_designer_file? ||
+      generated_postscript? ||
+      generated_protocol_buffer? ||
+      generated_jni_header? ||
+      composer_lock? ||
+      node_modules? ||
+      godeps? ||
+      vcr_cassette? ||
+      generated_by_zephir?
     end
 
     # Internal: Is the blob an Xcode file?
     #
-    # Generated if the file extension is an Xcode 
+    # Generated if the file extension is an Xcode
     # file extension.
     #
     # Returns true of false.
@@ -265,4 +264,3 @@ module Linguist
     end
   end
 end
-

lib/linguist/grammars.rb

@@ -0,0 +1,13 @@
+# Note: This file is included in the github-linguist-grammars gem, not the
+# github-linguist gem.
+
+module Linguist
+  module Grammars
+    # Get the path to the directory containing the language grammar JSON files.
+    #
+    # Returns a String.
+    def self.path
+      File.expand_path("../../../grammars", __FILE__)
+    end
+  end
+end

lib/linguist/heuristics.rb

@@ -13,17 +13,31 @@ module Linguist
     # Returns an array of Languages or []
     def self.find_by_heuristics(data, languages)
       if active?
+        result = []
+
         if languages.all? { |l| ["Perl", "Prolog"].include?(l) }
-          result = disambiguate_pl(data, languages)
+          result = disambiguate_pl(data)
         end
         if languages.all? { |l| ["ECL", "Prolog"].include?(l) }
-          result = disambiguate_ecl(data, languages)
+          result = disambiguate_ecl(data)
         end
         if languages.all? { |l| ["IDL", "Prolog"].include?(l) }
-          result = disambiguate_pro(data, languages)
+          result = disambiguate_pro(data)
         end
         if languages.all? { |l| ["Common Lisp", "OpenCL"].include?(l) }
-          result = disambiguate_cl(data, languages)
+          result = disambiguate_cl(data)
+        end
+        if languages.all? { |l| ["Hack", "PHP"].include?(l) }
+          result = disambiguate_hack(data)
+        end
+        if languages.all? { |l| ["Scala", "SuperCollider"].include?(l) }
+          result = disambiguate_sc(data)
+        end
+        if languages.all? { |l| ["AsciiDoc", "AGS Script"].include?(l) }
+          result = disambiguate_asc(data)
+        end
+        if languages.all? { |l| ["FORTRAN", "Forth"].include?(l) }
+          result = disambiguate_f(data)
         end
         return result
       end
@@ -33,28 +47,37 @@ module Linguist
     # We want to shortcut look for Objective-C _and_ now C++ too!
     #
     # Returns an array of Languages or []
-    def self.disambiguate_c(data, languages)
+    def self.disambiguate_c(data)
       matches = []
-      matches << Language["Objective-C"] if data.include?("@interface")
-      matches << Language["C++"] if data.include?("#include <cstdint>")
+      if data.include?("@interface")
+        matches << Language["Objective-C"]
+      elsif data.include?("#include <cstdint>")
+        matches << Language["C++"]
+      end
       matches
     end
 
-    def self.disambiguate_pl(data, languages)
+    def self.disambiguate_pl(data)
       matches = []
-      matches << Language["Prolog"] if data.include?(":-")
-      matches << Language["Perl"] if data.include?("use strict")
+      if data.include?("use strict")
+        matches << Language["Perl"]
+      elsif data.include?(":-")
+        matches << Language["Prolog"]
+      end
       matches
     end
 
-    def self.disambiguate_ecl(data, languages)
+    def self.disambiguate_ecl(data)
       matches = []
-      matches << Language["Prolog"] if data.include?(":-")
-      matches << Language["ECL"] if data.include?(":=")
+      if data.include?(":-")
+        matches << Language["Prolog"]
+      elsif data.include?(":=")
+        matches << Language["ECL"]
+      end
       matches
     end
 
-    def self.disambiguate_pro(data, languages)
+    def self.disambiguate_pro(data)
       matches = []
       if (data.include?(":-"))
         matches << Language["Prolog"]
@@ -64,7 +87,7 @@ module Linguist
       matches
     end
 
-    def self.disambiguate_ts(data, languages)
+    def self.disambiguate_ts(data)
       matches = []
       if (data.include?("</translation>"))
         matches << Language["XML"]
@@ -74,20 +97,60 @@ module Linguist
       matches
     end
 
-    def self.disambiguate_cl(data, languages)
+    def self.disambiguate_cl(data)
       matches = []
-      matches << Language["Common Lisp"] if data.include?("(defun ")
-      matches << Language["OpenCL"] if /\/\* |\/\/ |^\}/.match(data)
+      if data.include?("(defun ")
+        matches << Language["Common Lisp"]
+      elsif /\/\* |\/\/ |^\}/.match(data)
+        matches << Language["OpenCL"]
+      end
       matches
     end
 
-    def self.disambiguate_r(data, languages)
+    def self.disambiguate_r(data)
       matches = []
       matches << Language["Rebol"] if /\bRebol\b/i.match(data)
       matches << Language["R"] if data.include?("<-")
       matches
     end
 
+    def self.disambiguate_hack(data)
+      matches = []
+      if data.include?("<?hh")
+        matches << Language["Hack"]
+      elsif /<?[^h]/.match(data)
+        matches << Language["PHP"]
+      end
+      matches
+    end
+
+    def self.disambiguate_sc(data)
+      matches = []
+      if (/\^(this|super)\./.match(data) || /^\s*(\+|\*)\s*\w+\s*{/.match(data) || /^\s*~\w+\s*=\./.match(data))
+        matches << Language["SuperCollider"]
+      end
+      if (/^\s*import (scala|java)\./.match(data) || /^\s*val\s+\w+\s*=/.match(data) || /^\s*class\b/.match(data))
+        matches << Language["Scala"]
+      end
+      matches
+    end
+
+    def self.disambiguate_asc(data)
+      matches = []
+      matches << Language["AsciiDoc"] if /^=+(\s|\n)/.match(data)
+      matches
+    end
+
+    def self.disambiguate_f(data)
+      matches = []
+      if /^: /.match(data)
+        matches << Language["Forth"]
+      elsif /^([c*][^a-z]|      subroutine\s)/i.match(data)
+        matches << Language["FORTRAN"]
+      end
+      matches
+    end
+
     def self.active?
       !!ACTIVE
     end

lib/linguist/language.rb

@@ -1,8 +1,7 @@
 require 'escape_utils'
-require 'pygments'
 require 'yaml'
 begin
-  require 'json'
+  require 'yajl'
 rescue LoadError
 end
 
@@ -62,7 +61,7 @@ module Linguist
       end
 
       # Language name index
-      @index[language.name] = @name_index[language.name] = language
+      @index[language.name.downcase] = @name_index[language.name.downcase] = language
 
       language.aliases.each do |name|
         # All Language aliases should be unique. Raise if there is a duplicate.
@@ -70,7 +69,7 @@ module Linguist
           raise ArgumentError, "Duplicate alias: #{name}"
         end
 
-        @index[name] = @alias_index[name] = language
+        @index[name.downcase] = @alias_index[name.downcase] = language
       end
 
       language.extensions.each do |extension|
@@ -101,12 +100,8 @@ module Linguist
     def self.detect(blob)
       name = blob.name.to_s
 
-      # Check if the blob is possibly binary and bail early; this is a cheap
-      # test that uses the extension name to guess a binary binary mime type.
-      #
-      # We'll perform a more comprehensive test later which actually involves
-      # looking for binary characters in the blob
-      return nil if blob.likely_binary? || blob.binary?
+      # Bail early if the blob is binary or empty.
+      return nil if blob.likely_binary? || blob.binary? || blob.empty?
 
       # A bit of an elegant hack. If the file is executable but extensionless,
       # append a "magic" extension so it can be classified with other
@@ -125,16 +120,18 @@ module Linguist
       if possible_languages.length > 1
         data = blob.data
         possible_language_names = possible_languages.map(&:name)
+        heuristic_languages = Heuristics.find_by_heuristics(data, possible_language_names)
+
+        if heuristic_languages.size > 1
+          possible_language_names = heuristic_languages.map(&:name)
+        end
 
-        # Don't bother with binary contents or an empty file
-        if data.nil? || data == ""
-          nil
         # Check if there's a shebang line and use that as authoritative
-        elsif (result = find_by_shebang(data)) && !result.empty?
+        if (result = find_by_shebang(data)) && !result.empty?
           result.first
         # No shebang. Still more work to do. Try to find it with our heuristics.
-        elsif (determined = Heuristics.find_by_heuristics(data, possible_language_names)) && !determined.empty?
-          determined.first
+        elsif heuristic_languages.size == 1
+          heuristic_languages.first
         # Lastly, fall back to the probabilistic classifier.
         elsif classified = Classifier.classify(Samples.cache, data, possible_language_names).first
           # Return the actual Language object based of the string language name (i.e., first element of `#classify`)
@@ -164,7 +161,7 @@ module Linguist
     #
     # Returns the Language or nil if none was found.
     def self.find_by_name(name)
-      @name_index[name]
+      name && @name_index[name.downcase]
     end
 
     # Public: Look up Language by one of its aliases.
@@ -178,7 +175,7 @@ module Linguist
     #
     # Returns the Lexer or nil if none was found.
     def self.find_by_alias(name)
-      @alias_index[name]
+      name && @alias_index[name.downcase]
     end
 
     # Public: Look up Languages by filename.
@@ -194,9 +191,25 @@ module Linguist
     def self.find_by_filename(filename)
       basename = File.basename(filename)
       extname = FileBlob.new(filename).extension
-      langs = @filename_index[basename] +
-              @extension_index[extname]
-      langs.compact.uniq
+      (@filename_index[basename] + find_by_extension(extname)).compact.uniq
+    end
+
+    # Public: Look up Languages by file extension.
+    #
+    # extname - The extension String.
+    #
+    # Examples
+    #
+    #   Language.find_by_extension('.rb')
+    #   # => [#<Language name="Ruby">]
+    #
+    #   Language.find_by_extension('rb')
+    #   # => [#<Language name="Ruby">]
+    #
+    # Returns all matching Languages or [] if none were found.
+    def self.find_by_extension(extname)
+      extname = ".#{extname}" unless extname.start_with?(".")
+      @extension_index[extname]
     end
 
     # Public: Look up Languages by shebang line.
@@ -227,7 +240,7 @@ module Linguist
     #
     # Returns the Language or nil if none was found.
     def self.[](name)
-      @index[name]
+      name && @index[name.downcase]
     end
 
     # Public: A List of popular languages
@@ -286,10 +299,7 @@ module Linguist
       # Set aliases
       @aliases = [default_alias_name] + (attributes[:aliases] || [])
 
-      # Lookup Lexer object
-      @lexer = Pygments::Lexer.find_by_name(attributes[:lexer] || name) ||
-        raise(ArgumentError, "#{@name} is missing lexer")
-
+      # Load the TextMate scope name or try to guess one
       @tm_scope = attributes[:tm_scope] || begin
         context = case @type
                   when :data, :markup, :prose
@@ -421,11 +431,6 @@ module Linguist
     # Returns the extensions Array
     attr_reader :filenames
 
-    # Public: Return all possible extensions for language
-    def all_extensions
-      (extensions + [primary_extension]).uniq
-    end
-
     # Deprecated: Get primary extension
     #
     # Defaults to the first extension but can be overridden
@@ -533,8 +538,8 @@ module Linguist
   languages_yml = File.expand_path("../languages.yml", __FILE__)
   languages_json = File.expand_path("../languages.json", __FILE__)
 
-  if File.exist?(languages_json) && defined?(JSON)
-    languages = JSON.load(File.read(languages_json))
+  if File.exist?(languages_json) && defined?(Yajl)
+    languages = Yajl.load(File.read(languages_json))
   else
     languages = YAML.load_file(languages_yml)
   end
@@ -583,9 +588,9 @@ module Linguist
       :ace_mode          => options['ace_mode'],
       :wrap              => options['wrap'],
       :group_name        => options['group'],
-      :searchable        => options.key?('searchable') ? options['searchable'] : true,
+      :searchable        => options.fetch('searchable', true),
       :search_term       => options['search_term'],
-      :extensions        => [options['extensions'].first] + options['extensions'][1..-1].sort,
+      :extensions        => Array(options['extensions']),
       :interpreters      => options['interpreters'].sort,
       :filenames         => options['filenames'],
       :popular           => popular.include?(name)

lib/linguist/samples.rb

@@ -1,5 +1,5 @@
 begin
-  require 'json'
+  require 'yajl'
 rescue LoadError
   require 'yaml'
 end
@@ -19,7 +19,7 @@ module Linguist
     # Hash of serialized samples object
     def self.cache
       @cache ||= begin
-        serializer = defined?(JSON) ? JSON : YAML
+        serializer = defined?(Yajl) ? Yajl : YAML
         serializer.load(File.read(PATH))
       end
     end

lib/linguist/vendor.yml

@@ -36,15 +36,16 @@
 # Go dependencies
 - Godeps/_workspace/
 
-# Bootstrap minified css and js
-- (^|/)bootstrap([^.]*)(\.min)?\.(js|css)$
+# Minified JavaScript and CSS
+- (\.|-)min\.(js|css)$
+
+# Bootstrap css and js
+- (^|/)bootstrap([^.]*)\.(js|css)$
 
 # Font Awesome
-- font-awesome.min.css
 - font-awesome.css
 
 # Foundation css
-- foundation.min.css
 - foundation.css
 
 # Normalize.css
@@ -56,7 +57,6 @@
 
 # Animate.css
 - animate.css
-- animate.min.css
 
 # Vendored dependencies
 - third[-_]?party/
@@ -73,12 +73,12 @@
 ## Commonly Bundled JavaScript frameworks ##
 
 # jQuery
-- (^|/)jquery([^.]*)(\.min)?\.js$
-- (^|/)jquery\-\d\.\d+(\.\d+)?(\.min)?\.js$
+- (^|/)jquery([^.]*)\.js$
+- (^|/)jquery\-\d\.\d+(\.\d+)?\.js$
 
 # jQuery UI
-- (^|/)jquery\-ui(\-\d\.\d+(\.\d+)?)?(\.\w+)?(\.min)?\.(js|css)$
-- (^|/)jquery\.(ui|effects)\.([^.]*)(\.min)?\.(js|css)$
+- (^|/)jquery\-ui(\-\d\.\d+(\.\d+)?)?(\.\w+)?\.(js|css)$
+- (^|/)jquery\.(ui|effects)\.([^.]*)\.(js|css)$
 
 # Prototype
 - (^|/)prototype(.*)\.js$
@@ -110,27 +110,29 @@
 # MathJax
 - (^|/)MathJax/
 
+# Codemirror
+- (^|/)[Cc]ode[Mm]irror/(lib|mode|theme|addon|keymap)
+
 # SyntaxHighlighter - http://alexgorbatchev.com/
 - (^|/)shBrush([^.]*)\.js$
 - (^|/)shCore\.js$
 - (^|/)shLegacy\.js$
 
 # AngularJS
-- (^|/)angular([^.]*)(\.min)?\.js$
+- (^|/)angular([^.]*)\.js$
 
 # D3.js
-- (^|\/)d3(\.v\d+)?([^.]*)(\.min)?\.js$
+- (^|\/)d3(\.v\d+)?([^.]*)\.js$
 
 # React
-- (^|/)react(-[^.]*)?(\.min)?\.js$
+- (^|/)react(-[^.]*)?\.js$
 
 # Modernizr
-- (^|/)modernizr\-\d\.\d+(\.\d+)?(\.min)?\.js$
+- (^|/)modernizr\-\d\.\d+(\.\d+)?\.js$
 - (^|/)modernizr\.custom\.\d+\.js$
 
 # Knockout
 - (^|/)knockout-(\d+\.){3}(debug\.)?js$
-- knockout-min.js
 
 ## Python ##
 
@@ -168,8 +170,8 @@
 - \.intellisense\.js$
 
 # jQuery validation plugin (MS bundles this with asp.net mvc)
-- (^|/)jquery([^.]*)\.validate(\.unobtrusive)?(\.min)?\.js$
-- (^|/)jquery([^.]*)\.unobtrusive\-ajax(\.min)?\.js$
+- (^|/)jquery([^.]*)\.validate(\.unobtrusive)?\.js$
+- (^|/)jquery([^.]*)\.unobtrusive\-ajax\.js$
 
 # Microsoft Ajax
 - (^|/)[Mm]icrosoft([Mm]vc)?([Aa]jax|[Vv]alidation)(\.debug)?\.js$
@@ -196,7 +198,7 @@
 - (^|/)extjs/welcome/
 
 # Html5shiv
-- (^|/)html5shiv(\.min)?\.js$
+- (^|/)html5shiv\.js$
 
 # Samples folders
 - ^[Ss]amples/
@@ -215,8 +217,8 @@
 - ^[Tt]est/fixtures/
 
 # PhoneGap/Cordova
-- (^|/)cordova([^.]*)(\.min)?\.js$
-- (^|/)cordova\-\d\.\d(\.\d)?(\.min)?\.js$
+- (^|/)cordova([^.]*)\.js$
+- (^|/)cordova\-\d\.\d(\.\d)?\.js$
 
 # Foundation js
 - foundation(\..*)?\.js$
@@ -236,7 +238,6 @@
 
 # Octicons
 - octicons.css
-- octicons.min.css
 - sprockets-octicons.scss
 
 # Typesafe Activator

lib/linguist/version.rb

@@ -1,3 +1,3 @@
 module Linguist
-  VERSION = "3.4.1"
+  VERSION = "4.0.2"
 end

package.json

@@ -0,0 +1,6 @@
+{
+  "repository": "https://github.com/github/linguist",
+  "dependencies": {
+    "season": "~>3.0"
+  }
+}

samples/Ant Build System/filenames/ant.xml

@@ -0,0 +1,110 @@
+<?xml version="1.0" encoding="iso-8859-1"?>
+<project name="WebBuild">
+
+    <!-- generate timestamps -->
+    <tstamp />
+
+    <!-- Debugging Macro -->
+    <import file="echopath.xml" />
+
+    <!-- JS build files macro -->
+    <import file="rhinoscript.xml" />
+
+    <!-- Component Build Files -->
+    <import file="setup.xml" />
+    <import file="clean.xml" />
+    <import file="copy.xml" />
+    <import file="file.transform.xml" />
+    <import file="external.tools.xml" />
+    <import file="rename.xml" />
+    <import file="js.xml" />
+    <import file="css.xml" />
+    <import file="img.xml" />
+    <import file="png8.xml" />
+    <import file="yui.xml" />
+    <import file="cdn.xml" />
+    <import file="datauri.xml" />
+    <import file="devlive.xml" />
+
+    <!-- This dirname is the only complete path we know for sure, everything builds off of it -->
+    <dirname property="dir.build" file="${ant.file.WebBuild}" />
+
+    <!-- get name for newly built folder -->
+    <basename property="app.name"       file="${basedir}" />
+
+    <!-- read global properties file -->
+    <property file="${dir.build}\build.properties" />
+
+    <!-- Build Directories -->
+    <property name="dir.build.js"   location="${dir.build}/js" />
+
+    <!-- App Directories -->
+    <property name="dir.app"        location="${dir.result}/${app.name}" />
+    <property name="dir.app.temp"   location="${dir.temp}/${app.name}" />
+    <property name="dir.app.files"  location="${dir.app.temp}/${dir.files}" />
+
+    <!-- Files -->
+    <property name="mapping.js"     location="${dir.app.temp}/${mapping.file.js}" />
+    <property name="mapping.css"    location="${dir.app.temp}/${mapping.file.css}" />
+    <property name="mapping.img"    location="${dir.app.temp}/${mapping.file.img}" />
+    <property name="mapping.swf"    location="${dir.app.temp}/${mapping.file.swf}" />
+    <property name="mapping.fonts"  location="${dir.app.temp}/${mapping.file.fonts}" />
+
+    <!-- Tool Directories -->
+    <property name="dir.bin"    location="${dir.build}/Bin" />
+    <property name="dir.jar"    location="${dir.bin}/jar" />
+
+    <!-- Tool Files -->
+	<property name="tools.compressor"     location="${dir.jar}/${tools.file.compressor}" />
+	<property name="tools.cssembed"       location="${dir.jar}/${tools.file.cssembed}" />
+    <property name="tools.filetransform"  location="${dir.jar}/${tools.file.filetransform}" />
+    <property name="tools.optipng"        location="${dir.bin}/${tools.file.optipng}" />
+    <property name="tools.jpegtran"       location="${dir.bin}/${tools.file.jpegtran}" />
+
+
+    <!-- BUILD TARGETS -->
+
+    <!-- low level utility build targets -->
+
+    <!-- Build the tools -->
+    <target name="-setup.build.tools"
+            depends="-define.filetransform, -define.cssembed, -define.yuicompressor, -define.jsclasspath"
+    />
+
+    <!-- set up filesystem properties -->
+    <target
+        name="-setup"
+        depends="-setup.mode, -setup.conditions, -setup.js, -setup.css, -setup.swf, -setup.img, -setup.fonts, -setup.yui"
+    />
+
+    <!-- utility-ish targets -->
+    <target name="copy"         depends="clean, tools, -copy" />
+    <target name="tools"        depends="-setup.build.tools" />
+    <target name="finalize"     depends="copy, -finalize" />
+    <target name="-prepare"     depends="copy, -setup" />
+
+    <!-- individual component build targets (empty descriptions are to make sure they show in "ant -p") -->
+    <target name="devlive"      depends="-prepare, -devlive"            description="" />
+    <target name="js"           depends="-prepare, -js"                 description="" />
+    <target name="css"          depends="-prepare, -css"                description="" />
+    <target name="rename"       depends="-prepare, -rename"             description="" />
+    <target name="yui"          depends="-prepare, rename, -yui"        description="" />
+    <target name="cdn"          depends="-prepare, -cdn"                description="" />
+
+    <!-- high level build targets (Excluding of images is on purpose here, it's slow) -->
+    <target name="core"
+            depends="devlive, js, css, cdn, rename, yui, -js.inline"
+            description="Core build work"
+    />
+
+    <target name="prod"
+            depends="core, finalize"
+            description="Full Production Build"
+    />
+
+    <!-- debug target -->
+    <target name="debug" depends="-setup">
+        <echoproperties/>
+    </target>
+
+</project>

samples/Ant Build System/filenames/build.xml

@@ -0,0 +1 @@
+ant.xml

samples/C++/bar.hh

@@ -0,0 +1,10 @@
+class Bar
+{
+  protected:
+
+    char *name;
+
+  public:
+
+    void hello();
+}

samples/FORTRAN/sample1.f

@@ -0,0 +1,25 @@
+c comment
+* comment
+
+      program main
+
+      end
+
+      subroutine foo( i, x, b )
+      INTEGER            i
+      REAL               x
+      LOGICAL            b
+
+      if( i.ne.0 ) then
+         call bar( -i )
+      end if
+
+      return
+      end
+
+      double complex function baz()
+
+      baz = (0.0d0,0.0d0)
+
+      return 
+      end

samples/FORTRAN/sample1.for

@@ -0,0 +1,25 @@
+c comment
+* comment
+
+      program main
+
+      end
+
+      subroutine foo( i, x, b )
+      INTEGER            i
+      REAL               x
+      LOGICAL            b
+
+      if( i.ne.0 ) then
+         call bar( -i )
+      end if
+
+      return
+      end
+
+      double complex function baz()
+
+      baz = (0.0d0,0.0d0)
+
+      return 
+      end

samples/FORTRAN/sample2.f

@@ -0,0 +1,25 @@
+      PROGRAM MAIN
+
+      END
+
+C comment
+* comment
+
+      SUBROUTINE foo( i, x, b )
+      INTEGER            i
+      REAL               x
+      LOGICAL            b
+
+      IF( i.NE.0 ) THEN
+         CALL bar( -i )
+      END IF
+
+      RETURN
+      END
+
+      DOUBLE COMPLEX FUNCTION baz()
+
+      baz = (0.0d0,0.0d0)
+
+      RETURN 
+      END

samples/FORTRAN/sample3.F

@@ -0,0 +1,25 @@
+c comment
+* comment
+
+      program main
+
+      end
+
+      subroutine foo( i, x, b )
+      INTEGER            i
+      REAL               x
+      LOGICAL            b
+
+      if( i.ne.0 ) then
+         call bar( -i )
+      end if
+
+      return
+      end
+
+      double complex function baz()
+
+      baz = (0.0d0,0.0d0)
+
+      return 
+      end

samples/Forth/core.f

@@ -0,0 +1,252 @@
+: immediate   lastxt @ dup c@ negate swap c! ;
+
+: \   source nip >in ! ; immediate \ Copyright 2004, 2012 Lars Brinkhoff
+
+: char \ ( "word" -- char )
+    bl-word here 1+ c@ ;
+
+: ahead  here 0 , ;
+
+: resolve   here swap ! ;
+
+: '   bl-word here find 0branch [ ahead ] exit [ resolve ] 0 ;
+
+: postpone-nonimmediate   [ ' literal , ' compile, ] literal , ;
+
+: create   dovariable_code header, reveal ;
+
+create postponers
+    ' postpone-nonimmediate ,
+    ' abort ,
+    ' , ,
+
+: word \ ( char "<chars>string<char>" -- caddr )
+    drop bl-word here ;
+
+: postpone \ ( C: "word" -- )
+    bl word find 1+ cells  postponers + @ execute ; immediate
+
+: unresolved \ ( C: "word" -- orig )
+    postpone postpone  postpone ahead ; immediate
+
+: chars \ ( n1 -- n2 )
+    ;
+
+: else \ ( -- ) ( C: orig1 -- orig2 )
+    unresolved branch swap resolve ; immediate
+
+: if \ ( flag -- ) ( C: -- orig )
+    unresolved 0branch ; immediate
+
+: then \ ( -- ) ( C: orig -- )
+    resolve ; immediate
+
+: [char] \ ( "word" -- )
+    char  postpone literal ; immediate
+
+: (does>)   lastxt @ dodoes_code over >code ! r> swap >does ! ;
+
+: does>   postpone (does>) ; immediate
+
+: begin \ ( -- ) ( C: -- dest )
+    here ; immediate
+
+: while \ ( x -- ) ( C: dest -- orig dest )
+    unresolved 0branch swap ; immediate
+
+: repeat \ ( -- ) ( C: orig dest -- )
+    postpone branch ,  resolve ; immediate
+
+: until \ ( x -- ) ( C: dest -- )
+    postpone 0branch , ; immediate
+
+: recurse   lastxt @ compile, ; immediate
+
+: pad \ ( -- addr )
+    here 1024 + ;
+
+: parse \ ( char "string<char>" -- addr n )
+    pad >r  begin
+	source? if <source 2dup <> else 0 0 then
+    while
+	r@ c!  r> 1+ >r
+    repeat  2drop  pad r> over - ;
+
+: ( \ ( "string<paren>" -- )
+    [ char ) ] literal parse 2drop ; immediate
+    \ TODO: If necessary, refill and keep parsing.
+
+: string, ( addr n -- )
+    here over allot align  swap cmove ;
+
+: (s") ( -- addr n ) ( R: ret1 -- ret2 )
+    r> dup @ swap cell+ 2dup + aligned >r swap ;
+
+create squote   128 allot
+
+: s" ( "string<quote>" -- addr n )
+    state @ if
+	postpone (s")  [char] " parse  dup ,  string,
+    else
+	[char] " parse  >r squote r@ cmove  squote r>
+    then ; immediate
+
+: (abort") ( ... addr n -- ) ( R: ... -- )
+    cr type cr abort ;
+
+: abort" ( ... x "string<quote>" -- ) ( R: ... -- )
+    postpone if  postpone s"  postpone (abort")  postpone then ; immediate
+
+\ ----------------------------------------------------------------------
+
+( Core words. )
+
+\ TODO: #
+\ TODO: #>
+\ TODO: #s
+
+: and  ( x y -- x&y )   nand invert ;
+
+: *   1 2>r 0 swap begin r@ while
+         r> r> swap 2dup dup + 2>r and if swap over + swap then dup +
+      repeat r> r> 2drop drop ;
+
+\ TODO: */mod
+
+: +loop ( -- ) ( C: nest-sys -- )
+    postpone (+loop)  postpone 0branch  ,  postpone unloop ; immediate
+
+: space   bl emit ;
+
+: ?.-  dup 0 < if [char] - emit negate then ;
+
+: digit   [char] 0 + emit ;
+
+: (.)   base @ /mod  ?dup if recurse then  digit ;
+
+: ." ( "string<quote>" -- )   postpone s"  postpone type ; immediate
+
+: . ( x -- )   ?.- (.) space ;
+
+: postpone-number ( caddr -- )
+    0 0 rot count >number dup 0= if
+	2drop nip
+	postpone (literal)  postpone (literal)  postpone ,
+	postpone literal  postpone ,
+    else
+	." Undefined: " type cr abort
+    then ;
+
+' postpone-number  postponers cell+  !
+
+: / ( x y -- x/y )   /mod nip ;
+
+: 0< ( n -- flag )   0 < ;
+
+: 1- ( n -- n-1 )   -1 + ;
+
+: 2! ( x1 x2 addr -- )   swap over ! cell+ ! ;
+
+: 2* ( n -- 2n )   dup + ;
+
+\ Kernel: 2/
+
+: 2@ ( addr -- x1 x2 )   dup cell+ @ swap @ ;
+
+\ Kernel: 2drop
+\ Kernel: 2dup
+
+\ TODO: 2over ( x1 x2 x3 x4 -- x1 x2 x3 x4 x1 x2 )
+\           3 pick 3 pick ;
+
+\ TODO: 2swap
+
+\ TODO: <#
+
+: abs ( n -- |n| )
+    dup 0< if negate then ;
+
+\ TODO: accept
+
+: c, ( n -- )
+    here c!  1 chars allot ;
+
+: char+ ( n1 -- n2 )
+    1+ ;
+
+: constant   create , does> @ ;
+
+: decimal ( -- )
+    10 base ! ;
+
+: depth ( -- n )
+    data_stack 100 cells +  'SP @  - /cell /  2 - ;
+
+: do ( n1 n2 -- ) ( R: -- loop-sys ) ( C: -- do-sys )
+    postpone 2>r  here ; immediate
+
+\ TODO: environment?
+\ TODO: evaluate
+\ TODO: fill
+\ TODO: fm/mod )
+\ TODO: hold
+
+: j ( -- x1 ) ( R: x1 x2 x3 -- x1 x2 x3 )
+    'RP @ 3 cells + @ ;
+
+\ TODO: leave
+
+: loop ( -- ) ( C: nest-sys -- )
+    postpone 1  postpone (+loop)
+    postpone 0branch  ,
+    postpone unloop ; immediate
+
+: lshift   begin ?dup while 1- swap dup + swap repeat ;
+
+: rshift   1 begin over while dup + swap 1- swap repeat nip
+           2>r 0 1 begin r@ while
+              r> r> 2dup swap dup + 2>r and if swap over + swap then dup +
+           repeat r> r> 2drop drop ;
+
+: max ( x y -- max[x,y] )
+    2dup > if drop else nip then ;
+
+\ Kernel: min
+\ TODO:   mod
+\ TODO:   move
+
+: (quit) ( R: ... -- )
+    return_stack 100 cells + 'RP !
+    0 'source-id !  tib ''source !  #tib ''#source !
+    postpone [
+    begin
+	refill
+    while
+	interpret  state @ 0= if ." ok" cr then
+    repeat
+    bye ;
+
+' (quit)  ' quit >body cell+  !
+
+\ TODO: s>d
+\ TODO: sign
+\ TODO: sm/rem
+
+: spaces ( n -- )
+    0 do space loop ;
+
+\ TODO: u.
+
+: signbit ( -- n )   -1 1 rshift invert ;
+
+: xor ( x y -- x^y )    2dup nand >r r@ nand swap r> nand nand ;
+
+: u<  ( x y -- flag )  signbit xor swap signbit xor > ;
+
+\ TODO: um/mod
+
+: variable ( "word" -- )
+    create /cell allot ;
+
+: ['] \ ( C: "word" -- )
+    ' postpone literal ; immediate

samples/Forth/core.for

@@ -0,0 +1,252 @@
+: immediate   lastxt @ dup c@ negate swap c! ;
+
+: \   source nip >in ! ; immediate \ Copyright 2004, 2012 Lars Brinkhoff
+
+: char \ ( "word" -- char )
+    bl-word here 1+ c@ ;
+
+: ahead  here 0 , ;
+
+: resolve   here swap ! ;
+
+: '   bl-word here find 0branch [ ahead ] exit [ resolve ] 0 ;
+
+: postpone-nonimmediate   [ ' literal , ' compile, ] literal , ;
+
+: create   dovariable_code header, reveal ;
+
+create postponers
+    ' postpone-nonimmediate ,
+    ' abort ,
+    ' , ,
+
+: word \ ( char "<chars>string<char>" -- caddr )
+    drop bl-word here ;
+
+: postpone \ ( C: "word" -- )
+    bl word find 1+ cells  postponers + @ execute ; immediate
+
+: unresolved \ ( C: "word" -- orig )
+    postpone postpone  postpone ahead ; immediate
+
+: chars \ ( n1 -- n2 )
+    ;
+
+: else \ ( -- ) ( C: orig1 -- orig2 )
+    unresolved branch swap resolve ; immediate
+
+: if \ ( flag -- ) ( C: -- orig )
+    unresolved 0branch ; immediate
+
+: then \ ( -- ) ( C: orig -- )
+    resolve ; immediate
+
+: [char] \ ( "word" -- )
+    char  postpone literal ; immediate
+
+: (does>)   lastxt @ dodoes_code over >code ! r> swap >does ! ;
+
+: does>   postpone (does>) ; immediate
+
+: begin \ ( -- ) ( C: -- dest )
+    here ; immediate
+
+: while \ ( x -- ) ( C: dest -- orig dest )
+    unresolved 0branch swap ; immediate
+
+: repeat \ ( -- ) ( C: orig dest -- )
+    postpone branch ,  resolve ; immediate
+
+: until \ ( x -- ) ( C: dest -- )
+    postpone 0branch , ; immediate
+
+: recurse   lastxt @ compile, ; immediate
+
+: pad \ ( -- addr )
+    here 1024 + ;
+
+: parse \ ( char "string<char>" -- addr n )
+    pad >r  begin
+	source? if <source 2dup <> else 0 0 then
+    while
+	r@ c!  r> 1+ >r
+    repeat  2drop  pad r> over - ;
+
+: ( \ ( "string<paren>" -- )
+    [ char ) ] literal parse 2drop ; immediate
+    \ TODO: If necessary, refill and keep parsing.
+
+: string, ( addr n -- )
+    here over allot align  swap cmove ;
+
+: (s") ( -- addr n ) ( R: ret1 -- ret2 )
+    r> dup @ swap cell+ 2dup + aligned >r swap ;
+
+create squote   128 allot
+
+: s" ( "string<quote>" -- addr n )
+    state @ if
+	postpone (s")  [char] " parse  dup ,  string,
+    else
+	[char] " parse  >r squote r@ cmove  squote r>
+    then ; immediate
+
+: (abort") ( ... addr n -- ) ( R: ... -- )
+    cr type cr abort ;
+
+: abort" ( ... x "string<quote>" -- ) ( R: ... -- )
+    postpone if  postpone s"  postpone (abort")  postpone then ; immediate
+
+\ ----------------------------------------------------------------------
+
+( Core words. )
+
+\ TODO: #
+\ TODO: #>
+\ TODO: #s
+
+: and  ( x y -- x&y )   nand invert ;
+
+: *   1 2>r 0 swap begin r@ while
+         r> r> swap 2dup dup + 2>r and if swap over + swap then dup +
+      repeat r> r> 2drop drop ;
+
+\ TODO: */mod
+
+: +loop ( -- ) ( C: nest-sys -- )
+    postpone (+loop)  postpone 0branch  ,  postpone unloop ; immediate
+
+: space   bl emit ;
+
+: ?.-  dup 0 < if [char] - emit negate then ;
+
+: digit   [char] 0 + emit ;
+
+: (.)   base @ /mod  ?dup if recurse then  digit ;
+
+: ." ( "string<quote>" -- )   postpone s"  postpone type ; immediate
+
+: . ( x -- )   ?.- (.) space ;
+
+: postpone-number ( caddr -- )
+    0 0 rot count >number dup 0= if
+	2drop nip
+	postpone (literal)  postpone (literal)  postpone ,
+	postpone literal  postpone ,
+    else
+	." Undefined: " type cr abort
+    then ;
+
+' postpone-number  postponers cell+  !
+
+: / ( x y -- x/y )   /mod nip ;
+
+: 0< ( n -- flag )   0 < ;
+
+: 1- ( n -- n-1 )   -1 + ;
+
+: 2! ( x1 x2 addr -- )   swap over ! cell+ ! ;
+
+: 2* ( n -- 2n )   dup + ;
+
+\ Kernel: 2/
+
+: 2@ ( addr -- x1 x2 )   dup cell+ @ swap @ ;
+
+\ Kernel: 2drop
+\ Kernel: 2dup
+
+\ TODO: 2over ( x1 x2 x3 x4 -- x1 x2 x3 x4 x1 x2 )
+\           3 pick 3 pick ;
+
+\ TODO: 2swap
+
+\ TODO: <#
+
+: abs ( n -- |n| )
+    dup 0< if negate then ;
+
+\ TODO: accept
+
+: c, ( n -- )
+    here c!  1 chars allot ;
+
+: char+ ( n1 -- n2 )
+    1+ ;
+
+: constant   create , does> @ ;
+
+: decimal ( -- )
+    10 base ! ;
+
+: depth ( -- n )
+    data_stack 100 cells +  'SP @  - /cell /  2 - ;
+
+: do ( n1 n2 -- ) ( R: -- loop-sys ) ( C: -- do-sys )
+    postpone 2>r  here ; immediate
+
+\ TODO: environment?
+\ TODO: evaluate
+\ TODO: fill
+\ TODO: fm/mod )
+\ TODO: hold
+
+: j ( -- x1 ) ( R: x1 x2 x3 -- x1 x2 x3 )
+    'RP @ 3 cells + @ ;
+
+\ TODO: leave
+
+: loop ( -- ) ( C: nest-sys -- )
+    postpone 1  postpone (+loop)
+    postpone 0branch  ,
+    postpone unloop ; immediate
+
+: lshift   begin ?dup while 1- swap dup + swap repeat ;
+
+: rshift   1 begin over while dup + swap 1- swap repeat nip
+           2>r 0 1 begin r@ while
+              r> r> 2dup swap dup + 2>r and if swap over + swap then dup +
+           repeat r> r> 2drop drop ;
+
+: max ( x y -- max[x,y] )
+    2dup > if drop else nip then ;
+
+\ Kernel: min
+\ TODO:   mod
+\ TODO:   move
+
+: (quit) ( R: ... -- )
+    return_stack 100 cells + 'RP !
+    0 'source-id !  tib ''source !  #tib ''#source !
+    postpone [
+    begin
+	refill
+    while
+	interpret  state @ 0= if ." ok" cr then
+    repeat
+    bye ;
+
+' (quit)  ' quit >body cell+  !
+
+\ TODO: s>d
+\ TODO: sign
+\ TODO: sm/rem
+
+: spaces ( n -- )
+    0 do space loop ;
+
+\ TODO: u.
+
+: signbit ( -- n )   -1 1 rshift invert ;
+
+: xor ( x y -- x^y )    2dup nand >r r@ nand swap r> nand nand ;
+
+: u<  ( x y -- flag )  signbit xor swap signbit xor > ;
+
+\ TODO: um/mod
+
+: variable ( "word" -- )
+    create /cell allot ;
+
+: ['] \ ( C: "word" -- )
+    ' postpone literal ; immediate

samples/Forth/core1.F

@@ -0,0 +1,252 @@
+: immediate   lastxt @ dup c@ negate swap c! ;
+
+: \   source nip >in ! ; immediate \ Copyright 2004, 2012 Lars Brinkhoff
+
+: char \ ( "word" -- char )
+    bl-word here 1+ c@ ;
+
+: ahead  here 0 , ;
+
+: resolve   here swap ! ;
+
+: '   bl-word here find 0branch [ ahead ] exit [ resolve ] 0 ;
+
+: postpone-nonimmediate   [ ' literal , ' compile, ] literal , ;
+
+: create   dovariable_code header, reveal ;
+
+create postponers
+    ' postpone-nonimmediate ,
+    ' abort ,
+    ' , ,
+
+: word \ ( char "<chars>string<char>" -- caddr )
+    drop bl-word here ;
+
+: postpone \ ( C: "word" -- )
+    bl word find 1+ cells  postponers + @ execute ; immediate
+
+: unresolved \ ( C: "word" -- orig )
+    postpone postpone  postpone ahead ; immediate
+
+: chars \ ( n1 -- n2 )
+    ;
+
+: else \ ( -- ) ( C: orig1 -- orig2 )
+    unresolved branch swap resolve ; immediate
+
+: if \ ( flag -- ) ( C: -- orig )
+    unresolved 0branch ; immediate
+
+: then \ ( -- ) ( C: orig -- )
+    resolve ; immediate
+
+: [char] \ ( "word" -- )
+    char  postpone literal ; immediate
+
+: (does>)   lastxt @ dodoes_code over >code ! r> swap >does ! ;
+
+: does>   postpone (does>) ; immediate
+
+: begin \ ( -- ) ( C: -- dest )
+    here ; immediate
+
+: while \ ( x -- ) ( C: dest -- orig dest )
+    unresolved 0branch swap ; immediate
+
+: repeat \ ( -- ) ( C: orig dest -- )
+    postpone branch ,  resolve ; immediate
+
+: until \ ( x -- ) ( C: dest -- )
+    postpone 0branch , ; immediate
+
+: recurse   lastxt @ compile, ; immediate
+
+: pad \ ( -- addr )
+    here 1024 + ;
+
+: parse \ ( char "string<char>" -- addr n )
+    pad >r  begin
+	source? if <source 2dup <> else 0 0 then
+    while
+	r@ c!  r> 1+ >r
+    repeat  2drop  pad r> over - ;
+
+: ( \ ( "string<paren>" -- )
+    [ char ) ] literal parse 2drop ; immediate
+    \ TODO: If necessary, refill and keep parsing.
+
+: string, ( addr n -- )
+    here over allot align  swap cmove ;
+
+: (s") ( -- addr n ) ( R: ret1 -- ret2 )
+    r> dup @ swap cell+ 2dup + aligned >r swap ;
+
+create squote   128 allot
+
+: s" ( "string<quote>" -- addr n )
+    state @ if
+	postpone (s")  [char] " parse  dup ,  string,
+    else
+	[char] " parse  >r squote r@ cmove  squote r>
+    then ; immediate
+
+: (abort") ( ... addr n -- ) ( R: ... -- )
+    cr type cr abort ;
+
+: abort" ( ... x "string<quote>" -- ) ( R: ... -- )
+    postpone if  postpone s"  postpone (abort")  postpone then ; immediate
+
+\ ----------------------------------------------------------------------
+
+( Core words. )
+
+\ TODO: #
+\ TODO: #>
+\ TODO: #s
+
+: and  ( x y -- x&y )   nand invert ;
+
+: *   1 2>r 0 swap begin r@ while
+         r> r> swap 2dup dup + 2>r and if swap over + swap then dup +
+      repeat r> r> 2drop drop ;
+
+\ TODO: */mod
+
+: +loop ( -- ) ( C: nest-sys -- )
+    postpone (+loop)  postpone 0branch  ,  postpone unloop ; immediate
+
+: space   bl emit ;
+
+: ?.-  dup 0 < if [char] - emit negate then ;
+
+: digit   [char] 0 + emit ;
+
+: (.)   base @ /mod  ?dup if recurse then  digit ;
+
+: ." ( "string<quote>" -- )   postpone s"  postpone type ; immediate
+
+: . ( x -- )   ?.- (.) space ;
+
+: postpone-number ( caddr -- )
+    0 0 rot count >number dup 0= if
+	2drop nip
+	postpone (literal)  postpone (literal)  postpone ,
+	postpone literal  postpone ,
+    else
+	." Undefined: " type cr abort
+    then ;
+
+' postpone-number  postponers cell+  !
+
+: / ( x y -- x/y )   /mod nip ;
+
+: 0< ( n -- flag )   0 < ;
+
+: 1- ( n -- n-1 )   -1 + ;
+
+: 2! ( x1 x2 addr -- )   swap over ! cell+ ! ;
+
+: 2* ( n -- 2n )   dup + ;
+
+\ Kernel: 2/
+
+: 2@ ( addr -- x1 x2 )   dup cell+ @ swap @ ;
+
+\ Kernel: 2drop
+\ Kernel: 2dup
+
+\ TODO: 2over ( x1 x2 x3 x4 -- x1 x2 x3 x4 x1 x2 )
+\           3 pick 3 pick ;
+
+\ TODO: 2swap
+
+\ TODO: <#
+
+: abs ( n -- |n| )
+    dup 0< if negate then ;
+
+\ TODO: accept
+
+: c, ( n -- )
+    here c!  1 chars allot ;
+
+: char+ ( n1 -- n2 )
+    1+ ;
+
+: constant   create , does> @ ;
+
+: decimal ( -- )
+    10 base ! ;
+
+: depth ( -- n )
+    data_stack 100 cells +  'SP @  - /cell /  2 - ;
+
+: do ( n1 n2 -- ) ( R: -- loop-sys ) ( C: -- do-sys )
+    postpone 2>r  here ; immediate
+
+\ TODO: environment?
+\ TODO: evaluate
+\ TODO: fill
+\ TODO: fm/mod )
+\ TODO: hold
+
+: j ( -- x1 ) ( R: x1 x2 x3 -- x1 x2 x3 )
+    'RP @ 3 cells + @ ;
+
+\ TODO: leave
+
+: loop ( -- ) ( C: nest-sys -- )
+    postpone 1  postpone (+loop)
+    postpone 0branch  ,
+    postpone unloop ; immediate
+
+: lshift   begin ?dup while 1- swap dup + swap repeat ;
+
+: rshift   1 begin over while dup + swap 1- swap repeat nip
+           2>r 0 1 begin r@ while
+              r> r> 2dup swap dup + 2>r and if swap over + swap then dup +
+           repeat r> r> 2drop drop ;
+
+: max ( x y -- max[x,y] )
+    2dup > if drop else nip then ;
+
+\ Kernel: min
+\ TODO:   mod
+\ TODO:   move
+
+: (quit) ( R: ... -- )
+    return_stack 100 cells + 'RP !
+    0 'source-id !  tib ''source !  #tib ''#source !
+    postpone [
+    begin
+	refill
+    while
+	interpret  state @ 0= if ." ok" cr then
+    repeat
+    bye ;
+
+' (quit)  ' quit >body cell+  !
+
+\ TODO: s>d
+\ TODO: sign
+\ TODO: sm/rem
+
+: spaces ( n -- )
+    0 do space loop ;
+
+\ TODO: u.
+
+: signbit ( -- n )   -1 1 rshift invert ;
+
+: xor ( x y -- x^y )    2dup nand >r r@ nand swap r> nand nand ;
+
+: u<  ( x y -- flag )  signbit xor swap signbit xor > ;
+
+\ TODO: um/mod
+
+: variable ( "word" -- )
+    create /cell allot ;
+
+: ['] \ ( C: "word" -- )
+    ' postpone literal ; immediate

samples/Golo/adapters.golo

@@ -0,0 +1,67 @@
+# Copyright 2012-2014 Institut National des Sciences Appliquées de Lyon (INSA-Lyon)
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module samples.Adapters
+
+local function list_sample = |fabric| {
+  println(">>> list_sample()")
+  let carbonCopy = list[]
+  let conf = map[
+    ["extends", "java.util.ArrayList"],
+    ["overrides", map[
+      ["*", |super, name, args| {
+        if name == "add" {
+          if args: length() == 2 {
+            carbonCopy: add(args: get(1))
+          } else {
+            carbonCopy: add(args: get(1), args: get(2))
+          }
+        }
+        return super: invokeWithArguments(args)
+      }
+    ]]
+  ]]
+  let list = fabric: maker(conf): newInstance()
+  list: add("bar")
+  list: add(0, "foo")
+  list: add("baz")
+  println("      list: " + list + " " + list: getClass())
+  println("carbonCopy: " + carbonCopy + " " + carbonCopy: getClass())
+}
+
+local function runnable_sample = |fabric| {
+  println(">>> runnable_sample")
+  let result = array[1, 2, 3]
+  let conf = map[
+    ["interfaces", ["java.io.Serializable", "java.lang.Runnable"]],
+    ["implements", map[
+      ["run", |this| {
+        for (var i = 0, i < result: length(), i = i + 1) {
+          result: set(i, result: get(i) + 10)
+        }
+      }]
+    ]]
+  ]
+  let runner = fabric: maker(conf): newInstance()
+  runner: run()
+  println("      result: " + result: toString())
+  println("serializable? " + (runner oftype java.io.Serializable.class))
+  println("    runnable? " + (runner oftype java.lang.Runnable.class))
+}
+
+function main = |args| {
+  let fabric = AdapterFabric()
+  list_sample(fabric)
+  runnable_sample(fabric)
+}

samples/Golo/async.golo

@@ -0,0 +1,84 @@
+# Copyright 2012-2014 Institut National des Sciences Appliquées de Lyon (INSA-Lyon)
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module samples.AsyncHelpers
+
+import gololang.Async
+import java.util.concurrent.TimeUnit
+import java.util.concurrent.Executors
+
+local function fib = |n| {
+  if n <= 1 {
+    return n
+  } else {
+    return fib(n - 1) + fib(n - 2)
+  }
+}
+
+function main = |args| {
+
+  let executor = newCachedThreadPool()
+  println("Let's do some useless asynchronous operations...")
+
+  var f = executor: enqueue({
+    Thread.sleep(1000_L)
+    return 666
+  })
+  f:
+    onSet(|v| -> println(">>> #slow -> " + v)): 
+    onFail(|e| -> println(">>> #fail -> " + e))
+  f:
+    cancel(true)
+
+  f = executor: enqueue({
+    Thread.sleep(1000_L)
+    return 666
+  })
+  f:
+    onSet(|v| -> println(">>> #ok -> " + v)): 
+    onFail(|e| -> println(">>> #wtf? -> " + e))
+
+  let fib_10 = promise()
+  let fib_20 = promise()
+  let fib_30 = promise()
+  let fib_40 = promise()
+
+  let futures = [
+    fib_10: future(), fib_20: future(), 
+    fib_30: future(), fib_40: future()
+  ]
+
+  executor: submit(-> fib_10: set(fib(10)))
+  executor: submit(-> fib_20: set(fib(20)))
+  executor: submit(-> fib_30: set(fib(30)))
+  executor: submit(-> fib_40: set(fib(40)))
+
+  all(futures): onSet(|results| -> println(">>> Fibs: " + results))
+
+  let truth = promise()
+  truth: 
+    future(): 
+    map(|v| -> "truth=" + v): 
+    onSet(|v| -> executor: submit(-> println(">>> (another thread) " + v))):
+    onSet(|v| -> println(">>> (same thread) " + v))
+  executor: submit({
+    Thread.sleep(500_L)
+    truth: set(42)
+  })
+  
+  Thread.sleep(1000_L)
+  executor: shutdown()
+  executor: awaitTermination(2_L, SECONDS())
+  println("Bye!")
+}

samples/Golo/augmentations.golo

@@ -0,0 +1,37 @@
+# Copyright 2012-2014 Institut National des Sciences Appliquées de Lyon (INSA-Lyon)
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module samples.Augmentations
+
+import java.util.LinkedList
+
+augment java.util.List {
+  function with = |this, value| {
+    this: add(value)
+    return this
+  }
+}
+
+augment java.util.Collection {
+  function doToEach = |this, func| {
+    foreach (element in this) {
+      func(element)
+    }
+  }
+}
+
+function main = |args| {
+  let list = LinkedList(): with("foo"): with("bar"): with("baz")
+  list: doToEach(|value| -> println(">>> " + value))
+}

samples/Golo/closures.golo

@@ -0,0 +1,43 @@
+# Copyright 2012-2014 Institut National des Sciences Appliquées de Lyon (INSA-Lyon)
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module Closures
+
+local function sayHello = |who| -> "Hello " + who + "!"
+
+function main = |args| {
+  let adder = |a, b| -> a + b
+  println(adder: invokeWithArguments(1, 2))
+  println(adder(1, 2))
+
+  let addToTen = adder: bindTo(10)
+  println(addToTen: invokeWithArguments(2))
+  println(addToTen(2))
+
+  let adding = |x| -> |y| -> adder(x, y)
+  let addingTen = adding(10)
+  println(addingTen(4))
+  println(adding(2)(4))
+
+  println(sayHello("Julien"))
+
+  let list = java.util.LinkedList()
+  let pump_it = {
+    list: add("I heard you say")
+    list: add("Hey!")
+    list: add("Hey!")
+  }
+  pump_it()
+  println(list)
+}

samples/Golo/coin-change.golo

@@ -0,0 +1,34 @@
+# Copyright 2012-2014 Institut National des Sciences Appliquées de Lyon (INSA-Lyon)
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module CoinChange
+
+import java.util.LinkedList
+
+function change = |money, coins| -> match {
+  when money == 0 then 1
+  when (money < 0) or (coins: isEmpty()) then 0
+  otherwise change(money - coins: head(), coins) + change(money, coins: tail())
+}
+
+function main = |args| {
+  let coins = LinkedList(): append(1, 2, 5, 10, 20)
+  println("Coins: " + coins)
+  println("0: " + change(0, coins))
+  println("1: " + change(1, coins))
+  println("2: " + change(2, coins))
+  println("10: " + change(10, coins))
+  println("12: " + change(12, coins))
+  println("6: " + change(6, coins))
+}

samples/Golo/collection-literals.golo

@@ -0,0 +1,55 @@
+# Copyright 2012-2014 Institut National des Sciences Appliquées de Lyon (INSA-Lyon)
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module samples.CollectionLiterals
+
+local function play_with_tuples = {
+  let hello = ["Hello", "world", "!"]
+  foreach str in hello {
+    print(str + " ")
+  }
+  println("")
+
+  println(hello: get(0) + "-" + hello: get(1) + "-" + hello: get(2))
+
+  println(hello: join("/"))
+}
+
+local function play_with_literals = {
+  let data = [
+    [1, 2, 3],
+    tuple[1, 2, 3],
+    array[1, 2, 3],
+    set[1, 2, 3, 3, 1],
+    map[
+      ["a", 10],
+      ["b", 20]
+    ],
+    vector[1, 2, 3],
+    list[1, 2, 3]
+  ]
+
+  data: each(|element| {
+    println(element: toString())
+    println("  type: " + element: getClass())
+  })
+}
+
+function main = |args| {
+  println(">>> Literals")
+  play_with_literals()
+  println("\n>>> Tuples")
+  play_with_tuples()
+}
+

samples/Golo/context-decorator.golo

@@ -0,0 +1,53 @@
+# Copyright 2012-2014 Institut National des Sciences Appliquées de Lyon (INSA-Lyon)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module samples.ContextDecorator
+
+import gololang.Decorators
+
+let myContext = defaultContext():
+  count(0):
+  define("entry", |this, args| {
+    this: count(this: count() + 1)
+    println("hello:" + this: count())
+    return args
+  }):
+  define("exit", |this, result| {
+    require(result >= 3, "wrong value")
+    println("goobye")
+    return result
+  }):
+  define("catcher", |this, e| {
+    println("Caught " + e)
+    throw e
+  }):
+  define("finallizer", |this| {println("do some cleanup")})
+
+
+@withContext(myContext)
+function foo = |a, b| {
+  println("Hard computation")
+  return a + b
+}
+
+function main = |args| {
+  println(foo(1,2))
+  println("====")
+  println(withContext(myContext)(|a| -> 2*a)(3))
+  println("====")
+  try {
+    println(foo(1, 1))
+  } catch (e) { }
+}
+

samples/Golo/decorators.golo

@@ -0,0 +1,83 @@
+# Copyright 2012-2014 Institut National des Sciences Appliquées de Lyon (INSA-Lyon)
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module samples.Decorators
+
+import java.util.LinkedList
+
+function simple_decorator = |func| {
+  return |a,b| -> func(a+1,b+1)
+}
+
+@simple_decorator
+function simple_adder = |x,y| -> x + y
+
+function decorator_with_params = |param1, param2|{
+  return |func| {
+    return |a,b| -> func(a+param1,b+param2)
+  }
+}
+
+@decorator_with_params(10,2)
+function parametrized_adder = |x,y| -> x + y
+
+function generic_decorator = |func| {
+  return |args...| {
+    println("number of params : "+args: length())
+    return func: invokeWithArguments(args)
+  }
+}
+
+@generic_decorator
+function generic_adder0 = -> 42
+
+@generic_decorator
+function generic_adder1 = |x| -> x
+
+@generic_decorator
+function generic_adder2 = |x,y| -> x + y
+
+@generic_decorator
+function generic_adder3 = |x,y,z| -> x + y + z
+
+function list_sum_decorator = |func| {
+    return |this| -> func(this) - 8
+}
+
+augment java.util.List {
+
+  @list_sum_decorator
+  function sum = |this| {
+    var acc = 0
+    foreach elem in this {
+        acc = acc + elem
+    }
+    return acc
+  }
+}
+
+function main = |args| {
+  println(simple_adder(10,30))
+  println(parametrized_adder(10,20))
+  println(generic_adder0())
+  println(generic_adder1(42))
+  println(generic_adder2(20,22))
+  println(generic_adder3(10,12,20))
+  let list = LinkedList()
+  list: add(5)
+  list: add(10)
+  list: add(15)
+  list: add(20)
+  println(list: sum())
+}

samples/Golo/dynamic-evaluation.golo

@@ -0,0 +1,88 @@
+# Copyright 2012-2014 Institut National des Sciences Appliquées de Lyon (INSA-Lyon)
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module samples.DynamicEvaluation
+
+import gololang.EvaluationEnvironment
+
+local function test_asModule = |env| {
+  let code =
+"""
+module foo
+
+function a = -> "a!"
+function b = -> "b!"
+"""
+  let mod = env: asModule(code)
+  let a = fun("a", mod)
+  let b = fun("b", mod)
+  println(">>> asModule()")
+  println(a())
+  println(b())
+}
+
+local function test_anonymousModule = |env| {
+  let code =
+"""
+function a = -> "a."
+function b = -> "b."
+"""
+  let mod = env: anonymousModule(code)
+  let a = fun("a", mod)
+  let b = fun("b", mod)
+  println(">>> anonymousModule()")
+  println(a())
+  println(b())
+}
+
+local function test_asFunction = |env| {
+  let code = "return (a + b) * 2"
+  let f = env: asFunction(code, "a", "b")
+  println(">>> asFunction")
+  println(f(10, 20))
+}
+
+local function test_def = |env| {
+  let code = "|a, b| -> (a + b) * 2"
+  let f = env: def(code)
+  println(">>> def")
+  println(f(10, 20))
+}
+
+local function test_run = |env| {
+  let code = """println(">>> run")
+  foreach (i in range(0, 3)) {
+    println("w00t")
+  }"""
+  env: run(code)
+}
+
+local function test_run_map = |env| {
+  let code = """println(">>> run_map")
+  println(a)
+  println(b)
+  """
+  let values = java.util.TreeMap(): add("a", 1): add("b", 2)
+  env: run(code, values)
+}
+
+function main = |args| {
+  let env = EvaluationEnvironment()
+  test_asModule(env)
+  test_anonymousModule(env)
+  test_asFunction(env)
+  test_def(env)
+  test_run(env)
+  test_run_map(env)
+}

samples/Golo/dynamic-object-person.golo

@@ -0,0 +1,29 @@
+# Copyright 2012-2014 Institut National des Sciences Appliquées de Lyon (INSA-Lyon)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module samples.DynamicObjectPerson
+
+local function mrbean = -> DynamicObject():
+  name("Mr Bean"):
+  email("mrbean@gmail.com"):
+  define("toString", |this| -> this: name() + " <" + this: email() + ">")
+
+function main = |args| {
+
+  let bean = mrbean()
+  println(bean: toString())
+
+  bean: email("mrbean@outlook.com")
+  println(bean: toString())
+}

samples/Golo/echo-args.golo

@@ -0,0 +1,34 @@
+# Copyright 2012-2014 Institut National des Sciences Appliquées de Lyon (INSA-Lyon)
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module EchoArgs
+
+function main = |args| {
+  
+  println("With a for loop and an index:")
+  for (var i = 0, i < args: length(), i = i + 1) {
+    println("  #" + i + " -> " + args: get(i))
+  }
+
+  println("With a foreach loop:")
+  foreach arg in args {
+    println("  " + arg)
+  }
+
+  println("With a foreach over a range:")
+  foreach i in range(0, args: length()) {
+    println("  #" + i + " -> " + args: get(i))
+  }
+}
+

samples/Golo/enums-thread-state.golo

@@ -0,0 +1,31 @@
+# Copyright 2012-2014 Institut National des Sciences Appliquées de Lyon (INSA-Lyon)
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module sample.EnumsThreadState
+
+import java.lang.Thread$State
+
+function main = |args| {
+
+  # Call the enum entry like a function
+  let new = Thread$State.NEW()
+  println("name=" + new: name() + ", ordinal=" + new: ordinal())
+  println("-----------")
+
+  # Walk through all enum entries
+  foreach element in Thread$State.values() {
+    println("name=" + element: name() + ", ordinal=" + element: ordinal())
+  }
+}
+

samples/Golo/fibonacci.golo

@@ -0,0 +1,39 @@
+# Copyright 2012-2014 Institut National des Sciences Appliquées de Lyon (INSA-Lyon)
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module samples.Fibonacci
+
+import java.lang.System
+
+function fib = |n| {
+  if n <= 1 {
+    return n
+  } else {
+    return fib(n - 1) + fib(n - 2)
+  }
+}
+
+local function run = {
+  let start = System.currentTimeMillis()
+  let result = fib(40)
+  let duration = System.currentTimeMillis() - start
+  println(">>> " + result + " (took " + duration + "ms)")
+}
+
+function main = |args| {
+  while true {
+    run()
+  }
+}
+

samples/Golo/helloworld.golo

@@ -0,0 +1,20 @@
+# Copyright 2012-2014 Institut National des Sciences Appliquées de Lyon (INSA-Lyon)
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module hello.World
+
+function main = |args| {
+  println("Hello world!")
+}
+

samples/Golo/http-server.golo

@@ -0,0 +1,53 @@
+# Copyright 2012-2014 Institut National des Sciences Appliquées de Lyon (INSA-Lyon)
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module samples.WebServer
+
+import java.lang
+import java.net.InetSocketAddress
+import com.sun.net.httpserver
+import com.sun.net.httpserver.HttpServer
+
+function main = |args| {
+
+  let server = HttpServer.create(InetSocketAddress("localhost", 8081), 0)
+  
+  server: createContext("/", |exchange| {
+    let headers = exchange: getResponseHeaders()
+    let response = StringBuilder():
+      append("Requested URI: "):
+      append(exchange: getRequestURI()):
+      append("\n"):
+      append("Current time: "):
+      append(java.util.Date()):
+      append("\n"):
+      toString()
+    headers: set("Content-Type", "text/plain")
+    exchange: sendResponseHeaders(200, response: length())
+    exchange: getResponseBody(): write(response: getBytes())
+    exchange: close()
+  })
+
+  server: createContext("/shutdown", |exchange| {
+    let response = "Ok, thanks, bye!"
+    exchange: getResponseHeaders(): set("Content-Type", "text/plain")
+    exchange: sendResponseHeaders(200, response: length())
+    exchange: getResponseBody(): write(response: getBytes())
+    exchange: close()
+    server: stop(5)
+  })
+
+  server: start()
+  println(">>> http://localhost:8081/")
+}

samples/Golo/logdeco.golo

@@ -0,0 +1,65 @@
+
+module samples.LogDeco
+
+function log1 = |msg| {
+    return |fun| {
+        return |args...| {
+            println(msg)
+            return fun: invokeWithArguments(args)
+        }
+    }
+}
+
+@log1("calling foo")
+function foo = |a| {
+    println("foo got a " + a)
+}
+
+@log1("I'am a bar")
+function bar = |a| -> 2*a
+
+let sayHello = log1("Hello")
+
+@sayHello
+function baz = -> "Goodbye"
+
+function log2 = |msgBefore| -> |msgAfter| -> |func| -> |args...| {
+    println(msgBefore)
+    let res = func: invokeWithArguments(args)
+    println(msgAfter)
+    return res
+}
+
+@log2("enter foo")("exit foo")
+function spam = |a| {
+    println("foo: " + a)
+}
+
+function logEnterExit = |name| -> log2("# enter " + name)("# exit " + name)
+
+@logEnterExit("bar")
+function egg = { println("doing something...") }
+
+function main = |args| {
+
+    foo("bar")
+    
+    println("---")
+    println(bar(21))
+    
+    println("---")
+    println(baz())
+
+    println("---")
+    spam("bar")
+
+    println("---")
+    egg()
+
+    println("---")
+    let strange_use = log2("hello")("goodbye")({println(":p")})
+    strange_use()
+
+    println("---")
+    log2("another")("use")(|a|{println(a)})("strange")
+}

samples/Golo/matching-operator.golo

@@ -0,0 +1,40 @@
+# Copyright 2012-2014 Institut National des Sciences Appliquées de Lyon (INSA-Lyon)
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module Matching
+
+import java.util.LinkedList
+
+local function data = {
+  let list = LinkedList()
+  list: add("foo@bar.com")
+  list: add("+33.6.11.22.33")
+  list: add("http://golo-lang.org/")
+  list: add("def foo = bar(_._) with :> T")
+  return list
+}
+
+local function what_it_could_be = |item| -> match {
+  when item: contains("@") then "an email?"
+  when item: startsWith("+33") then "a French phone number?"
+  when item: startsWith("http://") then "a website URL?"
+  otherwise "I have no clue, mate!"
+}
+
+function main = |args| {
+  foreach item in data() {
+    println(item + " => " + what_it_could_be(item))
+  }
+}
+

samples/Golo/max-int.golo

@@ -0,0 +1,24 @@
+# Copyright 2012-2014 Institut National des Sciences Appliquées de Lyon (INSA-Lyon)
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module samples.MaxInt
+
+local function max_int = {
+  return java.lang.Integer.MAX_VALUE()
+}
+
+function main = |args| {
+  println(max_int())
+}
+

samples/Golo/memoize.golo

@@ -0,0 +1,55 @@
+# Copyright 2012-2014 Institut National des Sciences Appliquées de Lyon (INSA-Lyon)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module samples.MemoizeDecorator
+
+import gololang.Decorators
+
+import java.lang.System
+
+let memo = memoizer()
+
+@memo
+function fib = |n| {
+  if n <= 1 {
+    return n
+  } else {
+    return fib(n - 1) + fib(n - 2)
+  }
+}
+
+@memo
+function foo = |n| -> n
+
+local function run = {
+  let start = System.currentTimeMillis()
+  let result = fib(40)
+  let duration = System.currentTimeMillis() - start
+  println(">>> fib(40) = " + result + " (took " + duration + "ms)")
+}
+
+local function run2 = {
+  let start = System.currentTimeMillis()
+  let result = foo(40)
+  let duration = System.currentTimeMillis() - start
+  println(">>> foo(40) = " + result + " (took " + duration + "ms)")
+}
+
+function main = |args| {
+  foreach i in range(0, 5) {
+    println("run " + i)
+    run()
+    run2()
+  }
+}

samples/Golo/null-safety.golo

@@ -0,0 +1,43 @@
+# Copyright 2012-2014 Institut National des Sciences Appliquées de Lyon (INSA-Lyon)
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module DealingWithNull
+
+import java.util
+
+function main = |args| {
+
+  # Data model
+  let contacts = map[
+    ["mrbean", map[
+      ["email", "bean@gmail.com"],
+      ["url", "http://mrbean.com"]
+    ]],
+    ["larry", map[
+      ["email", "larry@iamricherthanyou.com"]
+    ]]
+  ]
+
+  # MrBean and Larry
+  let mrbean = contacts: get("mrbean")
+  let larry = contacts: get("larry")
+  
+  # Illustrates orIfNull
+  println(mrbean: get("url") orIfNull "n/a")
+  println(larry: get("url") orIfNull "n/a")
+
+  # Querying a non-existent data model because there is no 'address' entry
+  println(mrbean: get("address")?: street()?: number() orIfNull "n/a")
+}
+

samples/Golo/prepost-decorators.golo

@@ -0,0 +1,65 @@
+# Copyright 2012-2014 Institut National des Sciences Appliquées de Lyon (INSA-Lyon)
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module samples.PrepostDecorator
+
+import gololang.Decorators
+
+let isInteger = isOfType(Integer.class)
+
+@checkResult(isString(): andThen(lengthIs(2)))
+@checkArguments(isInteger: andThen(isPositive()), isString())
+function foo = |a, b| {
+    return b + a
+}
+
+let myCheck = checkArguments(isInteger: andThen(isPositive()))
+
+@myCheck
+function inv = |v| -> 1.0 / v
+
+let isPositiveInt = isInteger: andThen(isPositive())
+
+@checkArguments(isPositiveInt)
+function mul = |v| -> 10 * v
+
+@checkArguments(isNumber())
+function num = |v| -> "ok"
+
+@checkArguments(isNotNull())
+function notnull = |v| -> "ok"
+
+function main = |args| {
+    try { println(foo(1, "b")) } catch (e) { println(e) }
+    try { println(foo(-1, "b")) } catch (e) { println(e) }
+    try { println(foo("a", 2)) } catch (e) { println(e) }
+    try { println(foo(1, 2)) } catch (e) { println(e) }
+    try { println(foo(10, "ab")) } catch (e) { println(e) }
+
+    try { println(inv(10)) } catch (e) { println(e) }
+    try { println(inv(0)) } catch (e) { println(e) }
+
+    try { println(mul(5)) } catch (e) { println(e) }
+    try { println(mul(0)) } catch (e) { println(e) }
+
+    try { println(num(1)) } catch (e) { println(e) }
+    try { println(num(1_L)) } catch (e) { println(e) }
+    try { println(num(1.5)) } catch (e) { println(e) }
+    try { println(num(1.5_F)) } catch (e) { println(e) }
+    try { println(num("a")) } catch (e) { println(e) }
+    try { println(num('a')) } catch (e) { println(e) }
+
+    try { println(notnull('1')) } catch (e) { println(e) }
+    try { println(notnull(null)) } catch (e) { println(e) }
+}

samples/Golo/structs.golo

@@ -0,0 +1,69 @@
+# Copyright 2012-2014 Institut National des Sciences Appliquées de Lyon (INSA-Lyon)
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module StructDemo
+
+struct Point = { x, y }
+
+augment StructDemo.types.Point {
+
+  function move = |this, offsetX, offsetY| {
+    this: x(this: x() + offsetX)
+    this: y(this: y() + offsetY)
+    return this
+  }
+
+  function relative = |this, offsetX, offsetY| -> Point(this: x() + offsetX, this: y() + offsetY)
+}
+
+function main = |args| {
+  
+  let p1 = Point(1, 2)
+  let p2 = Point(): x(1): y(2)
+  let p3 = p1: frozenCopy()
+  let p4 = p1: frozenCopy()
+
+  println(p1)
+  println("x = " + p1: x())
+  println("y = " + p1: y())
+
+  println("p1 == p2 " + (p1 == p2))
+  println("p1 == p3 " + (p1 == p3))
+  println("p3 == p4 " + (p3 == p4))
+
+  println("#p1 " + p1: hashCode())
+  println("#p2 " + p2: hashCode())
+  println("#p3 " + p3: hashCode())
+  println("#p4 " + p4: hashCode())
+
+  println("p1: members() " + p1: members())
+  println("p1: values() " + p1: values())
+  foreach item in p1 {
+    println(item: get(0) + " -> " + item: get(1))
+  }
+
+  println("p1: set(\"x\", 10) " + p1: set("x", 10))
+  println("p1: move(10, 5) " + p1: move(10, 5))
+  println("p1: relative(11, 6) " + p1: relative(11, 6))
+
+  let p5 = ImmutablePoint(10, 20)
+  println("p5: " + p5)
+  try {
+    p5: x(100)
+  } catch (expected) {
+    println("p5 is immutable, so... " + expected: getMessage())
+  }
+}
+
+

samples/Golo/swing-actionlistener.golo

@@ -0,0 +1,43 @@
+# Copyright 2012-2014 Institut National des Sciences Appliquées de Lyon (INSA-Lyon)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module samples.SwingActionListener
+
+import java.awt.event
+import javax.swing
+import javax.swing.WindowConstants
+
+local function listener = |handler| -> asInterfaceInstance(ActionListener.class, handler)
+
+function main = |args| {
+
+  let frame = JFrame("Action listeners")
+  frame: setDefaultCloseOperation(EXIT_ON_CLOSE())
+
+  let button = JButton("Click me!")
+  button: setFont(button: getFont(): deriveFont(96.0_F))
+
+  # Using a helper function
+  button: addActionListener(listener(|event| -> println("Clicked!")))
+
+  # Using a standard augmentation: MethodHandle::to(Class)
+  button: addActionListener((|event| -> println("[click]")): to(ActionListener.class))
+
+  # Straight closure passing
+  button: addActionListener(|event| -> println("( )"))
+
+  frame: getContentPane(): add(button)
+  frame: pack()
+  frame: setVisible(true)
+}

samples/Golo/swing-helloworld.golo

@@ -0,0 +1,31 @@
+# Copyright 2012-2014 Institut National des Sciences Appliquées de Lyon (INSA-Lyon)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module samples.SwingHelloWorld
+
+import javax.swing
+import javax.swing.WindowConstants
+
+function main = |args| {
+
+  let frame = JFrame("Hello world")
+  frame: setDefaultCloseOperation(EXIT_ON_CLOSE())
+
+  let label = JLabel("Hello world")
+  label: setFont(label: getFont(): deriveFont(128.0_F))
+
+  frame: getContentPane(): add(label)
+  frame: pack()
+  frame: setVisible(true)
+}

samples/Golo/templates-chat-webapp.golo

@@ -0,0 +1,90 @@
+# Copyright 2012-2014 Institut National des Sciences Appliquées de Lyon (INSA-Lyon)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module samples.TemplatesChatWebapp
+
+import java.lang
+import java.io
+import java.net.InetSocketAddress
+import com.sun.net.httpserver
+import com.sun.net.httpserver.HttpServer
+
+local function redirect = |exchange, to| {
+  exchange: getResponseHeaders(): set("Location", to)
+  exchange: sendResponseHeaders(303, 0)
+  exchange: close()
+}
+
+local function respond = |exchange, body| {
+  exchange: getResponseHeaders(): set("Content-Type", "text/html")
+  exchange: sendResponseHeaders(200, body: length())
+  exchange: getResponseBody(): write(body: getBytes())
+  exchange: close()
+}
+
+# This is leaky and works with just 1 POST parameter...
+local function extract_post = |exchange, posts| {
+  let reader = BufferedReader(InputStreamReader(exchange: getRequestBody()))
+  var line = reader: readLine()
+  while line isnt null {
+    if line: startsWith("msg=") {
+      posts: add(java.net.URLDecoder.decode(line: substring(4), "UTF-8"))
+    }
+    line = reader: readLine()
+  }
+  reader: close()
+}
+
+
+local function index = |posts, template, exchange| {
+  if exchange: getRequestMethod() == "POST" {
+    extract_post(exchange, posts)
+    redirect(exchange, "/")
+  } else {
+    respond(exchange, template(posts))
+  }
+}
+
+local function index_template = -> """
+<%@params posts %>
+<!DOCTYPE html>
+<html>
+  <head>
+    <title>Golo Chat</title>
+  </head>
+  <body>
+  <form action="/" method="post">
+    <input type="text" name="msg">
+    <input type="submit" value="Send">
+  </form>
+  <div>
+    <h3>Last posts</h3>
+    <% foreach post in posts { %>
+      <div>
+        <%= post %>
+      </div>
+    <% } %>
+  </div>
+  </body>
+</html>
+"""
+
+function main = |args| {
+  let index_tpl = gololang.TemplateEngine(): compile(index_template())
+  let posts = java.util.concurrent.ConcurrentLinkedDeque()
+  let server = HttpServer.create(InetSocketAddress("localhost", 8081), 0)
+  server: createContext("/", ^index: bindTo(posts): bindTo(index_tpl))
+  server: start()
+  println(">>> http://localhost:8081/")
+}

samples/Golo/util-containers.golo

@@ -0,0 +1,51 @@
+# Copyright 2012-2014 Institut National des Sciences Appliquées de Lyon (INSA-Lyon)
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module MoreCoolContainers
+
+function main = |args| {
+  
+  println(">>> DynamicVariable")
+
+  let dyn = DynamicVariable("Foo")
+  println(dyn: value())
+  
+  let t1 = Thread({
+  dyn: withValue(666, {
+      println(dyn: value())
+    })
+  })
+  
+  let t2 = Thread({
+    dyn: withValue(69, {
+      println(dyn: value())
+    })
+  })
+
+  t1: start()
+  t2: start()
+  t1: join()
+  t2: join()
+  println(dyn: value())
+
+  println(">>> Observable")
+
+  let foo = Observable("Foo")
+  foo: onChange(|v| -> println("foo = " + v))
+
+  let mapped = foo: map(|v| -> v + "!")
+  mapped: onChange(|v| -> println("mapped = " + v))
+  
+  foo: set("69")
+}

samples/Golo/workers.golo

@@ -0,0 +1,48 @@
+# Copyright 2012-2014 Institut National des Sciences Appliquées de Lyon (INSA-Lyon)
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module Workers
+
+import java.lang.Thread
+import java.util.concurrent
+import gololang.concurrent.workers.WorkerEnvironment
+
+local function pusher = |queue, message| -> queue: offer(message)
+
+local function generator = |port, message| {
+  foreach i in range(0, 100) {
+    port: send(message)
+  }
+}
+
+function main = |args| {
+
+  let env = WorkerEnvironment.builder(): withFixedThreadPool()
+  let queue = ConcurrentLinkedQueue()
+
+  let pusherPort = env: spawn(^pusher: bindTo(queue))
+  let generatorPort = env: spawn(^generator: bindTo(pusherPort))
+
+  let finishPort = env: spawn(|any| -> env: shutdown())
+
+  foreach i in range(0, 10) {
+    generatorPort: send("[" + i + "]")
+  }
+  Thread.sleep(2000_L)
+  finishPort: send("Die!")
+
+  env: awaitTermination(2000)
+  println(queue: reduce("", |acc, next| -> acc + " " + next))
+}
+

samples/Graphviz (DOT)/annoying.DOT

@@ -0,0 +1,50 @@
+/* 
+  Huffman Tree DOT graph.
+
+  DOT Reference :  http://www.graphviz.org/doc/info/lang.html
+                   http://en.wikipedia.org/wiki/DOT_language
+  Timestamp     :  1415989074 
+  Phrase        :  'OH GOD WHY IS LINGUIST SO ANAL ABOUT THIS STUFF'
+
+  Generated on http://huffman.ooz.ie/
+*/
+
+digraph G {
+    edge [label=0];
+    graph [ranksep=0];
+    T [shape=record, label="{{T|4}|000}"];
+    S [shape=record, label="{{S|5}|001}"];
+    SPACE [shape=record, label="{{SPACE|9}|01}"];
+    A [shape=record, label="{{A|3}|1000}"];
+    H [shape=record, label="{{H|3}|1001}"];
+    U [shape=record, label="{{U|3}|1010}"];
+    L [shape=record, label="{{L|2}|10110}"];
+    N [shape=record, label="{{N|2}|10111}"];
+    I [shape=record, label="{{I|4}|1100}"];
+    O [shape=record, label="{{O|4}|1101}"];
+    G [shape=record, label="{{G|2}|11100}"];
+    F [shape=record, label="{{F|2}|11101}"];
+    GF [label=4];
+    W [shape=record, label="{{W|1}|111100}"];
+    Y [shape=record, label="{{Y|1}|111101}"];
+    B [shape=record, label="{{B|1}|111110}"];
+    D [shape=record, label="{{D|1}|111111}"];
+    BD [label=2];
+    WYBD [label=4];
+    GFWYBD [label=8];
+    47 -> 18 -> 9 -> T;
+    29 -> 13 -> 6 -> A;
+    7 -> U;
+    4 -> L;
+    16 -> 8 -> I;
+    GFWYBD -> GF -> G;
+    WYBD -> 2 -> W;
+    BD -> B;9 -> S [label=1];
+    18 -> SPACE [label=1];
+    6 -> H [label=1];
+    13 -> 7 -> 4 -> N [label=1];
+    8 -> O [label=1];
+    GF -> F [label=1];
+    2 -> Y [label=1];
+    47 -> 29 -> 16 -> GFWYBD -> WYBD -> BD -> D [label=1];
+}

samples/Graphviz (DOT)/sample.dot

@@ -0,0 +1,74 @@
+/* 
+  Huffman Tree DOT graph.
+
+  DOT Reference :  http://www.graphviz.org/doc/info/lang.html
+                   http://en.wikipedia.org/wiki/DOT_language
+  Timestamp     :  1415988139 
+  Phrase        :  'SERIAL KILLER AND SEX OFFENDER ANGUS SINCLAIR IS JAILED FOR A MINIMUM OF 37 YEARS FOR THE 1977 WORLDS END MURDERS OF HELEN SCOTT AND CHRISTINE EADIE.'
+
+  Generated on http://huffman.ooz.ie/
+*/
+
+digraph G {
+    edge [label=0];
+    graph [ranksep=0];
+    node [shape=record];
+    U [label="{{U|3}|00000}"];
+    G [label="{{G|1}|0000100}"];
+    K [label="{{K|1}|0000101}"];
+    _3 [label="{{3|1}|0000110}"];
+    _9 [label="{{9|1}|0000111}"];
+    _39 [label=2];
+    L [label="{{L|7}|0001}"];
+    O [label="{{O|7}|0010}"];
+    Y [label="{{Y|1}|0011000}"];
+    X [label="{{X|1}|0011001}"];
+    YX [label=2];
+    J [label="{{J|1}|0011010}"];
+    W [label="{{W|1}|0011011}"];
+    JW [label=2];
+    YXJW [label=4];
+    M [label="{{M|4}|00111}"];
+    E [label="{{E|15}|010}"];
+    D [label="{{D|8}|0110}"];
+    T [label="{{T|4}|01110}"];
+    DOT [label="{{DOT|1}|0111100}"];
+    _1 [label="{{1|1}|0111101}"];
+    DOT1 [label=2];
+    _7 [label="{{7|3}|011111}"];
+    A [label="{{A|9}|1000}"];
+    N [label="{{N|9}|1001}"];
+    S [label="{{S|10}|1010}"];
+    I [label="{{I|11}|1011}"];
+    R [label="{{R|11}|1100}"];
+    C [label="{{C|3}|110100}"];
+    H [label="{{H|3}|110101}"];
+    F [label="{{F|6}|11011}"];
+    SPACE [label="{{SPACE|26}|111}"];
+    149 -> 61 -> 29 -> 14 -> 7 -> U;
+    4 -> 2 -> G;
+    _39 -> _3;
+    15 -> O;
+    8 -> YXJW -> YX -> Y;
+    JW -> J;
+    32 -> E;
+    17 -> D;
+    9 -> T;
+    5 -> DOT1 -> DOT;
+    88 -> 39 -> 18 -> A;
+    21 -> S;
+    49 -> 23 -> R;
+    12 -> 6 -> C;2 -> K [label=1];
+    7 -> 4 -> _39 -> _9 [label=1];
+    14 -> L [label=1];
+    YX -> X [label=1];
+    YXJW -> JW -> W [label=1];
+    29 -> 15 -> 8 -> M [label=1];
+    DOT1 -> _1 [label=1];
+    61 -> 32 -> 17 -> 9 -> 5 -> _7 [label=1];
+    18 -> N [label=1];
+    39 -> 21 -> I [label=1];
+    6 -> H [label=1];
+    23 -> 12 -> F [label=1];
+    149 -> 88 -> 49 -> SPACE [label=1];
+}

samples/Hack/Assert.hh

@@ -0,0 +1,55 @@
+<?hh // strict
+/**
+ * Copyright (c) 2014, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree. An additional grant
+ * of patent rights can be found in the PATENTS file in the same directory.
+ *
+ */
+
+final class AssertException extends Exception {}
+
+final class Assert {
+  public static function isNum(mixed $x): num {
+    if (is_float($x)) {
+      return $x;
+    } else if (is_int($x)) {
+      return $x;
+    }
+    throw new AssertException('Expected an int or float value');
+  }
+
+  public static function isInt(mixed $x): int {
+    if (is_int($x)) {
+      return $x;
+    }
+    throw new AssertException('Expected an int');
+  }
+
+  public static function isFloat(mixed $x): float {
+    if (is_float($x)) {
+      return $x;
+    }
+    throw new AssertException('Expected a float');
+  }
+
+  public static function isString(mixed $x): string {
+    if (is_string($x)) {
+      return $x;
+    }
+    throw new AssertException('Expected a string');
+  }
+
+  // For arrays you need to check every element
+  public static function isArrayOf<T>(
+    (function(mixed): T) $fn,
+    mixed $x,
+  ): array<T> {
+    if (is_array($x)) {
+      return array_map($fn, $x);
+    }
+    throw new AssertException('Expected an array');
+  }
+}

samples/Hack/AssertRecipe.hh

@@ -0,0 +1,52 @@
+<?hh // strict
+/**
+ * Copyright (c) 2014, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree. An additional grant
+ * of patent rights can be found in the PATENTS file in the same directory.
+ *
+ */
+
+require_once $_SERVER['DOCUMENT_ROOT'].'/core/controller/recipe/init.php';
+require_once "demo.php";
+
+class AssertRecipe extends Recipe implements RecipeWithDemo {
+
+  protected function getName(): string {
+    return 'Assert';
+  }
+
+  <<Override>>
+  protected function getDescription(): ?string {
+    return 'When you have values with unknown types, it is useful to make '.
+      'some runtime assertions and have the type checker understand. This '.
+      'recipe demonstrates one approach.';
+  }
+
+  protected function getFilenames(): Vector<string> {
+    return Vector {
+      'Assert.php',
+    };
+  }
+
+  protected function getDocs(): Vector<(string, string)> {
+    return Vector{
+      tuple ('Mixed Types', 'hack.annotations.mixedtypes'),
+      tuple ('Type Inference', 'hack.otherrulesandfeatures.typeinference'),
+    };
+  }
+
+  public function getDemoFilename(): string {
+    return 'demo.php';
+  }
+
+  public function getDemoResult(): string {
+    return assert_main();
+  }
+
+  public function getDemoXHP(): ?:xhp {
+    return null;
+  }
+}

samples/Hack/Controller.hh

@@ -0,0 +1,39 @@
+<?hh // strict
+/**
+ * Copyright (c) 2014, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree. An additional grant
+ * of patent rights can be found in the PATENTS file in the same directory.
+ *
+ */
+
+require_once $_SERVER['DOCUMENT_ROOT'].'/core/startup/init.php';
+
+abstract class Controller {
+  protected function __construct() {
+    startup();
+  }
+
+  abstract protected function getCSS(): Set<string>;
+  abstract protected function getJS(): Set<string>;
+  abstract protected function getTitle(): string;
+  abstract protected function render(): :xhp;
+
+  final protected function getHead(): :xhp {
+    $css = $this->getCSS()->toVector()->map(
+      ($css) ==> <link rel="stylesheet" type="text/css" href={$css} />
+    );
+    $js = $this->getJS()->toVector()->map(
+      ($js) ==> <script src={$js} />
+    );
+    return
+      <head>
+      <meta http-equiv="content-type" content="text/html; charset=UTF-8"/>
+        <title>{$this->getTitle()}</title>
+        {$css->toArray()}
+        {$js->toArray()}
+      </head>;
+  }
+}

samples/Hack/DBResultRecipe.hh

@@ -0,0 +1,52 @@
+<?hh // strict
+/**
+ * Copyright (c) 2014, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree. An additional grant
+ * of patent rights can be found in the PATENTS file in the same directory.
+ *
+ */
+
+require_once $_SERVER['DOCUMENT_ROOT'].'/core/controller/recipe/init.php';
+require_once "demo.php";
+
+class DBResultRecipe extends Recipe implements RecipeWithDemo {
+
+  protected function getName(): string {
+    return 'DB Result';
+  }
+
+  <<Override>>
+  protected function getDescription(): ?string {
+    return 'Fetching data from a DB introduces a few typing challenges. '.
+      'First, the data comes back untyped. Second, a row in a DB generally '.
+      'contains columns of different types.';
+  }
+
+  protected function getFilenames(): Vector<string> {
+    return Vector {
+      'FakeDB.php',
+    };
+  }
+
+  protected function getDocs(): Vector<(string, string)> {
+    return Vector{
+      tuple ('Hack Shapes', 'hack.shapes'),
+      tuple ('Mixed Types', 'hack.annotations.mixedtypes'),
+    };
+  }
+
+  public function getDemoFilename(): string {
+    return 'demo.php';
+  }
+
+  public function getDemoResult(): string {
+    return db_result_main();
+  }
+
+  public function getDemoXHP(): ?:xhp {
+    return null;
+  }
+}

samples/Hack/Documentation.hh

@@ -0,0 +1,22 @@
+<?hh // strict
+/**
+ * Copyright (c) 2014, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree. An additional grant
+ * of patent rights can be found in the PATENTS file in the same directory.
+ *
+ */
+
+require_once $_SERVER['DOCUMENT_ROOT'].'/vendor/hhvm/xhp/src/init.php';
+
+final class :documentation extends :x:element {
+  attribute string name;
+
+  protected function render(): :xhp {
+    $name = implode('.', explode(' ', $this->getAttribute('name'))).".php";
+    $href = "http://hhvm.com/manual/en/$name";
+    return <a class="docs button" href={$href} target="_blank">docs &rarr;</a>;
+  }
+}

samples/Hack/FakeDB.hh

@@ -0,0 +1,65 @@
+<?hh // strict
+
+type DBResultExtra = shape('age' => int);
+type DBResult = shape(
+  'id' => int,
+  'name' => string,
+  'extra' => DBResultExtra,
+);
+
+final class FakeDB {
+  public function getRawRows(): array<array<string, mixed>> {
+    $good_extra = json_encode(array('age' => 40));
+    $bad_extra = 'corrupt data';
+    // Real code would query a DB, but for now let's hardcode it
+    return array(
+      array(
+        'id' => 123,
+        'name' => 'Alice',
+        'extra' => $good_extra,
+      ),
+      array(
+        'id' => 456,
+        'name' => 'Bob',
+        'extra' => $bad_extra,
+      ),
+    );
+  }
+
+  /**
+   * When processing untyped data you need to check each piece of data and
+   * figure out whether to give up or recover when the data is bad
+   */
+  public function processRow(array<string, mixed> $row): ?DBResult {
+    $row = Map::fromArray($row);
+    $id = $row->contains('id') ? $row['id'] : null;
+    $name = $row->contains('name') ? $row['name'] : null;
+    $extra = $row->contains('extra') ? json_decode($row['extra'], true) : null;
+
+    // Ignore rows with invalid IDs or names
+    if (!is_int($id) || !is_string($name)) {
+      return null;
+    }
+
+    // Try to recover from a bad extra column
+    if (!is_array($extra)) {
+      $extra = shape('age' => 0);
+    } else {
+      $extra = Map::fromArray($extra);
+      $extra = shape('age' => $extra->contains('age') ? $extra['age'] : 0);
+    }
+
+    return shape('id' => $id, 'name' => $name, 'extra' => $extra);
+  }
+
+  public function getDBResults(): Vector<DBResult> {
+    $ret = Vector {};
+    foreach ($this->getRawRows() as $raw_row) {
+      $row = $this->processRow($raw_row);
+      if ($row !== null) {
+        $ret->add($row);
+      }
+    }
+    return $ret;
+  }
+}

samples/Hack/GetAndPostRecipe.hh

@@ -0,0 +1,72 @@
+<?hh // strict
+/**
+ * Copyright (c) 2014, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree. An additional grant
+ * of patent rights can be found in the PATENTS file in the same directory.
+ *
+ */
+
+require_once $_SERVER['DOCUMENT_ROOT'].'/core/controller/recipe/init.php';
+require_once "demo.php";
+
+class GetAndPostRecipe extends Recipe implements RecipeWithDemo {
+
+  protected function getName(): string {
+    return '$_GET and $_POST';
+  }
+
+  <<Override>>
+  protected function getDescription(): ?string {
+    return 'A small example of how to interact with superglobals and the '.
+      'untyped data they can contain.';
+  }
+
+  protected function getFilenames(): Vector<string> {
+    return Vector {
+      'NonStrictFile.php',
+      'StrictFile.php',
+    };
+  }
+
+  protected function getDocs(): Vector<(string, string)> {
+    return Vector {
+      tuple('invariant()', 'hack.otherrulesandfeatures.invariant'),
+    };
+  }
+
+  public function getDemoFilename(): string {
+    return 'demo.php';
+  }
+
+  public function getDemoResult(): string {
+    return get_and_post_main();
+  }
+
+  public function getDemoXHP(): :xhp {
+    $url = '/recipes/get-and-post/';
+    return
+      <x:frag>
+        <div>
+          <a href={"$url?myIntParam=8675309#demo"} class="button">GET myIntParam=8675309</a>
+        </div>
+        <div>
+        <a href={"$url?myIntParam=boom#demo"} class="button">GET myIntParam=boom</a>
+        </div>
+        <div>
+        <form action={"$url#demo"} method="post">
+          <input type="hidden" name="myIntParam" value="5551234"/>
+          <input type="submit" class="button" value="POST myIntParam=5551234"/>
+        </form>
+        </div>
+        <div>
+        <form action={"$url#demo"} method="post">
+          <input type="hidden" name="myIntParam" value="boom"/>
+          <input type="submit" class="button" value="POST myIntParam=boom"/>
+        </form>
+        </div>
+      </x:frag>;
+  }
+}

samples/Hack/GetController.hh

@@ -0,0 +1,30 @@
+<?hh // strict
+/**
+ * Copyright (c) 2014, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree. An additional grant
+ * of patent rights can be found in the PATENTS file in the same directory.
+ *
+ */
+
+abstract class GetController extends Controller {
+  final protected function __construct(private Request $request) {
+    parent::__construct();
+  }
+
+  final protected function getRequest(): Request {
+    return $this->request;
+  }
+
+  final public function go(array<mixed, mixed> $get): void {
+    $request = new Request(Map::fromArray($get));
+    $controller = new static($request);
+    echo "<!DOCTYPE html>";
+    $head = $controller->getHead();
+    $body = $controller->render();
+    echo (string)$head;
+    echo (string)$body;
+  }
+}

samples/Hack/HomeController.hh

@@ -0,0 +1,38 @@
+<?hh // strict
+/**
+ * Copyright (c) 2014, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree. An additional grant
+ * of patent rights can be found in the PATENTS file in the same directory.
+ *
+ */
+
+require_once $_SERVER['DOCUMENT_ROOT'].'/core/controller/init.php';
+require_once $_SERVER['DOCUMENT_ROOT'].'/core/controller/standard-page/init.php';
+require_once $_SERVER['DOCUMENT_ROOT'].'/vendor/hhvm/xhp/src/init.php';
+
+class HomeController extends GetController {
+  use StandardPage;
+
+  protected function getTitle(): string {
+    return 'Hack Cookbook';
+  }
+
+  protected function renderMainColumn(): :xhp {
+    return <div>
+      <h1>Cookbook</h1>
+      <p>
+        The Hack Cookbook helps you write Hack code by giving you examples of
+        Hack code. It is written in Hack and is open source. If you
+        <a href="http://github.com/facebook/hack-example-site">
+          head over to GitHub,
+        </a>
+        you can read the code, check out the repository, and run it
+        yourself. The recipes in this cookbook are small examples that
+        illustrate how to use Hack to solve common and interesting problems.
+      </p>
+    </div>;
+  }
+}

samples/Hack/MySecureRequest.hh

@@ -0,0 +1,13 @@
+<?hh // strict
+
+require_once $_SERVER['DOCUMENT_ROOT'].'/core/funs/init.php';
+
+final class MySecureRequest {
+  public function __construct(private Map<string, mixed> $GETParams) {}
+  public function stringParam(string $name): UNESCAPED_STRING {
+    invariant($this->GETParams->contains($name), 'Unknown GET param: '.$name);
+    $raw_string = $this->GETParams[$name];
+    invariant(is_string($raw_string), $name.' is not a string');
+    return unescaped_string($raw_string);
+  }
+}

samples/Hack/Nav.hh

@@ -0,0 +1,104 @@
+<?hh // strict
+/**
+ * Copyright (c) 2014, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree. An additional grant
+ * of patent rights can be found in the PATENTS file in the same directory.
+ *
+ */
+
+require_once $_SERVER['DOCUMENT_ROOT'].'/vendor/hhvm/xhp/src/init.php';
+
+type NavItem = shape(
+  'name' => string,
+  'location' => string,
+);
+
+type NavSection = shape(
+  'name' => string,
+  'location' => ?string,
+  'items' => Vector<NavItem>,
+);
+
+final class :hack:nav extends :x:element {
+  private function getNavSections(): Vector<NavSection> {
+    return Vector{
+      shape(
+        'name' => 'Home',
+        'location' => '/',
+        'items' => Vector {},
+      ),
+      shape(
+        'name' => 'GitHub',
+        'location' => 'http://github.com/facebook/hack-example-site',
+        'items' => Vector {},
+      ),
+      shape(
+        'name' => 'Recipes',
+        'location' => null,
+        'items' => Vector {
+          shape(
+            'name' => '$_GET and $_POST',
+            'location' => '/recipes/get-and-post/',
+          ),
+          shape(
+            'name' => 'Assert',
+            'location' => '/recipes/assert/',
+          ),
+          shape(
+            'name' => 'DB Result',
+            'location' => '/recipes/db-result/',
+          ),
+          shape(
+            'name' => 'Unescaped String',
+            'location' => '/recipes/unescaped-string/',
+          ),
+          shape(
+            'name' => 'User ID',
+            'location' => '/recipes/user-id/',
+          ),
+        },
+      ),
+    };
+  }
+
+  private function renderNavItems(Vector<NavItem> $items): :xhp {
+    $render_item = $item ==>
+      <li>
+        <a class="navItem" href={$item['location']}>
+          {$item['name']}
+        </a>
+      </li>;
+    return
+      <x:frag>
+        {$items->map($render_item)->toArray()}
+      </x:frag>;
+  }
+
+  private function renderNavSection(NavSection $section): :xhp {
+    $section_item = <h3 class="navItem">{$section['name']}</h3>;
+    if ($section['location'] !== null) {
+      $section_item = <a href={$section['location']}>{$section_item}</a>;
+    }
+    return
+      <li class="navSectionItem">
+        {$section_item}
+        <ul class="navItems">
+          {$this->renderNavItems($section['items'])}
+        </ul>
+      </li>;
+  }
+
+  public function render(): :xhp {
+    $sections = $this->getNavSections()
+      ->map($section ==> $this->renderNavSection($section));
+    return
+      <div class="nav">
+        <ul class="navSections">
+          {$sections->toArray()}
+        </ul>
+      </div>;
+  }
+}

samples/Hack/NonStrictFile.hh

@@ -0,0 +1,27 @@
+<?hh
+/**
+ * Copyright (c) 2014, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree. An additional grant
+ * of patent rights can be found in the PATENTS file in the same directory.
+ *
+ */
+
+function getGETParams(): Map<string, mixed> {
+  // $_GET is not defined in code so Hack doesn't know about it and you can't
+  // use it in strict mode. You can interact with it outside of strict mode,
+  // though.
+  return Map::fromArray($_GET);
+}
+
+function getPOSTParams(): Map<string, mixed> {
+  // Same deal with $_POST and other magically defined globals
+  return Map::fromArray($_POST);
+}
+
+// Same deal with $_SERVER
+function isGET(): bool {
+  return $_SERVER['REQUEST_METHOD'] === 'GET';
+}

samples/Hack/Recipe.hh

@@ -0,0 +1,93 @@
+<?hh // strict
+/**
+ * Copyright (c) 2014, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree. An additional grant
+ * of patent rights can be found in the PATENTS file in the same directory.
+ *
+ */
+
+require_once $_SERVER['DOCUMENT_ROOT'].'/core/controller/init.php';
+require_once $_SERVER['DOCUMENT_ROOT'].'/core/controller/standard-page/init.php';
+require_once $_SERVER['DOCUMENT_ROOT'].'/core/myxhp/init.php';
+
+abstract class Recipe extends GetController {
+  use StandardPage;
+
+  abstract protected function getName(): string;
+  abstract protected function getFilenames(): Vector<string>;
+  abstract protected function getDocs(): Vector<(string, string)>;
+
+  protected function getDescription(): ?string {
+    return null;
+  }
+
+  final protected function getTitle(): string {
+    return $this->getName().' - Hack Cookbook';
+  }
+
+  final protected function renderMainColumn(): :xhp {
+    $main_column =
+      <x:frag>
+        <h1>{$this->getName()}</h1>
+      </x:frag>;
+    $description = $this->getDescription();
+    if ($description !== null) {
+      $main_column->appendChild(<p>{$description}</p>);
+    }
+    foreach ($this->getFilenames() as $filename) {
+      $file =
+        <div class="file">
+          <div class="filename">{$filename}</div>
+          <phpfile filename={$filename}/>
+        </div>;
+      $main_column->appendChild($file);
+    }
+    $recipe = $this;
+    if ($recipe instanceof RecipeWithDemo) {
+      try {
+        $result = $recipe->getDemoResult();
+      } catch (Exception $e) {
+        $result = sprintf(
+          "Demo threw an %s:\n%s",
+          get_class($e),
+          $e->getMessage(),
+        );
+      }
+      $result = explode("\n", trim($result));
+      $result = array_map($x ==> <x:frag>{$x}<br/></x:frag>, $result);
+      $demo =
+        <x:frag>
+          <div class="demo" id="demo">
+            <h3>Demo</h3>
+            {$recipe->getDemoXHP()}
+            <div class="filename">{$recipe->getDemoFilename()}</div>
+            <phpfile filename={$recipe->getDemoFilename()}/>
+            <div class="filename">Output</div>
+            <div class="demoResult">
+              {$result}
+            </div>
+          </div>
+        </x:frag>;
+      $main_column->appendChild($demo);
+    }
+    if (!$this->getDocs()->isEmpty()) {
+      $render_doc_link = function($doc) {
+        list($name, $link) = $doc;
+        $link = "http://hhvm.com/manual/en/$link.php";
+        return <li><a href={$link}>{$name}</a></li>;
+      };
+      $main_column->appendChild(
+        <div class="docs">
+          <h3>Relevant Official Documentation</h3>
+          <ul>
+            {$this->getDocs()->map($render_doc_link)->toArray()}
+          </ul>
+        </div>
+      );
+    }
+    return $main_column;
+  }
+}

samples/Hack/RecipeWithDemo.hh

@@ -0,0 +1,16 @@
+<?hh // strict
+/**
+ * Copyright (c) 2014, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree. An additional grant
+ * of patent rights can be found in the PATENTS file in the same directory.
+ *
+ */
+
+interface RecipeWithDemo {
+  public function getDemoFilename(): string;
+  public function getDemoResult(): string;
+  public function getDemoXHP(): ?:xhp;
+}

samples/Hack/Request.hh

@@ -0,0 +1,15 @@
+<?hh // strict
+/**
+ * Copyright (c) 2014, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree. An additional grant
+ * of patent rights can be found in the PATENTS file in the same directory.
+ *
+ */
+
+final class Request {
+  public function __construct(private Map<string, mixed> $params) {}
+}
+

samples/Hack/StandardPage.hh

@@ -0,0 +1,81 @@
+<?hh // strict
+/**
+ * Copyright (c) 2014, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree. An additional grant
+ * of patent rights can be found in the PATENTS file in the same directory.
+ *
+ */
+
+trait StandardPage {
+  require extends Controller;
+
+  abstract protected function renderMainColumn(): :xhp;
+
+  protected function getExtraCSS(): Set<string> {
+    return Set {};
+  }
+
+  protected function getExtraJS(): Set<string> {
+    return Set {};
+  }
+
+  final protected function getCSS(): Set<string> {
+    return (Set {
+      '/css/base.css',
+    })->addAll($this->getExtraCSS());
+  }
+
+  final protected function getJS(): Set<string> {
+    return (Set {
+    })->addAll($this->getExtraJS());
+  }
+
+  final private function renderHeader(): :xhp {
+    return
+      <div class="hackHeader">
+        <div class="width">
+          <a href="http://hacklang.org/">
+            <div class="logo">Hack</div>
+          </a>
+          <div class="headerNav">
+          <ul>
+            <li>
+              <a href="http://hacklang.org/install/">Install</a>
+            </li>
+            <li>
+              <a href="http://hacklang.org/tutorial/">Tutorial</a>
+            </li>
+            <li>
+              <a href="/">Cookbook</a>
+            </li>
+            <li>
+              <a href="http://hhvm.com/manual">Docs</a>
+            </li>
+            <li>
+              <a href="http://github.com/facebook/hhvm">GitHub</a>
+            </li>
+            <li>
+              <a href="http://hhvm.com/">HHVM</a>
+            </li>
+          </ul>
+          </div>
+        </div>
+      </div>;
+  }
+
+  final protected function render(): :xhp {
+    return
+      <div>
+        {$this->renderHeader()}
+        <div class="width">
+          <div class="mainContainer">
+            <div class="mainColumn">{$this->renderMainColumn()}</div>
+            <hack:nav/>
+          </div>
+        </div>
+    </div>;
+  }
+}

samples/Hack/StrictFile.hh

@@ -0,0 +1,46 @@
+<?hh // strict
+/**
+ * Copyright (c) 2014, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree. An additional grant
+ * of patent rights can be found in the PATENTS file in the same directory.
+ *
+ */
+
+require_once $_SERVER['DOCUMENT_ROOT'].'/core/funs/init.php';
+
+abstract class MyRequest {
+  abstract public function getParams(): Map<string, mixed>;
+
+  // Throws when things go wrong
+  public function intParamX(string $name): int {
+    $params = $this->getParams();
+    invariant($params->contains($name), sprintf('Unknown param: %s', $name));
+    $param = $params[$name];
+    invariant(is_numeric($param), sprintf('Param %s is not an int', $name));
+    return (int)$param;
+  }
+
+  // A lenient version
+  public function intParam(string $name): ?int {
+    $params = $this->getParams();
+    if (!$params->contains($name)) { return null; }
+    $param = $params[$name];
+    if (!is_numeric($param)) { return null; }
+    return (int)$param;
+  }
+}
+
+final class MyGETRequest extends MyRequest {
+  public function getParams(): Map<string, mixed> {
+    return getGETParams();
+  }
+}
+
+final class MyPOSTRequest extends MyRequest {
+  public function getParams(): Map<string, mixed> {
+    return getPOSTParams();
+  }
+}

samples/Hack/UnescapedString.hh

@@ -0,0 +1,16 @@
+<?hh // strict
+
+// Outside of this file, no one knows that UNESCAPED_STRING is a string
+newtype UNESCAPED_STRING = string;
+
+// This is how we initially taint a string.
+function unescaped_string(string $s): UNESCAPED_STRING {
+  return $s;
+}
+
+// This is the only thing you can do with an UNESCAPED_STRING (other than pass
+// it around)
+function escape_unescaped_string(UNESCAPED_STRING $s): string {
+  // Your use case will decide how you want to escape your strings
+  return sprintf('Escaped ---> "%s" <--- Escaped', $s);
+}

samples/Hack/UnescapedStringRecipe.hh

@@ -0,0 +1,59 @@
+<?hh // strict
+/**
+ * Copyright (c) 2014, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree. An additional grant
+ * of patent rights can be found in the PATENTS file in the same directory.
+ *
+ */
+
+require_once $_SERVER['DOCUMENT_ROOT'].'/core/controller/recipe/init.php';
+require_once "demo.php";
+
+class UnescapedStringRecipe extends Recipe implements RecipeWithDemo {
+
+  protected function getName(): string {
+    return 'Unescaped string';
+  }
+
+  <<Override>>
+  protected function getDescription(): ?string {
+    return 'Forgetting to properly escape the strings you get from your users '.
+      'can lead to serious security holes. Hack can help by forcing you to '.
+      'escape these strings before using them as strings.';
+  }
+
+  protected function getFilenames(): Vector<string> {
+    return Vector {
+      'UnescapedString.php',
+      'MySecureRequest.php',
+    };
+  }
+
+  protected function getDocs(): Vector<(string, string)> {
+    return Vector{
+      tuple('Opaque Type Aliasing',  'hack.typealiasing.opaquetypealiasing'),
+    };
+  }
+
+  public function getDemoFilename(): string {
+    return 'demo.php';
+  }
+
+  public function getDemoResult(): string {
+    return unescaped_string_main();
+  }
+
+  public function getDemoXHP(): ?:xhp {
+    $url = '/recipes/unescaped-string/';
+    return
+      <x:frag>
+        Try setting the myStrParam GET param to something nice and innocent with this button...
+        <div>
+          <a href={"$url?myStrParam='); DROP TABLE important_stuff; --#demo"} class="button">GET myStrParam=Hello world</a>
+        </div>
+      </x:frag>;
+  }
+}

samples/Hack/UserID.hh

@@ -0,0 +1,33 @@
+<?hh // strict
+/**
+ * Copyright (c) 2014, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree. An additional grant
+ * of patent rights can be found in the PATENTS file in the same directory.
+ *
+ */
+require_once $_SERVER['DOCUMENT_ROOT'].'/core/funs/init.php';
+
+// Outside of this file, no one knows that these types are ints. They do know
+// that USER_ID is an ID and COW_ID is an ID
+newtype ID = int;
+newtype USER_ID as ID = ID;
+newtype COW_ID as ID = ID;
+
+function assert_user_id(int $x): USER_ID {
+  // Everyone knows all user ids are odd
+  invariant($x % 2, sprintf('Invalid user ID: %d', $x));
+  return $x;
+}
+
+function assert_cow_id(int $x): COW_ID {
+  // Everyone knows all cow ids are even
+  invariant($x % 2 === 0, sprintf('Invalid cow ID: %d', $x));
+  return $x;
+}
+
+function id_to_int(ID $id): int {
+  return $id;
+}

samples/Hack/UserIDRecipe.hh

@@ -0,0 +1,54 @@
+<?hh // strict
+/**
+ * Copyright (c) 2014, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree. An additional grant
+ * of patent rights can be found in the PATENTS file in the same directory.
+ *
+ */
+
+require_once $_SERVER['DOCUMENT_ROOT'].'/core/controller/recipe/init.php';
+require_once "demo.php";
+
+class UserIDRecipe extends Recipe implements RecipeWithDemo {
+
+  protected function getName(): string {
+    return 'User ID';
+  }
+
+  <<Override>>
+  protected function getDescription(): ?string {
+    return 'Protect your user IDs from being confused with normal ints';
+  }
+
+  protected function getFilenames(): Vector<string> {
+    return Vector {
+      'UserID.php',
+      'UsingUserID.php',
+    };
+  }
+
+  protected function getDocs(): Vector<(string, string)> {
+    return Vector {
+      tuple('Opaque Type Aliasing',  'hack.typealiasing.opaquetypealiasing'),
+      tuple(
+        'Opaque Type Aliasing with Constraints',
+        'hack.typealiasing.opaquewithconstraints',
+      ),
+    };
+  }
+
+  public function getDemoFilename(): string {
+    return 'demo.php';
+  }
+
+  public function getDemoResult(): string {
+    return user_id_main();
+  }
+
+  public function getDemoXHP(): ?:xhp {
+    return null;
+  }
+}

samples/Hack/UsingUserID.hh

@@ -0,0 +1,22 @@
+<?hh // strict
+/**
+ * Copyright (c) 2014, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree. An additional grant
+ * of patent rights can be found in the PATENTS file in the same directory.
+ *
+ */
+
+function get_something_string(ID $id, string $something): string {
+  return sprintf("Awesome %s #%d\n", $something, id_to_int($id));
+}
+
+function get_user_string(USER_ID $id): string {
+  return get_something_string($id, 'user');
+}
+
+function get_cow_string(COW_ID $id): string {
+  return get_something_string($id, 'cow');
+}

samples/Hack/error.hh

@@ -0,0 +1,43 @@
+<?hh
+/**
+ * Copyright (c) 2014, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree. An additional grant
+ * of patent rights can be found in the PATENTS file in the same directory.
+ *
+ */
+
+final class TypehintViolationException extends Exception {
+}
+
+function setup_errors(): void {
+  set_error_handler('handle_error', E_ALL);
+}
+
+/**
+ * I want to turn failed typehints into exceptions so that I can handle them in
+ * my example code
+ */
+function handle_error(
+  $errno,
+  $errstr,
+  $errfile,
+  $errline,
+  $errcontext = array(),
+  $errtrace = array(),
+): bool {
+  if (E_RECOVERABLE_ERROR == $errno) {
+    // Transform typehint failures into an exception.
+    if (strpos($errstr, 'must be an instance of ') !== false) {
+      throw new TypehintViolationException($errstr);
+    }
+    // Transform nullable type violations to exceptions.
+    if ((strpos($errstr, 'must be of type ?') !== false) &&
+        (strpos($errstr, 'Value returned from') === false)) {
+      throw new TypehintViolationException($errstr);
+    }
+  }
+  return false;
+}

samples/Hack/funs.hh

@@ -0,0 +1,32 @@
+<?hh
+/**
+ * Copyright (c) 2014, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree. An additional grant
+ * of patent rights can be found in the PATENTS file in the same directory.
+ *
+ */
+
+/**
+ * This file contains a bunch of php stubs for functions that have been added
+ * to hhvm (though aren't in a release yet). These are important because the
+ * Hack typechecker can understand them
+ */
+
+class InvariantViolationException extends Exception {}
+
+function invariant(mixed $test, string $message): void {
+  if (!$test) {
+    invariant_violation($message);
+  }
+}
+
+function invariant_violation(string $message): void {
+  throw new InvariantViolationException($message);
+}
+
+function class_meth(string $class, string $method) {
+  return array($class, $method);
+}

samples/Hack/funs.php

@@ -0,0 +1,32 @@
+<?hh
+/**
+ * Copyright (c) 2014, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree. An additional grant
+ * of patent rights can be found in the PATENTS file in the same directory.
+ *
+ */
+
+/**
+ * This file contains a bunch of php stubs for functions that have been added
+ * to hhvm (though aren't in a release yet). These are important because the
+ * Hack typechecker can understand them
+ */
+
+class InvariantViolationException extends Exception {}
+
+function invariant(mixed $test, string $message): void {
+  if (!$test) {
+    invariant_violation($message);
+  }
+}
+
+function invariant_violation(string $message): void {
+  throw new InvariantViolationException($message);
+}
+
+function class_meth(string $class, string $method) {
+  return array($class, $method);
+}

samples/Hack/index.hh

@@ -0,0 +1,14 @@
+<?hh
+/**
+ * Copyright (c) 2014, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree. An additional grant
+ * of patent rights can be found in the PATENTS file in the same directory.
+ *
+ */
+
+require_once 'HomeController.php';
+
+HomeController::go($_GET);

samples/Hack/phpfile.hh

@@ -0,0 +1,31 @@
+<?hh // strict
+/**
+ * Copyright (c) 2014, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree. An additional grant
+ * of patent rights can be found in the PATENTS file in the same directory.
+ *
+ */
+
+require_once $_SERVER['DOCUMENT_ROOT'].'/vendor/hhvm/xhp/src/init.php';
+
+final class :phpfile extends :x:primitive {
+  category %flow;
+
+  attribute string filename;
+
+  /**
+   * Ok, I'll admit this is kind of gross. I don't really want to implement
+   * syntax highlighting, so I'm relying on the built-in PHP support. XHP
+   * makes html strings sort of difficult to use (which is good cause they're
+   * dangerous). Anyway, this is one way around it :)
+   */
+  protected function stringify(): string {
+    return
+      '<div class="code">'.
+      (string)highlight_file($this->getAttribute('filename'), /*ret*/ true).
+      '</div>';
+  }
+}

samples/Hack/startup.hh

@@ -0,0 +1,14 @@
+<?hh // strict
+/**
+ * Copyright (c) 2014, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree. An additional grant
+ * of patent rights can be found in the PATENTS file in the same directory.
+ *
+ */
+
+function startup(): void {
+  setup_errors();
+}

samples/JavaScript/empty.js

@@ -1,3 +0,0 @@
-(function() {
-
-}).call(this);

samples/Maven POM/filenames/pom.xml

@@ -0,0 +1,207 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+	<modelVersion>4.0.0</modelVersion>
+	<groupId>renpengben</groupId>
+	<artifactId>spring4mvc-jpa</artifactId>
+	<packaging>war</packaging>
+	<version>0.0.1-SNAPSHOT</version>
+	<name>spring4mvc-jpa Maven Webapp</name>
+
+	<url>https://renpengben.github.io</url>
+
+	<description>spring4mvc-jpa</description>
+
+	<properties>
+		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+		<java.version>1.7</java.version>
+		<junit.version>4.11</junit.version>
+		<slf4j.version>1.7.7</slf4j.version>
+		<log4j.version>1.2.17</log4j.version>
+
+		<spring.version>4.0.5.RELEASE</spring.version>
+		<spring.data.jpa.version>1.6.0.RELEASE</spring.data.jpa.version>
+		<cglib.version>2.1_3</cglib.version>
+
+		<mysql.version>5.1.31</mysql.version>
+		<hibernate.version>4.3.5.Final</hibernate.version>
+		<hibernate-validator.version>5.1.1.Final</hibernate-validator.version>
+		<druid-version>1.0.6</druid-version>
+
+	</properties>
+
+
+	<dependencies>
+
+		<dependency>
+			<groupId>junit</groupId>
+			<artifactId>junit</artifactId>
+			<version>${junit.version}</version>
+			<scope>test</scope>
+		</dependency>
+
+		<dependency>
+			<groupId>org.slf4j</groupId>
+			<artifactId>slf4j-api</artifactId>
+			<version>${slf4j.version}</version>
+		</dependency>
+		<dependency>
+			<groupId>org.slf4j</groupId>
+			<artifactId>slf4j-log4j12</artifactId>
+			<version>${slf4j.version}</version>
+		</dependency>
+		<dependency>
+			<groupId>log4j</groupId>
+			<artifactId>log4j</artifactId>
+			<version>${log4j.version}</version>
+		</dependency>
+
+
+		<!-- Spring -->
+		<dependency>
+			<groupId>org.springframework</groupId>
+			<artifactId>spring-core</artifactId>
+			<version>${spring.version}</version>
+			<exclusions>
+				<exclusion>
+					<groupId>commons-logging</groupId>
+					<artifactId>commons-logging</artifactId>
+				</exclusion>
+			</exclusions>
+		</dependency>
+
+		<dependency>
+			<groupId>org.springframework</groupId>
+			<artifactId>spring-beans</artifactId>
+			<version>${spring.version}</version>
+		</dependency>
+		<dependency>
+			<groupId>org.springframework</groupId>
+			<artifactId>spring-context</artifactId>
+			<version>${spring.version}</version>
+		</dependency>
+
+		<dependency>
+			<groupId>org.springframework</groupId>
+			<artifactId>spring-aop</artifactId>
+			<version>${spring.version}</version>
+		</dependency>
+
+		<dependency>
+			<groupId>org.springframework</groupId>
+			<artifactId>spring-expression</artifactId>
+			<version>${spring.version}</version>
+		</dependency>
+		<dependency>
+			<groupId>org.springframework</groupId>
+			<artifactId>spring-tx</artifactId>
+			<version>${spring.version}</version>
+		</dependency>
+
+
+		<dependency>
+			<groupId>org.springframework</groupId>
+			<artifactId>spring-aspects</artifactId>
+			<version>${spring.version}</version>
+		</dependency>
+		<dependency>
+			<groupId>org.springframework</groupId>
+			<artifactId>spring-context-support</artifactId>
+			<version>${spring.version}</version>
+		</dependency>
+		<dependency>
+			<groupId>org.springframework</groupId>
+			<artifactId>spring-jdbc</artifactId>
+			<version>${spring.version}</version>
+		</dependency>
+		<dependency>
+			<groupId>org.springframework</groupId>
+			<artifactId>spring-orm</artifactId>
+			<version>${spring.version}</version>
+		</dependency>
+
+		<dependency>
+			<groupId>org.springframework</groupId>
+			<artifactId>spring-web</artifactId>
+			<version>${spring.version}</version>
+		</dependency>
+		<dependency>
+			<groupId>org.springframework</groupId>
+			<artifactId>spring-webmvc</artifactId>
+			<version>${spring.version}</version>
+		</dependency>
+		<dependency>
+			<groupId>org.springframework</groupId>
+			<artifactId>spring-test</artifactId>
+			<version>${spring.version}</version>
+			<scope>test</scope>
+		</dependency>
+
+		<dependency>
+			<groupId>org.springframework.data</groupId>
+			<artifactId>spring-data-jpa</artifactId>
+			<version>${spring.data.jpa.version}</version>
+			<exclusions>
+				<exclusion>
+					<artifactId>junit-dep</artifactId>
+					<groupId>junit</groupId>
+				</exclusion>
+			</exclusions>
+		</dependency>
+		<dependency>
+			<groupId>cglib</groupId>
+			<artifactId>cglib-nodep</artifactId>
+			<version>${cglib.version}</version>
+		</dependency>
+
+
+
+		<!-- JPA -->
+
+		<dependency>
+			<groupId>org.hibernate</groupId>
+			<artifactId>hibernate-core</artifactId>
+			<version>${hibernate.version}</version>
+		</dependency>
+		<dependency>
+			<groupId>org.hibernate</groupId>
+			<artifactId>hibernate-entitymanager</artifactId>
+			<version>${hibernate.version}</version>
+		</dependency>
+		<dependency>
+			<groupId>org.hibernate</groupId>
+			<artifactId>hibernate-validator</artifactId>
+			<version>${hibernate-validator.version}</version>
+			<scope>compile</scope>
+
+		</dependency>
+
+
+		<dependency>
+			<groupId>mysql</groupId>
+			<artifactId>mysql-connector-java</artifactId>
+			<version>${mysql.version}</version>
+			<scope>runtime</scope>
+		</dependency>
+		<dependency>
+			<groupId>com.alibaba</groupId>
+			<artifactId>druid</artifactId>
+			<version>${druid-version}</version>
+		</dependency>
+
+
+	</dependencies>
+	<build>
+		<plugins>
+			<plugin>
+				<groupId>org.apache.maven.plugins</groupId>
+				<artifactId>maven-compiler-plugin</artifactId>
+				<version>2.0.2</version>
+				<configuration>
+					<source>1.7</source>
+					<target>1.7</target>
+				</configuration>
+			</plugin>
+		</plugins>
+	</build>
+</project>

samples/Papyrus/CAMTEST_OverShoulderME.psc

@@ -0,0 +1,68 @@
+Scriptname CAMTEST_OverShoulderME extends activemagiceffect  
+{Play with camera effects}
+
+;--=== Imports ===--
+
+Import Utility
+Import Game
+
+;--=== Properties ===--
+
+Actor Property PlayerRef Auto
+ActorBase Property CAMTEST_CameraActor Auto
+
+;--=== Variables ===--
+
+Actor Player
+
+Actor Camera
+
+Actor Target
+
+Float PosX
+Float PosY
+Float PosZ
+Float SpeedMult
+
+ObjectReference Mist
+ObjectReference Fog
+
+;--=== Events ===--
+
+Event OnInit()
+	Player = PlayerRef
+EndEvent
+
+Event onEffectStart(Actor akTarget, Actor akCaster)
+	Camera = Player.PlaceActorAtMe(CAMTEST_CameraActor)
+	Camera.EnableAI(False)
+	Camera.SetScale(0.1)
+	Camera.TranslateTo(Player.X + 40,Player.Y,Player.Z,0,0,0,800,30)
+	DisablePlayerControls(abMovement = true, abFighting = true, abCamSwitch = true,  abLooking = true, abSneaking = true, abMenu = true, abActivate = true, abJournalTabs = false)
+	SetPlayerAIDriven(True)
+	ForceThirdPerson()
+	SetHUDCartMode()
+	SetInChargen(True, True, False)
+	SetCameraTarget(Camera)
+	ForceFirstPerson()
+	Wait(1)
+	Camera.SplineTranslateTo(Player.X + 4000,Player.Y,Player.Z + 1000,15,0,Camera.GetHeadingAngle(Player) + Camera.GetAngleZ(),1800,800,100)
+;	Camera.SetLookAt(Player)
+	Wait(10)
+	Camera.SplineTranslateTo(Player.X + 1000,Player.Y - 500,Player.Z + 500,25,0,Camera.GetHeadingAngle(Player) + Camera.GetAngleZ(),1800,800,100)
+	Wait(10)
+	SetHUDCartMode(False)
+	SetCameraTarget(Player)
+	SetInChargen(False, False, False)
+	EnablePlayerControls()
+	SetPlayerAIDriven(False)
+EndEvent
+
+Event onUpdate()
+EndEvent
+
+Event onEffectFinish(Actor akTarget, Actor akCaster)
+EndEvent
+
+;--=== Functions ===--
+

samples/Papyrus/vMFX_FXPlugin.psc

@@ -0,0 +1 @@
+Scriptname vMFX_FXPlugin extends Quest  

samples/Papyrus/vSCM_MetaQuestScript.psc

@@ -0,0 +1,120 @@
+Scriptname vSCM_MetaQuestScript extends Quest  
+{Do initialization and track variables for scripts}
+
+;--=== Imports ===--
+
+Import Utility
+Import Game
+
+;--=== Properties ===--
+
+Actor Property PlayerRef Auto
+
+Float Property ModVersion Auto Hidden
+
+String Property ModName = "Smarter Combat Music" Auto Hidden
+
+Message Property vSCM_ModLoadedMSG Auto
+Message Property vSCM_ModUpdatedMSG Auto
+
+;--=== Variables ===--
+
+Float _CurrentVersion
+String _sCurrentVersion
+
+Bool _Running
+
+Float _ScriptLatency
+Float _StartTime
+Float _EndTime
+
+;--=== Events ===--
+
+Event OnInit()
+	If ModVersion == 0
+		DoUpkeep(True)
+	EndIf
+EndEvent
+
+Event OnReset()
+	Debug.Trace("SCM: Metaquest event: OnReset")
+EndEvent
+
+Event OnGameReloaded()
+	Debug.Trace("SCM: Metaquest event: OnGameReloaded")
+EndEvent
+
+;--=== Functions ===--
+
+Function DoUpkeep(Bool DelayedStart = True)
+	;FIXME: CHANGE THIS WHEN UPDATING!
+	_CurrentVersion = 0.01
+	_sCurrentVersion = GetVersionString(_CurrentVersion)
+	String sErrorMessage
+	If DelayedStart
+		Wait(RandomFloat(2,4))
+	EndIf
+	Debug.Trace("SCM: " + ModName)
+	Debug.Trace("SCM: Performing upkeep...")
+	Debug.Trace("SCM: Loaded version is " + GetVersionString(ModVersion) + ", Current version is " + _sCurrentVersion)
+	If ModVersion == 0
+		Debug.Trace("SCM: Newly installed, doing initialization...")
+		DoInit()
+		If ModVersion == _CurrentVersion
+			Debug.Trace("SCM: Initialization succeeded.")
+		Else
+			Debug.Trace("SCM: WARNING! Initialization had a problem!")
+		EndIf
+	ElseIf ModVersion < _CurrentVersion
+		Debug.Trace("SCM: Installed version is older. Starting the upgrade...")
+		DoUpgrade()
+		If ModVersion != _CurrentVersion
+			Debug.Trace("SCM: WARNING! Upgrade failed!")
+			Debug.MessageBox("WARNING! " + ModName + " upgrade failed for some reason. You should report this to the mod author.")
+		EndIf
+		Debug.Trace("SCM: Upgraded to " + _CurrentVersion)
+		vSCM_ModUpdatedMSG.Show(_CurrentVersion)
+	Else
+		Debug.Trace("SCM: Loaded, no updates.")
+		;CheckForOrphans()
+	EndIf
+	CheckForExtras()
+	UpdateConfig()
+	Debug.Trace("SCM: Upkeep complete!")
+EndFunction
+
+Function DoInit()
+	Debug.Trace("SCM: Initializing...")
+	_Running = True
+	ModVersion = _CurrentVersion
+	vSCM_ModLoadedMSG.Show(_CurrentVersion)
+EndFunction
+
+Function DoUpgrade()
+	_Running = False
+	If ModVersion < 0.01
+		Debug.Trace("SCM: Upgrading to 0.01...")
+		ModVersion = 0.01
+	EndIf
+	_Running = True
+	Debug.Trace("SCM: Upgrade complete!")
+EndFunction
+
+Function UpdateConfig()
+	Debug.Trace("SCM: Updating configuration...")
+
+	Debug.Trace("SCM: Updated configuration values, some scripts may update in the background!")
+EndFunction
+
+String Function GetVersionString(Float fVersion)
+	Int Major = Math.Floor(fVersion) as Int
+	Int Minor = ((fVersion - (Major as Float)) * 100.0) as Int
+	If Minor < 10
+		Return Major + ".0" + Minor
+	Else
+		Return Major + "." + Minor
+	EndIf
+EndFunction
+
+Function CheckForExtras()
+EndFunction

samples/Ruby/filenames/Gemfile.lock

@@ -0,0 +1,42 @@
+PATH
+  remote: .
+  specs:
+    github-linguist (4.0.1)
+      charlock_holmes (~> 0.7.3)
+      escape_utils (~> 1.0.1)
+      mime-types (>= 1.19)
+      rugged (~> 0.22.0b1)
+    github-linguist-grammars (4.0.1)
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    charlock_holmes (0.7.3)
+    coderay (1.1.0)
+    escape_utils (1.0.1)
+    metaclass (0.0.4)
+    method_source (0.8.2)
+    mime-types (2.4.3)
+    mocha (1.1.0)
+      metaclass (~> 0.0.1)
+    plist (3.1.0)
+    pry (0.10.1)
+      coderay (~> 1.1.0)
+      method_source (~> 0.8.1)
+      slop (~> 3.4)
+    rake (10.3.2)
+    rugged (0.22.0b1)
+    slop (3.6.0)
+    yajl-ruby (1.2.1)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  github-linguist!
+  github-linguist-grammars!
+  mocha
+  plist (~> 3.1)
+  pry
+  rake
+  yajl-ruby

samples/SuperCollider/WarpPreset.sc

@@ -0,0 +1,19 @@
+WarpPreset {
+	*new {|path|
+		if(path.notNil) {
+			^Object.readArchive(path);
+		};
+
+		^super.new.init();
+	}
+
+	init {
+
+	}
+
+	save {
+		Dialog.savePanel({|path|
+			this.writeArchive(path);
+		});
+	}
+}

samples/SuperCollider/WarpTate.sc

@@ -0,0 +1,286 @@
+WarpTate {
+	classvar <numSections = 8;
+	// classvar <sectionDur = 3 * 60;
+	classvar <sectionDur = 60;
+
+	var <sensorKeys;
+	var <clock;
+	var <tempo;
+	var <>tempoChannel;
+	var <>tempoControl;
+	var <out;
+	var <tracks;
+	var <>sections;
+	var <availableControls;
+	var <controls;
+	var <sensorVals;
+	var <sensorPrevs;
+	var <sensorMins;
+	var <sensorMaxs;
+	var <>sensorMinAdj;
+	var <>sensorMaxAdj;
+	var <doAdjusts;
+	var <playRout;
+
+	*new {
+		^super.new.init;
+	}
+
+	init {
+		tempo = 120;
+		tempoChannel = 15;
+		tempoControl = 3;
+		clock = TempoClock.default
+			.tempo_(2)
+			.permanent_(true);
+
+		MIDIClient.init;
+		out = MIDIOut.newByName("IAC Driver", "Bus 1");
+		out.latency = 0;
+
+		tracks = IdentityDictionary[];
+		sections = Array.newClear(WarpTate.numSections);
+		// sections is a List of IdentityDictionary to be mapped to
+		// WarpTrack settings var
+		// e.g. List[IdentityDictionary[
+		//	 			'303_1' -> IdentityDictionary['notes' -> List[42]],
+		// 				'808_1'	-> IdentityDictionary['notes' -> List[24]]
+		// 			]
+		// 		]
+		sensorKeys = ['303a', '303b', '808a', '808b'];
+
+		// channel 16 reserved for tempo changes
+		availableControls = 15.collect {|channel|
+			(0..120).reject({|item, i|
+				[ 0, 1, 2, 4, 5, 6, 7, 8, 10, 11, 12, 13, 64, 65, 66, 67, 68,
+				69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 84, 91, 92, 93, 94,
+				95, 96, 97, 98, 99, 100, 102 ].includes(item)
+			});
+		};
+
+		controls = (0..127).collect { IdentityDictionary[] };
+
+		this.addOSCdefs();
+
+		CmdPeriod.add({this.stop});
+	}
+
+	tempo_ {|argTempo|
+		if(argTempo >= 50 && argTempo <= 177) {
+			tempo = argTempo;
+			clock.tempo = tempo / 60;
+
+			out.control(tempoChannel, tempoControl, tempo - 50);
+		} {
+			"Tempo out of Logic's range :(".postln;
+		};
+	}
+
+	addTrack {|trackKey, channel, type|
+		tracks[trackKey] = WarpTrack(this, trackKey, channel, type);
+		^tracks[trackKey];
+	}
+
+	loadTrack {|preset, checkAvailable|
+		var track = WarpTrack.load(this, preset, checkAvailable),
+			trackKey = track.settings['key'];
+
+		tracks[trackKey] = track;
+		^tracks[trackKey];
+	}
+
+	readTrack {|path|
+		var track = WarpTrack.read(this, path),
+			trackKey = track.settings['key'];
+
+		tracks[trackKey] = track;
+		^tracks[trackKey];
+	}
+
+	removeTrack {|trackKey|
+		tracks[trackKey].allOff();
+		tracks[trackKey] = nil;
+	}
+
+	removeAllTracks {
+		tracks = IdentityDictionary[];
+	}
+
+	on {|trackKey, note|
+		tracks[trackKey].on(note);
+	}
+
+	off {|trackKey, note|
+		tracks[trackKey].off(note);
+	}
+
+	hit {|trackKey, note=60, vel=127, dur=1|
+		tracks[trackKey].hit(note, vel, dur);
+	}
+
+	noteOn {|midiChannel, note, vel|
+		out.noteOn(midiChannel, note, vel);
+	}
+
+	noteOff {|midiChannel, note, vel|
+		out.noteOff(midiChannel, note, vel);
+	}
+
+	control {|midiChannel, num, val|
+		out.control(midiChannel, num, val);
+	}
+
+	isControlAvailable {|channel, controlNum|
+		^controls[channel].keys.includes(controlNum.asSymbol).not;
+	}
+
+	setControl {|channel, num, key|
+		controls[channel][num.asSymbol] = key;
+		availableControls[channel].remove(num);
+	}
+
+	assign {|trackKey, paramKey, controlNum, learn=false|
+		var channel = tracks[trackKey].settings['midiChannel'];
+
+		if(controlNum.notNil && this.isControlAvailable(channel, controlNum)) {
+			tracks[trackKey].assign(paramKey, controlNum, learn);
+			availableControls[channel].removeAt(0);
+
+		} {
+			if(availableControls[channel].size > 0) {
+				tracks[trackKey].assign(paramKey, availableControls[channel][0], learn);
+				availableControls[channel].removeAt(0);
+			} {
+				"no controls left!".postln;
+			};
+		};
+
+		"Don't forget to turn off MIDI learn".postln
+	}
+
+	setParam {|trackKey, paramKey, val|
+		var track, param;
+
+		if((track = tracks[trackKey]).notNil) {
+			if(track.params[paramKey].notNil) {
+				track.setParam(paramKey, val);
+			} {
+				"paramKey doesn't exist".postln;
+			};
+		} {
+			"track key doesn't exist".postln;
+		};
+	}
+
+	// sec takes IdentityDictionarys of WarpTrack settings
+	addSection {|index, tempo=120, presets|
+		// Add section with tempo
+		sections[index] = IdentityDictionary[
+			'tempo' 	-> tempo,
+			'tracks'	-> presets
+		];
+
+		presets.do {|preset, i|
+			// create track if there ain't one with this key
+			if(tracks.includesKey(preset['key']).not) {
+				this.loadTrack(preset, true);
+			};
+
+			// store the preset
+			// sections[index]['tracks'][preset['key']] = preset;
+		};
+
+		^sections;
+	}
+
+	play {
+
+		if(sections.any {|item, i| item.notNil; }) {
+			playRout = Routine {
+				inf.do {|i|
+					sections.do {|section|
+						if(section.notNil) {
+							if(i !== 0) {
+								tracks.do {|track|
+									track.allOff();
+								};
+								this.removeAllTracks();
+							};
+
+							this.tempo = section['tempo'];
+
+							section['tracks'].do {|track, i|
+								var newTrack = this.loadTrack(track, false);
+								newTrack.play();
+							};
+							sectionDur.wait;
+						};
+					};
+				}
+			};
+			clock.playNextBar(playRout);
+		} {
+			"no sections added!".postln;
+		};
+
+	}
+
+	stop {
+		playRout.stop;
+
+		tracks.do {|track|
+			track.allOff();
+			out.allNotesOff(track.settings['midiChannel']);
+		}
+	}
+
+	addOSCdefs {
+		sensorVals = 0!sensorKeys.size;
+		sensorPrevs = 0!sensorKeys.size;
+		sensorMins = 9999!sensorKeys.size;
+		sensorMaxs = 0!sensorKeys.size;
+		sensorMinAdj = sensorMinAdj ?? { 0.005 };
+		sensorMaxAdj = sensorMaxAdj ?? { 0.01 };
+		doAdjusts = false!sensorKeys.size;
+
+		sensorKeys.do {|sensorKey, i|
+			OSCdef(("sensor_" ++ sensorKey).asSymbol, {|msg, time, addr, recvPort|
+				var val = msg[1];
+
+
+				sensorPrevs[i] = sensorVals[i];
+				sensorVals[i] = val;
+
+				if(doAdjusts[i]) {
+					sensorMins[i] = min(val, sensorMins[i]);
+					sensorMaxs[i] = max(val, sensorMaxs[i]);
+
+					if(val < sensorMaxs[i]) {
+						sensorMaxs[i] = sensorMaxs[i] - sensorMaxAdj;
+					};
+
+					if(val > sensorMins[i]) {
+						sensorMins[i] = sensorMins[i] + sensorMinAdj;
+					};
+				} {
+					val = val.clip(sensorMins[i], sensorMaxs[i]);
+				};
+
+
+				tracks.do {|track, j|
+					if(track.settings['sensorFuncs'].includesKey(sensorKey)) {
+						track.sensor(
+							sensorKey,
+							val.linlin(
+								sensorMins[i],
+								sensorMaxs[i],
+								127,
+								0
+							)
+						);
+					};
+				}
+			}, ("/prox/" ++ sensorKey).asSymbol);
+		}
+	}
+}

samples/SuperCollider/WarpTrack.sc

@@ -0,0 +1,263 @@
+WarpTrack {
+	classvar <defaults;
+
+	var <parent;
+	var <settings;
+
+	*initClass {
+		defaults['303_1'] = defaults['303'].copy;
+		defaults['303_1']['paramControls'].putPairs([
+			'Bus 1', 46,
+			'Bus 2', 45,
+			'Bus 3', 48,
+			'echovol', 47
+		]);
+		defaults['303_1']['params'].putPairs([
+			'Bus 1', 0,
+			'Bus 2', 0,
+			'Bus 3', 0,
+			'echovol', 0
+		]);
+
+		defaults['303_2'] = defaults['303_1'].copy;
+		defaults['303_2']['paramControls'].putPairs([
+		]);
+		defaults['303_2']['params'].putPairs([
+		]);
+
+		defaults['808_1'] = defaults['808'].copy;
+		defaults['808_1']['paramControls'].putPairs([
+			'bitcrusher', 88,
+			'Send 1', 89,
+			'Send 2', 90,
+			'Send 3', 101
+		]);
+		defaults['808_1']['params'].putPairs([
+			'bitcrusher', 0,
+			'Send 1', 0,
+			'Send 2', 0,
+			'Send 3', 0
+		]);
+
+		defaults['808_2'] = defaults['808_1'].copy;
+		defaults['808_2']['paramControls'].putPairs([]);
+		defaults['808_2']['params'].putPairs([]);
+	}
+
+	*new {|argParent, argKey, argMidiChannel, argType|
+		^super.new.init(argParent, argKey, argMidiChannel, argType);
+	}
+
+	*read {|argParent, path|
+		^super.new.init(argParent).readPreset(path);
+	}
+
+	*load {|argParent, preset, checkAvailable|
+		^super.new.init(argParent).loadPreset(preset, checkAvailable);
+	}
+
+	init {|argParent, argKey, argMidiChannel, argType|
+		parent = argParent;
+
+		settings = IdentityDictionary[
+			'key'			-> argKey,
+			'midiChannel'	-> argMidiChannel,
+			'notes'			-> Set[],
+			'params'		-> IdentityDictionary[],
+			'paramControls'	-> IdentityDictionary[],
+			'patternTrack'	-> true,
+			'sensorFuncs'	-> IdentityDictionary[]
+		];
+
+		if(argType.notNil) {
+			settings['type'] = argType;
+			if(WarpTrack.defaults.keys.includes(argType)) {
+				['paramControls', 'params'].do {|key, i|
+					settings[key] = WarpTrack.defaults[argType][key];
+				}
+			};
+
+			this.initParams();
+		};
+	}
+
+	on {|note|
+		var clock = parent.clock;
+		var sub = 1 / (parent.clock.tempo * 16); // one sub division
+
+		clock.schedAbs(clock.nextBar - sub, {
+			if(settings['patternTrack']) {
+				this.allOff();
+			} {
+				this.off(note);
+			};
+		});
+
+		parent.clock.playNextBar({
+			settings['notes'].add(note);
+			parent.noteOn(settings['midiChannel'], note, 127);
+		});
+	}
+
+	off {|note|
+		// settings['notes'].remove(note);
+		parent.noteOff(settings['midiChannel'], note, 0);
+	}
+
+	hit {|note=60, vel=127, dur=1, quant=0|
+		{
+			parent.noteOn(settings['midiChannel'], note, vel);
+			dur.wait;
+			parent.noteOff(settings['midiChannel'], note, vel);
+		}.fork(parent.clock, quant:quant);
+	}
+
+	allOff {
+		settings['notes'].do {|note, i|
+			this.off(note);
+		};
+	}
+
+	assign {|paramKey, num, learn=false, init=true, checkAvailable=true|
+		if(num.notNil) {
+			this.assignAll(IdentityDictionary[paramKey -> num], learn, init, checkAvailable);
+		} {
+			parent.assign(settings['key'], paramKey, nil, learn);
+		};
+	}
+
+	assignAll {|paramControls, learn=false, init=true, checkAvailable=true|
+		var action = {
+			var channel = settings['midiChannel'];
+
+			paramControls.keysValuesDo { |paramKey, num|
+				if(checkAvailable.not ||
+					(checkAvailable && parent.isControlAvailable(channel, num))) {
+					settings['paramControls'][paramKey] = num;
+					if(init) {
+						settings['params'][paramKey] = 0;
+					};
+					parent.setControl(channel, num, paramKey);
+					if(learn) {
+						parent.control(channel, num, 127);
+						0.05.wait;
+						parent.control(channel, num, 0);
+					};
+					paramKey ++ " assigned to controlNum " ++ num;
+				} {
+					("this controlNum " ++ num ++ " is already assigned!").postln;
+				};
+			};
+		};
+
+		if(learn) {
+			action.fork;
+		} {
+			action.();
+		};
+
+	}
+
+	initParams {
+		settings['params'].keysValuesDo { |key, value|
+			this.setParam(key, value);
+		};
+		settings['paramControls'].keysValuesDo { |key, value|
+			parent.setControl(settings['midiChannel'], value, key);
+		};
+	}
+
+	setParam {|paramKey, val, quant|
+		var func = {
+			parent.control(
+				settings['midiChannel'],
+				settings['paramControls'][paramKey],
+				val
+			);
+
+			settings['params'][paramKey] = val;
+		};
+
+		if(quant.notNil) {
+			{
+				func.();
+			}.fork(parent.clock, quant:quant);
+		} {
+			func.();
+		};
+	}
+
+	readPreset {|path, checkAvailable=true|
+		this.loadPreset(Object.readArchive(path), checkAvailable);
+	}
+
+	loadPreset {|preset, checkAvailable=true|
+		// copy all settings except notes and paramControls
+		preset.keys.reject({|settingKey, i|
+			['notes', 'paramControls'].includes(settingKey);
+		}).do {|presetKey, i|
+			settings[presetKey] = preset[presetKey];
+		};
+
+		// copy notes if it's a patternTrack
+		if(preset['patternTrack']) {
+			settings['notes'] = preset['notes'];
+		};
+
+		// assign all without learn or init
+		this.assignAll(
+			preset['paramControls'],
+			false,
+			false,
+			checkAvailable
+		);
+
+		this.initParams();
+
+		// if(settings['notes'].size > 0) {
+		// 	this.on(settings['notes'].asArray[0]);
+		// };
+	}
+
+	sensor {|sensorKey, val|
+		settings['sensorFuncs'][sensorKey].do {|func, i|
+			func.(this, val);
+		}
+	}
+
+	addFunc {|sensorKey, funcKey, func|
+		if(parent.sensorKeys.includes(sensorKey)) {
+			if(settings['sensorFuncs'].includesKey(sensorKey).not) {
+				settings['sensorFuncs'][sensorKey] = IdentityDictionary[];
+			};
+
+			settings['sensorFuncs'][sensorKey][funcKey] = func;
+		} {
+			"parent doesn't have that sensor key".postln;
+		};
+	}
+
+	removeFunc {|sensorKey, funcKey|
+		settings['sensorFuncs'][sensorKey].removeAt(funcKey);
+
+		if(settings['sensorFuncs'][sensorKey].isEmpty) {
+			settings['sensorFuncs'].removeAt(sensorKey);
+		};
+	}
+
+	availableControls {
+		^parent.availableControls[settings['midiChannel']].copy;
+	}
+
+	save {
+		Dialog.savePanel({|path|
+			settings.writeArchive(path);
+		});
+	}
+
+	play {
+		if(settings['patternTrack'] && settings['notes'].notEmpty) {
+			this.on(settings['notes'].choose);
+		};
+	}
+}

samples/SuperCollider/WarpUtil.sc

@@ -0,0 +1,127 @@
+WarpUtil {
+	var <>parent;
+	var <curSensor;
+
+	var <win;
+	var <texts;
+	var <sensorSlider;
+	var <sliders;
+	var <updateRout;
+
+	*new {|argParent|
+		^super.new.init(argParent);
+	}
+
+	init {|argParent|
+		parent = argParent;
+		texts = IdentityDictionary[];
+
+		this.makeView();
+		this.startUpdate();
+	}
+
+	calibrate {
+		{
+			parent.doAdjusts.size.do {|i|
+				parent.sensorMaxs[i] = 0;
+				parent.sensorMins[i] = 9999;
+				parent.doAdjusts[i] = true;
+			};
+
+			sliders.do {|slider, i|
+				if(parent.sensorKeys[i] !== curSensor) {
+					slider.valueAction_(0);
+					0.1.wait;
+					slider.valueAction_(1);
+				};
+			}
+		}.fork(AppClock);
+	}
+
+	stopCalibrate {
+		parent.doAdjusts.size.do {|i|
+			parent.doAdjusts[i] = false;
+		}
+	}
+
+	makeView {
+		var width = 370;
+
+		if(win.notNil) {
+			win.close;
+		};
+
+		win = Window("Sensor Inspector", Rect(256, 10, width, 424)).front;
+		win.view.addFlowLayout;
+
+		['sensorVals', 'sensorPrevs', 'sensorMins', 'sensorMaxs'].do {|label, i|
+			StaticText(win, (width * 0.5)@20)
+				.string_(label.asString);
+
+			win.view.decorator.nextLine;
+
+
+			texts[label] = StaticText(win, (width / 2)@40);
+
+			if(curSensor.notNil) {
+				texts[label].string_(
+					"\t" ++ parent.perform(label).at(
+						parent.sensorKeys.indexOf(curSensor)
+					);
+				);
+			};
+
+			win.view.decorator.nextLine;
+		};
+
+		sensorSlider = EZSlider(win, 280@20, label: curSensor.asString);
+
+		sliders = parent.sensorKeys.collect { |sensorKey, i|
+			EZSlider(win, 280@20, label: sensorKey.asString)
+				.action_({|ez|
+					NetAddr.localAddr.sendMsg(
+						"/prox/" ++ sensorKey,
+						ez.value * 100
+					);
+				});
+		};
+	}
+
+	startUpdate {
+		updateRout = Routine {
+			inf.do {|i|
+				var index;
+
+				if(curSensor.notNil) {
+					index = parent.sensorKeys.indexOf(curSensor);
+
+					['sensorVals', 'sensorPrevs', 'sensorMins',
+						'sensorMaxs'].do {|label, i|
+
+							texts[label]
+							.string_("\t" ++ parent.perform(label)[index]);
+					};
+
+					sensorSlider.value = parent.sensorVals[index].linlin(
+						parent.sensorMins[index],
+						parent.sensorMaxs[index],
+						0,
+						1
+					);
+				};
+				0.05.wait;
+			}
+		};
+
+		updateRout.play(AppClock);
+	}
+
+	stopUpdate {
+		updateRout.stop();
+	}
+
+	curSensor_ {|argCurSensor|
+		curSensor = argCurSensor;
+		this.makeView();
+	}
+}

script/download-grammars

@@ -0,0 +1,221 @@
+#!/usr/bin/env ruby
+
+require 'json'
+require 'net/http'
+require 'plist'
+require 'set'
+require 'tmpdir'
+require 'uri'
+require 'yaml'
+
+GRAMMARS_PATH = File.expand_path("../../grammars", __FILE__)
+SOURCES_FILE = File.expand_path("../../grammars.yml", __FILE__)
+CSONC = File.expand_path("../../node_modules/.bin/csonc", __FILE__)
+
+class TarballPackage
+  def self.fetch(tmp_dir, url)
+    `curl --silent --location --max-time 10 --output "#{tmp_dir}/archive" "#{url}"`
+    raise "Failed to fetch GH package: #{url} #{$?.to_s}" unless $?.success?
+
+    output = File.join(tmp_dir, 'extracted')
+    Dir.mkdir(output)
+    `tar -C "#{output}" -xf "#{tmp_dir}/archive"`
+    raise "Failed to uncompress tarball: #{tmp_dir}/archive (from #{url}) #{$?.to_s}" unless $?.success?
+
+    Dir["#{output}/**/*"].select do |path|
+      case File.extname(path.downcase)
+      when '.plist'
+        path.split('/')[-2] == 'Syntaxes'
+      when '.tmlanguage'
+        true
+      when '.cson'
+        path.split('/')[-2] == 'grammars'
+      else
+        false
+      end
+    end
+  end
+
+  attr_reader :url
+
+  def initialize(url)
+    @url = url
+  end
+
+  def fetch(tmp_dir)
+    self.class.fetch(tmp_dir, url)
+  end
+end
+
+class SingleGrammar
+  attr_reader :url
+
+  def initialize(url)
+    @url = url
+  end
+
+  def fetch(tmp_dir)
+    filename = File.join(tmp_dir, File.basename(url))
+    `curl --silent --location --max-time 10 --output "#{filename}" "#{url}"`
+    raise "Failed to fetch grammar: #{url}: #{$?.to_s}" unless $?.success?
+    [filename]
+  end
+end
+
+class SVNPackage
+  attr_reader :url
+
+  def initialize(url)
+    @url = url
+  end
+
+  def fetch(tmp_dir)
+    `svn export -q "#{url}/Syntaxes" "#{tmp_dir}/Syntaxes"`
+    raise "Failed to export SVN repository: #{url}: #{$?.to_s}" unless $?.success?
+    Dir["#{tmp_dir}/Syntaxes/*.{plist,tmLanguage,tmlanguage}"]
+  end
+end
+
+class GitHubPackage
+  def self.parse_url(url)
+    url, ref = url.split("@", 2)
+    path = URI.parse(url).path.split('/')
+    [path[1], path[2].chomp('.git'), ref || "master"]
+  end
+
+  attr_reader :user
+  attr_reader :repo
+  attr_reader :ref
+
+  def initialize(url)
+    @user, @repo, @ref = self.class.parse_url(url)
+  end
+
+  def url
+    suffix = "@#{ref}" unless ref == "master"
+    "https://github.com/#{user}/#{repo}#{suffix}"
+  end
+
+  def fetch(tmp_dir)
+    url = "https://github.com/#{user}/#{repo}/archive/#{ref}.tar.gz"
+    TarballPackage.fetch(tmp_dir, url)
+  end
+end
+
+def load_grammar(path)
+  case File.extname(path.downcase)
+  when '.plist', '.tmlanguage'
+    Plist::parse_xml(path)
+  when '.cson'
+    cson = `"#{CSONC}" "#{path}"`
+    raise "Failed to convert CSON grammar '#{path}': #{$?.to_s}" unless $?.success?
+    JSON.parse(cson)
+  else
+    raise "Invalid document type #{path}"
+  end
+end
+
+def install_grammar(tmp_dir, source, all_scopes)
+  p = if source.end_with?('.tmLanguage', '.plist')
+        SingleGrammar.new(source)
+      elsif source.start_with?('https://github.com')
+        GitHubPackage.new(source)
+      elsif source.start_with?('http://svn.textmate.org')
+        SVNPackage.new(source)
+      elsif source.end_with?('.tar.gz')
+        TarballPackage.new(source)
+      else
+        nil
+      end
+
+  raise "Unsupported source: #{source}" unless p
+
+  installed = []
+
+  p.fetch(tmp_dir).each do |path|
+    grammar = load_grammar(path)
+    scope = grammar['scopeName']
+
+    if all_scopes.key?(scope)
+      $stderr.puts "WARN: Duplicated scope #{scope}\n" +
+        "  Current package: #{p.url}\n" +
+      "  Previous package: #{all_scopes[scope]}"
+      next
+    end
+
+    File.write(File.join(GRAMMARS_PATH, "#{scope}.json"), JSON.pretty_generate(grammar))
+    all_scopes[scope] = p.url
+    installed << scope
+  end
+
+  $stderr.puts("OK #{p.url} (#{installed.join(', ')})")
+end
+
+def run_thread(queue, all_scopes)
+  Dir.mktmpdir do |tmpdir|
+    loop do
+      source, index = begin
+        queue.pop(true)
+      rescue ThreadError
+        # The queue is empty.
+        break
+      end
+
+      dir = "#{tmpdir}/#{index}"
+      Dir.mkdir(dir)
+
+      install_grammar(dir, source, all_scopes)
+    end
+  end
+end
+
+def generate_yaml(all_scopes, base)
+  yaml = all_scopes.each_with_object(base) do |(key,value),out|
+    out[value] ||= []
+    out[value] << key
+  end
+
+  yaml = yaml.sort.to_h
+  yaml.each { |k, v| v.sort! }
+  yaml
+end
+
+def main(sources)
+  begin
+    Dir.mkdir(GRAMMARS_PATH)
+  rescue Errno::EEXIST
+  end
+
+  `npm install`
+
+  all_scopes = {}
+
+  if ARGV[0] == '--add'
+    Dir.mktmpdir do |tmpdir|
+      install_grammar(tmpdir, ARGV[1], all_scopes)
+    end
+    generate_yaml(all_scopes, sources)
+  else
+    queue = Queue.new
+
+    sources.each do |url, scopes|
+      queue.push([url, queue.length])
+    end
+
+    threads = 8.times.map do
+      Thread.new { run_thread(queue, all_scopes) }
+    end
+    threads.each(&:join)
+    generate_yaml(all_scopes, {})
+  end
+end
+
+sources = File.open(SOURCES_FILE) do |file|
+  YAML.load(file)
+end
+
+yaml = main(sources)
+
+File.write(SOURCES_FILE, YAML.dump(yaml))
+
+$stderr.puts("Done")