Add calculuswhiz/Assembly-Syntax-Definition grammar and use it for Unix Assembly (#4096)

Each strategy takes as candidates the language outputted by the
previous strategy if any. This was already the case for the
Classifier and Heuristic strategies as these couldn't generate new
candidate languages (as opposed to the Modeline, Filename, Shebang,
and Extension strategies).

In practice, this signifies that if, for example, the Shebang
strategy finds two possible languages for a given file (as is
currently possible with the perl interpreter), the next strategy, the
Extension strategy, will use this information and further reduce the
set of possible language.
Currently, without this commit, the Extension strategy would discard
the results from the previous strategy and start anew, possibly
returning a different language from those returned by the Shebang
strategy.

.github/ISSUE_TEMPLATE.md

@@ -0,0 +1,26 @@
+<!--- Provide a general summary of the issue in the Title above -->
+
+## Preliminary Steps
+
+Please confirm you have...
+- [ ] reviewed [How Linguist Works](https://github.com/github/linguist#how-linguist-works),
+- [ ] reviewed the [Troubleshooting](https://github.com/github/linguist#troubleshooting) docs,
+- [ ] considered implementing an [override](https://github.com/github/linguist#overrides),
+- [ ] verified an issue has not already been logged for your issue ([linguist issues](https://github.com/issues?utf8=%E2%9C%93&q=is%3Aissue+repo%3Agithub/linguist)).
+
+<!-- Please review these preliminary steps before logging your issue. You may find the information referenced may answer or explain the behaviour you are seeing. It'll help us to know you've reviewed this information. -->
+
+## Problem Description
+
+<!--- Provide a more detailed introduction to the issue itself, and why you consider it to be a bug -->
+
+### URL of the affected repository:
+
+### Last modified on:
+<!-- YYYY-MM-DD -->
+
+### Expected language:
+<!-- expected language -->
+
+### Detected language:
+<!-- detected language -->

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,46 @@
+<!--- Briefly describe what you're changing. -->
+
+## Description
+<!--- If necessary, go into depth of what this pull request is doing. -->
+
+## Checklist:
+<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
+<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
+- [ ] **I am associating a language with a new file extension.**
+  - [ ] The new extension is used in hundreds of repositories on GitHub.com
+    - Search results for each extension:
+      <!-- Replace FOOBAR with the new extension, and KEYWORDS with keywords unique to the language. Repeat for each extension added. -->
+      - https://github.com/search?utf8=%E2%9C%93&type=Code&ref=searchresults&q=extension%3AFOOBAR+KEYWORDS+NOT+nothack
+  - [ ] I have included a real-world usage sample for all extensions added in this PR:
+    - Sample source(s):
+      - [URL to each sample source, if applicable]
+    - Sample license(s):
+  - [ ] I have included a change to the heuristics to distinguish my language from others using the same extension.
+
+- [ ] **I am adding a new language.**
+  - [ ] The extension of the new language is used in hundreds of repositories on GitHub.com.
+    - Search results for each extension:
+      <!-- Replace FOOBAR with the new extension, and KEYWORDS with keywords unique to the language. Repeat for each extension added. -->
+      -  https://github.com/search?utf8=%E2%9C%93&type=Code&ref=searchresults&q=extension%3AFOOBAR+KEYWORDS+NOT+nothack
+  - [ ] I have included a real-world usage sample for all extensions added in this PR:
+    - Sample source(s):
+      - [URL to each sample source, if applicable]
+    - Sample license(s):
+  - [ ] I have included a syntax highlighting grammar.
+  - [ ] I have included a change to the heuristics to distinguish my language from others using the same extension.
+
+- [ ] **I am fixing a misclassified language**
+  - [ ] I have included a new sample for the misclassified language:
+    - Sample source(s):
+      - [URL to each sample source, if applicable]
+    - Sample license(s):
+  - [ ] I have included a change to the heuristics to distinguish my language from others using the same extension.
+
+- [ ] **I am changing the source of a syntax highlighting grammar**
+  <!-- Update the Lightshow URLs below to show the new and old grammars in action. -->
+  - Old: https://github-lightshow.herokuapp.com/
+  - New: https://github-lightshow.herokuapp.com/
+
+- [ ] **I am adding new or changing current functionality**
+  <!-- This includes modifying the vendor, documentation, and generated lists. -->
+  - [ ] I have added or updated the tests for the new or changed functionality.

.gitignore

@@ -1,3 +1,4 @@
+*.gem
 /Gemfile.lock
 .bundle/
 .idea
@@ -7,3 +8,6 @@ lib/linguist/samples.json
 /node_modules
 test/fixtures/ace_modes.json
 /vendor/gems/
+/tmp
+*.bundle
+*.so

.travis.yml

@@ -1,20 +1,31 @@
 language: ruby
 sudo: false
+
 addons:
   apt:
     packages:
     - libicu-dev
-    - libicu48
+    - libicu52
+
 before_install: script/travis/before_install
+
 script:
   - bundle exec rake
-  - script/licensed verify
+  - script/licensed status
+
 rvm:
-  - 2.0.0
-  - 2.1
-  - 2.2
+  - 2.3.3
+  - 2.4.0
+  - 2.5.0
+
 notifications:
   disabled: true
+
 git:
   submodules: false
+  depth: 3
+
 cache: bundler
+dist: trusty
+
+bundler_args: --without debug

CONTRIBUTING.md

@@ -1,24 +1,53 @@
 # Contributing
 
-Hi there! We're thrilled that you'd like to contribute to this project. Your help is essential for keeping it great. This project adheres to the [Contributor Covenant Code of Conduct](http://contributor-covenant.org/). By participating, you are expected to uphold this code.
+Hi there! We're thrilled that you'd like to contribute to this project. Your help is essential for keeping it great.
+
+Contributions to this project are [released](https://help.github.com/articles/github-terms-of-service/#6-contributions-under-repository-license) to the public under the [project's open source license](LICENSE).
+
+This project adheres to the [Contributor Covenant Code of Conduct](http://contributor-covenant.org/). By participating, you are expected to uphold this code.
 
 The majority of contributions won't need to touch any Ruby code at all.
 
+## Getting started
+
+Before you can start contributing to Linguist, you'll need to set up your environment first. Clone the repo and run `script/bootstrap` to install its dependencies.
+
+    git clone https://github.com/github/linguist.git
+    cd linguist/
+    script/bootstrap
+
+To run Linguist from the cloned repository, you will need to generate the code samples first:
+
+    bundle exec rake samples
+
+Run this command each time a [sample][samples] has been modified.
+
+To run Linguist from the cloned repository:
+
+    bundle exec bin/linguist --breakdown
+
+### Dependencies
+
+Linguist uses the [`charlock_holmes`](https://github.com/brianmario/charlock_holmes) character encoding detection library which in turn uses [ICU](http://site.icu-project.org/), and the libgit2 bindings for Ruby provided by [`rugged`](https://github.com/libgit2/rugged). [Docker](https://www.docker.com/) is also required when adding or updating grammars. These components have their own dependencies - `icu4c`, and `cmake` and `pkg-config` respectively - which you may need to install before you can install Linguist.
+
+For example, on macOS with [Homebrew](http://brew.sh/): `brew install cmake pkg-config icu4c docker` and on Ubuntu: `apt-get install cmake pkg-config libicu-dev docker-ce`.
+
 ## Adding an extension to a language
 
 We try only to add new extensions once they have some usage on GitHub. In most cases we prefer that extensions be in use in hundreds of repositories before supporting them in Linguist.
 
 To add support for a new extension:
 
-0. Add your extension to the language entry in [`languages.yml`][languages], keeping the extensions in alphabetical order.
-0. Add at least one sample for your extension to the [samples directory][samples] in the correct subdirectory.
-0. Open a pull request, linking to a [GitHub search result](https://github.com/search?utf8=%E2%9C%93&q=extension%3Aboot+NOT+nothack&type=Code&ref=searchresults) showing in-the-wild usage.
+1. Add your extension to the language entry in [`languages.yml`][languages], keeping the extensions in alphabetical and case-sensitive (uppercase before lowercase) order, with the exception of the primary extension; the primary extension should be first.
+1. Add at least one sample for your extension to the [samples directory][samples] in the correct subdirectory. We'd prefer examples of real-world code showing common usage. The more representative of the structure of the language, the better.
+1. Open a pull request, linking to a [GitHub search result](https://github.com/search?utf8=%E2%9C%93&q=extension%3Aboot+NOT+nothack&type=Code&ref=searchresults) showing in-the-wild usage.  
+  If you are adding a sample, please state clearly the license covering the code in the sample, and if possible, link to the original source of the sample.
 
-In addition, if this extension is already listed in [`languages.yml`][languages] then sometimes a few more steps will need to be taken:
+Additionally, if this extension is already listed in [`languages.yml`][languages] and associated with another language, then sometimes a few more steps will need to be taken:
 
-0. Make sure that example `.yourextension` files are present in the [samples directory][samples] for each language that uses `.yourextension`.
-0. Test the performance of the Bayesian classifier with a relatively large number (1000s) of sample `.yourextension` files. (ping @arfon or @bkeepers to help with this) to ensure we're not misclassifying files.
-0. If the Bayesian classifier does a bad job with the sample `.yourextension` files then a [heuristic](https://github.com/github/linguist/blob/master/lib/linguist/heuristics.rb) may need to be written to help.
+1. Make sure that example `.yourextension` files are present in the [samples directory][samples] for each language that uses `.yourextension`.
+1. Test the performance of the Bayesian classifier with a relatively large number (1000s) of sample `.yourextension` files. (ping **@lildude** to help with this) to ensure we're not misclassifying files.
+1. If the Bayesian classifier does a bad job with the sample `.yourextension` files then a [heuristic](https://github.com/github/linguist/blob/master/lib/linguist/heuristics.rb) may need to be written to help.
 
 
 ## Adding a language
@@ -27,50 +56,61 @@ We try only to add languages once they have some usage on GitHub. In most cases
 
 To add support for a new language:
 
-0. Add an entry for your language to [`languages.yml`][languages].
-0. Add a grammar for your language. Please only add grammars that have [one of these licenses](https://github.com/github/linguist/blob/257425141d4e2a5232786bf0b13c901ada075f93/vendor/licenses/config.yml#L2-L11).
-  0. Add your grammar as a submodule: `git submodule add https://github.com/JaneSmith/MyGrammar vendor/grammars/MyGrammar`.
-  0. Add your grammar to [`grammars.yml`][grammars] by running `script/convert-grammars --add vendor/grammars/MyGrammar`.
-  0. Download the license for the grammar: `script/licensed`. Be careful to only commit the file for the new grammar, as this script may update licenses for other grammars as well.
-0. Add samples for your language to the [samples directory][samples] in the correct subdirectory.
-0. Add a `language_id` for your language. See `script/set-language-ids` for more information. **You should only ever need to run `script/set-language-ids --update`. Anything other than this risks breaking GitHub search :cry:**
-0. Open a pull request, linking to a [GitHub search result](https://github.com/search?utf8=%E2%9C%93&q=extension%3Aboot+NOT+nothack&type=Code&ref=searchresults) showing in-the-wild usage.
+1. Add an entry for your language to [`languages.yml`][languages]. Omit the `language_id` field for now.
+1. Add a syntax-highlighting grammar for your language using: `script/add-grammar https://github.com/JaneSmith/MyGrammar`  
+  This command will analyze the grammar and, if no problems are found, add it to the repository. If problems are found, please report them to the grammar maintainer as you will not be able to add the grammar if problems are found.  
+  **Please only add grammars that have [one of these licenses][licenses].**
+1. Add samples for your language to the [samples directory][samples] in the correct subdirectory.
+1. Add a `language_id` for your language using `script/set-language-ids`.  
+  **You should only ever need to run `script/set-language-ids --update`. Anything other than this risks breaking GitHub search :cry:**
+1. Open a pull request, linking to a [GitHub search results](https://github.com/search?utf8=%E2%9C%93&q=extension%3Aboot+NOT+nothack&type=Code&ref=searchresults) showing in-the-wild usage.  
+  Please state clearly the license covering the code in the samples. Link directly to the original source if possible.
 
 In addition, if your new language defines an extension that's already listed in [`languages.yml`][languages] (such as `.foo`) then sometimes a few more steps will need to be taken:
 
-0. Make sure that example `.foo` files are present in the [samples directory][samples] for each language that uses `.foo`.
-0. Test the performance of the Bayesian classifier with a relatively large number (1000s) of sample `.foo` files. (ping @arfon or @bkeepers to help with this) to ensure we're not misclassifying files.
-0. If the Bayesian classifier does a bad job with the sample `.foo` files then a [heuristic](https://github.com/github/linguist/blob/master/lib/linguist/heuristics.rb) may need to be written to help.
+1. Make sure that example `.foo` files are present in the [samples directory][samples] for each language that uses `.foo`.
+1. Test the performance of the Bayesian classifier with a relatively large number (1000s) of sample `.foo` files. (ping **@lildude** to help with this) to ensure we're not misclassifying files.
+1. If the Bayesian classifier does a bad job with the sample `.foo` files then a [heuristic](https://github.com/github/linguist/blob/master/lib/linguist/heuristics.rb) may need to be written to help.
 
 Remember, the goal here is to try and avoid false positives!
 
 
 ## Fixing a misclassified language
 
-Most languages are detected by their file extension defined in [languages.yml][languages].  For disambiguating between files with common extensions, linguist applies some [heuristics](/lib/linguist/heuristics.rb) and a [statistical classifier](lib/linguist/classifier.rb). This process can help differentiate between, for example, `.h` files which could be either C, C++, or Obj-C.
+Most languages are detected by their file extension defined in [`languages.yml`][languages].  For disambiguating between files with common extensions, Linguist applies some [heuristics](/lib/linguist/heuristics.rb) and a [statistical classifier](lib/linguist/classifier.rb). This process can help differentiate between, for example, `.h` files which could be either C, C++, or Obj-C.
 
 Misclassifications can often be solved by either adding a new filename or extension for the language or adding more [samples][samples] to make the classifier smarter.
 
 
 ## Fixing syntax highlighting
 
-Syntax highlighting in GitHub is performed using TextMate-compatible grammars. These are the same grammars that TextMate, Sublime Text and Atom use. Every language in [languages.yml][languages] is mapped to its corresponding TM `scope`. This scope will be used when picking up a grammar for highlighting.
+Syntax highlighting in GitHub is performed using TextMate-compatible grammars. These are the same grammars that TextMate, Sublime Text and Atom use. Every language in [`languages.yml`][languages] is mapped to its corresponding TextMate `scopeName`. This scope name will be used when picking up a grammar for highlighting.
 
-Assuming your code is being detected as the right language, in most cases this is due to a bug in the language grammar rather than a bug in Linguist. [`grammars.yml`][grammars] lists all the grammars we use for syntax highlighting on github.com. Find the one corresponding to your code's programming language and submit a bug report upstream. If you can, try to reproduce the highlighting problem in the text editor that the grammar is designed for (TextMate, Sublime Text, or Atom) and include that information in your bug report.
+Assuming your code is being detected as the right language, in most cases syntax highlighting problems are due to a bug in the language grammar rather than a bug in Linguist. [`vendor/README.md`][grammars] lists all the grammars we use for syntax highlighting on GitHub.com. Find the one corresponding to your code's programming language and submit a bug report upstream. If you can, try to reproduce the highlighting problem in the text editor that the grammar is designed for (TextMate, Sublime Text, or Atom) and include that information in your bug report.
 
 You can also try to fix the bug yourself and submit a Pull Request. [TextMate's documentation](https://manual.macromates.com/en/language_grammars) offers a good introduction on how to work with TextMate-compatible grammars. You can test grammars using [Lightshow](https://github-lightshow.herokuapp.com).
 
 Once the bug has been fixed upstream, we'll pick it up for GitHub in the next release of Linguist.
 
+
+## Changing the source of a syntax highlighting grammar
+
+We'd like to ensure Linguist and GitHub.com are using the latest and greatest grammars that are consistent with the current usage but understand that sometimes a grammar can lag behind the evolution of a language or even stop being developed. This often results in someone grasping the opportunity to create a newer and better and more actively maintained grammar, and we'd love to use it and pass on it's functionality to our users.
+
+Switching the source of a grammar is really easy:
+
+    script/add-grammar --replace MyGrammar https://github.com/PeterPan/MyGrammar
+
+This command will analyze the grammar and, if no problems are found, add it to the repository. If problems are found, please report these problems to the grammar maintainer as you will not be able to add the grammar if problems are found.  
+
+**Please only add grammars that have [one of these licenses][licenses].**
+
+Please then open a pull request for the updated grammar.
+
+
 ## Testing
 
-For development you are going to want to checkout out the source. To get it, clone the repo and run [Bundler](http://gembundler.com/) to install its dependencies.
-
-    git clone https://github.com/github/linguist.git
-    cd linguist/
-    script/bootstrap
-
-To run the tests:
+You can run the tests locally with:
 
     bundle exec rake test
 
@@ -82,36 +122,46 @@ Here's our current build status: [![Build Status](https://api.travis-ci.org/gith
 
 Linguist is maintained with :heart: by:
 
-- @arfon (GitHub Staff)
-- @larsbrinkhoff
-- @pchaigno
+- **@Alhadis**
+- **@BenEddy** (GitHub staff)
+- **@Caged** (GitHub staff)
+- **@grantr** (GitHub staff)
+- **@kivikakk** (GitHub staff)
+- **@larsbrinkhoff**
+- **@lildude** (GitHub staff)
+- **@pchaigno**
+- **@rafer** (GitHub staff)
+- **@shreyasjoshis** (GitHub staff)
 
 As Linguist is a production dependency for GitHub we have a couple of workflow restrictions:
 
-- Anyone with commit rights can merge Pull Requests provided that there is a :+1: from a GitHub member of staff
+- Anyone with commit rights can merge Pull Requests provided that there is a :+1: from a GitHub staff member.
 - Releases are performed by GitHub staff so we can ensure GitHub.com always stays up to date with the latest release of Linguist and there are no regressions in production.
 
 ### Releasing
 
 If you are the current maintainer of this gem:
 
-0. Create a branch for the release: `git checkout -b cut-release-vxx.xx.xx`
-0. Make sure your local dependencies are up to date: `script/bootstrap`
-0. If grammar submodules have not been updated recently, update them: `git submodule update --remote && git commit -a`
-0. Ensure that samples are updated: `bundle exec rake samples`
-0. Ensure that tests are green: `bundle exec rake test`
-0. Bump gem version in `lib/linguist/version.rb`, [like this](https://github.com/github/linguist/commit/8d2ea90a5ba3b2fe6e1508b7155aa4632eea2985).
-0. Make a PR to github/linguist, [like this](https://github.com/github/linguist/pull/1238).
-0. Build a local gem: `bundle exec rake build_gem`
-0. Test the gem:
-  0. Bump the Gemfile and Gemfile.lock versions for an app which relies on this gem
-  0. Install the new gem locally
-  0. Test behavior locally, branch deploy, whatever needs to happen
-0. Merge github/linguist PR
-0. Tag and push: `git tag vx.xx.xx; git push --tags`
-0. Push to rubygems.org -- `gem push github-linguist-3.0.0.gem`
+1. Create a branch for the release: `git checkout -b cut-release-vxx.xx.xx`
+1. Make sure your local dependencies are up to date: `script/bootstrap`
+1. If grammar submodules have not been updated recently, update them: `git submodule update --remote && git commit -a`
+1. Ensure that samples are updated: `bundle exec rake samples`
+1. Ensure that tests are green: `bundle exec rake test`
+1. Bump gem version in `lib/linguist/version.rb`, [like this](https://github.com/github/linguist/commit/8d2ea90a5ba3b2fe6e1508b7155aa4632eea2985).
+1. Make a PR to github/linguist, [like this](https://github.com/github/linguist/pull/1238).
+1. Build a local gem: `bundle exec rake build_gem`
+1. Test the gem:
+  1. Bump the Gemfile and Gemfile.lock versions for an app which relies on this gem
+  1. Install the new gem locally
+  1. Test behavior locally, branch deploy, whatever needs to happen
+1. Merge github/linguist PR
+1. Tag and push: `git tag vx.xx.xx; git push --tags`
+1. Create a GitHub release with the pushed tag (https://github.com/github/linguist/releases/new)
+1. Build a grammars tarball (`./script/build-grammars-tarball`) and attach it to the GitHub release
+1. Push to rubygems.org -- `gem push github-linguist-3.0.0.gem`
 
-[grammars]: /grammars.yml
+[grammars]: /vendor/README.md
 [languages]: /lib/linguist/languages.yml
+[licenses]: https://github.com/github/linguist/blob/257425141d4e2a5232786bf0b13c901ada075f93/vendor/licenses/config.yml#L2-L11
 [samples]: /samples
 [new-issue]: https://github.com/github/linguist/issues/new

Gemfile

@@ -1,3 +1,6 @@
 source 'https://rubygems.org'
 gemspec :name => "github-linguist"
-gem 'byebug' if RUBY_VERSION >= '2.0'
+
+group :debug do
+  gem 'byebug' if RUBY_VERSION >= '2.2'
+end

LICENSE

@@ -1,4 +1,4 @@
-Copyright (c) 2011-2016 GitHub, Inc.
+Copyright (c) 2017 GitHub, Inc.
 
 Permission is hereby granted, free of charge, to any person
 obtaining a copy of this software and associated documentation

README.md

@@ -1,39 +1,139 @@
 # Linguist
 
+[![Build Status](https://travis-ci.org/github/linguist.svg?branch=master)](https://travis-ci.org/github/linguist)
+
 [issues]: https://github.com/github/linguist/issues
 [new-issue]: https://github.com/github/linguist/issues/new
 
 This library is used on GitHub.com to detect blob languages, ignore binary or vendored files, suppress generated files in diffs, and generate language breakdown graphs.
 
-See [Troubleshooting](#troubleshooting) and [`CONTRIBUTING.md`](/CONTRIBUTING.md) before filing an issue or creating a pull request.
+See [Troubleshooting](#troubleshooting) and [`CONTRIBUTING.md`](CONTRIBUTING.md) before filing an issue or creating a pull request.
+
+## How Linguist works
+
+Linguist takes the list of languages it knows from [`languages.yml`](/lib/linguist/languages.yml) and uses a number of methods to try and determine the language used by each file, and the overall repository breakdown.
+
+Linguist starts by going through all the files in a repository and excludes all files that it determines to be binary data, [vendored code](#vendored-code), [generated code](#generated-code), [documentation](#documentation), or are defined as `data` (e.g. SQL) or `prose` (e.g. Markdown) languages, whilst taking into account any [overrides](#overrides).
+
+If an [explicit language override](#using-gitattributes) has been used, that language is used for the matching files. The language of each remaining file is then determined using the following strategies, in order, with each step either identifying the precise language or reducing the number of likely languages passed down to the next strategy:
+
+- Vim or Emacs modeline,
+- commonly used filename,
+- shell shebang,
+- file extension,
+- heuristics,
+- naïve Bayesian classification
+
+The result of this analysis is used to produce the language stats bar which displays the languages percentages for the files in the repository. The percentages are calculated based on the bytes of code for each language as reported by the [List Languages](https://developer.github.com/v3/repos/#list-languages) API.
+
+![language stats bar](https://cloud.githubusercontent.com/assets/173/5562290/48e24654-8ddf-11e4-8fe7-735b0ce3a0d3.png)
+
+### How Linguist works on GitHub.com
+
+When you push changes to a repository on GitHub.com, a low priority background job is enqueued to analyze your repository as explained above. The results of this analysis are cached for the lifetime of your repository and are only updated when the repository is updated. As this analysis is performed by a low priority background job, it can take a while, particularly during busy periods, for your language statistics bar to reflect your changes.
+
+
+## Usage
+
+Install the gem:
+
+    $ gem install github-linguist
+
+#### Dependencies
+
+Linguist uses the [`charlock_holmes`](https://github.com/brianmario/charlock_holmes) character encoding detection library which in turn uses [ICU](http://site.icu-project.org/), and the libgit2 bindings for Ruby provided by [`rugged`](https://github.com/libgit2/rugged). These components have their own dependencies - `icu4c`, and `cmake` and `pkg-config` respectively - which you may need to install before you can install Linguist.
+
+For example, on macOS with [Homebrew](http://brew.sh/): `brew install cmake pkg-config icu4c` and on Ubuntu: `apt-get install cmake pkg-config libicu-dev`.
+
+### Application usage
+
+Linguist can be used in your application as follows:
+
+```ruby
+require 'rugged'
+require 'linguist'
+
+repo = Rugged::Repository.new('.')
+project = Linguist::Repository.new(repo, repo.head.target_id)
+project.language       #=> "Ruby"
+project.languages      #=> { "Ruby" => 119387 }
+```
+
+### Command line usage
+
+A repository's languages stats can also be assessed from the command line using the `linguist` executable. Without any options, `linguist` will output the breakdown that correlates to what is shown in the language stats bar. The `--breakdown` flag will additionally show the breakdown of files by language.
+
+You can try running `linguist` on the root directory in this repository itself:
+
+```console
+$ bundle exec bin/linguist --breakdown
+68.57%  Ruby
+22.90%  C
+6.93%   Go
+1.21%   Lex
+0.39%   Shell
+
+Ruby:
+Gemfile
+Rakefile
+bin/git-linguist
+bin/linguist
+ext/linguist/extconf.rb
+github-linguist.gemspec
+lib/linguist.rb
+…
+```
+
 
 ## Troubleshooting
 
 ### My repository is detected as the wrong language
 
-![language stats bar](https://cloud.githubusercontent.com/assets/173/5562290/48e24654-8ddf-11e4-8fe7-735b0ce3a0d3.png)
+If the language stats bar is reporting a language that you don't expect:
 
-The Language stats bar displays languages percentages for the files in the repository. The percentages are calculated based on the bytes of code for each language as reported by the [List Languages](https://developer.github.com/v3/repos/#list-languages) API. If the bar is reporting a language that you don't expect:
+1. Click on the name of the language in the stats bar to see a list of the files that are identified as that language.  
+  Keep in mind this performs a search so the [code search restrictions](https://help.github.com/articles/searching-code/#considerations-for-code-search) may result in files identified in the language statistics not appearing in the search results. [Installing Linguist locally](#usage) and running it from the [command line](#command-line-usage) will give you accurate results.
+1. If you see files that you didn't write in the search results, consider moving the files into one of the [paths for vendored code](/lib/linguist/vendor.yml), or use the [manual overrides](#overrides) feature to ignore them.
+1. If the files are misclassified, search for [open issues][issues] to see if anyone else has already reported the issue. Any information you can add, especially links to public repositories, is helpful. You can also use the [manual overrides](#overrides) feature to correctly classify them in your repository.
+1. If there are no reported issues of this misclassification, [open an issue][new-issue] and include a link to the repository or a sample of the code that is being misclassified.
+
+Keep in mind that the repository language stats are only [updated when you push changes](#how-linguist-works-on-githubcom), and the results are cached for the lifetime of your repository. If you have not made any changes to your repository in a while, you may find pushing another change will correct the stats.
+
+### My repository isn't showing my language
+
+Linguist does not consider [vendored code](#vendored-code), [generated code](#generated-code), [documentation](#documentation), or `data` (e.g. SQL) or `prose` (e.g. Markdown) languages (as defined by the `type` attribute in [`languages.yml`](/lib/linguist/languages.yml)) when calculating the repository language statistics.
+
+If the language statistics bar is not showing your language at all, it could be for a few reasons:
+
+1. Linguist doesn't know about your language.
+1. The extension you have chosen is not associated with your language in [`languages.yml`](/lib/linguist/languages.yml).
+1. All the files in your repository fall into one of the categories listed above that Linguist excludes by default.
+
+If Linguist doesn't know about the language or the extension you're using, consider [contributing](CONTRIBUTING.md) to Linguist by opening a pull request to add support for your language or extension. For everything else, you can use the [manual overrides](#overrides) feature to tell Linguist to include your files in the language statistics.
+
+### There's a problem with the syntax highlighting of a file
+
+Linguist detects the language of a file but the actual syntax-highlighting is powered by a set of language grammars which are included in this project as a set of submodules [as listed here](/vendor/README.md).
+
+If you experience an issue with the syntax-highlighting on GitHub, **please report the issue to the upstream grammar repository, not here.** Grammars are updated every time we build the Linguist gem so upstream bug fixes are automatically incorporated as they are fixed.
 
-0. Click on the name of the language in the stats bar to see a list of the files that are identified as that language.
-0. If you see files that you didn't write, consider moving the files into one of the [paths for vendored  code](/lib/linguist/vendor.yml), or use the [manual overrides](#overrides) feature to ignore them.
-0. If the files are being misclassified, search for [open issues][issues] to see if anyone else has already reported the issue. Any information you can add, especially links to public repositories, is helpful.
-0. If there are no reported issues of this misclassification, [open an issue][new-issue] and include a link to the repository or a sample of the code that is being misclassified.
 
 ## Overrides
 
-Linguist supports a number of different custom overrides strategies for language definitions and vendored paths.
+Linguist supports a number of different custom override strategies for language definitions and file paths.
 
 ### Using gitattributes
 
-Add a `.gitattributes` file to your project and use standard git-style path matchers for the files you want to override to set `linguist-documentation`, `linguist-language`, and `linguist-vendored`. `.gitattributes` will be used to determine language statistics, but will not be used to syntax highlight files. To manually set syntax highlighting, use [Vim or Emacs modelines](#using-emacs-or-vim-modelines).
+Add a `.gitattributes` file to your project and use standard git-style path matchers for the files you want to override using the `linguist-documentation`, `linguist-language`, `linguist-vendored`, `linguist-generated`  and `linguist-detectable` attributes. `.gitattributes` will be used to determine language statistics and will be used to syntax highlight files. You can also manually set syntax highlighting using [Vim or Emacs modelines](#using-emacs-or-vim-modelines).
 
 ```
 $ cat .gitattributes
 *.rb linguist-language=Java
 ```
 
-Checking code you didn't write, such as JavaScript libraries, into your git repo is a common practice, but this often inflates your project's language stats and may even cause your project to be labeled as another language. By default, Linguist treats all of the paths defined in [lib/linguist/vendor.yml](https://github.com/github/linguist/blob/master/lib/linguist/vendor.yml) as vendored and therefore doesn't include them in the language statistics for a repository.
+#### Vendored code
+
+Checking code you didn't write, such as JavaScript libraries, into your git repo is a common practice, but this often inflates your project's language stats and may even cause your project to be labeled as another language. By default, Linguist treats all of the paths defined in [`vendor.yml`](/lib/linguist/vendor.yml) as vendored and therefore doesn't include them in the language statistics for a repository.
 
 Use the `linguist-vendored` attribute to vendor or un-vendor paths.
 
@@ -43,7 +143,9 @@ special-vendored-path/* linguist-vendored
 jquery.js linguist-vendored=false
 ```
 
-Just like vendored files, Linguist excludes documentation files from your project's language stats. [lib/linguist/documentation.yml](lib/linguist/documentation.yml) lists common documentation paths and excludes them from the language statistics for your repository.
+#### Documentation
+
+Just like vendored files, Linguist excludes documentation files from your project's language stats. [`documentation.yml`](/lib/linguist/documentation.yml) lists common documentation paths and excludes them from the language statistics for your repository.
 
 Use the `linguist-documentation` attribute to mark or unmark paths as documentation.
 
@@ -53,19 +155,33 @@ project-docs/* linguist-documentation
 docs/formatter.rb linguist-documentation=false
 ```
 
-#### Generated file detection
+#### Generated code
 
-Not all plain text files are true source files. Generated files like minified js and compiled CoffeeScript can be detected and excluded from language stats. As an added bonus, unlike vendored and documentation files, these files are suppressed in diffs.
+Not all plain text files are true source files. Generated files like minified JavaScript and compiled CoffeeScript can be detected and excluded from language stats. As an added bonus, unlike vendored and documentation files, these files are suppressed in diffs. [`generated.rb`](/lib/linguist/generated.rb) lists common generated paths and excludes them from the language statistics of your repository.
 
-```ruby
-Linguist::FileBlob.new("underscore.min.js").generated? # => true
+Use the `linguist-generated` attribute to mark or unmark paths as generated.
+
+```
+$ cat .gitattributes
+Api.elm linguist-generated=true
 ```
 
-See [Linguist::Generated#generated?](https://github.com/github/linguist/blob/master/lib/linguist/generated.rb).
+#### Detectable
+
+Only programming languages are included in the language statistics. Languages of a different type (as defined in [`languages.yml`](/lib/linguist/languages.yml)) are not "detectable" causing them not to be included in the language statistics.
+
+Use the `linguist-detectable` attribute to mark or unmark paths as detectable.
+
+```
+$ cat .gitattributes
+*.kicad_pcb linguist-detectable=true
+*.sch linguist-detectable=true
+tools/export_bom.py linguist-detectable=false
+```
 
 ### Using Emacs or Vim modelines
 
-Alternatively, you can use Vim or Emacs style modelines to set the language for a single file. Modelines can be placed anywhere within a file and are respected when determining how to syntax-highlight a file on GitHub.com
+If you do not want to use `.gitattributes` to override the syntax highlighting used on GitHub.com, you can use Vim or Emacs style modelines to set the language for a single file. Modelines can be placed anywhere within a file and are respected when determining how to syntax-highlight a file on GitHub.com
 
 ##### Vim
 ```
@@ -81,52 +197,15 @@ vim: set ft=cpp:
 -*- mode: php;-*-
 ```
 
-## Usage
-
-Install the gem:
-
-```
-$ gem install github-linguist
-```
-
-Then use it in your application:
-
-```ruby
-require 'rugged'
-require 'linguist'
-
-repo = Rugged::Repository.new('.')
-project = Linguist::Repository.new(repo, repo.head.target_id)
-project.language       #=> "Ruby"
-project.languages      #=> { "Ruby" => 119387 }
-```
-
-These stats are also printed out by the `linguist` executable. You can use the
-`--breakdown` flag, and the binary will also output the breakdown of files by language.
-
-You can try running `linguist` on the root directory in this repository itself:
-
-```
-$ bundle exec linguist --breakdown
-
-100.00% Ruby
-
-Ruby:
-Gemfile
-Rakefile
-bin/linguist
-github-linguist.gemspec
-lib/linguist.rb
-…
-```
 
 ## Contributing
 
 Please check out our [contributing guidelines](CONTRIBUTING.md).
 
+
 ## License
 
 The language grammars included in this gem are covered by their repositories'
-respective licenses. `grammars.yml` specifies the repository for each grammar.
+respective licenses. [`vendor/README.md`](/vendor/README.md) lists the repository for each grammar.
 
 All other files are covered by the MIT license, see `LICENSE`.

Rakefile

@@ -1,16 +1,24 @@
 require 'bundler/setup'
 require 'rake/clean'
 require 'rake/testtask'
+require 'rake/extensiontask'
 require 'yaml'
 require 'yajl'
 require 'open-uri'
+require 'json'
 
 task :default => :test
 
 Rake::TestTask.new
 
+gem_spec = Gem::Specification.load('github-linguist.gemspec')
+
+Rake::ExtensionTask.new('linguist', gem_spec) do |ext|
+  ext.lib_dir = File.join('lib', 'linguist')
+end
+
 # Extend test task to check for samples and fetch latest Ace modes
-task :test => [:check_samples, :fetch_ace_modes]
+task :test => [:compile, :check_samples, :fetch_ace_modes]
 
 desc "Check that we have samples.json generated"
 task :check_samples do
@@ -33,15 +41,22 @@ task :fetch_ace_modes do
   end
 end
 
-task :samples do
+task :samples => :compile do
   require 'linguist/samples'
   json = Yajl.dump(Linguist::Samples.data, :pretty => true)
   File.write 'lib/linguist/samples.json', json
 end
 
+task :flex do
+  if `flex -V` !~ /^flex \d+\.\d+\.\d+/
+    fail "flex not detected"
+  end
+  system "cd ext/linguist && flex tokenizer.l"
+end
+
 task :build_gem => :samples do
   rm_rf "grammars"
-  sh "script/convert-grammars"
+  sh "script/grammar-compiler compile -o grammars || true"
   languages = YAML.load_file("lib/linguist/languages.yml")
   File.write("lib/linguist/languages.json", Yajl.dump(languages))
   `gem build github-linguist.gemspec`

bin/git-linguist

@@ -1,5 +1,7 @@
 #!/usr/bin/env ruby
 
+$LOAD_PATH[0, 0] = File.join(File.dirname(__FILE__), '..', 'lib')
+
 require 'linguist'
 require 'rugged'
 require 'optparse'
@@ -102,16 +104,21 @@ def git_linguist(args)
   commit = nil
 
   parser = OptionParser.new do |opts|
-    opts.banner = "Usage: git-linguist [OPTIONS] stats|breakdown|dump-cache|clear|disable"
+    opts.banner = <<-HELP
+    Linguist v#{Linguist::VERSION}
+    Detect language type and determine language breakdown for a given Git repository.
+
+    Usage:
+    git-linguist [OPTIONS] stats|breakdown|dump-cache|clear|disable"
+    HELP
 
     opts.on("-f", "--force", "Force a full rescan") { incremental = false }
-    opts.on("--commit=COMMIT", "Commit to index") { |v| commit = v}
+    opts.on("-c", "--commit=COMMIT", "Commit to index") { |v| commit = v}
   end
 
   parser.parse!(args)
-
   git_dir = `git rev-parse --git-dir`.strip
-  raise "git-linguist must be run in a Git repository (#{Dir.pwd})" unless $?.success?
+  raise "git-linguist must be run in a Git repository" unless $?.success?
   wrapper = GitLinguist.new(git_dir, commit, incremental)
 
   case args.pop
@@ -133,6 +140,10 @@ def git_linguist(args)
     $stderr.print(parser.help)
     exit 1
   end
+rescue Exception => e
+  $stderr.puts e.message
+  $stderr.puts e.backtrace
+  exit 1
 end
 
 git_linguist(ARGV)

bin/linguist

@@ -1,29 +1,37 @@
 #!/usr/bin/env ruby
 
-# linguist — detect language type for a file, or, given a directory, determine language breakdown
-#     usage: linguist <path> [<--breakdown>]
-#
+$LOAD_PATH[0, 0] = File.join(File.dirname(__FILE__), '..', 'lib')
+
 require 'linguist'
 require 'rugged'
+require 'json'
+require 'optparse'
 
 path = ARGV[0] || Dir.pwd
 
-# special case if not given a directory but still given the --breakdown option
+# special case if not given a directory
+# but still given the --breakdown or --json options/
 if path == "--breakdown"
   path = Dir.pwd
   breakdown = true
+elsif path == "--json"
+  path = Dir.pwd
+  json_breakdown = true
 end
 
 ARGV.shift
 breakdown = true if ARGV[0] == "--breakdown"
+json_breakdown = true if ARGV[0] == "--json"
 
 if File.directory?(path)
   rugged = Rugged::Repository.new(path)
   repo = Linguist::Repository.new(rugged, rugged.head.target_id)
-  repo.languages.sort_by { |_, size| size }.reverse.each do |language, size|
-    percentage = ((size / repo.size.to_f) * 100)
-    percentage = sprintf '%.2f' % percentage
-    puts "%-7s %s" % ["#{percentage}%", language]
+  if !json_breakdown
+    repo.languages.sort_by { |_, size| size }.reverse.each do |language, size|
+      percentage = ((size / repo.size.to_f) * 100)
+      percentage = sprintf '%.2f' % percentage
+      puts "%-7s %s" % ["#{percentage}%", language]
+    end
   end
   if breakdown
     puts
@@ -35,6 +43,8 @@ if File.directory?(path)
       end
       puts
     end
+  elsif json_breakdown
+    puts JSON.dump(repo.breakdown_by_file)
   end
 elsif File.file?(path)
   blob = Linguist::FileBlob.new(path, Dir.pwd)
@@ -63,5 +73,12 @@ elsif File.file?(path)
     puts "  appears to be a vendored file"
   end
 else
-  abort "usage: linguist <path>"
+  abort <<-HELP
+  Linguist v#{Linguist::VERSION}
+  Detect language type for a file, or, given a repository, determine language breakdown.
+
+  Usage: linguist <path>
+         linguist <path> [--breakdown] [--json]
+         linguist [--breakdown] [--json]
+  HELP
 end

ext/linguist/extconf.rb

@@ -0,0 +1,3 @@
+require 'mkmf'
+dir_config('linguist')
+create_makefile('linguist/linguist')

ext/linguist/lex.linguist_yy.h

@@ -0,0 +1,336 @@
+#ifndef linguist_yyHEADER_H
+#define linguist_yyHEADER_H 1
+#define linguist_yyIN_HEADER 1
+
+#line 6 "lex.linguist_yy.h"
+
+#define  YY_INT_ALIGNED short int
+
+/* A lexical scanner generated by flex */
+
+#define FLEX_SCANNER
+#define YY_FLEX_MAJOR_VERSION 2
+#define YY_FLEX_MINOR_VERSION 5
+#define YY_FLEX_SUBMINOR_VERSION 35
+#if YY_FLEX_SUBMINOR_VERSION > 0
+#define FLEX_BETA
+#endif
+
+/* First, we deal with  platform-specific or compiler-specific issues. */
+
+/* begin standard C headers. */
+#include <stdio.h>
+#include <string.h>
+#include <errno.h>
+#include <stdlib.h>
+
+/* end standard C headers. */
+
+/* flex integer type definitions */
+
+#ifndef FLEXINT_H
+#define FLEXINT_H
+
+/* C99 systems have <inttypes.h>. Non-C99 systems may or may not. */
+
+#if defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L
+
+/* C99 says to define __STDC_LIMIT_MACROS before including stdint.h,
+ * if you want the limit (max/min) macros for int types. 
+ */
+#ifndef __STDC_LIMIT_MACROS
+#define __STDC_LIMIT_MACROS 1
+#endif
+
+#include <inttypes.h>
+typedef int8_t flex_int8_t;
+typedef uint8_t flex_uint8_t;
+typedef int16_t flex_int16_t;
+typedef uint16_t flex_uint16_t;
+typedef int32_t flex_int32_t;
+typedef uint32_t flex_uint32_t;
+typedef uint64_t flex_uint64_t;
+#else
+typedef signed char flex_int8_t;
+typedef short int flex_int16_t;
+typedef int flex_int32_t;
+typedef unsigned char flex_uint8_t; 
+typedef unsigned short int flex_uint16_t;
+typedef unsigned int flex_uint32_t;
+#endif /* ! C99 */
+
+/* Limits of integral types. */
+#ifndef INT8_MIN
+#define INT8_MIN               (-128)
+#endif
+#ifndef INT16_MIN
+#define INT16_MIN              (-32767-1)
+#endif
+#ifndef INT32_MIN
+#define INT32_MIN              (-2147483647-1)
+#endif
+#ifndef INT8_MAX
+#define INT8_MAX               (127)
+#endif
+#ifndef INT16_MAX
+#define INT16_MAX              (32767)
+#endif
+#ifndef INT32_MAX
+#define INT32_MAX              (2147483647)
+#endif
+#ifndef UINT8_MAX
+#define UINT8_MAX              (255U)
+#endif
+#ifndef UINT16_MAX
+#define UINT16_MAX             (65535U)
+#endif
+#ifndef UINT32_MAX
+#define UINT32_MAX             (4294967295U)
+#endif
+
+#endif /* ! FLEXINT_H */
+
+#ifdef __cplusplus
+
+/* The "const" storage-class-modifier is valid. */
+#define YY_USE_CONST
+
+#else	/* ! __cplusplus */
+
+/* C99 requires __STDC__ to be defined as 1. */
+#if defined (__STDC__)
+
+#define YY_USE_CONST
+
+#endif	/* defined (__STDC__) */
+#endif	/* ! __cplusplus */
+
+#ifdef YY_USE_CONST
+#define yyconst const
+#else
+#define yyconst
+#endif
+
+/* An opaque pointer. */
+#ifndef YY_TYPEDEF_YY_SCANNER_T
+#define YY_TYPEDEF_YY_SCANNER_T
+typedef void* yyscan_t;
+#endif
+
+/* For convenience, these vars (plus the bison vars far below)
+   are macros in the reentrant scanner. */
+#define yyin yyg->yyin_r
+#define yyout yyg->yyout_r
+#define yyextra yyg->yyextra_r
+#define yyleng yyg->yyleng_r
+#define yytext yyg->yytext_r
+#define yylineno (YY_CURRENT_BUFFER_LVALUE->yy_bs_lineno)
+#define yycolumn (YY_CURRENT_BUFFER_LVALUE->yy_bs_column)
+#define yy_flex_debug yyg->yy_flex_debug_r
+
+/* Size of default input buffer. */
+#ifndef YY_BUF_SIZE
+#define YY_BUF_SIZE 16384
+#endif
+
+#ifndef YY_TYPEDEF_YY_BUFFER_STATE
+#define YY_TYPEDEF_YY_BUFFER_STATE
+typedef struct yy_buffer_state *YY_BUFFER_STATE;
+#endif
+
+#ifndef YY_TYPEDEF_YY_SIZE_T
+#define YY_TYPEDEF_YY_SIZE_T
+typedef size_t yy_size_t;
+#endif
+
+#ifndef YY_STRUCT_YY_BUFFER_STATE
+#define YY_STRUCT_YY_BUFFER_STATE
+struct yy_buffer_state
+	{
+	FILE *yy_input_file;
+
+	char *yy_ch_buf;		/* input buffer */
+	char *yy_buf_pos;		/* current position in input buffer */
+
+	/* Size of input buffer in bytes, not including room for EOB
+	 * characters.
+	 */
+	yy_size_t yy_buf_size;
+
+	/* Number of characters read into yy_ch_buf, not including EOB
+	 * characters.
+	 */
+	yy_size_t yy_n_chars;
+
+	/* Whether we "own" the buffer - i.e., we know we created it,
+	 * and can realloc() it to grow it, and should free() it to
+	 * delete it.
+	 */
+	int yy_is_our_buffer;
+
+	/* Whether this is an "interactive" input source; if so, and
+	 * if we're using stdio for input, then we want to use getc()
+	 * instead of fread(), to make sure we stop fetching input after
+	 * each newline.
+	 */
+	int yy_is_interactive;
+
+	/* Whether we're considered to be at the beginning of a line.
+	 * If so, '^' rules will be active on the next match, otherwise
+	 * not.
+	 */
+	int yy_at_bol;
+
+    int yy_bs_lineno; /**< The line count. */
+    int yy_bs_column; /**< The column count. */
+    
+	/* Whether to try to fill the input buffer when we reach the
+	 * end of it.
+	 */
+	int yy_fill_buffer;
+
+	int yy_buffer_status;
+
+	};
+#endif /* !YY_STRUCT_YY_BUFFER_STATE */
+
+void linguist_yyrestart (FILE *input_file ,yyscan_t yyscanner );
+void linguist_yy_switch_to_buffer (YY_BUFFER_STATE new_buffer ,yyscan_t yyscanner );
+YY_BUFFER_STATE linguist_yy_create_buffer (FILE *file,int size ,yyscan_t yyscanner );
+void linguist_yy_delete_buffer (YY_BUFFER_STATE b ,yyscan_t yyscanner );
+void linguist_yy_flush_buffer (YY_BUFFER_STATE b ,yyscan_t yyscanner );
+void linguist_yypush_buffer_state (YY_BUFFER_STATE new_buffer ,yyscan_t yyscanner );
+void linguist_yypop_buffer_state (yyscan_t yyscanner );
+
+YY_BUFFER_STATE linguist_yy_scan_buffer (char *base,yy_size_t size ,yyscan_t yyscanner );
+YY_BUFFER_STATE linguist_yy_scan_string (yyconst char *yy_str ,yyscan_t yyscanner );
+YY_BUFFER_STATE linguist_yy_scan_bytes (yyconst char *bytes,yy_size_t len ,yyscan_t yyscanner );
+
+void *linguist_yyalloc (yy_size_t ,yyscan_t yyscanner );
+void *linguist_yyrealloc (void *,yy_size_t ,yyscan_t yyscanner );
+void linguist_yyfree (void * ,yyscan_t yyscanner );
+
+/* Begin user sect3 */
+
+#define yytext_ptr yytext_r
+
+#ifdef YY_HEADER_EXPORT_START_CONDITIONS
+#define INITIAL 0
+#define sgml 1
+#define c_comment 2
+#define xml_comment 3
+#define haskell_comment 4
+#define ocaml_comment 5
+#define python_dcomment 6
+#define python_scomment 7
+
+#endif
+
+#ifndef YY_NO_UNISTD_H
+/* Special case for "unistd.h", since it is non-ANSI. We include it way
+ * down here because we want the user's section 1 to have been scanned first.
+ * The user has a chance to override it with an option.
+ */
+#include <unistd.h>
+#endif
+
+#define YY_EXTRA_TYPE struct tokenizer_extra *
+
+int linguist_yylex_init (yyscan_t* scanner);
+
+int linguist_yylex_init_extra (YY_EXTRA_TYPE user_defined,yyscan_t* scanner);
+
+/* Accessor methods to globals.
+   These are made visible to non-reentrant scanners for convenience. */
+
+int linguist_yylex_destroy (yyscan_t yyscanner );
+
+int linguist_yyget_debug (yyscan_t yyscanner );
+
+void linguist_yyset_debug (int debug_flag ,yyscan_t yyscanner );
+
+YY_EXTRA_TYPE linguist_yyget_extra (yyscan_t yyscanner );
+
+void linguist_yyset_extra (YY_EXTRA_TYPE user_defined ,yyscan_t yyscanner );
+
+FILE *linguist_yyget_in (yyscan_t yyscanner );
+
+void linguist_yyset_in  (FILE * in_str ,yyscan_t yyscanner );
+
+FILE *linguist_yyget_out (yyscan_t yyscanner );
+
+void linguist_yyset_out  (FILE * out_str ,yyscan_t yyscanner );
+
+yy_size_t linguist_yyget_leng (yyscan_t yyscanner );
+
+char *linguist_yyget_text (yyscan_t yyscanner );
+
+int linguist_yyget_lineno (yyscan_t yyscanner );
+
+void linguist_yyset_lineno (int line_number ,yyscan_t yyscanner );
+
+/* Macros after this point can all be overridden by user definitions in
+ * section 1.
+ */
+
+#ifndef YY_SKIP_YYWRAP
+#ifdef __cplusplus
+extern "C" int linguist_yywrap (yyscan_t yyscanner );
+#else
+extern int linguist_yywrap (yyscan_t yyscanner );
+#endif
+#endif
+
+#ifndef yytext_ptr
+static void yy_flex_strncpy (char *,yyconst char *,int ,yyscan_t yyscanner);
+#endif
+
+#ifdef YY_NEED_STRLEN
+static int yy_flex_strlen (yyconst char * ,yyscan_t yyscanner);
+#endif
+
+#ifndef YY_NO_INPUT
+
+#endif
+
+/* Amount of stuff to slurp up with each read. */
+#ifndef YY_READ_BUF_SIZE
+#define YY_READ_BUF_SIZE 8192
+#endif
+
+/* Number of entries by which start-condition stack grows. */
+#ifndef YY_START_STACK_INCR
+#define YY_START_STACK_INCR 25
+#endif
+
+/* Default declaration of generated scanner - a define so the user can
+ * easily add parameters.
+ */
+#ifndef YY_DECL
+#define YY_DECL_IS_OURS 1
+
+extern int linguist_yylex (yyscan_t yyscanner);
+
+#define YY_DECL int linguist_yylex (yyscan_t yyscanner)
+#endif /* !YY_DECL */
+
+/* yy_get_previous_state - get the state just before the EOB char was reached */
+
+#undef YY_NEW_FILE
+#undef YY_FLUSH_BUFFER
+#undef yy_set_bol
+#undef yy_new_buffer
+#undef yy_set_interactive
+#undef YY_DO_BEFORE_ACTION
+
+#ifdef YY_DECL_IS_OURS
+#undef YY_DECL_IS_OURS
+#undef YY_DECL
+#endif
+
+#line 118 "tokenizer.l"
+
+
+#line 335 "lex.linguist_yy.h"
+#undef linguist_yyIN_HEADER
+#endif /* linguist_yyHEADER_H */

ext/linguist/linguist.c

@@ -0,0 +1,75 @@
+#include "ruby.h"
+#include "linguist.h"
+#include "lex.linguist_yy.h"
+
+// Anything longer is unlikely to be useful.
+#define MAX_TOKEN_LEN 32
+
+int linguist_yywrap(yyscan_t yyscanner) {
+	return 1;
+}
+
+static VALUE rb_tokenizer_extract_tokens(VALUE self, VALUE rb_data) {
+	YY_BUFFER_STATE buf;
+	yyscan_t scanner;
+	struct tokenizer_extra extra;
+	VALUE ary, s;
+	long len;
+	int r;
+
+	Check_Type(rb_data, T_STRING);
+
+	len = RSTRING_LEN(rb_data);
+	if (len > 100000)
+		len = 100000;
+
+	linguist_yylex_init_extra(&extra, &scanner);
+	buf = linguist_yy_scan_bytes(RSTRING_PTR(rb_data), (int) len, scanner);
+
+	ary = rb_ary_new();
+	do {
+		extra.type = NO_ACTION;
+		extra.token = NULL;
+		r = linguist_yylex(scanner);
+		switch (extra.type) {
+		case NO_ACTION:
+			break;
+		case REGULAR_TOKEN:
+			len = strlen(extra.token);
+			if (len <= MAX_TOKEN_LEN)
+				rb_ary_push(ary, rb_str_new(extra.token, len));
+			free(extra.token);
+			break;
+		case SHEBANG_TOKEN:
+			len = strlen(extra.token);
+			if (len <= MAX_TOKEN_LEN) {
+				s = rb_str_new2("SHEBANG#!");
+				rb_str_cat(s, extra.token, len);
+				rb_ary_push(ary, s);
+			}
+			free(extra.token);
+			break;
+		case SGML_TOKEN:
+			len = strlen(extra.token);
+			if (len <= MAX_TOKEN_LEN) {
+				s = rb_str_new(extra.token, len);
+				rb_str_cat2(s, ">");
+				rb_ary_push(ary, s);
+			}
+			free(extra.token);
+			break;
+		}
+	} while (r);
+
+	linguist_yy_delete_buffer(buf, scanner);
+	linguist_yylex_destroy(scanner);
+
+	return ary;
+}
+
+__attribute__((visibility("default"))) void Init_linguist() {
+	VALUE rb_mLinguist = rb_define_module("Linguist");
+	VALUE rb_cTokenizer = rb_define_class_under(rb_mLinguist, "Tokenizer", rb_cObject);
+
+	rb_define_method(rb_cTokenizer, "extract_tokens", rb_tokenizer_extract_tokens, 1);
+}

ext/linguist/linguist.h

@@ -0,0 +1,11 @@
+enum tokenizer_type {
+  NO_ACTION,
+  REGULAR_TOKEN,
+  SHEBANG_TOKEN,
+  SGML_TOKEN,
+};
+
+struct tokenizer_extra {
+  char *token;
+  enum tokenizer_type type;
+};

ext/linguist/tokenizer.l

@@ -0,0 +1,119 @@
+%{
+
+#include "linguist.h"
+
+#define feed_token(tok, typ) do { \
+    yyextra->token = (tok); \
+    yyextra->type = (typ); \
+  } while (0)
+
+#define eat_until_eol() do { \
+    int c; \
+    while ((c = input(yyscanner)) != '\n' && c != EOF && c); \
+    if (c == EOF || !c) \
+      return 0; \
+  } while (0)
+
+#define eat_until_unescaped(q) do { \
+    int c; \
+    while ((c = input(yyscanner)) != EOF && c) { \
+      if (c == '\n') \
+        break; \
+      if (c == '\\') { \
+        c = input(yyscanner); \
+        if (c == EOF || !c) \
+          return 0; \
+      } else if (c == q) \
+        break; \
+    } \
+    if (c == EOF || !c) \
+      return 0; \
+  } while (0)
+
+%}
+
+%option never-interactive yywrap reentrant nounput warn nodefault header-file="lex.linguist_yy.h" extra-type="struct tokenizer_extra *" prefix="linguist_yy"
+%x sgml c_comment xml_comment haskell_comment ocaml_comment python_dcomment python_scomment
+
+%%
+
+^#![ \t]*([[:alnum:]_\/]*\/)?env([ \t]+([^ \t=]*=[^ \t]*))*[ \t]+[[:alpha:]_]+ {
+    const char *off = strrchr(yytext, ' ');
+    if (!off)
+      off = yytext;
+    else
+      ++off;
+    feed_token(strdup(off), SHEBANG_TOKEN);
+    eat_until_eol();
+    return 1;
+  }
+
+^#![ \t]*[[:alpha:]_\/]+ {
+    const char *off = strrchr(yytext, '/');
+    if (!off)
+      off = yytext;
+    else
+      ++off;
+    if (strcmp(off, "env") == 0) {
+      eat_until_eol();
+    } else {
+      feed_token(strdup(off), SHEBANG_TOKEN);
+      eat_until_eol();
+      return 1;
+    }
+  }
+
+^[ \t]*(\/\/|--|\#|%|\")" ".*   { /* nothing */ }
+
+"/*"                              { BEGIN(c_comment); }
+  /* See below for xml_comment start. */
+"{-"                              { BEGIN(haskell_comment); }
+"(*"                              { BEGIN(ocaml_comment); }
+"\"\"\""                          { BEGIN(python_dcomment); }
+"'''"                             { BEGIN(python_scomment); }
+
+<c_comment,xml_comment,haskell_comment,ocaml_comment,python_dcomment,python_scomment>.|\n { /* nothing */ }
+<c_comment>"*/"                   { BEGIN(INITIAL); }
+<xml_comment>"-->"                { BEGIN(INITIAL); }
+<haskell_comment>"-}"             { BEGIN(INITIAL); }
+<ocaml_comment>"*)"               { BEGIN(INITIAL); }
+<python_dcomment>"\"\"\""         { BEGIN(INITIAL); }
+<python_scomment>"'''"            { BEGIN(INITIAL); }
+
+\"\"|''                           { /* nothing */ }
+\"                                { eat_until_unescaped('"'); }
+'                                 { eat_until_unescaped('\''); }
+(0x[0-9a-fA-F]([0-9a-fA-F]|\.)*|[0-9]([0-9]|\.)*)([uU][lL]{0,2}|([eE][-+][0-9]*)?[fFlL]*) { /* nothing */ }
+\<[[:alnum:]_!./?-]+              {
+    if (strcmp(yytext, "<!--") == 0) {
+     BEGIN(xml_comment);
+    } else {
+      feed_token(strdup(yytext), SGML_TOKEN);
+      BEGIN(sgml);
+      return 1;
+    }
+  }
+<sgml>[[:alnum:]_]+=\"            { feed_token(strndup(yytext, strlen(yytext) - 1), REGULAR_TOKEN); eat_until_unescaped('"'); return 1; }
+<sgml>[[:alnum:]_]+='             { feed_token(strndup(yytext, strlen(yytext) - 1), REGULAR_TOKEN); eat_until_unescaped('\''); return 1; }
+<sgml>[[:alnum:]_]+=[[:alnum:]_]* { feed_token(strdup(yytext), REGULAR_TOKEN); *(strchr(yyextra->token, '=') + 1) = 0; return 1; }
+<sgml>[[:alnum:]_]+               { feed_token(strdup(yytext), REGULAR_TOKEN); return 1; }
+<sgml>\>                          { BEGIN(INITIAL); }
+<sgml>.|\n                        { /* nothing */ }
+;|\{|\}|\(|\)|\[|\]               { feed_token(strdup(yytext), REGULAR_TOKEN); return 1; }
+[[:alnum:]_.@#/*]+                {
+    if (strncmp(yytext, "/*", 2) == 0) {
+      if (strlen(yytext) >= 4 && strcmp(yytext + strlen(yytext) - 2, "*/") == 0) {
+        /* nothing */
+      } else {
+        BEGIN(c_comment);
+      }
+    } else {
+      feed_token(strdup(yytext), REGULAR_TOKEN);
+      return 1;
+    }
+  }
+\<\<?|\+|\-|\*|\/|%|&&?|\|\|?     { feed_token(strdup(yytext), REGULAR_TOKEN); return 1; }
+.|\n                              { /* nothing */ }
+
+%%
+

github-linguist.gemspec

@@ -2,7 +2,7 @@ require File.expand_path('../lib/linguist/version', __FILE__)
 
 Gem::Specification.new do |s|
   s.name    = 'github-linguist'
-  s.version = Linguist::VERSION
+  s.version = ENV['GEM_VERSION'] || Linguist::VERSION
   s.summary = "GitHub Language detection"
   s.description = 'We use this library at GitHub to detect blob languages, highlight code, ignore binary files, suppress generated files in diffs, and generate language breakdown graphs.'
 
@@ -10,22 +10,23 @@ Gem::Specification.new do |s|
   s.homepage = "https://github.com/github/linguist"
   s.license  = "MIT"
 
-  s.files = Dir['lib/**/*'] + Dir['grammars/*'] + ['LICENSE']
+  s.files = Dir['lib/**/*'] + Dir['ext/**/*'] + Dir['grammars/*'] + ['LICENSE']
   s.executables = ['linguist', 'git-linguist']
+  s.extensions = ['ext/linguist/extconf.rb']
 
-  s.add_dependency 'charlock_holmes', '~> 0.7.3'
-  s.add_dependency 'escape_utils',    '~> 1.1.0'
+  s.add_dependency 'charlock_holmes', '~> 0.7.6'
+  s.add_dependency 'escape_utils',    '~> 1.2.0'
   s.add_dependency 'mime-types',      '>= 1.19'
-  s.add_dependency 'rugged',          '>= 0.23.0b'
+  s.add_dependency 'rugged',          '>= 0.25.1'
 
   s.add_development_dependency 'minitest', '>= 5.0'
+  s.add_development_dependency 'rake-compiler', '~> 0.9'
   s.add_development_dependency 'mocha'
   s.add_development_dependency 'plist', '~>3.1'
   s.add_development_dependency 'pry'
   s.add_development_dependency 'rake'
   s.add_development_dependency 'yajl-ruby'
   s.add_development_dependency 'color-proximity', '~> 0.2.1'
-  s.add_development_dependency 'licensed'
-  s.add_development_dependency 'licensee', '>= 8.3.0'
-
+  s.add_development_dependency 'licensed', '~> 1.0.0'
+  s.add_development_dependency 'licensee'
 end

grammars.yml

@@ -1,13 +1,16 @@
----
-http://svn.edgewall.org/repos/genshi/contrib/textmate/Genshi.tmbundle/Syntaxes/Markup%20Template%20%28XML%29.tmLanguage:
-- text.xml.genshi
 https://bitbucket.org/Clams/sublimesystemverilog/get/default.tar.gz:
 - source.systemverilog
 - source.ucfconstraints
+https://svn.edgewall.org/repos/genshi/contrib/textmate/Genshi.tmbundle/Syntaxes/Markup%20Template%20%28XML%29.tmLanguage:
+- text.xml.genshi
+vendor/grammars/ABNF.tmbundle:
+- source.abnf
 vendor/grammars/Agda.tmbundle:
 - source.agda
 vendor/grammars/Alloy.tmbundle:
 - source.alloy
+vendor/grammars/Assembly-Syntax-Definition:
+- source.assembly.unix
 vendor/grammars/AutoHotkey:
 - source.ahk
 vendor/grammars/BrightScript.tmbundle:
@@ -20,6 +23,8 @@ vendor/grammars/ColdFusion:
 - text.html.cfm
 vendor/grammars/Docker.tmbundle:
 - source.dockerfile
+vendor/grammars/EBNF.tmbundle:
+- source.ebnf
 vendor/grammars/Elm/Syntaxes:
 - source.elm
 - text.html.mediawiki.elm-build-output
@@ -41,15 +46,19 @@ vendor/grammars/Isabelle.tmbundle:
 - source.isabelle.theory
 vendor/grammars/JSyntax:
 - source.j
-vendor/grammars/Julia.tmbundle:
-- source.julia
 vendor/grammars/Lean.tmbundle:
 - source.lean
 vendor/grammars/LiveScript.tmbundle:
 - source.livescript
+vendor/grammars/MATLAB-Language-grammar:
+- source.matlab
+vendor/grammars/MQL5-sublime:
+- source.mql5
 vendor/grammars/MagicPython:
 - source.python
 - source.regexp.python
+- text.python.console
+- text.python.traceback
 vendor/grammars/Modelica:
 - source.modelica
 vendor/grammars/NSIS:
@@ -92,6 +101,8 @@ vendor/grammars/Sublime-Modula-2:
 - source.modula2
 vendor/grammars/Sublime-Nit:
 - source.nit
+vendor/grammars/Sublime-Pep8/:
+- source.pep8
 vendor/grammars/Sublime-QML:
 - source.qml
 vendor/grammars/Sublime-REBOL:
@@ -107,7 +118,9 @@ vendor/grammars/SublimeBrainfuck:
 - source.bf
 vendor/grammars/SublimeClarion:
 - source.clarion
-vendor/grammars/SublimeGDB:
+vendor/grammars/SublimeEthereum:
+- source.solidity
+vendor/grammars/SublimeGDB/:
 - source.disasm
 - source.gdb
 - source.gdb.session
@@ -118,12 +131,22 @@ vendor/grammars/SublimePuppet:
 - source.puppet
 vendor/grammars/SublimeXtend:
 - source.xtend
+vendor/grammars/Syntax-highlighting-for-PostCSS:
+- source.css.postcss.sugarss
+- source.postcss
 vendor/grammars/TLA:
 - source.tla
 vendor/grammars/TXL:
 - source.txl
+vendor/grammars/Terraform.tmLanguage:
+- source.terraform
 vendor/grammars/Textmate-Gosu-Bundle:
 - source.gosu.2
+vendor/grammars/TypeScript-TmLanguage:
+- source.ts
+- source.tsx
+- text.error-list
+- text.find-refs
 vendor/grammars/UrWeb-Language-Definition:
 - source.ur
 vendor/grammars/VBDotNetSyntax:
@@ -134,7 +157,7 @@ vendor/grammars/X10:
 - source.x10
 vendor/grammars/abap.tmbundle:
 - source.abap
-vendor/grammars/actionscript3-tmbundle:
+vendor/grammars/actionscript3-tmbundle/:
 - source.actionscript.3
 - text.html.asdoc
 - text.xml.flex-config
@@ -172,8 +195,24 @@ vendor/grammars/atom-language-1c-bsl:
 - source.sdbl
 vendor/grammars/atom-language-clean:
 - source.clean
+- text.restructuredtext.clean
+vendor/grammars/atom-language-julia:
+- source.julia
+- source.julia.console
+vendor/grammars/atom-language-nextflow:
+- source.nextflow
+- source.nextflow-groovy
+vendor/grammars/atom-language-p4:
+- source.p4
+vendor/grammars/atom-language-perl6:
+- source.meta-info
+- source.perl6fe
+- source.quoting.perl6fe
+- source.regexp.perl6fe
 vendor/grammars/atom-language-purescript:
 - source.purescript
+vendor/grammars/atom-language-rust:
+- source.rust
 vendor/grammars/atom-language-srt:
 - text.srt
 vendor/grammars/atom-language-stan:
@@ -205,13 +244,14 @@ vendor/grammars/capnproto.tmbundle:
 vendor/grammars/carto-atom:
 - source.css.mss
 vendor/grammars/ceylon-sublimetext:
-- module.ceylon
 - source.ceylon
 vendor/grammars/chapel-tmbundle:
 - source.chapel
 vendor/grammars/cmake.tmbundle:
 - source.cache.cmake
 - source.cmake
+vendor/grammars/conllu-linguist-grammar:
+- text.conllu
 vendor/grammars/cool-tmbundle:
 - source.cool
 vendor/grammars/cpp-qt.tmbundle:
@@ -219,8 +259,6 @@ vendor/grammars/cpp-qt.tmbundle:
 - source.qmake
 vendor/grammars/creole:
 - text.html.creole
-vendor/grammars/css.tmbundle:
-- source.css
 vendor/grammars/cucumber-tmbundle:
 - source.ruby.rspec.cucumber.steps
 - text.gherkin.feature
@@ -228,11 +266,11 @@ vendor/grammars/cython:
 - source.cython
 vendor/grammars/d.tmbundle:
 - source.d
-vendor/grammars/dart-sublime-bundle:
+vendor/grammars/dartlang:
 - source.dart
-- source.pubspec
-- text.dart-analysis-output
-- text.dart-doccomments
+- source.yaml-ext
+vendor/grammars/data-weave-tmLanguage:
+- source.data-weave
 vendor/grammars/desktop.tmbundle:
 - source.desktop
 vendor/grammars/diff.tmbundle:
@@ -284,12 +322,9 @@ vendor/grammars/graphviz.tmbundle:
 - source.dot
 vendor/grammars/groovy.tmbundle:
 - source.groovy
-vendor/grammars/haxe-sublime-bundle:
-- source.erazor
-- source.haxe.2
-- source.hss.1
+vendor/grammars/haxe-TmLanguage:
+- source.hx
 - source.hxml
-- source.nmml
 vendor/grammars/html.tmbundle:
 - text.html.basic
 vendor/grammars/idl.tmbundle:
@@ -314,14 +349,14 @@ vendor/grammars/java.tmbundle:
 - source.java-properties
 - text.html.jsp
 - text.junit-test-report
+vendor/grammars/javadoc.tmbundle:
+- text.html.javadoc
 vendor/grammars/javascript-objective-j.tmbundle:
 - source.js.objj
 vendor/grammars/jflex.tmbundle:
 - source.jflex
 vendor/grammars/json.tmbundle:
 - source.json
-vendor/grammars/kotlin-sublime-package:
-- source.Kotlin
 vendor/grammars/language-agc:
 - source.agc
 vendor/grammars/language-apl:
@@ -331,6 +366,8 @@ vendor/grammars/language-asn1:
 vendor/grammars/language-babel:
 - source.js.jsx
 - source.regexp.babel
+vendor/grammars/language-ballerina:
+- source.ballerina
 vendor/grammars/language-batchfile:
 - source.batchfile
 vendor/grammars/language-blade:
@@ -339,6 +376,8 @@ vendor/grammars/language-click:
 - source.click
 vendor/grammars/language-clojure:
 - source.clojure
+vendor/grammars/language-closure-templates:
+- text.html.soy
 vendor/grammars/language-coffee-script:
 - source.coffee
 - source.litcoffee
@@ -349,18 +388,32 @@ vendor/grammars/language-csharp:
 - source.cake
 - source.cs
 - source.csx
-- source.nant-build
 vendor/grammars/language-csound:
 - source.csound
 - source.csound-document
 - source.csound-score
+vendor/grammars/language-css:
+- source.css
+vendor/grammars/language-cwl:
+- source.cwl
 vendor/grammars/language-emacs-lisp:
 - source.emacs.lisp
+vendor/grammars/language-fontforge:
+- source.afm
+- source.fontforge
+- source.opentype
+- text.sfd
 vendor/grammars/language-gfm:
 - source.gfm
+vendor/grammars/language-gn:
+- source.gn
 vendor/grammars/language-graphql:
 - source.graphql
+vendor/grammars/language-haml:
+- text.haml
+- text.hamlc
 vendor/grammars/language-haskell:
+- annotation.liquidhaskell.haskell
 - hint.haskell
 - hint.message.haskell
 - hint.type.haskell
@@ -368,34 +421,60 @@ vendor/grammars/language-haskell:
 - source.cabal
 - source.haskell
 - source.hsc2hs
+- source.hsig
 - text.tex.latex.haskell
-vendor/grammars/language-hy:
-- source.hy
 vendor/grammars/language-inform7:
 - source.inform7
 vendor/grammars/language-javascript:
 - source.js
 - source.js.regexp
 - source.js.regexp.replacement
+- source.jsdoc
+vendor/grammars/language-jison:
+- source.jison
+- source.jisonlex
+- source.jisonlex-injection
+vendor/grammars/language-jolie:
+- source.jolie
 vendor/grammars/language-jsoniq:
 - source.jq
 - source.xq
+vendor/grammars/language-kotlin:
+- source.kotlin
 vendor/grammars/language-less:
 - source.css.less
 vendor/grammars/language-maxscript:
 - source.maxscript
+vendor/grammars/language-meson:
+- source.meson
 vendor/grammars/language-ncl:
 - source.ncl
+vendor/grammars/language-ninja:
+- source.ninja
+vendor/grammars/language-pan:
+- source.pan
+vendor/grammars/language-pcb:
+- source.gerber
+- source.pcb.board
+- source.pcb.schematic
+- source.pcb.sexp
 vendor/grammars/language-povray:
 - source.pov-ray sdl
-vendor/grammars/language-python:
-- text.python.console
-- text.python.traceback
+vendor/grammars/language-reason:
+- source.reason
+- source.reason.hover.type
+vendor/grammars/language-regexp:
+- source.regexp
+- source.regexp.extended
 vendor/grammars/language-renpy:
 - source.renpy
 vendor/grammars/language-restructuredtext:
 - text.restructuredtext
+vendor/grammars/language-ring:
+- source.ring
 vendor/grammars/language-roff:
+- source.ditroff
+- source.ditroff.desc
 - source.ideal
 - source.pic
 - text.roff
@@ -403,6 +482,12 @@ vendor/grammars/language-roff:
 vendor/grammars/language-rpm-spec:
 - source.changelogs.rpm-spec
 - source.rpm-spec
+vendor/grammars/language-ruby:
+- source.ruby
+- source.ruby.gemfile
+- text.html.erb
+vendor/grammars/language-sed:
+- source.sed
 vendor/grammars/language-shellscript:
 - source.shell
 - text.shell-session
@@ -412,17 +497,25 @@ vendor/grammars/language-toc-wow:
 - source.toc
 vendor/grammars/language-turing:
 - source.turing
+vendor/grammars/language-typelanguage:
+- source.tl
 vendor/grammars/language-viml:
 - source.viml
 vendor/grammars/language-wavefront:
 - source.wavefront.mtl
 - source.wavefront.obj
+vendor/grammars/language-webassembly:
+- source.webassembly
 vendor/grammars/language-xbase:
 - source.harbour
+vendor/grammars/language-xcompose:
+- config.xcompose
 vendor/grammars/language-yaml:
 - source.yaml
 vendor/grammars/language-yang:
 - source.yang
+vendor/grammars/language-yara:
+- source.yara
 vendor/grammars/latex.tmbundle:
 - text.bibtex
 - text.log.latex
@@ -448,11 +541,10 @@ vendor/grammars/make.tmbundle:
 - source.makefile
 vendor/grammars/mako-tmbundle:
 - text.html.mako
+vendor/grammars/marko-tmbundle:
+- text.marko
 vendor/grammars/mathematica-tmbundle:
 - source.mathematica
-vendor/grammars/matlab.tmbundle:
-- source.matlab
-- source.octave
 vendor/grammars/maven.tmbundle:
 - text.xml.pom
 vendor/grammars/mediawiki.tmbundle:
@@ -467,8 +559,6 @@ vendor/grammars/nemerle.tmbundle:
 - source.nemerle
 vendor/grammars/nesC:
 - source.nesc
-vendor/grammars/ninja.tmbundle:
-- source.ninja
 vendor/grammars/nix:
 - source.nix
 vendor/grammars/nu.tmbundle:
@@ -487,7 +577,9 @@ vendor/grammars/ooc.tmbundle:
 - source.ooc
 vendor/grammars/opa.tmbundle:
 - source.opa
-vendor/grammars/oz-tmbundle/Syntaxes/Oz.tmLanguage:
+vendor/grammars/openscad.tmbundle:
+- source.scad
+vendor/grammars/oz-tmbundle:
 - source.oz
 vendor/grammars/parrot:
 - source.parrot.pir
@@ -498,10 +590,6 @@ vendor/grammars/pawn-sublime-language:
 vendor/grammars/perl.tmbundle:
 - source.perl
 - source.perl.6
-vendor/grammars/perl6fe:
-- source.meta-info
-- source.perl6fe
-- source.regexp.perl6fe
 vendor/grammars/php-smarty.tmbundle:
 - text.html.smarty
 vendor/grammars/php.tmbundle:
@@ -524,13 +612,10 @@ vendor/grammars/python-django.tmbundle:
 vendor/grammars/r.tmbundle:
 - source.r
 - text.tex.latex.rd
-vendor/grammars/ruby-haml.tmbundle:
-- text.haml
+vendor/grammars/rascal-syntax-highlighting:
+- source.rascal
 vendor/grammars/ruby-slim.tmbundle:
 - text.slim
-vendor/grammars/ruby.tmbundle:
-- source.ruby
-- text.html.erb
 vendor/grammars/sas.tmbundle:
 - source.SASLog
 - source.sas
@@ -538,13 +623,15 @@ vendor/grammars/sass-textmate-bundle:
 - source.sass
 vendor/grammars/scala.tmbundle:
 - source.sbt
-- source.scala
 vendor/grammars/scheme.tmbundle:
 - source.scheme
 vendor/grammars/scilab.tmbundle:
 - source.scilab
 vendor/grammars/secondlife-lsl:
 - source.lsl
+vendor/grammars/shaders-tmLanguage:
+- source.hlsl
+- source.shaderlab
 vendor/grammars/smali-sublime:
 - source.smali
 vendor/grammars/smalltalk-tmbundle:
@@ -553,6 +640,8 @@ vendor/grammars/sourcepawn:
 - source.sp
 vendor/grammars/sql.tmbundle:
 - source.sql
+vendor/grammars/squirrel-language:
+- source.nut
 vendor/grammars/st2-zonefile:
 - text.zone_file
 vendor/grammars/standard-ml.tmbundle:
@@ -560,6 +649,8 @@ vendor/grammars/standard-ml.tmbundle:
 - source.ml
 vendor/grammars/sublime-MuPAD:
 - source.mupad
+vendor/grammars/sublime-angelscript:
+- source.angelscript
 vendor/grammars/sublime-aspectj:
 - source.aspectj
 vendor/grammars/sublime-autoit:
@@ -572,6 +663,8 @@ vendor/grammars/sublime-cirru:
 - source.cirru
 vendor/grammars/sublime-clips:
 - source.clips
+vendor/grammars/sublime-fantom:
+- source.fan
 vendor/grammars/sublime-glsl:
 - source.essl
 - source.glsl
@@ -579,6 +672,8 @@ vendor/grammars/sublime-golo:
 - source.golo
 vendor/grammars/sublime-mask:
 - source.mask
+vendor/grammars/sublime-nearley:
+- source.ne
 vendor/grammars/sublime-netlinx:
 - source.netlinx
 - source.netlinx.erb
@@ -593,8 +688,8 @@ vendor/grammars/sublime-rexx:
 - source.rexx
 vendor/grammars/sublime-robot-plugin:
 - text.robot
-vendor/grammars/sublime-rust:
-- source.rust
+vendor/grammars/sublime-shen:
+- source.shen
 vendor/grammars/sublime-spintools:
 - source.regexp.spin
 - source.spin
@@ -604,11 +699,6 @@ vendor/grammars/sublime-terra:
 - source.terra
 vendor/grammars/sublime-text-ox:
 - source.ox
-vendor/grammars/sublime-typescript:
-- source.ts
-- source.tsx
-- text.error-list
-- text.find-refs
 vendor/grammars/sublime-varnish:
 - source.varnish.vcl
 vendor/grammars/sublime_cobol:
@@ -639,14 +729,16 @@ vendor/grammars/verilog.tmbundle:
 - source.verilog
 vendor/grammars/vhdl:
 - source.vhdl
+vendor/grammars/vscode-scala-syntax:
+- source.scala
 vendor/grammars/vue-syntax-highlight:
 - text.html.vue
+vendor/grammars/wdl-sublime-syntax-highlighter:
+- source.wdl
 vendor/grammars/xc.tmbundle:
 - source.xc
 vendor/grammars/xml.tmbundle:
 - text.xml
 - text.xml.xsl
-vendor/grammars/xquery:
-- source.xquery
 vendor/grammars/zephir-sublime:
 - source.php.zephir

lib/linguist.rb

@@ -15,9 +15,9 @@ class << Linguist
   #       see Linguist::LazyBlob and Linguist::FileBlob for examples
   #
   # Returns Language or nil.
-  def detect(blob)
+  def detect(blob, allow_empty: false)
     # Bail early if the blob is binary or empty.
-    return nil if blob.likely_binary? || blob.binary? || blob.empty?
+    return nil if blob.likely_binary? || blob.binary? || (!allow_empty && blob.empty?)
 
     Linguist.instrument("linguist.detection", :blob => blob) do
       # Call each strategy until one candidate is returned.
@@ -59,8 +59,9 @@ class << Linguist
   # Strategies are called in turn until a single Language is returned.
   STRATEGIES = [
     Linguist::Strategy::Modeline,
-    Linguist::Shebang,
     Linguist::Strategy::Filename,
+    Linguist::Shebang,
+    Linguist::Strategy::Extension,
     Linguist::Heuristics,
     Linguist::Classifier
   ]
@@ -73,7 +74,7 @@ class << Linguist
   #       end
   #     end
   #
-  #     Linguist.instrumenter = CustomInstrumenter
+  #     Linguist.instrumenter = CustomInstrumenter.new
   #
   # The instrumenter must conform to the `ActiveSupport::Notifications`
   # interface, which defines `#instrument` and accepts:

lib/linguist/blob.rb

@@ -11,11 +11,13 @@ module Linguist
     #
     # path    - A path String (does not necessarily exists on the file system).
     # content - Content of the file.
+    # symlink - Whether the file is a symlink.
     #
     # Returns a Blob.
-    def initialize(path, content)
+    def initialize(path, content, symlink: false)
       @path = path
       @content = content
+      @symlink = symlink
     end
 
     # Public: Filename
@@ -63,11 +65,18 @@ module Linguist
     #
     # Returns an Array
     def extensions
-      _, *segments = name.downcase.split(".")
+      _, *segments = name.downcase.split(".", -1)
 
       segments.map.with_index do |segment, index|
         "." + segments[index..-1].join(".")
       end
     end
+
+    # Public: Is this a symlink?
+    #
+    # Returns true or false.
+    def symlink?
+      @symlink
+    end
   end
 end

lib/linguist/blob_helper.rb

@@ -275,10 +275,8 @@ module Linguist
           # also--importantly--without having to duplicate many (potentially
           # large) strings.
           begin
-            encoded_newlines = ["\r\n", "\r", "\n"].
-              map { |nl| nl.encode(ruby_encoding, "ASCII-8BIT").force_encoding(data.encoding) }
-
-            data.split(Regexp.union(encoded_newlines), -1)
+            
+            data.split(encoded_newlines_re, -1)
           rescue Encoding::ConverterNotFoundError
             # The data is not splittable in the detected encoding.  Assume it's
             # one big line.
@@ -289,6 +287,51 @@ module Linguist
         end
     end
 
+    def encoded_newlines_re
+      @encoded_newlines_re ||= Regexp.union(["\r\n", "\r", "\n"].
+                                              map { |nl| nl.encode(ruby_encoding, "ASCII-8BIT").force_encoding(data.encoding) })
+
+    end
+
+    def first_lines(n)
+      return lines[0...n] if defined? @lines
+      return [] unless viewable? && data
+
+      i, c = 0, 0
+      while c < n && j = data.index(encoded_newlines_re, i)
+        i = j + $&.length
+        c += 1
+      end
+      data[0...i].split(encoded_newlines_re, -1)
+    end
+
+    def last_lines(n)
+      if defined? @lines
+        if n >= @lines.length
+          @lines
+        else
+          lines[-n..-1]
+        end
+      end
+      return [] unless viewable? && data
+
+      no_eol = true
+      i, c = data.length, 0
+      k = i
+      while c < n && j = data.rindex(encoded_newlines_re, i - 1)
+        if c == 0 && j + $&.length == i
+          no_eol = false
+          n += 1
+        end
+        i = j
+        k = j + $&.length
+        c += 1
+      end
+      r = data[k..-1].split(encoded_newlines_re, -1)
+      r.pop if !no_eol
+      r
+    end
+
     # Public: Get number of lines of code
     #
     # Requires Blob#data
@@ -340,7 +383,10 @@ module Linguist
       !vendored? &&
       !documentation? &&
       !generated? &&
-      language && DETECTABLE_TYPES.include?(language.type)
+      language && ( defined?(detectable?) && !detectable?.nil? ?
+        detectable? :
+        DETECTABLE_TYPES.include?(language.type)
+      )
     end
   end
 end

lib/linguist/classifier.rb

@@ -3,6 +3,8 @@ require 'linguist/tokenizer'
 module Linguist
   # Language bayesian classifier.
   class Classifier
+    CLASSIFIER_CONSIDER_BYTES = 50 * 1024
+
     # Public: Use the classifier to detect language of the blob.
     #
     # blob               - An object that quacks like a blob.
@@ -17,7 +19,7 @@ module Linguist
     # Returns an Array of Language objects, most probable first.
     def self.call(blob, possible_languages)
       language_names = possible_languages.map(&:name)
-      classify(Samples.cache, blob.data, language_names).map do |name, _|
+      classify(Samples.cache, blob.data[0...CLASSIFIER_CONSIDER_BYTES], language_names).map do |name, _|
         Language[name] # Return the actual Language objects
       end
     end
@@ -95,7 +97,7 @@ module Linguist
     # Returns sorted Array of result pairs. Each pair contains the
     # String language name and a Float score.
     def classify(tokens, languages)
-      return [] if tokens.nil?
+      return [] if tokens.nil? || languages.empty?
       tokens = Tokenizer.tokenize(tokens) if tokens.is_a?(String)
       scores = {}
 

lib/linguist/documentation.yml

@@ -9,11 +9,13 @@
 
 ## Documentation directories ##
 
-- ^docs?/
+- ^[Dd]ocs?/
 - (^|/)[Dd]ocumentation/
-- (^|/)javadoc/
-- ^man/
+- (^|/)[Jj]avadoc/
+- ^[Mm]an/
 - ^[Ee]xamples/
+- ^[Dd]emos?/
+- (^|/)inst/doc/
 
 ## Documentation files ##
 
@@ -27,4 +29,4 @@
 - (^|/)[Rr]eadme(\.|$)
 
 # Samples folders
-- ^[Ss]amples/
+- ^[Ss]amples?/

lib/linguist/file_blob.rb

@@ -23,21 +23,26 @@ module Linguist
     #
     # Returns a String like '100644'
     def mode
-      File.stat(@fullpath).mode.to_s(8)
+      @mode ||= File.stat(@fullpath).mode.to_s(8)
+    end
+
+    def symlink?
+      return @symlink if defined? @symlink
+      @symlink = (File.symlink?(@fullpath) rescue false)
     end
 
     # Public: Read file contents.
     #
     # Returns a String.
     def data
-      File.read(@fullpath)
+      @data ||= File.read(@fullpath)
     end
 
     # Public: Get byte size
     #
     # Returns an Integer.
     def size
-      File.size(@fullpath)
+      @size ||= File.size(@fullpath)
     end
   end
 end

lib/linguist/generated.rb

@@ -3,7 +3,7 @@ module Linguist
     # Public: Is the blob a generated file?
     #
     # name - String filename
-    # data - String blob data. A block also maybe passed in for lazy
+    # data - String blob data. A block also may be passed in for lazy
     #        loading. This behavior is deprecated and you should always
     #        pass in a String.
     #
@@ -52,11 +52,15 @@ module Linguist
     # Return true or false
     def generated?
       xcode_file? ||
+      cocoapods? ||
+      carthage_build? ||
       generated_net_designer_file? ||
       generated_net_specflow_feature_file? ||
       composer_lock? ||
+      cargo_lock? ||
       node_modules? ||
-      npm_shrinkwrap? ||
+      go_vendor? ||
+      npm_shrinkwrap_or_package_lock? ||
       godeps? ||
       generated_by_zephir? ||
       minified_files? ||
@@ -69,6 +73,7 @@ module Linguist
       compiled_cython_file? ||
       generated_go? ||
       generated_protocol_buffer? ||
+      generated_javascript_protocol_buffer? ||
       generated_apache_thrift? ||
       generated_jni_header? ||
       vcr_cassette? ||
@@ -76,7 +81,11 @@ module Linguist
       generated_unity3d_meta? ||
       generated_racc? ||
       generated_jflex? ||
-      generated_grammarkit?
+      generated_grammarkit? ||
+      generated_roxygen2? ||
+      generated_jison? ||
+      generated_yarn_lock? ||
+      generated_grpc_cpp?
     end
 
     # Internal: Is the blob an Xcode file?
@@ -89,6 +98,20 @@ module Linguist
       ['.nib', '.xcworkspacedata', '.xcuserstate'].include?(extname)
     end
 
+    # Internal: Is the blob part of Pods/, which contains dependencies not meant for humans in pull requests.
+    #
+    # Returns true or false.
+    def cocoapods?
+      !!name.match(/(^Pods|\/Pods)\//)
+    end
+
+    # Internal: Is the blob part of Carthage/Build/, which contains dependencies not meant for humans in pull requests.
+    #
+    # Returns true or false.
+    def carthage_build?
+      !!name.match(/(^|\/)Carthage\/Build\//)
+    end
+
     # Internal: Is the blob minified files?
     #
     # Consider a file minified if the average line length is
@@ -200,7 +223,7 @@ module Linguist
     #
     # Returns true or false
     def generated_net_designer_file?
-      name.downcase =~ /\.designer\.cs$/
+      name.downcase =~ /\.designer\.(cs|vb)$/
     end
 
     # Internal: Is this a codegen file for Specflow feature file?
@@ -238,7 +261,11 @@ module Linguist
     #
     # Returns true or false.
     def generated_postscript?
-      return false unless ['.ps', '.eps'].include? extname
+      return false unless ['.ps', '.eps', '.pfa'].include? extname
+
+      # Type 1 and Type 42 fonts converted to PostScript are stored as hex-encoded byte streams; these
+      # streams are always preceded the `eexec` operator (if Type 1), or the `/sfnts` key (if Type 42).
+      return true if data =~ /(\n|\r\n|\r)\s*(?:currentfile eexec\s+|\/sfnts\s+\[\1<)\h{8,}\1/
 
       # We analyze the "%%Creator:" comment, which contains the author/generator
       # of the file. If there is one, it should be in one of the first few lines.
@@ -248,10 +275,12 @@ module Linguist
       # Most generators write their version number, while human authors' or companies'
       # names don't contain numbers. So look if the line contains digits. Also
       # look for some special cases without version numbers.
-      return creator =~ /[0-9]/ ||
-        creator.include?("mpage") ||
-        creator.include?("draw") ||
-        creator.include?("ImageMagick")
+      return true if creator =~ /[0-9]|draw|mpage|ImageMagick|inkscape|MATLAB/ ||
+        creator =~ /PCBNEW|pnmtops|\(Unknown\)|Serif Affinity|Filterimage -tops/
+
+      # EAGLE doesn't include a version number when it generates PostScript.
+      # However, it does prepend its name to the document's "%%Title" field.
+      !!creator.include?("EAGLE") and lines[0..4].find {|line| line =~ /^%%Title: EAGLE Drawing /}
     end
 
     def generated_go?
@@ -274,16 +303,25 @@ module Linguist
       return lines[0].include?("Generated by the protocol buffer compiler.  DO NOT EDIT!")
     end
 
-    APACHE_THRIFT_EXTENSIONS = ['.rb', '.py', '.go', '.js', '.m', '.java', '.h', '.cc', '.cpp']
+    # Internal: Is the blob a Javascript source file generated by the
+    # Protocol Buffer compiler?
+    #
+    # Returns true of false.
+    def generated_javascript_protocol_buffer?
+      return false unless extname == ".js"
+      return false unless lines.count > 6
+
+      return lines[5].include?("GENERATED CODE -- DO NOT EDIT!")
+    end
+
+    APACHE_THRIFT_EXTENSIONS = ['.rb', '.py', '.go', '.js', '.m', '.java', '.h', '.cc', '.cpp', '.php']
 
     # Internal: Is the blob generated by Apache Thrift compiler?
     #
     # Returns true or false
     def generated_apache_thrift?
       return false unless APACHE_THRIFT_EXTENSIONS.include?(extname)
-      return false unless lines.count > 1
-
-      return lines[0].include?("Autogenerated by Thrift Compiler") || lines[1].include?("Autogenerated by Thrift Compiler")
+      return lines.first(6).any? { |l| l.include?("Autogenerated by Thrift Compiler") }
     end
 
     # Internal: Is the blob a C/C++ header generated by the Java JNI tool javah?
@@ -304,11 +342,19 @@ module Linguist
       !!name.match(/node_modules\//)
     end
 
-    # Internal: Is the blob a generated npm shrinkwrap file.
+    # Internal: Is the blob part of the Go vendor/ tree,
+    # not meant for humans in pull requests.
     #
     # Returns true or false.
-    def npm_shrinkwrap?
-      !!name.match(/npm-shrinkwrap\.json/)
+    def go_vendor?
+      !!name.match(/vendor\/((?!-)[-0-9A-Za-z]+(?<!-)\.)+(com|edu|gov|in|me|net|org|fm|io)/)
+    end
+
+    # Internal: Is the blob a generated npm shrinkwrap or package lock file?
+    #
+    # Returns true or false.
+    def npm_shrinkwrap_or_package_lock?
+      name.match(/npm-shrinkwrap\.json/) || name.match(/package-lock\.json/)
     end
 
     # Internal: Is the blob part of Godeps/,
@@ -326,13 +372,20 @@ module Linguist
       !!name.match(/composer\.lock/)
     end
 
-    # Internal: Is the blob a generated by Zephir
+    # Internal: Is the blob generated by Zephir?
     #
     # Returns true or false.
     def generated_by_zephir?
       !!name.match(/.\.zep\.(?:c|h|php)$/)
     end
 
+    # Internal: Is the blob a generated Rust Cargo lock file?
+    #
+    # Returns true or false.
+    def cargo_lock?
+      !!name.match(/Cargo\.lock/)
+    end
+
     # Is the blob a VCR Cassette file?
     #
     # Returns true or false
@@ -426,5 +479,59 @@ module Linguist
       return false unless lines.count > 1
       return lines[0].start_with?("// This is a generated file. Not intended for manual editing.")
     end
+
+    # Internal: Is this a roxygen2-generated file?
+    #
+    # A roxygen2-generated file typically contain:
+    # % Generated by roxygen2: do not edit by hand
+    # on the first line.
+    #
+    # Return true or false
+    def generated_roxygen2?
+      return false unless extname == '.Rd'
+      return false unless lines.count > 1
+
+      return lines[0].include?("% Generated by roxygen2: do not edit by hand")
+    end
+
+    # Internal: Is this a Jison-generated file?
+    #
+    # Jison-generated parsers typically contain:
+    # /* parser generated by jison
+    # on the first line.
+    #
+    # Jison-generated lexers typically contain:
+    # /* generated by jison-lex
+    # on the first line.
+    #
+    # Return true or false
+    def generated_jison?
+      return false unless extname == '.js'
+      return false unless lines.count > 1
+      return lines[0].start_with?("/* parser generated by jison ") ||
+             lines[0].start_with?("/* generated by jison-lex ")
+    end
+
+    # Internal: Is the blob a generated yarn lockfile?
+    #
+    # Returns true or false.
+    def generated_yarn_lock?
+      return false unless name.match(/yarn\.lock/)
+      return false unless lines.count > 0
+      return lines[0].include?("# THIS IS AN AUTOGENERATED FILE")
+    end
+
+    # Internal: Is this a protobuf/grpc-generated C++ file?
+    #
+    # A generated file contains:
+    # // Generated by the gRPC C++ plugin.
+    # on the first line.
+    #
+    # Return true or false
+    def generated_grpc_cpp?
+      return false unless %w{.cpp .hpp .h .cc}.include? extname
+      return false unless lines.count > 1
+      return lines[0].start_with?("// Generated by the gRPC")
+    end
   end
 end

lib/linguist/heuristics.rb

@@ -1,6 +1,8 @@
 module Linguist
   # A collection of simple heuristics that can be used to better analyze languages.
   class Heuristics
+    HEURISTICS_CONSIDER_BYTES = 50 * 1024
+
     # Public: Use heuristics to detect language of the blob.
     #
     # blob               - An object that quacks like a blob.
@@ -14,12 +16,13 @@ module Linguist
     #
     # Returns an Array of languages, or empty if none matched or were inconclusive.
     def self.call(blob, candidates)
-      data = blob.data
+      return [] if blob.symlink?
+
+      data = blob.data[0...HEURISTICS_CONSIDER_BYTES]
 
       @heuristics.each do |heuristic|
-        if heuristic.matches?(blob.name)
-          languages = Array(heuristic.call(data))
-          return languages if languages.any? || languages.all? { |l| candidates.include?(l) }
+        if heuristic.matches?(blob.name, candidates)
+          return Array(heuristic.call(data))
         end
       end
 
@@ -28,7 +31,8 @@ module Linguist
 
     # Internal: Define a new heuristic.
     #
-    # languages - String names of languages to disambiguate.
+    # exts_and_langs - String names of file extensions and languages to
+    #                  disambiguate.
     # heuristic - Block which takes data as an argument and returns a Language or nil.
     #
     # Examples
@@ -41,23 +45,28 @@ module Linguist
     #       end
     #     end
     #
-    def self.disambiguate(*extensions, &heuristic)
-      @heuristics << new(extensions, &heuristic)
+    def self.disambiguate(*exts_and_langs, &heuristic)
+      @heuristics << new(exts_and_langs, &heuristic)
     end
 
     # Internal: Array of defined heuristics
     @heuristics = []
 
     # Internal
-    def initialize(extensions, &heuristic)
-      @extensions = extensions
+    def initialize(exts_and_langs, &heuristic)
+      @exts_and_langs, @candidates = exts_and_langs.partition {|e| e =~ /\A\./}
       @heuristic = heuristic
     end
 
-    # Internal: Check if this heuristic matches the candidate languages.
-    def matches?(filename)
+    # Internal: Check if this heuristic matches the candidate filenames or
+    # languages.
+    def matches?(filename, candidates)
       filename = filename.downcase
-      @extensions.any? { |ext| filename.end_with?(ext) }
+      candidates = candidates.compact.map(&:name)
+      @exts_and_langs.any? { |ext| filename.end_with?(ext) } ||
+        (candidates.any? &&
+         (@candidates - candidates == [] &&
+          candidates - @candidates == []))
     end
 
     # Internal: Perform the heuristic
@@ -66,7 +75,25 @@ module Linguist
     end
 
     # Common heuristics
+    CPlusPlusRegex = Regexp.union(
+        /^\s*#\s*include <(cstdint|string|vector|map|list|array|bitset|queue|stack|forward_list|unordered_map|unordered_set|(i|o|io)stream)>/,
+        /^\s*template\s*</,
+        /^[ \t]*try/,
+        /^[ \t]*catch\s*\(/,
+        /^[ \t]*(class|(using[ \t]+)?namespace)\s+\w+/,
+        /^[ \t]*(private|public|protected):$/,
+        /std::\w+/)
     ObjectiveCRegex = /^\s*(@(interface|class|protocol|property|end|synchronised|selector|implementation)\b|#import\s+.+\.h[">])/
+    Perl5Regex = /\buse\s+(?:strict\b|v?5\.)/
+    Perl6Regex = /^\s*(?:use\s+v6\b|\bmodule\b|\b(?:my\s+)?class\b)/
+
+    disambiguate ".as" do |data|
+      if /^\s*(package\s+[a-z0-9_\.]+|import\s+[a-zA-Z0-9_\.]+;|class\s+[A-Za-z0-9_]+\s+extends\s+[A-Za-z0-9_]+)/.match(data)
+        Language["ActionScript"]
+      else
+        Language["AngelScript"]
+      end
+    end
 
     disambiguate ".asc" do |data|
       if /^(----[- ]BEGIN|ssh-(rsa|dss)) /.match(data)
@@ -110,6 +137,12 @@ module Linguist
       end
     end
 
+    disambiguate ".cls" do |data|
+      if /\\\w+{/.match(data)
+        Language["TeX"]
+      end
+    end
+
     disambiguate ".cs" do |data|
       if /![\w\s]+methodsFor: /.match(data)
         Language["Smalltalk"]
@@ -119,11 +152,18 @@ module Linguist
     end
 
     disambiguate ".d" do |data|
-      if /^module /.match(data)
+      # see http://dlang.org/spec/grammar
+      # ModuleDeclaration | ImportDeclaration | FuncDeclaration | unittest
+      if /^module\s+[\w.]*\s*;|import\s+[\w\s,.:]*;|\w+\s+\w+\s*\(.*\)(?:\(.*\))?\s*{[^}]*}|unittest\s*(?:\(.*\))?\s*{[^}]*}/.match(data)
         Language["D"]
-      elsif /^((dtrace:::)?BEGIN|provider |#pragma (D (option|attributes)|ident)\s)/.match(data)
+      # see http://dtrace.org/guide/chp-prog.html, http://dtrace.org/guide/chp-profile.html, http://dtrace.org/guide/chp-opt.html
+      elsif /^(\w+:\w*:\w*:\w*|BEGIN|END|provider\s+|(tick|profile)-\w+\s+{[^}]*}|#pragma\s+D\s+(option|attributes|depends_on)\s|#pragma\s+ident\s)/.match(data)
         Language["DTrace"]
-      elsif /(\/.*:( .* \\)$| : \\$|^ : |: \\$)/.match(data)
+      # path/target : dependency \
+      # target : \
+      #  : dependency
+      # path/file.ext1 : some/path/../file.ext2
+      elsif /([\/\\].*:\s+.*\s\\$|: \\$|^ : |^[\w\s\/\\.]+\w+\.\w+\s*:\s+[\w\s\/\\.]+\w+\.\w+)/.match(data)
         Language["Makefile"]
       end
     end
@@ -152,7 +192,7 @@ module Linguist
       elsif data.include?("flowop")
         Language["Filebench WML"]
       elsif fortran_rx.match(data)
-        Language["FORTRAN"]
+        Language["Fortran"]
       end
     end
 
@@ -160,7 +200,7 @@ module Linguist
       if /^: /.match(data)
         Language["Forth"]
       elsif fortran_rx.match(data)
-        Language["FORTRAN"]
+        Language["Fortran"]
       end
     end
 
@@ -193,8 +233,7 @@ module Linguist
     disambiguate ".h" do |data|
       if ObjectiveCRegex.match(data)
         Language["Objective-C"]
-      elsif (/^\s*#\s*include <(cstdint|string|vector|map|list|array|bitset|queue|stack|forward_list|unordered_map|unordered_set|(i|o|io)stream)>/.match(data) ||
-        /^\s*template\s*</.match(data) || /^[ \t]*try/.match(data) || /^[ \t]*catch\s*\(/.match(data) || /^[ \t]*(class|(using[ \t]+)?namespace)\s+\w+/.match(data) || /^[ \t]*(private|public|protected):$/.match(data) || /std::\w+/.match(data))
+      elsif CPlusPlusRegex.match(data)
         Language["C++"]
       end
     end
@@ -202,6 +241,8 @@ module Linguist
     disambiguate ".inc" do |data|
       if /^<\?(?:php)?/.match(data)
         Language["PHP"]
+      elsif /^\s*#(declare|local|macro|while)\s/.match(data)
+        Language["POV-Ray SDL"]
       end
     end
 
@@ -211,7 +252,7 @@ module Linguist
       elsif /^(%[%{}]xs|<.*>)/.match(data)
         Language["Lex"]
       elsif /^\.[a-z][a-z](\s|$)/i.match(data)
-        Language["Groff"]
+        Language["Roff"]
       elsif /^\((de|class|rel|code|data|must)\s/.match(data)
         Language["PicoLisp"]
       end
@@ -242,7 +283,7 @@ module Linguist
         Language["MUF"]
       elsif /^\s*;/.match(data)
         Language["M"]
-      elsif /^\s*\(\*/.match(data)
+      elsif /\*\)$/.match(data)
         Language["Mathematica"]
       elsif /^\s*%/.match(data)
         Language["Matlab"]
@@ -252,10 +293,12 @@ module Linguist
     end
 
     disambiguate ".md" do |data|
-      if /^[-a-z0-9=#!\*\[|]/i.match(data)
+      if /(^[-a-z0-9=#!\*\[|>])|<\//i.match(data) || data.empty?
         Language["Markdown"]
       elsif /^(;;|\(define_)/.match(data)
-        Language["GCC machine description"]
+        Language["GCC Machine Description"]
+      else
+        Language["Markdown"]
       end
     end
 
@@ -270,7 +313,7 @@ module Linguist
     disambiguate ".mod" do |data|
       if data.include?('<!ENTITY ')
         Language["XML"]
-      elsif /MODULE\s\w+\s*;/i.match(data) || /^\s*END \w+;$/i.match(data)
+      elsif /^\s*MODULE [\w\.]+;/i.match(data) || /^\s*END [\w\.]+;/i.match(data)
         Language["Modula-2"]
       else
         [Language["Linux Kernel Module"], Language["AMPL"]]
@@ -279,9 +322,9 @@ module Linguist
 
     disambiguate ".ms" do |data|
       if /^[.'][a-z][a-z](\s|$)/i.match(data)
-        Language["Groff"]
-      elsif /(?<!\S)\.(include|globa?l)\s/.match(data) || /(?<!\/\*)(\A|\n)\s*\.[A-Za-z]/.match(data.gsub(/"([^\\"]|\\.)*"|'([^\\']|\\.)*'|\\\s*(?:--.*)?\n/, ""))
-        Language["GAS"]
+        Language["Roff"]
+      elsif /(?<!\S)\.(include|globa?l)\s/.match(data) || /(?<!\/\*)(\A|\n)\s*\.[A-Za-z][_A-Za-z0-9]*:/.match(data.gsub(/"([^\\"]|\\.)*"|'([^\\']|\\.)*'|\\\s*(?:--.*)?\n/, ""))
+        Language["Unix Assembly"]
       else
         Language["MAXScript"]
       end
@@ -289,7 +332,7 @@ module Linguist
 
     disambiguate ".n" do |data|
       if /^[.']/.match(data)
-        Language["Groff"]
+        Language["Roff"]
       elsif /^(module|namespace|using)\s/.match(data)
         Language["Nemerle"]
       end
@@ -318,33 +361,27 @@ module Linguist
     end
 
     disambiguate ".pl" do |data|
-      if /^[^#]+:-/.match(data)
+      if /^[^#]*:-/.match(data)
         Language["Prolog"]
-      elsif /use strict|use\s+v?5\./.match(data)
+      elsif Perl5Regex.match(data)
         Language["Perl"]
-      elsif /^(use v6|(my )?class|module)/.match(data)
-        Language["Perl6"]
+      elsif Perl6Regex.match(data)
+        Language["Perl 6"]
       end
     end
 
-    disambiguate ".pm", ".t" do |data|
-      if /use strict|use\s+v?5\./.match(data)
-        Language["Perl"]
-      elsif /^(use v6|(my )?class|module)/.match(data)
-        Language["Perl6"]
-      end
-    end
-
-    disambiguate ".pod" do |data|
-      if /^=\w+$/.match(data)
-        Language["Pod"]
-      else
+    disambiguate ".pm" do |data|
+      if Perl5Regex.match(data)
         Language["Perl"]
+      elsif Perl6Regex.match(data)
+        Language["Perl 6"]
+      elsif /^\s*\/\* XPM \*\//.match(data)
+        Language["XPM"]
       end
     end
 
     disambiguate ".pro" do |data|
-      if /^[^#]+:-/.match(data)
+      if /^[^\[#]+:-/.match(data)
         Language["Prolog"]
       elsif data.include?("last_client=")
         Language["INI"]
@@ -375,7 +412,7 @@ module Linguist
       if /^\.!|^\.end lit(?:eral)?\b/i.match(data)
         Language["RUNOFF"]
       elsif /^\.\\" /.match(data)
-        Language["Groff"]
+        Language["Roff"]
       end
     end
 
@@ -426,10 +463,12 @@ module Linguist
     end
     
     disambiguate ".t" do |data|
-      if /^\s*%|^\s*var\s+\w+\s*:\s*\w+/.match(data)
+      if Perl5Regex.match(data)
+        Language["Perl"]
+      elsif Perl6Regex.match(data)
+        Language["Perl 6"]
+      elsif /^\s*%[ \t]+|^\s*var\s+\w+\s*:=\s*\w+/.match(data)
         Language["Turing"]
-      elsif /^\s*use\s+v6\s*;/.match(data)
-        Language["Perl6"]
       end
     end
     
@@ -442,7 +481,7 @@ module Linguist
     end
 
     disambiguate ".ts" do |data|
-      if data.include?("<TS")
+      if /<TS\b/.match(data)
         Language["XML"]
       else
         Language["TypeScript"]
@@ -457,5 +496,32 @@ module Linguist
         Language["Scilab"]
       end
     end
+
+    disambiguate ".tsx" do |data|
+      if /^\s*(import.+(from\s+|require\()['"]react|\/\/\/\s*<reference\s)/.match(data)
+        Language["TypeScript"]
+      elsif /^\s*<\?xml\s+version/i.match(data)
+        Language["XML"]
+      end
+    end
+  
+    disambiguate ".w" do |data|
+      if (data.include?("&ANALYZE-SUSPEND _UIB-CODE-BLOCK _CUSTOM _DEFINITIONS"))
+        Language["OpenEdge ABL"]
+      elsif /^@(<|\w+\.)/.match(data)
+        Language["CWeb"]
+      end
+    end
+  
+    disambiguate ".x" do |data|
+      if /\b(program|version)\s+\w+\s*{|\bunion\s+\w+\s+switch\s*\(/.match(data)
+        Language["RPC"]
+      elsif /^%(end|ctor|hook|group)\b/.match(data)
+        Language["Logos"]
+      elsif /OUTPUT_ARCH\(|OUTPUT_FORMAT\(|SECTIONS/.match(data)
+        Language["Linker Script"]
+      end
+    end
+
   end
 end

lib/linguist/language.rb

@@ -11,6 +11,7 @@ require 'linguist/samples'
 require 'linguist/file_blob'
 require 'linguist/blob_helper'
 require 'linguist/strategy/filename'
+require 'linguist/strategy/extension'
 require 'linguist/strategy/modeline'
 require 'linguist/shebang'
 
@@ -90,17 +91,6 @@ module Linguist
       language
     end
 
-    # Public: Detects the Language of the blob.
-    #
-    # blob - an object that includes the Linguist `BlobHelper` interface;
-    #       see Linguist::LazyBlob and Linguist::FileBlob for examples
-    #
-    # Returns Language or nil.
-    def self.detect(blob)
-      warn "[DEPRECATED] `Linguist::Language.detect` is deprecated. Use `Linguist.detect`. #{caller[0]}"
-      Linguist.detect(blob)
-    end
-
     # Public: Get all Languages
     #
     # Returns an Array of Languages
@@ -119,8 +109,8 @@ module Linguist
     #
     # Returns the Language or nil if none was found.
     def self.find_by_name(name)
-      return nil if name.to_s.empty?
-      name && (@name_index[name.downcase] || @name_index[name.split(',').first.downcase])
+      return nil if !name.is_a?(String) || name.to_s.empty?
+      name && (@name_index[name.downcase] || @name_index[name.split(',', 2).first.downcase])
     end
 
     # Public: Look up Language by one of its aliases.
@@ -134,52 +124,52 @@ module Linguist
     #
     # Returns the Language or nil if none was found.
     def self.find_by_alias(name)
-      return nil if name.to_s.empty?
-      name && (@alias_index[name.downcase] || @alias_index[name.split(',').first.downcase])
+      return nil if !name.is_a?(String) || name.to_s.empty?
+      name && (@alias_index[name.downcase] || @alias_index[name.split(',', 2).first.downcase])
     end
 
     # Public: Look up Languages by filename.
     #
+    # The behaviour of this method recently changed.
+    # See the second example below.
+    #
     # filename - The path String.
     #
     # Examples
     #
+    #   Language.find_by_filename('Cakefile')
+    #   # => [#<Language name="CoffeeScript">]
     #   Language.find_by_filename('foo.rb')
-    #   # => [#<Language name="Ruby">]
+    #   # => []
     #
     # Returns all matching Languages or [] if none were found.
     def self.find_by_filename(filename)
       basename = File.basename(filename)
-
-      # find the first extension with language definitions
-      extname = FileBlob.new(filename).extensions.detect do |e|
-        !@extension_index[e].empty?
-      end
-
-      (@filename_index[basename] + @extension_index[extname]).compact.uniq
+      @filename_index[basename]
     end
 
     # Public: Look up Languages by file extension.
     #
-    # extname - The extension String.
+    # The behaviour of this method recently changed.
+    # See the second example below.
+    #
+    # filename - The path String.
     #
     # Examples
     #
-    #   Language.find_by_extension('.rb')
+    #   Language.find_by_extension('dummy.rb')
     #   # => [#<Language name="Ruby">]
-    #
     #   Language.find_by_extension('rb')
-    #   # => [#<Language name="Ruby">]
+    #   # => []
     #
     # Returns all matching Languages or [] if none were found.
-    def self.find_by_extension(extname)
-      extname = ".#{extname}" unless extname.start_with?(".")
-      @extension_index[extname.downcase]
-    end
+    def self.find_by_extension(filename)
+      # find the first extension with language definitions
+      extname = FileBlob.new(filename.downcase).extensions.detect do |e|
+        !@extension_index[e].empty?
+      end
 
-    # DEPRECATED
-    def self.find_by_shebang(data)
-      @interpreter_index[Shebang.interpreter(data)]
+      @extension_index[extname]
     end
 
     # Public: Look up Languages by interpreter.
@@ -224,8 +214,12 @@ module Linguist
     #
     # Returns the Language or nil if none was found.
     def self.[](name)
-      return nil if name.to_s.empty?
-      name && (@index[name.downcase] || @index[name.split(',').first.downcase])
+      return nil if !name.is_a?(String) || name.to_s.empty?
+
+      lang = @index[name.downcase]
+      return lang if lang
+
+      @index[name.split(',', 2).first.downcase]
     end
 
     # Public: A List of popular languages
@@ -259,17 +253,6 @@ module Linguist
       @colors ||= all.select(&:color).sort_by { |lang| lang.name.downcase }
     end
 
-    # Public: A List of languages compatible with Ace.
-    #
-    # TODO: Remove this method in a 5.x release. Every language now needs an ace_mode
-    # key, so this function isn't doing anything unique anymore.
-    #
-    # Returns an Array of Languages.
-    def self.ace_modes
-      warn "This method will be deprecated in a future 5.x release. Every language now has an `ace_mode` set."
-      @ace_modes ||= all.select(&:ace_mode).sort_by { |lang| lang.name.downcase }
-    end
-
     # Internal: Initialize a new Language
     #
     # attributes - A hash of attributes
@@ -286,7 +269,7 @@ module Linguist
       @color = attributes[:color]
 
       # Set aliases
-      @aliases = [default_alias_name] + (attributes[:aliases] || [])
+      @aliases = [default_alias] + (attributes[:aliases] || [])
 
       # Load the TextMate scope name or try to guess one
       @tm_scope = attributes[:tm_scope] || begin
@@ -301,12 +284,10 @@ module Linguist
 
       @ace_mode = attributes[:ace_mode]
       @codemirror_mode = attributes[:codemirror_mode]
+      @codemirror_mime_type = attributes[:codemirror_mime_type]
       @wrap = attributes[:wrap] || false
 
-      # Set legacy search term
-      @search_term = attributes[:search_term] || default_alias_name
-
-      # Set the language_id 
+      # Set the language_id
       @language_id = attributes[:language_id]
 
       # Set extensions or default to [].
@@ -360,17 +341,6 @@ module Linguist
     # Returns an Array of String names
     attr_reader :aliases
 
-    # Deprecated: Get code search term
-    #
-    # Examples
-    #
-    #   # => "ruby"
-    #   # => "python"
-    #   # => "perl"
-    #
-    # Returns the name String
-    attr_reader :search_term
-
     # Public: Get language_id (used in GitHub search)
     #
     # Examples
@@ -398,7 +368,10 @@ module Linguist
     # Returns a String name or nil
     attr_reader :ace_mode
 
-    # Public: Get Codemirror mode
+    # Public: Get CodeMirror mode
+    #
+    # Maps to a directory in the `mode/` source code.
+    #   https://github.com/codemirror/CodeMirror/tree/master/mode
     #
     # Examples
     #
@@ -409,6 +382,17 @@ module Linguist
     # Returns a String name or nil
     attr_reader :codemirror_mode
 
+    # Public: Get CodeMirror MIME type mode
+    #
+    # Examples
+    #
+    #  # => "nil"
+    #  # => "text/x-javascript"
+    #  # => "text/x-csrc"
+    #
+    # Returns a String name or nil
+    attr_reader :codemirror_mime_type
+
     # Public: Should language lines be wrapped
     #
     # Returns true or false
@@ -441,22 +425,6 @@ module Linguist
     # Returns the extensions Array
     attr_reader :filenames
 
-    # Deprecated: Get primary extension
-    #
-    # Defaults to the first extension but can be overridden
-    # in the languages.yml.
-    #
-    # The primary extension can not be nil. Tests should verify this.
-    #
-    # This method is only used by app/helpers/gists_helper.rb for creating
-    # the language dropdown. It really should be using `name` instead.
-    # Would like to drop primary extension.
-    #
-    # Returns the extension String.
-    def primary_extension
-      extensions.first
-    end
-
     # Public: Get URL escaped name.
     #
     # Examples
@@ -470,12 +438,13 @@ module Linguist
       EscapeUtils.escape_url(name).gsub('+', '%20')
     end
 
-    # Internal: Get default alias name
+    # Public: Get default alias name
     #
     # Returns the alias name String
-    def default_alias_name
+    def default_alias
       name.downcase.gsub(/\s/, '-')
     end
+    alias_method :default_alias_name, :default_alias
 
     # Public: Get Language group
     #
@@ -570,14 +539,6 @@ module Linguist
       end
     end
 
-    if fns = filenames[name]
-      fns.each do |filename|
-        if !options['filenames'].include?(filename)
-          options['filenames'] << filename
-        end
-      end
-    end
-
     Language.create(
       :name              => name,
       :color             => options['color'],
@@ -586,10 +547,10 @@ module Linguist
       :tm_scope          => options['tm_scope'],
       :ace_mode          => options['ace_mode'],
       :codemirror_mode   => options['codemirror_mode'],
+      :codemirror_mime_type => options['codemirror_mime_type'],
       :wrap              => options['wrap'],
       :group_name        => options['group'],
       :searchable        => options.fetch('searchable', true),
-      :search_term       => options['search_term'],
       :language_id       => options['language_id'],
       :extensions        => Array(options['extensions']),
       :interpreters      => options['interpreters'].sort,

lib/linguist/lazy_blob.rb

@@ -7,7 +7,8 @@ module Linguist
     GIT_ATTR = ['linguist-documentation',
                 'linguist-language',
                 'linguist-vendored',
-                'linguist-generated']
+                'linguist-generated',
+                'linguist-detectable']
 
     GIT_ATTR_OPTS = { :priority => [:index], :skip_system => true }
     GIT_ATTR_FLAGS = Rugged::Repository::Attributes.parse_opts(GIT_ATTR_OPTS)
@@ -70,6 +71,14 @@ module Linguist
       end
     end
 
+    def detectable?
+      if attr = git_attributes['linguist-detectable']
+        return boolean_attribute(attr)
+      else
+        nil
+      end
+    end
+
     def data
       load_blob!
       @data
@@ -80,6 +89,11 @@ module Linguist
       @size
     end
 
+    def symlink?
+      # We don't create LazyBlobs for symlinks.
+      false
+    end
+
     def cleanup!
       @data.clear if @data
     end

lib/linguist/popular.yml

@@ -26,4 +26,4 @@
 - Shell
 - Swift
 - TeX
-- VimL
+- Vim script

lib/linguist/shebang.rb

@@ -3,15 +3,20 @@ module Linguist
     # Public: Use shebang to detect language of the blob.
     #
     # blob               - An object that quacks like a blob.
+    # candidates         - A list of candidate languages.
     #
     # Examples
     #
     #   Shebang.call(FileBlob.new("path/to/file"))
     #
-    # Returns an Array with one Language if the blob has a shebang with a valid
-    # interpreter, or empty if there is no shebang.
-    def self.call(blob, _ = nil)
-      Language.find_by_interpreter interpreter(blob.data)
+    # Returns an array of languages from the candidate list for which the
+    # blob's shebang is valid. Returns an empty list if there is no shebang.
+    # If the candidate list is empty, any language is a valid candidate.
+    def self.call(blob, candidates)
+      return [] if blob.symlink?
+
+      languages = Language.find_by_interpreter interpreter(blob.data)
+      candidates.any? ? candidates & languages : languages
     end
 
     # Public: Get the interpreter from the shebang

lib/linguist/strategy/extension.rb

@@ -0,0 +1,23 @@
+module Linguist
+  module Strategy
+    # Detects language based on extension
+    class Extension
+      # Public: Use the file extension to detect the blob's language.
+      #
+      # blob               - An object that quacks like a blob.
+      # candidates         - A list of candidate languages.
+      #
+      # Examples
+      #
+      #   Extension.call(FileBlob.new("path/to/file"))
+      #
+      # Returns an array of languages associated with a blob's file extension.
+      # Selected languages must be in the candidate list, except if it's empty,
+      # in which case any language is a valid candidate.
+      def self.call(blob, candidates)
+        languages = Language.find_by_extension(blob.name.to_s)
+        candidates.any? ? candidates & languages : languages
+      end
+    end
+  end
+end

lib/linguist/strategy/filename.rb

@@ -1,9 +1,23 @@
 module Linguist
   module Strategy
-    # Detects language based on filename and/or extension
+    # Detects language based on filename
     class Filename
-      def self.call(blob, _)
-        Language.find_by_filename(blob.name.to_s)
+      # Public: Use the filename to detect the blob's language.
+      #
+      # blob               - An object that quacks like a blob.
+      # candidates         - A list of candidate languages.
+      #
+      # Examples
+      #
+      #   Filename.call(FileBlob.new("path/to/file"))
+      #
+      # Returns an array of languages with a associated blob's filename.
+      # Selected languages must be in the candidate list, except if it's empty,
+      # in which case any language is a valid candidate.
+      def self.call(blob, candidates)
+        name = blob.name.to_s
+        languages = Language.find_by_filename(name)
+        candidates.any? ? candidates & languages : languages
       end
     end
   end

lib/linguist/strategy/modeline.rb

@@ -109,8 +109,10 @@ module Linguist
       # Returns an Array with one Language if the blob has a Vim or Emacs modeline
       # that matches a Language name or alias. Returns an empty array if no match.
       def self.call(blob, _ = nil)
-        header = blob.lines.first(SEARCH_SCOPE).join("\n")
-        footer = blob.lines.last(SEARCH_SCOPE).join("\n")
+        return [] if blob.symlink?
+
+        header = blob.first_lines(SEARCH_SCOPE).join("\n")
+        footer = blob.last_lines(SEARCH_SCOPE).join("\n")
         Array(Language.find_by_alias(modeline(header + footer)))
       end
 

lib/linguist/tokenizer.rb

@@ -1,4 +1,5 @@
 require 'strscan'
+require 'linguist/linguist'
 
 module Linguist
   # Generic programming language tokenizer.
@@ -15,191 +16,5 @@ module Linguist
     def self.tokenize(data)
       new.extract_tokens(data)
     end
-
-    # Read up to 100KB
-    BYTE_LIMIT = 100_000
-
-    # Start state on token, ignore anything till the next newline
-    SINGLE_LINE_COMMENTS = [
-      '//', # C
-      '--', # Ada, Haskell, AppleScript
-      '#',  # Ruby
-      '%',  # Tex
-      '"',  # Vim
-    ]
-
-    # Start state on opening token, ignore anything until the closing
-    # token is reached.
-    MULTI_LINE_COMMENTS = [
-      ['/*', '*/'],    # C
-      ['<!--', '-->'], # XML
-      ['{-', '-}'],    # Haskell
-      ['(*', '*)'],    # Coq
-      ['"""', '"""'],  # Python
-      ["'''", "'''"]   # Python
-    ]
-
-    START_SINGLE_LINE_COMMENT =  Regexp.compile(SINGLE_LINE_COMMENTS.map { |c|
-      "\s*#{Regexp.escape(c)} "
-    }.join("|"))
-
-    START_MULTI_LINE_COMMENT =  Regexp.compile(MULTI_LINE_COMMENTS.map { |c|
-      Regexp.escape(c[0])
-    }.join("|"))
-
-    # Internal: Extract generic tokens from data.
-    #
-    # data - String to scan.
-    #
-    # Examples
-    #
-    #   extract_tokens("printf('Hello')")
-    #   # => ['printf', '(', ')']
-    #
-    # Returns Array of token Strings.
-    def extract_tokens(data)
-      s = StringScanner.new(data)
-
-      tokens = []
-      until s.eos?
-        break if s.pos >= BYTE_LIMIT
-
-        if token = s.scan(/^#!.+$/)
-          if name = extract_shebang(token)
-            tokens << "SHEBANG#!#{name}"
-          end
-
-        # Single line comment
-        elsif s.beginning_of_line? && token = s.scan(START_SINGLE_LINE_COMMENT)
-          # tokens << token.strip
-          s.skip_until(/\n|\Z/)
-
-        # Multiline comments
-        elsif token = s.scan(START_MULTI_LINE_COMMENT)
-          # tokens << token
-          close_token = MULTI_LINE_COMMENTS.assoc(token)[1]
-          s.skip_until(Regexp.compile(Regexp.escape(close_token)))
-          # tokens << close_token
-
-        # Skip single or double quoted strings
-        elsif s.scan(/"/)
-          if s.peek(1) == "\""
-            s.getch
-          else
-            s.skip_until(/(?<!\\)"/)
-          end
-        elsif s.scan(/'/)
-          if s.peek(1) == "'"
-            s.getch
-          else
-            s.skip_until(/(?<!\\)'/)
-          end
-
-        # Skip number literals
-        elsif s.scan(/(0x\h(\h|\.)*|\d(\d|\.)*)([uU][lL]{0,2}|([eE][-+]\d*)?[fFlL]*)/)
-
-        # SGML style brackets
-        elsif token = s.scan(/<[^\s<>][^<>]*>/)
-          extract_sgml_tokens(token).each { |t| tokens << t }
-
-        # Common programming punctuation
-        elsif token = s.scan(/;|\{|\}|\(|\)|\[|\]/)
-          tokens << token
-
-        # Regular token
-        elsif token = s.scan(/[\w\.@#\/\*]+/)
-          tokens << token
-
-        # Common operators
-        elsif token = s.scan(/<<?|\+|\-|\*|\/|%|&&?|\|\|?/)
-          tokens << token
-
-        else
-          s.getch
-        end
-      end
-
-      tokens
-    end
-
-    # Internal: Extract normalized shebang command token.
-    #
-    # Examples
-    #
-    #   extract_shebang("#!/usr/bin/ruby")
-    #   # => "ruby"
-    #
-    #   extract_shebang("#!/usr/bin/env node")
-    #   # => "node"
-    #
-    #   extract_shebang("#!/usr/bin/env A=B foo=bar awk -f")
-    #   # => "awk"
-    #
-    # Returns String token or nil it couldn't be parsed.
-    def extract_shebang(data)
-      s = StringScanner.new(data)
-
-      if path = s.scan(/^#!\s*\S+/)
-        script = path.split('/').last
-        if script == 'env'
-          s.scan(/\s+/)
-          s.scan(/.*=[^\s]+\s+/)
-          script = s.scan(/\S+/)
-        end
-        script = script[/[^\d]+/, 0] if script
-        return script
-      end
-
-      nil
-    end
-
-    # Internal: Extract tokens from inside SGML tag.
-    #
-    # data - SGML tag String.
-    #
-    # Examples
-    #
-    #   extract_sgml_tokens("<a href='' class=foo>")
-    #   # => ["<a>", "href="]
-    #
-    # Returns Array of token Strings.
-    def extract_sgml_tokens(data)
-      s = StringScanner.new(data)
-
-      tokens = []
-
-      until s.eos?
-        # Emit start token
-        if token = s.scan(/<\/?[^\s>]+/)
-          tokens << "#{token}>"
-
-        # Emit attributes with trailing =
-        elsif token = s.scan(/\w+=/)
-          tokens << token
-
-          # Then skip over attribute value
-          if s.scan(/"/)
-            s.skip_until(/[^\\]"/)
-          elsif s.scan(/'/)
-            s.skip_until(/[^\\]'/)
-          else
-            s.skip_until(/\w+/)
-          end
-
-        # Emit lone attributes
-        elsif token = s.scan(/\w+/)
-          tokens << token
-
-        # Stop at the end of the tag
-        elsif s.scan(/>/)
-          s.terminate
-
-        else
-          s.getch
-        end
-      end
-
-      tokens
-    end
   end
 end

lib/linguist/vendor.yml

@@ -15,10 +15,11 @@
 # Dependencies
 - ^[Dd]ependencies/
 
+# Distributions
+- (^|/)dist/
+
 # C deps
-#  https://github.com/joyent/node
 - ^deps/
-- ^tools/
 - (^|/)configure$
 - (^|/)config.guess$
 - (^|/)config.sub$
@@ -47,6 +48,9 @@
 # Go dependencies
 - Godeps/_workspace/
 
+# GNU indent profiles
+- .indent.pro
+
 # Minified JavaScript and CSS
 - (\.|-)min\.(js|css)$
 
@@ -59,6 +63,7 @@
 
 # Font Awesome
 - (^|/)font-awesome\.(css|less|scss|styl)$
+- (^|/)font-awesome/.*\.(css|less|scss|styl)$
 
 # Foundation css
 - (^|/)foundation\.(css|less|scss|styl)$
@@ -66,12 +71,21 @@
 # Normalize.css
 - (^|/)normalize\.(css|less|scss|styl)$
 
+# Skeleton.css
+- (^|/)skeleton\.(css|less|scss|styl)$
+
 # Bourbon css
 - (^|/)[Bb]ourbon/.*\.(css|less|scss|styl)$
 
 # Animate.css
 - (^|/)animate\.(css|less|scss|styl)$
 
+# Materialize.css
+- (^|/)materialize\.(css|less|scss|styl|js)$
+
+# Select2
+- (^|/)select2/.*\.(css|scss|js)$
+
 # Vendored dependencies
 - third[-_]?party/
 - 3rd[-_]?party/
@@ -110,6 +124,15 @@
 # jQuery File Upload
 - (^|/)jquery\.fileupload(-\w+)?\.js$
 
+# jQuery dataTables
+- jquery.dataTables.js
+
+# bootboxjs
+- bootbox.js
+
+# pdf-worker
+- pdf.worker.js
+
 # Slick
 - (^|/)slick\.\w+.js$
 
@@ -126,6 +149,9 @@
 - .sublime-project
 - .sublime-workspace
 
+# VS Code workspace files
+- .vscode
+
 # Prototype
 - (^|/)prototype(.*)\.js$
 - (^|/)effects\.js$
@@ -165,7 +191,7 @@
 # Chart.js
 - (^|/)Chart\.js$
 
-# Codemirror
+# CodeMirror
 - (^|/)[Cc]ode[Mm]irror/(\d+\.\d+/)?(lib|mode|theme|addon|keymap|demo)
 
 # SyntaxHighlighter - http://alexgorbatchev.com/
@@ -182,6 +208,9 @@
 # React
 - (^|/)react(-[^.]*)?\.js$
 
+# flow-typed
+- (^|/)flow-typed/.*\.js$
+
 # Modernizr
 - (^|/)modernizr\-\d\.\d+(\.\d+)?\.js$
 - (^|/)modernizr\.custom\.\d+\.js$
@@ -215,10 +244,7 @@
 - \.imageset/
 
 # Carthage
-- ^Carthage/
-
-# Cocoapods
-- ^Pods/
+- (^|/)Carthage/
 
 # Sparkle
 - (^|/)Sparkle/
@@ -229,6 +255,15 @@
 # Fabric
 - Fabric.framework/
 
+# BuddyBuild
+- BuddyBuildSDK.framework/
+
+# Realm
+- Realm.framework
+
+# RealmSwift
+- RealmSwift.framework
+
 # git config files
 - gitattributes$
 - gitignore$
@@ -241,6 +276,13 @@
 - (^|/)gradlew\.bat$
 - (^|/)gradle/wrapper/
 
+## Java ##
+
+# Maven
+- (^|/)mvnw$
+- (^|/)mvnw\.cmd$
+- (^|/)\.mvn/wrapper/
+
 ## .NET ##
 
 # Visual Studio IntelliSense

lib/linguist/version.rb

@@ -1,3 +1,3 @@
 module Linguist
-  VERSION = "4.8.13"
+  VERSION = "6.0.1"
 end

package.json

@@ -1,7 +0,0 @@
-{
-  "repository": "https://github.com/github/linguist",
-  "dependencies": {
-    "season": "~>5.0"
-  },
-  "license": "MIT"
-}

samples/ABNF/toml.abnf

@@ -0,0 +1,190 @@
+; Source:  https://github.com/toml-lang/toml
+; License: MIT
+
+;; This is an attempt to define TOML in ABNF according to the grammar defined
+;; in RFC 4234 (http://www.ietf.org/rfc/rfc4234.txt).
+
+;; TOML
+
+toml = expression *( newline expression )
+expression = (
+  ws /
+  ws comment /
+  ws keyval ws [ comment ] /
+  ws table ws [ comment ]
+)
+
+;; Newline
+
+newline = (
+  %x0A /              ; LF
+  %x0D.0A             ; CRLF
+)
+
+newlines = 1*newline
+
+;; Whitespace
+
+ws = *(
+  %x20 /              ; Space
+  %x09                ; Horizontal tab
+)
+
+;; Comment
+
+comment-start-symbol = %x23 ; #
+non-eol = %x09 / %x20-10FFFF
+comment = comment-start-symbol *non-eol
+
+;; Key-Value pairs
+
+keyval-sep = ws %x3D ws ; =
+keyval = key keyval-sep val
+
+key = unquoted-key / quoted-key
+unquoted-key = 1*( ALPHA / DIGIT / %x2D / %x5F ) ; A-Z / a-z / 0-9 / - / _
+quoted-key = quotation-mark 1*basic-char quotation-mark ; See Basic Strings
+
+val = integer / float / string / boolean / date-time / array / inline-table
+
+;; Table
+
+table = std-table / array-table
+
+;; Standard Table
+
+std-table-open  = %x5B ws     ; [ Left square bracket
+std-table-close = ws %x5D     ; ] Right square bracket
+table-key-sep   = ws %x2E ws  ; . Period
+
+std-table = std-table-open key *( table-key-sep key) std-table-close
+
+;; Array Table
+
+array-table-open  = %x5B.5B ws  ; [[ Double left square bracket
+array-table-close = ws %x5D.5D  ; ]] Double right square bracket
+
+array-table = array-table-open key *( table-key-sep key) array-table-close
+
+;; Integer
+
+integer = [ minus / plus ] int
+minus = %x2D                       ; -
+plus = %x2B                        ; +
+digit1-9 = %x31-39                 ; 1-9
+underscore = %x5F                  ; _
+int = DIGIT / digit1-9 1*( DIGIT / underscore DIGIT )
+
+;; Float
+
+float = integer ( frac / frac exp / exp )
+zero-prefixable-int = DIGIT *( DIGIT / underscore DIGIT )
+frac = decimal-point zero-prefixable-int
+decimal-point = %x2E               ; .
+exp = e integer
+e = %x65 / %x45                    ; e E
+
+;; String
+
+string = basic-string / ml-basic-string / literal-string / ml-literal-string
+
+;; Basic String
+
+basic-string = quotation-mark *basic-char quotation-mark
+
+quotation-mark = %x22            ; "
+
+basic-char = basic-unescaped / escaped
+escaped = escape ( %x22 /          ; "    quotation mark  U+0022
+                   %x5C /          ; \    reverse solidus U+005C
+                   %x2F /          ; /    solidus         U+002F
+                   %x62 /          ; b    backspace       U+0008
+                   %x66 /          ; f    form feed       U+000C
+                   %x6E /          ; n    line feed       U+000A
+                   %x72 /          ; r    carriage return U+000D
+                   %x74 /          ; t    tab             U+0009
+                   %x75 4HEXDIG /  ; uXXXX                U+XXXX
+                   %x55 8HEXDIG )  ; UXXXXXXXX            U+XXXXXXXX
+
+basic-unescaped = %x20-21 / %x23-5B / %x5D-10FFFF
+
+escape = %x5C                    ; \
+
+;; Multiline Basic String
+
+ml-basic-string-delim = quotation-mark quotation-mark quotation-mark
+ml-basic-string = ml-basic-string-delim ml-basic-body ml-basic-string-delim
+ml-basic-body = *( ml-basic-char / newline / ( escape newline ))
+
+ml-basic-char = ml-basic-unescaped / escaped
+ml-basic-unescaped = %x20-5B / %x5D-10FFFF
+
+;; Literal String
+
+literal-string = apostraphe *literal-char apostraphe
+
+apostraphe = %x27 ; ' Apostrophe
+
+literal-char = %x09 / %x20-26 / %x28-10FFFF
+
+;; Multiline Literal String
+
+ml-literal-string-delim = apostraphe apostraphe apostraphe
+ml-literal-string = ml-literal-string-delim ml-literal-body ml-literal-string-delim
+
+ml-literal-body = *( ml-literal-char / newline )
+ml-literal-char = %x09 / %x20-10FFFF
+
+;; Boolean
+
+boolean = true / false
+true    = %x74.72.75.65     ; true
+false   = %x66.61.6C.73.65  ; false
+
+;; Datetime (as defined in RFC 3339)
+
+date-fullyear  = 4DIGIT
+date-month     = 2DIGIT  ; 01-12
+date-mday      = 2DIGIT  ; 01-28, 01-29, 01-30, 01-31 based on month/year
+time-hour      = 2DIGIT  ; 00-23
+time-minute    = 2DIGIT  ; 00-59
+time-second    = 2DIGIT  ; 00-58, 00-59, 00-60 based on leap second rules
+time-secfrac   = "." 1*DIGIT
+time-numoffset = ( "+" / "-" ) time-hour ":" time-minute
+time-offset    = "Z" / time-numoffset
+
+partial-time   = time-hour ":" time-minute ":" time-second [time-secfrac]
+full-date      = date-fullyear "-" date-month "-" date-mday
+full-time      = partial-time time-offset
+
+date-time      = full-date "T" full-time
+
+;; Array
+
+array-open  = %x5B ws  ; [
+array-close = ws %x5D  ; ]
+
+array = array-open array-values array-close
+
+array-values = [ val [ array-sep ] [ ( comment newlines) / newlines ] /
+                 val array-sep [ ( comment newlines) / newlines ] array-values ]
+
+array-sep = ws %x2C ws  ; , Comma
+
+;; Inline Table
+
+inline-table-open  = %x7B ws     ; {
+inline-table-close = ws %x7D     ; }
+inline-table-sep   = ws %x2C ws  ; , Comma
+
+inline-table = inline-table-open inline-table-keyvals inline-table-close
+
+inline-table-keyvals = [ inline-table-keyvals-non-empty ]
+inline-table-keyvals-non-empty = key keyval-sep val /
+                                 key keyval-sep val inline-table-sep inline-table-keyvals-non-empty
+
+;; Built-in ABNF terms, reproduced here for clarity
+
+; ALPHA = %x41-5A / %x61-7A ; A-Z / a-z
+; DIGIT = %x30-39 ; 0-9
+; HEXDIG = DIGIT / "A" / "B" / "C" / "D" / "E" / "F"

samples/ActionScript/FooBar.as

@@ -0,0 +1,35 @@
+// A sample for Actionscript.
+
+package foobar
+{
+	import flash.display.MovieClip;
+
+	class Bar
+	{
+		public function getNumber():Number
+		{
+			return 10;
+		}
+	}
+
+	class Foo extends Bar
+	{
+		private var ourNumber:Number = 25;
+
+		override public function getNumber():Number
+		{
+			return ourNumber;
+		}
+	}
+
+	class Main extends MovieClip
+	{
+		public function Main()
+		{
+			var x:Bar = new Bar();
+			var y:Foo = new Foo();
+			trace(x.getNumber());
+			trace(y.getNumber());
+		}
+	}
+}

samples/ActionScript/HelloWorld.as

@@ -0,0 +1,13 @@
+package mypackage
+{
+	public class Hello
+	{
+		/* Let's say hello!
+		 * This is just a test script for Linguist's Actionscript detection.
+		 */
+		public function sayHello():void
+		{
+			trace("Hello, world");
+		}
+	}
+}

samples/Adobe Font Metrics/OpenSansCondensed-Bold.afm

@@ -0,0 +1,69 @@
+StartFontMetrics 2.0
+Comment Generated by FontForge 20170719
+Comment Creation Date: Sun Jul 23 19:47:25 2017
+FontName OpenSansCondensed-Bold
+FullName Open Sans Condensed Bold
+FamilyName Open Sans Condensed
+Weight Bold
+Notice (Digitized data copyright (c) 2010-2011, Google Corporation.)
+ItalicAngle 0
+IsFixedPitch false
+UnderlinePosition -205
+UnderlineThickness 102
+Version 1.11
+EncodingScheme ISO10646-1
+FontBBox -667 -290 1046 1062
+CapHeight 714
+XHeight 544
+Ascender 760
+Descender -240
+StartCharMetrics 939
+C 32 ; WX 247 ; N space ; B 0 0 0 0 ;
+C 33 ; WX 270 ; N exclam ; B 54 -14 216 714 ;
+C 34 ; WX 445 ; N quotedbl ; B 59 456 388 714 ;
+C 35 ; WX 543 ; N numbersign ; B 20 0 525 714 ;
+C 36 ; WX 462 ; N dollar ; B 36 -59 427 760 ;
+C 37 ; WX 758 ; N percent ; B 30 -9 729 725 ;
+C 38 ; WX 581 ; N ampersand ; B 28 -10 572 725 ;
+C 39 ; WX 246 ; N quotesingle ; B 59 456 188 714 ;
+C -1 ; WX 462 ; N six.os ; B 36 -10 427 724 ;
+C -1 ; WX 420 ; N seven.os ; B 19 -170 402 544 ;
+C -1 ; WX 462 ; N eight.os ; B 35 -10 429 724 ;
+C -1 ; WX 461 ; N nine.os ; B 33 -182 424 564 ;
+C -1 ; WX 496 ; N g.alt ; B 36 -241 442 555 ;
+C -1 ; WX 496 ; N gcircumflex.alt ; B 36 -241 442 767 ;
+C -1 ; WX 496 ; N gbreve.alt ; B 36 -241 442 766 ;
+C -1 ; WX 496 ; N gdot.alt ; B 36 -241 442 756 ;
+C -1 ; WX 496 ; N gcommaaccent.alt ; B 36 -241 442 767 ;
+C -1 ; WX 0 ; N cyrotmarkcomb ; B -203 591 203 714 ;
+EndCharMetrics
+StartKernData
+StartKernPairs 15878
+KPX quotedbl uni1ECA 20
+KPX quotedbl uni1EC8 20
+KPX quotedbl Idotaccent 20
+KPX quotedbl Iogonek 20
+KPX quotedbl Imacron 20
+KPX quotedbl Idieresis 20
+KPX quotedbl Icircumflex 20
+KPX quotedbl Iacute 20
+KPX quotedbl Igrave 20
+KPX quotedbl I 20
+KPX quotedbl uni1EF9 20
+KPX quoteleft q -20
+KPX quoteleft o -20
+KPX quoteleft g -9
+KPX quoteleft e -20
+KPX quoteleft d -20
+KPX quoteleft c -20
+KPX quoteleft Z 20
+KPX Delta C -9
+KPX Delta A -20
+KPX Delta question 20
+KPX Delta period -41
+KPX Delta comma -41
+KPX Delta quotesingle 41
+KPX Delta quotedbl 41
+EndKernPairs
+EndKernData
+EndFontMetrics

samples/Adobe Font Metrics/SpecialElite.afm

@@ -0,0 +1,464 @@
+StartFontMetrics 2.0
+Comment Generated by FontForge 20170719
+Comment Creation Date: Sun Jul 23 19:52:19 2017
+FontName SpecialElite-Regular
+FullName Special Elite
+FamilyName Special Elite
+Weight Book
+Notice (Copyright (c) 2010 by Brian J. Bonislawsky DBA Astigmatic (AOETI). All rights reserved. Available under the Apache 2.0 licence.http://www.apache.org/licenses/LICENSE-2.0.html)
+ItalicAngle 0
+IsFixedPitch false
+UnderlinePosition -133
+UnderlineThickness 20
+Version 1.000
+EncodingScheme ISO10646-1
+FontBBox -33 -322 1052 959
+CapHeight 714
+XHeight 487
+Ascender 688
+Descender -225
+StartCharMetrics 371
+C 32 ; WX 292 ; N space ; B 0 0 0 0 ;
+C 33 ; WX 276 ; N exclam ; B 73 0 207 702 ;
+C 34 ; WX 352 ; N quotedbl ; B 48 449 295 704 ;
+C 35 ; WX 554 ; N numbersign ; B 31 -2 524 713 ;
+C 36 ; WX 526 ; N dollar ; B 31 -201 498 919 ;
+C 37 ; WX 666 ; N percent ; B 32 -186 642 872 ;
+C 38 ; WX 676 ; N ampersand ; B 31 -5 645 705 ;
+C 39 ; WX 196 ; N quotesingle ; B 48 449 143 703 ;
+C 40 ; WX 279 ; N parenleft ; B 55 -71 243 757 ;
+C 41 ; WX 281 ; N parenright ; B 37 -59 229 770 ;
+C 42 ; WX 522 ; N asterisk ; B 32 276 493 707 ;
+C 43 ; WX 496 ; N plus ; B 29 131 465 560 ;
+C 44 ; WX 336 ; N comma ; B 39 -197 290 251 ;
+C 45 ; WX 636 ; N hyphen ; B 63 273 573 397 ;
+C 46 ; WX 349 ; N period ; B 52 -3 298 245 ;
+C 47 ; WX 557 ; N slash ; B 23 -41 536 760 ;
+C 48 ; WX 610 ; N zero ; B 55 0 560 720 ;
+C 49 ; WX 569 ; N one ; B 27 -12 572 712 ;
+C 50 ; WX 573 ; N two ; B 50 -25 541 680 ;
+C 51 ; WX 557 ; N three ; B 44 -25 514 694 ;
+C 52 ; WX 612 ; N four ; B 15 4 584 708 ;
+C 53 ; WX 537 ; N five ; B 47 0 505 690 ;
+C 54 ; WX 588 ; N six ; B 48 -10 548 707 ;
+C 55 ; WX 555 ; N seven ; B 15 -34 549 734 ;
+C 56 ; WX 598 ; N eight ; B 51 1 551 720 ;
+C 57 ; WX 584 ; N nine ; B 48 -2 539 715 ;
+C 58 ; WX 343 ; N colon ; B 51 -3 297 518 ;
+C 59 ; WX 328 ; N semicolon ; B 45 -197 297 518 ;
+C 60 ; WX 463 ; N less ; B 31 120 401 565 ;
+C 61 ; WX 636 ; N equal ; B 63 186 573 513 ;
+C 62 ; WX 463 ; N greater ; B 62 120 433 565 ;
+C 63 ; WX 470 ; N question ; B 34 2 442 729 ;
+C 64 ; WX 665 ; N at ; B 46 -4 618 680 ;
+C 65 ; WX 549 ; N A ; B -1 -16 550 703 ;
+C 66 ; WX 604 ; N B ; B 29 -6 557 704 ;
+C 67 ; WX 579 ; N C ; B 46 -13 531 700 ;
+C 68 ; WX 622 ; N D ; B 36 -17 579 713 ;
+C 69 ; WX 638 ; N E ; B 38 -16 587 691 ;
+C 70 ; WX 605 ; N F ; B 29 -9 595 709 ;
+C 71 ; WX 615 ; N G ; B 45 -3 586 710 ;
+C 72 ; WX 652 ; N H ; B 40 -20 622 690 ;
+C 73 ; WX 495 ; N I ; B 26 -24 469 710 ;
+C 74 ; WX 541 ; N J ; B 16 -3 539 703 ;
+C 75 ; WX 582 ; N K ; B 28 -5 584 711 ;
+C 76 ; WX 602 ; N L ; B 23 -14 583 718 ;
+C 77 ; WX 697 ; N M ; B 46 -10 655 704 ;
+C 78 ; WX 627 ; N N ; B 41 -15 595 700 ;
+C 79 ; WX 616 ; N O ; B 42 -30 574 702 ;
+C 80 ; WX 553 ; N P ; B 30 -12 527 689 ;
+C 81 ; WX 602 ; N Q ; B 42 -98 571 711 ;
+C 82 ; WX 636 ; N R ; B 14 -9 624 706 ;
+C 83 ; WX 588 ; N S ; B 51 -13 547 690 ;
+C 84 ; WX 594 ; N T ; B 25 1 564 707 ;
+C 85 ; WX 621 ; N U ; B 24 -6 611 710 ;
+C 86 ; WX 611 ; N V ; B -1 -15 614 726 ;
+C 87 ; WX 643 ; N W ; B 8 0 614 689 ;
+C 88 ; WX 582 ; N X ; B 3 -1 580 697 ;
+C 89 ; WX 561 ; N Y ; B -21 -2 562 719 ;
+C 90 ; WX 592 ; N Z ; B 49 -1 551 709 ;
+C 91 ; WX 312 ; N bracketleft ; B 85 -72 297 754 ;
+C 92 ; WX 557 ; N backslash ; B 21 -41 534 760 ;
+C 249 ; WX 639 ; N ugrave ; B 5 -28 624 679 ;
+C 250 ; WX 639 ; N uacute ; B 5 -28 624 682 ;
+C 251 ; WX 639 ; N ucircumflex ; B 5 -28 624 691 ;
+C 252 ; WX 639 ; N udieresis ; B 5 -28 624 649 ;
+C 253 ; WX 592 ; N yacute ; B 0 -232 596 666 ;
+C 254 ; WX 552 ; N thorn ; B -33 -221 512 699 ;
+C 255 ; WX 592 ; N ydieresis ; B 0 -232 596 643 ;
+C -1 ; WX 549 ; N Amacron ; B -1 -16 550 809 ;
+C -1 ; WX 565 ; N amacron ; B 38 -6 561 619 ;
+C -1 ; WX 549 ; N Abreve ; B -1 -16 550 890 ;
+C -1 ; WX 565 ; N abreve ; B 38 -6 561 686 ;
+C -1 ; WX 549 ; N Aogonek ; B -1 -138 589 703 ;
+C -1 ; WX 565 ; N aogonek ; B 38 -118 624 502 ;
+C -1 ; WX 579 ; N Cacute ; B 46 -13 531 900 ;
+C -1 ; WX 547 ; N cacute ; B 39 -22 506 693 ;
+C -1 ; WX 579 ; N Ccircumflex ; B 46 -13 531 890 ;
+C -1 ; WX 547 ; N ccircumflex ; B 39 -22 506 689 ;
+C -1 ; WX 579 ; N Cdotaccent ; B 46 -13 531 859 ;
+C -1 ; WX 547 ; N cdotaccent ; B 39 -22 506 657 ;
+C -1 ; WX 579 ; N Ccaron ; B 46 -13 531 918 ;
+C -1 ; WX 547 ; N ccaron ; B 39 -22 506 710 ;
+C -1 ; WX 622 ; N Dcaron ; B 36 -17 579 924 ;
+C -1 ; WX 750 ; N dcaron ; B 40 -26 716 704 ;
+C -1 ; WX 623 ; N Dcroat ; B 36 -17 580 713 ;
+C -1 ; WX 603 ; N dcroat ; B 40 -26 597 714 ;
+C -1 ; WX 638 ; N Emacron ; B 38 -16 587 798 ;
+C -1 ; WX 543 ; N emacron ; B 40 -23 501 616 ;
+C -1 ; WX 638 ; N Ebreve ; B 38 -16 587 876 ;
+C -1 ; WX 543 ; N ebreve ; B 40 -23 501 683 ;
+C -1 ; WX 638 ; N Edotaccent ; B 38 -16 587 848 ;
+C -1 ; WX 543 ; N edotaccent ; B 40 -23 501 659 ;
+C -1 ; WX 638 ; N Eogonek ; B 38 -113 610 691 ;
+C -1 ; WX 543 ; N eogonek ; B 40 -145 501 499 ;
+C -1 ; WX 638 ; N Ecaron ; B 38 -16 587 913 ;
+C -1 ; WX 543 ; N ecaron ; B 40 -23 501 714 ;
+C -1 ; WX 615 ; N Gcircumflex ; B 45 -3 586 906 ;
+C -1 ; WX 583 ; N gcircumflex ; B 42 -224 562 676 ;
+C -1 ; WX 615 ; N Gbreve ; B 45 -3 586 899 ;
+C -1 ; WX 583 ; N gbreve ; B 42 -224 562 667 ;
+C -1 ; WX 615 ; N Gdotaccent ; B 45 -3 586 871 ;
+C -1 ; WX 583 ; N gdotaccent ; B 42 -224 562 637 ;
+C -1 ; WX 615 ; N Gcommaaccent ; B 45 -253 586 710 ;
+C -1 ; WX 583 ; N gcommaaccent ; B 42 -224 562 734 ;
+C -1 ; WX 652 ; N Hcircumflex ; B 40 -20 622 897 ;
+C -1 ; WX 616 ; N hcircumflex ; B 5 -29 601 688 ;
+C -1 ; WX 652 ; N Hbar ; B 40 -20 622 690 ;
+C -1 ; WX 616 ; N hbar ; B 5 -29 601 683 ;
+C -1 ; WX 495 ; N Itilde ; B 26 -24 469 859 ;
+C -1 ; WX 568 ; N itilde ; B 36 -42 568 615 ;
+C -1 ; WX 495 ; N Imacron ; B 26 -24 469 819 ;
+C -1 ; WX 568 ; N imacron ; B 36 -42 568 585 ;
+C -1 ; WX 495 ; N Ibreve ; B 26 -24 469 901 ;
+C -1 ; WX 568 ; N ibreve ; B 36 -42 568 661 ;
+C -1 ; WX 495 ; N Iogonek ; B 26 -154 469 710 ;
+C -1 ; WX 568 ; N iogonek ; B 36 -149 568 674 ;
+C -1 ; WX 495 ; N Idotaccent ; B 26 -24 469 873 ;
+C -1 ; WX 568 ; N dotlessi ; B 36 -42 568 468 ;
+C -1 ; WX 1036 ; N IJ ; B 26 -24 1034 710 ;
+C -1 ; WX 983 ; N ij ; B 36 -236 913 683 ;
+C -1 ; WX 541 ; N Jcircumflex ; B 16 -3 539 913 ;
+C -1 ; WX 415 ; N jcircumflex ; B -12 -236 405 699 ;
+C -1 ; WX 582 ; N Kcommaaccent ; B 28 -253 584 711 ;
+C -1 ; WX 620 ; N kcommaaccent ; B 11 -253 600 683 ;
+C -1 ; WX 620 ; N kgreenlandic ; B 11 -28 600 482 ;
+C -1 ; WX 602 ; N Lacute ; B 23 -14 583 923 ;
+C -1 ; WX 540 ; N lacute ; B 4 -28 538 902 ;
+C -1 ; WX 602 ; N Lcommaaccent ; B 23 -267 583 718 ;
+C -1 ; WX 540 ; N lcommaaccent ; B 4 -267 538 682 ;
+C -1 ; WX 602 ; N Lcaron ; B 23 -14 583 794 ;
+C -1 ; WX 582 ; N lcaron ; B 4 -28 549 704 ;
+C -1 ; WX 781 ; N Ldot ; B 23 -14 748 718 ;
+C -1 ; WX 571 ; N ldotaccent ; B 4 -28 538 682 ;
+C -1 ; WX 603 ; N Lslash ; B 24 -14 584 718 ;
+C -1 ; WX 541 ; N lslash ; B 4 -28 538 682 ;
+C -1 ; WX 627 ; N Nacute ; B 41 -15 595 894 ;
+C -1 ; WX 632 ; N nacute ; B 32 -23 612 696 ;
+C -1 ; WX 627 ; N Ncommaaccent ; B 41 -268 595 700 ;
+C -1 ; WX 632 ; N ncommaaccent ; B 32 -268 612 491 ;
+C -1 ; WX 627 ; N Ncaron ; B 41 -15 595 900 ;
+C -1 ; WX 632 ; N ncaron ; B 32 -23 612 712 ;
+C -1 ; WX 815 ; N napostrophe ; B 34 -23 795 704 ;
+C -1 ; WX 627 ; N Eng ; B 41 -320 595 700 ;
+C -1 ; WX 605 ; N eng ; B 32 -322 534 491 ;
+C -1 ; WX 616 ; N Omacron ; B 42 -30 574 815 ;
+C -1 ; WX 583 ; N omacron ; B 40 -34 543 598 ;
+C -1 ; WX 616 ; N Obreve ; B 42 -30 574 891 ;
+C -1 ; WX 583 ; N obreve ; B 40 -34 543 675 ;
+C -1 ; WX 616 ; N Ohungarumlaut ; B 42 -30 574 907 ;
+C -1 ; WX 583 ; N ohungarumlaut ; B 40 -34 545 693 ;
+C -1 ; WX 1018 ; N OE ; B 42 -30 967 702 ;
+C -1 ; WX 958 ; N oe ; B 40 -34 916 499 ;
+C -1 ; WX 636 ; N Racute ; B 14 -9 624 910 ;
+C -1 ; WX 579 ; N racute ; B 28 -16 566 693 ;
+C -1 ; WX 636 ; N Rcommaaccent ; B 14 -268 624 706 ;
+C -1 ; WX 579 ; N rcommaaccent ; B 28 -272 566 495 ;
+C -1 ; WX 636 ; N Rcaron ; B 14 -9 624 927 ;
+C -1 ; WX 579 ; N rcaron ; B 28 -16 566 698 ;
+C -1 ; WX 588 ; N Sacute ; B 51 -13 547 900 ;
+C -1 ; WX 519 ; N sacute ; B 48 -31 481 713 ;
+C -1 ; WX 588 ; N Scircumflex ; B 51 -13 547 904 ;
+C -1 ; WX 519 ; N scircumflex ; B 48 -31 481 710 ;
+C -1 ; WX 588 ; N Scedilla ; B 51 -145 547 690 ;
+C -1 ; WX 519 ; N scedilla ; B 48 -145 481 496 ;
+C -1 ; WX 588 ; N Scaron ; B 51 -13 547 904 ;
+C -1 ; WX 519 ; N scaron ; B 48 -31 481 710 ;
+C -1 ; WX 594 ; N Tcommaaccent ; B 25 -263 564 707 ;
+C -1 ; WX 510 ; N tcommaaccent ; B 0 -282 488 694 ;
+C -1 ; WX 594 ; N Tcaron ; B 25 1 564 920 ;
+C -1 ; WX 713 ; N tcaron ; B 0 -34 680 704 ;
+C -1 ; WX 594 ; N Tbar ; B 25 1 564 707 ;
+C -1 ; WX 510 ; N tbar ; B 0 -34 488 694 ;
+C -1 ; WX 621 ; N Utilde ; B 24 -6 611 850 ;
+C -1 ; WX 638 ; N utilde ; B 5 -28 624 636 ;
+C -1 ; WX 621 ; N Umacron ; B 24 -6 611 811 ;
+C -1 ; WX 638 ; N umacron ; B 5 -28 624 587 ;
+C -1 ; WX 621 ; N Ubreve ; B 24 -6 611 888 ;
+C -1 ; WX 638 ; N ubreve ; B 5 -28 624 665 ;
+C -1 ; WX 621 ; N Uring ; B 24 -6 611 959 ;
+C -1 ; WX 638 ; N uring ; B 5 -28 624 738 ;
+C -1 ; WX 621 ; N Uhungarumlaut ; B 24 -6 611 918 ;
+C -1 ; WX 638 ; N uhungarumlaut ; B 5 -28 624 691 ;
+C -1 ; WX 621 ; N Uogonek ; B 24 -136 611 710 ;
+C -1 ; WX 638 ; N uogonek ; B 5 -147 671 487 ;
+C -1 ; WX 643 ; N Wcircumflex ; B 8 0 614 901 ;
+C -1 ; WX 678 ; N wcircumflex ; B 5 -10 674 685 ;
+C -1 ; WX 561 ; N Ycircumflex ; B -21 -2 562 934 ;
+C -1 ; WX 592 ; N ycircumflex ; B 0 -232 596 691 ;
+C -1 ; WX 561 ; N Ydieresis ; B -21 -2 562 885 ;
+C -1 ; WX 592 ; N Zacute ; B 49 -1 551 905 ;
+C -1 ; WX 528 ; N zacute ; B 45 -22 487 684 ;
+C -1 ; WX 592 ; N Zdotaccent ; B 49 -1 551 866 ;
+C -1 ; WX 528 ; N zdotaccent ; B 45 -22 487 632 ;
+C -1 ; WX 592 ; N Zcaron ; B 49 -1 551 917 ;
+C -1 ; WX 528 ; N zcaron ; B 45 -22 487 688 ;
+C -1 ; WX 915 ; N AEacute ; B -11 -16 864 904 ;
+C -1 ; WX 888 ; N aeacute ; B 38 -23 846 670 ;
+C -1 ; WX 617 ; N Oslashacute ; B 43 -41 574 912 ;
+C -1 ; WX 583 ; N oslashacute ; B 40 -73 543 697 ;
+C -1 ; WX 415 ; N dotlessj ; B -12 -236 344 478 ;
+C -1 ; WX 281 ; N circumflex ; B 0 558 282 746 ;
+C -1 ; WX 281 ; N caron ; B 0 558 282 746 ;
+C -1 ; WX 281 ; N breve ; B 0 585 282 746 ;
+C -1 ; WX 132 ; N dotaccent ; B 0 600 133 729 ;
+C -1 ; WX 214 ; N ring ; B 0 547 215 780 ;
+C -1 ; WX 211 ; N ogonek ; B 0 -145 212 13 ;
+C -1 ; WX 283 ; N tilde ; B 0 583 284 701 ;
+C -1 ; WX 352 ; N hungarumlaut ; B 0 591 353 763 ;
+C -1 ; WX 185 ; N uni0312 ; B 28 474 152 694 ;
+C -1 ; WX 185 ; N uni0315 ; B 38 470 162 690 ;
+C -1 ; WX 192 ; N uni0326 ; B 32 -253 156 -33 ;
+C -1 ; WX 666 ; N mu ; B 24 -219 643 487 ;
+C -1 ; WX 643 ; N Wgrave ; B 8 0 614 895 ;
+C -1 ; WX 678 ; N wgrave ; B 5 -10 674 688 ;
+C -1 ; WX 643 ; N Wacute ; B 8 0 614 898 ;
+C -1 ; WX 678 ; N wacute ; B 5 -10 674 682 ;
+C -1 ; WX 643 ; N Wdieresis ; B 8 0 614 868 ;
+C -1 ; WX 678 ; N wdieresis ; B 5 -10 674 649 ;
+C -1 ; WX 561 ; N Ygrave ; B -21 -2 562 900 ;
+C -1 ; WX 592 ; N ygrave ; B 0 -232 596 666 ;
+C -1 ; WX 611 ; N endash ; B 50 270 551 391 ;
+C -1 ; WX 1113 ; N emdash ; B 51 270 1052 391 ;
+C -1 ; WX 265 ; N quoteleft ; B 41 390 217 704 ;
+C -1 ; WX 264 ; N quoteright ; B 54 390 230 704 ;
+C -1 ; WX 274 ; N quotesinglbase ; B 46 -138 223 176 ;
+C -1 ; WX 470 ; N quotedblleft ; B 41 390 422 704 ;
+C -1 ; WX 469 ; N quotedblright ; B 54 390 436 704 ;
+C -1 ; WX 479 ; N quotedblbase ; B 46 -138 428 176 ;
+C -1 ; WX 389 ; N dagger ; B 30 -16 359 724 ;
+C -1 ; WX 396 ; N daggerdbl ; B 35 -16 364 728 ;
+C -1 ; WX 316 ; N bullet ; B 50 246 266 479 ;
+C -1 ; WX 1063 ; N ellipsis ; B 52 -3 1016 245 ;
+C -1 ; WX 897 ; N perthousand ; B 33 -230 873 828 ;
+C -1 ; WX 296 ; N guilsinglleft ; B 44 149 232 434 ;
+C -1 ; WX 295 ; N guilsinglright ; B 63 149 251 434 ;
+C -1 ; WX 486 ; N fraction ; B -11 -53 501 748 ;
+C -1 ; WX 732 ; N Euro ; B 31 71 683 590 ;
+C -1 ; WX 757 ; N trademark ; B 60 303 703 693 ;
+C -1 ; WX 585 ; N partialdiff ; B 36 -47 553 772 ;
+C -1 ; WX 564 ; N product ; B 26 -17 534 707 ;
+C -1 ; WX 577 ; N minus ; B 63 282 514 395 ;
+C -1 ; WX 565 ; N approxequal ; B 59 137 513 522 ;
+C -1 ; WX 593 ; N notequal ; B 44 71 554 644 ;
+C -1 ; WX 1041 ; N fi ; B 20 -42 1041 702 ;
+C -1 ; WX 1013 ; N fl ; B 20 -29 1011 702 ;
+C -1 ; WX 292 ; N .notdef ; B 0 0 0 0 ;
+C -1 ; WX 0 ; N .null ; B 0 0 0 0 ;
+C -1 ; WX 292 ; N nonmarkingreturn ; B 0 0 0 0 ;
+EndCharMetrics
+StartKernData
+StartKernPairs 6408
+KPX quotedbl period -104
+KPX quotedbl comma -103
+KPX quotedbl Jcircumflex -34
+KPX quotedbl Aogonek -31
+KPX quotedbl Abreve -31
+KPX quotedbl Amacron -31
+KPX quotedbl AEacute -31
+KPX quotedbl Aacute -31
+KPX quotedbl Acircumflex -31
+KPX quotedbl Atilde -31
+KPX quotedbl Agrave -31
+KPX quotedbl Aring -31
+KPX quotedbl Adieresis -31
+KPX quotedbl AE -31
+KPX quotedbl J -34
+KPX quotedbl A -31
+KPX quotedbl quotedblbase -117
+KPX quotedbl quotesinglbase -117
+KPX quotedbl ellipsis -104
+KPX quotedbl slash -73
+KPX quotedbl ampersand -22
+KPX quotedbl four -27
+KPX ampersand Ycircumflex -40
+KPX ampersand Ygrave -40
+KPX ampersand Ydieresis -40
+KPX ampersand Yacute -40
+KPX ampersand Y -40
+KPX ampersand V -36
+KPX quotesingle period -97
+KPX quotesingle comma -97
+KPX quotesingle Jcircumflex -34
+KPX quotesingle Aogonek -31
+KPX quotesingle Abreve -31
+KPX quotesingle Amacron -31
+KPX hyphen T -28
+KPX hyphen one -68
+KPX hyphen B -25
+KPX hyphen seven -56
+KPX slash rcommaaccent -27
+KPX slash ncommaaccent -29
+KPX slash gcommaaccent -61
+KPX slash Jcircumflex -29
+KPX slash iogonek -26
+KPX slash ibreve -26
+KPX slash imacron -26
+KPX slash itilde -26
+KPX slash oslashacute -54
+KPX slash nacute -29
+KPX slash eng -29
+KPX slash ncaron -29
+KPX slash racute -27
+KPX slash scedilla -43
+KPX slash scircumflex -43
+KPX slash sacute -43
+KPX slash rcaron -27
+KPX slash ohungarumlaut -54
+KPX slash obreve -54
+KPX slash omacron -54
+KPX slash wgrave -23
+KPX slash wcircumflex -23
+KPX slash wdieresis -23
+KPX slash wacute -23
+KPX slash zdotaccent -41
+KPX J ebreve -32
+KPX J emacron -32
+KPX J edieresis -32
+KPX J ecircumflex -32
+KPX J egrave -32
+KPX J eacute -32
+KPX J e -32
+KPX J Aogonek -34
+KPX J Abreve -34
+KPX J Amacron -34
+KPX J AEacute -34
+KPX J Aacute -34
+KPX J Acircumflex -34
+KPX J Atilde -34
+KPX J Agrave -34
+KPX J Aring -34
+KPX J Adieresis -34
+KPX J AE -34
+KPX J A -34
+KPX J comma -29
+KPX J period -30
+KPX J v -29
+KPX J hyphen -30
+KPX J quotedblbase -34
+KPX J quotesinglbase -34
+KPX J guilsinglright -25
+KPX J guilsinglleft -25
+KPX J emdash -30
+KPX J endash -30
+KPX J guillemotright -25
+KPX J guillemotleft -25
+KPX J germandbls -36
+KPX J ellipsis -30
+KPX J slash -34
+KPX J p -28
+KPX J m -35
+KPX J b 54
+KPX K ycircumflex -60
+KPX K ygrave -60
+KPX K ydieresis -60
+KPX K yacute -60
+KPX K y -60
+KPX K wgrave -36
+KPX K wcircumflex -36
+KPX K wdieresis -36
+KPX K wacute -36
+KPX K w -36
+KPX K uogonek -25
+KPX K uhungarumlaut -25
+KPX K uring -25
+KPX K ubreve -25
+KPX K umacron -25
+KPX K utilde -25
+KPX K udieresis -25
+KPX K ucircumflex -25
+KPX K ugrave -25
+KPX K uacute -25
+KPX K u -25
+KPX K q -23
+KPX K oslashacute -28
+KPX K ohungarumlaut -28
+KPX K obreve -28
+KPX K omacron -28
+KPX K otilde -28
+KPX K odieresis -28
+KPX K ocircumflex -28
+KPX K ograve -28
+KPX K oacute -28
+KPX K eth -28
+KPX K oe -28
+KPX K oslash -28
+KPX K o -28
+KPX K dcaron -24
+KPX K d -24
+KPX K ccaron -27
+KPX K cdotaccent -27
+KPX K ccircumflex -27
+KPX K cacute -27
+KPX K ccedilla -27
+KPX K c -27
+KPX K ecaron -27
+KPX K eogonek -27
+KPX K edotaccent -27
+KPX K ebreve -27
+KPX K emacron -27
+KPX K edieresis -27
+KPX K ecircumflex -27
+KPX K egrave -27
+KPX K eacute -27
+KPX K e -27
+KPX K v -49
+KPX K hyphen -38
+KPX K guilsinglleft -24
+KPX K emdash -38
+KPX K endash -38
+KPX K guillemotleft -24
+KPX K b 49
+KPX L ycircumflex -36
+KPX L ygrave -36
+KPX L ydieresis -36
+KPX L yacute -36
+KPX L y -36
+KPX L wgrave -23
+KPX L wcircumflex -23
+KPX L wdieresis -23
+KPX L wacute -23
+KPX L w -23
+KPX L V -43
+KPX L Tcommaaccent -36
+KPX L Tbar -36
+KPX L Tcaron -36
+KPX L T -36
+KPX L quoteright -49
+KPX L v -32
+KPX L quoteleft -54
+KPX L quotedblright -49
+KPX L quotedblleft -54
+KPX L trademark -29
+KPX L backslash -50
+KPX L asterisk -30
+KPX trademark Aring -24
+KPX trademark Adieresis -24
+KPX trademark Yacute 29
+KPX trademark AE -24
+KPX trademark Y 29
+KPX trademark A -24
+KPX trademark b 31
+EndKernPairs
+EndKernData
+EndFontMetrics

samples/Adobe Font Metrics/lambda.afm

@@ -0,0 +1,23 @@
+StartFontMetrics 2.0
+Comment Generated by FontForge 20170719
+Comment Creation Date: Sun Jul 23 23:14:02 2017
+FontName Greek_Lambda_Character-Regular
+FullName Greek_Lambda_Character Regular
+FamilyName Greek_Lambda_Character
+Weight Regular
+Notice (NONE. NADA. PUBLIC DOMAIN, BOI)
+ItalicAngle 0
+IsFixedPitch false
+UnderlinePosition -175
+UnderlineThickness 90
+Version 020.017
+EncodingScheme ISO10646-1
+FontBBox 33 -177 566 760
+StartCharMetrics 5
+C 13 ; WX 602 ; N uni000D ; B 0 0 0 0 ;
+C 32 ; WX 602 ; N space ; B 0 0 0 0 ;
+C -1 ; WX 602 ; N lambda ; B 33 0 566 760 ;
+C -1 ; WX 602 ; N .notdef ; B 50 -177 551 706 ;
+C -1 ; WX 0 ; N NULL ; B 0 0 0 0 ;
+EndCharMetrics
+EndFontMetrics

samples/AngelScript/botmanager.as

@@ -0,0 +1,77 @@
+/*
+*	This is a sample script.
+*/
+
+#include "BotManagerInterface.acs"
+
+BotManager::BotManager g_BotManager( @CreateDumbBot );
+
+CConCommand@ m_pAddBot;
+
+void PluginInit()
+{
+	g_BotManager.PluginInit();
+	
+	@m_pAddBot = @CConCommand( "addbot", "Adds a new bot with the given name", @AddBotCallback );
+}
+
+void AddBotCallback( const CCommand@ args )
+{
+	if( args.ArgC() < 2 )
+	{
+		g_Game.AlertMessage( at_console, "Usage: addbot <name>" );
+		return;
+	}
+	
+	BotManager::BaseBot@ pBot = g_BotManager.CreateBot( args[ 1 ] );
+	
+	if( pBot !is null )
+	{
+		g_Game.AlertMessage( at_console, "Created bot " + args[ 1 ] + "\n" );
+	}
+	else
+	{
+		g_Game.AlertMessage( at_console, "Could not create bot\n" );
+	}
+}
+
+final class DumbBot : BotManager::BaseBot
+{	
+	DumbBot( CBasePlayer@ pPlayer )
+	{
+		super( pPlayer );
+	}
+	
+	void Think()
+	{
+		BotManager::BaseBot::Think();
+		
+		// If the bot is dead and can be respawned, send a button press
+		if( Player.pev.deadflag >= DEAD_RESPAWNABLE )
+		{
+			Player.pev.button |= IN_ATTACK;
+		}
+		else
+			Player.pev.button &= ~IN_ATTACK;
+		
+		KeyValueBuffer@ pInfoBuffer = g_EngineFuncs.GetInfoKeyBuffer( Player.edict() );
+		
+		pInfoBuffer.SetValue( "topcolor", Math.RandomLong( 0, 255 ) );
+		pInfoBuffer.SetValue( "bottomcolor", Math.RandomLong( 0, 255 ) );
+		
+		if( Math.RandomLong( 0, 100 ) > 10 )
+			Player.pev.button |= IN_ATTACK;
+		else
+			Player.pev.button &= ~IN_ATTACK;
+			
+		for( uint uiIndex = 0; uiIndex < 3; ++uiIndex )
+		{
+			m_vecVelocity[ uiIndex ] = Math.RandomLong( -50, 50 );
+		}
+	}
+}
+
+BotManager::BaseBot@ CreateDumbBot( CBasePlayer@ pPlayer )
+{
+	return @DumbBot( pPlayer );
+}

samples/AngelScript/payload.as

@@ -0,0 +1,396 @@
+// Sample script.
+// Source: https://github.com/codecat/ssbd-payload
+
+array<WorldScript::PayloadBeginTrigger@> g_payloadBeginTriggers;
+array<WorldScript::PayloadTeamForcefield@> g_teamForceFields;
+
+[GameMode]
+class Payload : TeamVersusGameMode
+{
+	[Editable]
+	UnitFeed PayloadUnit;
+
+	[Editable]
+	UnitFeed FirstNode;
+
+	[Editable default=10]
+	int PrepareTime;
+
+	[Editable default=300]
+	int TimeLimit;
+
+	[Editable default=90]
+	int TimeAddCheckpoint;
+
+	[Editable default=2]
+	float TimeOvertime;
+
+	[Editable default=1000]
+	int TimePayloadHeal;
+
+	[Editable default=1]
+	int PayloadHeal;
+
+	PayloadBehavior@ m_payload;
+
+	int m_tmStarting;
+	int m_tmStarted;
+	int m_tmLimitCustom;
+	int m_tmOvertime;
+	int m_tmInOvertime;
+
+	PayloadHUD@ m_payloadHUD;
+	PayloadClassSwitchWindow@ m_switchClass;
+
+	array<SValue@>@ m_switchedSidesData;
+
+	Payload(Scene@ scene)
+	{
+		super(scene);
+
+		m_tmRespawnCountdown = 5000;
+
+		@m_payloadHUD = PayloadHUD(m_guiBuilder);
+		@m_switchTeam = PayloadTeamSwitchWindow(m_guiBuilder);
+		@m_switchClass = PayloadClassSwitchWindow(m_guiBuilder);
+	}
+
+	void UpdateFrame(int ms, GameInput& gameInput, MenuInput& menuInput) override
+	{
+		TeamVersusGameMode::UpdateFrame(ms, gameInput, menuInput);
+
+		m_payloadHUD.Update(ms);
+
+		if (Network::IsServer())
+		{
+			uint64 tmNow = CurrPlaytimeLevel();
+
+			if (m_tmStarting == 0)
+			{
+				if (GetPlayersInTeam(0) > 0 && GetPlayersInTeam(1) > 0)
+				{
+					m_tmStarting = tmNow;
+					(Network::Message("GameStarting") << m_tmStarting).SendToAll();
+				}
+			}
+
+			if (m_tmStarting > 0 && m_tmStarted == 0 && tmNow - m_tmStarting > PrepareTime * 1000)
+			{
+				m_tmStarted = tmNow;
+				(Network::Message("GameStarted") << m_tmStarted).SendToAll();
+
+				for (uint i = 0; i < g_payloadBeginTriggers.length(); i++)
+				{
+					WorldScript@ ws = WorldScript::GetWorldScript(g_scene, g_payloadBeginTriggers[i]);
+					ws.Execute();
+				}
+			}
+		}
+
+		if (!m_ended && m_tmStarted > 0)
+			CheckTimeReached(ms);
+	}
+
+	string NameForTeam(int index) override
+	{
+		if (index == 0)
+			return "Defenders";
+		else if (index == 1)
+			return "Attackers";
+
+		return "Unknown";
+	}
+
+	void CheckTimeReached(int dt)
+	{
+		// Check if time limit is not reached yet
+		if (m_tmLimitCustom - (CurrPlaytimeLevel() - m_tmStarted) > 0)
+		{
+			// Don't need to continue checking
+			m_tmOvertime = 0;
+			m_tmInOvertime = 0;
+			return;
+		}
+
+		// Count how long we're in overtime for later time limit fixing when we reach a checkpoint
+		if (m_tmOvertime > 0)
+			m_tmInOvertime += dt;
+
+		// Check if there are any attackers still inside
+		if (m_payload.AttackersInside() > 0)
+		{
+			// We have overtime
+			m_tmOvertime = int(TimeOvertime * 1000);
+			return;
+		}
+
+		// If we have overtime
+		if (m_tmOvertime > 0)
+		{
+			// Decrease timer
+			m_tmOvertime -= dt;
+			if (m_tmOvertime <= 0)
+			{
+				// Overtime countdown reached, time limit reached
+				TimeReached();
+			}
+		}
+		else
+		{
+			// No overtime, so time limit is reached
+			TimeReached();
+		}
+	}
+
+	void TimeReached()
+	{
+		if (!Network::IsServer())
+			return;
+
+		(Network::Message("TimeReached")).SendToAll();
+		SetWinner(false);
+	}
+
+	bool ShouldFreezeControls() override
+	{
+		return m_switchClass.m_visible
+		    || TeamVersusGameMode::ShouldFreezeControls();
+	}
+
+	bool ShouldDisplayCursor() override
+	{
+		return m_switchClass.m_visible
+		    || TeamVersusGameMode::ShouldDisplayCursor();
+	}
+
+	bool CanSwitchTeams() override
+	{
+		return m_tmStarted == 0;
+	}
+
+	PlayerRecord@ CreatePlayerRecord() override
+	{
+		return PayloadPlayerRecord();
+	}
+
+	int GetPlayerClassCount(PlayerClass playerClass, TeamVersusScore@ team)
+	{
+		if (team is null)
+			return 0;
+
+		int ret = 0;
+		for (uint i = 0; i < team.m_players.length(); i++)
+		{
+			if (team.m_players[i].peer == 255)
+				continue;
+			auto record = cast<PayloadPlayerRecord>(team.m_players[i]);
+			if (record.playerClass == playerClass)
+				ret++;
+		}
+		return ret;
+	}
+
+	void PlayerClassesUpdated()
+	{
+		m_switchClass.PlayerClassesUpdated();
+	}
+
+	void SetWinner(bool attackers)
+	{
+		if (attackers)
+			print("Attackers win!");
+		else
+			print("Defenders win!");
+
+		m_payloadHUD.Winner(attackers);
+		EndMatch();
+	}
+
+	void DisplayPlayerName(int idt, SpriteBatch& sb, PlayerRecord@ record, PlayerHusk@ plr, vec2 pos) override
+	{
+		TeamVersusGameMode::DisplayPlayerName(idt, sb, record, plr, pos);
+
+		m_payloadHUD.DisplayPlayerName(idt, sb, cast<PayloadPlayerRecord>(record), plr, pos);
+	}
+
+	void RenderFrame(int idt, SpriteBatch& sb) override
+	{
+		Player@ player = GetLocalPlayer();
+		if (player !is null)
+		{
+			PlayerHealgun@ healgun = cast<PlayerHealgun>(player.m_currWeapon);
+			if (healgun !is null)
+				healgun.RenderMarkers(idt, sb);
+		}
+
+		TeamVersusGameMode::RenderFrame(idt, sb);
+	}
+
+	void RenderWidgets(PlayerRecord@ player, int idt, SpriteBatch& sb) override
+	{
+		m_payloadHUD.Draw(sb, idt);
+
+		TeamVersusGameMode::RenderWidgets(player, idt, sb);
+
+		m_switchClass.Draw(sb, idt);
+	}
+
+	void GoNextMap() override
+	{
+		if (m_switchedSidesData !is null)
+		{
+			TeamVersusGameMode::GoNextMap();
+			return;
+		}
+
+		ChangeLevel(GetCurrentLevelFilename());
+	}
+
+	void SpawnPlayers() override
+	{
+		if (m_switchedSidesData is null)
+		{
+			TeamVersusGameMode::SpawnPlayers();
+			return;
+		}
+
+		if (Network::IsServer())
+		{
+			for (uint i = 0; i < m_switchedSidesData.length(); i += 2)
+			{
+				uint peer = uint(m_switchedSidesData[i].GetInteger());
+				uint team = uint(m_switchedSidesData[i + 1].GetInteger());
+
+				TeamVersusScore@ joinScore = FindTeamScore(team);
+				if (joinScore is m_teamScores[0])
+					@joinScore = m_teamScores[1];
+				else
+					@joinScore = m_teamScores[0];
+
+				for (uint j = 0; j < g_players.length(); j++)
+				{
+					if (g_players[j].peer != peer)
+						continue;
+					SpawnPlayer(j, vec2(), 0, joinScore.m_team);
+					break;
+				}
+			}
+		}
+	}
+
+	void Save(SValueBuilder& builder) override
+	{
+		if (m_switchedSidesData is null)
+		{
+			builder.PushArray("teams");
+			for (uint i = 0; i < g_players.length(); i++)
+			{
+				if (g_players[i].peer == 255)
+					continue;
+				builder.PushInteger(g_players[i].peer);
+				builder.PushInteger(g_players[i].team);
+			}
+			builder.PopArray();
+		}
+
+		TeamVersusGameMode::Save(builder);
+	}
+
+	void Start(uint8 peer, SValue@ save, StartMode sMode) override
+	{
+		if (save !is null)
+			@m_switchedSidesData = GetParamArray(UnitPtr(), save, "teams", false);
+
+		TeamVersusGameMode::Start(peer, save, sMode);
+
+		m_tmLimit = 0; // infinite time limit as far as VersusGameMode is concerned
+		m_tmLimitCustom = TimeLimit * 1000; // 5 minutes by default
+
+		@m_payload = cast<PayloadBehavior>(PayloadUnit.FetchFirst().GetScriptBehavior());
+
+		if (m_payload is null)
+			PrintError("PayloadUnit is not a PayloadBehavior!");
+
+		UnitPtr unitFirstNode = FirstNode.FetchFirst();
+		if (unitFirstNode.IsValid())
+		{
+			auto node = cast<WorldScript::PayloadNode>(unitFirstNode.GetScriptBehavior());
+			if (node !is null)
+				@m_payload.m_targetNode = node;
+			else
+				PrintError("First target node is not a PayloadNode script!");
+		}
+		else
+			PrintError("First target node was not set!");
+
+		WorldScript::PayloadNode@ prevNode;
+
+		float totalDistance = 0.0f;
+
+		UnitPtr unitNode = unitFirstNode;
+		while (unitNode.IsValid())
+		{
+			auto node = cast<WorldScript::PayloadNode>(unitNode.GetScriptBehavior());
+			if (node is null)
+				break;
+
+			unitNode = node.NextNode.FetchFirst();
+
+			@node.m_prevNode = prevNode;
+			@node.m_nextNode = cast<WorldScript::PayloadNode>(unitNode.GetScriptBehavior());
+
+			if (prevNode !is null)
+				totalDistance += dist(prevNode.Position, node.Position);
+
+			@prevNode = node;
+		}
+
+		float currDistance = 0.0f;
+
+		auto distNode = cast<WorldScript::PayloadNode>(unitFirstNode.GetScriptBehavior());
+		while (distNode !is null)
+		{
+			if (distNode.m_prevNode is null)
+				distNode.m_locationFactor = 0.0f;
+			else
+			{
+				currDistance += dist(distNode.m_prevNode.Position, distNode.Position);
+				distNode.m_locationFactor = currDistance / totalDistance;
+			}
+
+			@distNode = distNode.m_nextNode;
+		}
+
+		m_payloadHUD.AddCheckpoints();
+	}
+
+	void SpawnPlayer(int i, vec2 pos = vec2(), int unitId = 0, uint team = 0) override
+	{
+		TeamVersusGameMode::SpawnPlayer(i, pos, unitId, team);
+
+		PayloadPlayerRecord@ record = cast<PayloadPlayerRecord>(g_players[i]);
+		record.HandlePlayerClass();
+
+		if (g_players[i].local)
+		{
+			//TODO: This doesn't work well
+			bool localAttackers = (team == HashString("player_1"));
+			for (uint j = 0; j < g_teamForceFields.length(); j++)
+			{
+				bool hasCollision = (localAttackers != g_teamForceFields[j].Attackers);
+
+				auto units = g_teamForceFields[j].Units.FetchAll();
+				for (uint k = 0; k < units.length(); k++)
+				{
+					PhysicsBody@ body = units[k].GetPhysicsBody();
+					if (body is null)
+					{
+						PrintError("PhysicsBody for unit " + units[k].GetDebugName() + "is null");
+						continue;
+					}
+					body.SetActive(hasCollision);
+				}
+			}
+		}
+	}
+}

samples/Ballerina/hello-world-service.bal

@@ -0,0 +1,16 @@
+import ballerina.lang.messages;
+import ballerina.net.http;
+import ballerina.doc;
+
+@doc:Description {value:"By default Ballerina assumes that the service is to be exposed via HTTP/1.1 using the system default port and that all requests coming to the HTTP server will be delivered to this service."}
+service<http> helloWorld {
+    @doc:Description {value:"All resources are invoked with an argument of type message, the built-in reference type representing a network invocation."}
+    resource sayHello (message m) {
+        // Creates an empty message.
+        message response = {};
+        // A util method that can be used to set string payload.
+        messages:setStringPayload(response, "Hello, World!");
+        // Reply keyword sends the response back to the client.
+        reply response;
+    }
+}

samples/Ballerina/hello-world.bal

@@ -0,0 +1,6 @@
+import ballerina.lang.system;
+
+function main (string[] args) {
+    system:println("Hello, World!");
+}
+

samples/Ballerina/json.bal

@@ -0,0 +1,31 @@
+import ballerina.lang.system;
+
+function main (string[] args) {
+    // JSON string value.
+    json j1 = "Apple";
+    system:println(j1);
+
+    // JSON number value.
+    json j2 = 5.36;
+    system:println(j2);
+
+    // JSON true value.
+    json j3 = true;
+    system:println(j3);
+
+    // JSON false value.
+    json j4 = false;
+    system:println(j4);
+
+    // JSON null value.
+    json j5 = null;
+
+    //JSON Objects.
+    json j6 = {name:"apple", color:"red", price:j2};
+    system:println(j6);
+
+    //JSON Arrays. They are arrays of any JSON value.
+    json j7 = [1, false, null, "foo",
+               {first:"John", last:"Pala"}];
+    system:println(j7);
+}

samples/Ballerina/var.bal

@@ -0,0 +1,28 @@
+import ballerina.lang.system;
+
+function divideBy10 (int d) (int, int) {
+    return d / 10, d % 10;
+}
+
+function main (string[] args) {
+    //Here the variable type is inferred type from the initial value. This is same as "int k = 5";
+    var k = 5;
+    system:println(10 + k);
+
+    //Here the type of the 'strVar' is 'string'.
+    var strVar = "Hello!";
+    system:println(strVar);
+
+    //Multiple assignment with 'var' allows you to define the variable then and there.
+    //Variable type is inferred from the right-hand side.
+    var q, r = divideBy10(6);
+    system:println("06/10: " + "quotient=" + q + " " +
+                   "remainder=" + r);
+
+    //To ignore a particular return value in a multiple assignment statement, use '_'.
+    var q1, _ = divideBy10(57);
+    system:println("57/10: " + "quotient=" + q1);
+
+    var _, r1 = divideBy10(9);
+    system:println("09/10: " + "remainder=" + r1);
+}

samples/Ballerina/xml.bal

@@ -0,0 +1,26 @@
+import ballerina.lang.system;
+
+function main (string[] args) {
+
+	// XML element. Can only have one root element.
+    xml x1 = xml `<book>The Lost World</book>`;
+    system:println(x1);
+
+    // XML text
+    xml x2 = xml `Hello, world!`;
+    system:println(x2);
+
+    // XML comment
+    xml x3 = xml `<!--I am a comment-->`;
+    system:println(x3);
+
+    // XML processing instruction
+    xml x4 = xml `<?target data?>`;
+    system:println(x4);
+
+    // Multiple XML items can be combined to form a sequence of XML. The resulting sequence is again an XML on its own.
+    xml x5 = x1 + x2 + x3 + x4;
+    system:println("\nResulting XML sequence:");
+    system:println(x5);
+
+}

samples/C++/bug1163046.--skeleton.re

@@ -0,0 +1,46 @@
+#include <iostream>
+
+#define YYCTYPE unsigned char
+#define YYCURSOR cursor
+#define YYLIMIT cursor
+#define YYMARKER marker
+#define YYFILL(n)
+
+bool scan(const char *text)
+{
+	YYCTYPE *start = (YYCTYPE *)text;
+	YYCTYPE *cursor = (YYCTYPE *)text;
+	YYCTYPE *marker = (YYCTYPE *)text;
+next:
+	YYCTYPE *token = cursor;
+/*!re2c
+'(This file must be converted with BinHex 4.0)'
+	{
+		if (token == start || *(token - 1) == '\n')
+		return true; else goto next;
+	}
+[\001-\377]
+	{ goto next; }
+[\000]
+	{ return false; }
+*/
+	return false;
+}
+
+#define do_scan(str, expect) \
+	res = scan(str) == expect ? 0 : 1; \
+	std::cerr << str << "\t-\t" << (res ? "fail" : "ok") << std::endl; \
+	result += res
+
+/*!max:re2c */
+
+int main(int,void**)
+{
+	int res, result = 0;
+	do_scan("(This file must be converted with BinHex 4.0)", 1);
+	do_scan("x(This file must be converted with BinHex 4.0)", 0);
+	do_scan("(This file must be converted with BinHex 4.0)x", 1);
+	do_scan("x(This file must be converted with BinHex 4.0)x", 0);
+	
+	return result;
+}

samples/C++/cnokw.re

@@ -0,0 +1,239 @@
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>
+
+#define	ADDEQ	257
+#define	ANDAND	258
+#define	ANDEQ	259
+#define	ARRAY	260
+#define	ASM	261
+#define	AUTO	262
+#define	BREAK	263
+#define	CASE	264
+#define	CHAR	265
+#define	CONST	266
+#define	CONTINUE	267
+#define	DECR	268
+#define	DEFAULT	269
+#define	DEREF	270
+#define	DIVEQ	271
+#define	DO	272
+#define	DOUBLE	273
+#define	ELLIPSIS	274
+#define	ELSE	275
+#define	ENUM	276
+#define	EQL	277
+#define	EXTERN	278
+#define	FCON	279
+#define	FLOAT	280
+#define	FOR	281
+#define	FUNCTION	282
+#define	GEQ	283
+#define	GOTO	284
+#define	ICON	285
+#define	ID	286
+#define	IF	287
+#define	INCR	288
+#define	INT	289
+#define	LEQ	290
+#define	LONG	291
+#define	LSHIFT	292
+#define	LSHIFTEQ	293
+#define	MODEQ	294
+#define	MULEQ	295
+#define	NEQ	296
+#define	OREQ	297
+#define	OROR	298
+#define	POINTER	299
+#define	REGISTER	300
+#define	RETURN	301
+#define	RSHIFT	302
+#define	RSHIFTEQ	303
+#define	SCON	304
+#define	SHORT	305
+#define	SIGNED	306
+#define	SIZEOF	307
+#define	STATIC	308
+#define	STRUCT	309
+#define	SUBEQ	310
+#define	SWITCH	311
+#define	TYPEDEF	312
+#define	UNION	313
+#define	UNSIGNED	314
+#define	VOID	315
+#define	VOLATILE	316
+#define	WHILE	317
+#define	XOREQ	318
+#define	EOI	319
+
+typedef unsigned int uint;
+typedef unsigned char uchar;
+
+#define	BSIZE	8192
+
+#define	YYCTYPE		uchar
+#define	YYCURSOR	cursor
+#define	YYLIMIT		s->lim
+#define	YYMARKER	s->ptr
+#define	YYFILL(n)	{cursor = fill(s, cursor);}
+
+#define	RET(i)	{s->cur = cursor; return i;}
+
+typedef struct Scanner {
+    int			fd;
+    uchar		*bot, *tok, *ptr, *cur, *pos, *lim, *top, *eof;
+    uint		line;
+} Scanner;
+
+uchar *fill(Scanner *s, uchar *cursor){
+    if(!s->eof){
+	uint cnt = s->tok - s->bot;
+	if(cnt){
+	    memcpy(s->bot, s->tok, s->lim - s->tok);
+	    s->tok = s->bot;
+	    s->ptr -= cnt;
+	    cursor -= cnt;
+	    s->pos -= cnt;
+	    s->lim -= cnt;
+	}
+	if((s->top - s->lim) < BSIZE){
+	    uchar *buf = (uchar*) malloc(((s->lim - s->bot) + BSIZE)*sizeof(uchar));
+	    memcpy(buf, s->tok, s->lim - s->tok);
+	    s->tok = buf;
+	    s->ptr = &buf[s->ptr - s->bot];
+	    cursor = &buf[cursor - s->bot];
+	    s->pos = &buf[s->pos - s->bot];
+	    s->lim = &buf[s->lim - s->bot];
+	    s->top = &s->lim[BSIZE];
+	    free(s->bot);
+	    s->bot = buf;
+	}
+	if((cnt = read(s->fd, (char*) s->lim, BSIZE)) != BSIZE){
+	    s->eof = &s->lim[cnt]; *(s->eof)++ = '\n';
+	}
+	s->lim += cnt;
+    }
+    return cursor;
+}
+
+int scan(Scanner *s){
+	uchar *cursor = s->cur;
+std:
+	s->tok = cursor;
+/*!re2c
+any	= [\000-\377];
+O	= [0-7];
+D	= [0-9];
+L	= [a-zA-Z_];
+H	= [a-fA-F0-9];
+E	= [Ee] [+-]? D+;
+FS	= [fFlL];
+IS	= [uUlL]*;
+ESC	= [\\] ([abfnrtv?'"\\] | "x" H+ | O+);
+*/
+
+/*!re2c
+	"/*"			{ goto comment; }
+	
+	L (L|D)*		{ RET(ID); }
+	
+	("0" [xX] H+ IS?) | ("0" D+ IS?) | (D+ IS?) |
+	(['] (ESC|any\[\n\\'])* ['])
+				{ RET(ICON); }
+	
+	(D+ E FS?) | (D* "." D+ E? FS?) | (D+ "." D* E? FS?)
+				{ RET(FCON); }
+	
+	(["] (ESC|any\[\n\\"])* ["])
+				{ RET(SCON); }
+	
+	"..."                   { RET(ELLIPSIS); }
+	">>="			{ RET(RSHIFTEQ); }
+	"<<="			{ RET(LSHIFTEQ); }
+	"+="			{ RET(ADDEQ); }
+	"-="			{ RET(SUBEQ); }
+	"*="			{ RET(MULEQ); }
+	"/="			{ RET(DIVEQ); }
+	"%="			{ RET(MODEQ); }
+	"&="			{ RET(ANDEQ); }
+	"^="			{ RET(XOREQ); }
+	"|="			{ RET(OREQ); }
+	">>"			{ RET(RSHIFT); }
+	"<<"			{ RET(LSHIFT); }
+	"++"			{ RET(INCR); }
+	"--"			{ RET(DECR); }
+	"->"			{ RET(DEREF); }
+	"&&"			{ RET(ANDAND); }
+	"||"			{ RET(OROR); }
+	"<="			{ RET(LEQ); }
+	">="			{ RET(GEQ); }
+	"=="			{ RET(EQL); }
+	"!="			{ RET(NEQ); }
+	";"			{ RET(';'); }
+	"{"			{ RET('{'); }
+	"}"			{ RET('}'); }
+	","			{ RET(','); }
+	":"			{ RET(':'); }
+	"="			{ RET('='); }
+	"("			{ RET('('); }
+	")"			{ RET(')'); }
+	"["			{ RET('['); }
+	"]"			{ RET(']'); }
+	"."			{ RET('.'); }
+	"&"			{ RET('&'); }
+	"!"			{ RET('!'); }
+	"~"			{ RET('~'); }
+	"-"			{ RET('-'); }
+	"+"			{ RET('+'); }
+	"*"			{ RET('*'); }
+	"/"			{ RET('/'); }
+	"%"			{ RET('%'); }
+	"<"			{ RET('<'); }
+	">"			{ RET('>'); }
+	"^"			{ RET('^'); }
+	"|"			{ RET('|'); }
+	"?"			{ RET('?'); }
+
+
+	[ \t\v\f]+		{ goto std; }
+
+	"\n"
+	    {
+		if(cursor == s->eof) RET(EOI);
+		s->pos = cursor; s->line++;
+		goto std;
+	    }
+
+	any
+	    {
+		printf("unexpected character: %c\n", *s->tok);
+		goto std;
+	    }
+*/
+
+comment:
+/*!re2c
+	"*/"			{ goto std; }
+	"\n"
+	    {
+		if(cursor == s->eof) RET(EOI);
+		s->tok = s->pos = cursor; s->line++;
+		goto comment;
+	    }
+        any			{ goto comment; }
+*/
+}
+
+main(){
+    Scanner in;
+    int t;
+    memset((char*) &in, 0, sizeof(in));
+    in.fd = 0;
+    while((t = scan(&in)) != EOI){
+/*
+	printf("%d\t%.*s\n", t, in.cur - in.tok, in.tok);
+	printf("%d\n", t);
+*/
+    }
+    close(in.fd);
+}

samples/C++/cvsignore.re

@@ -0,0 +1,63 @@
+
+#define YYFILL(n) if (cursor >= limit) break;
+#define YYCTYPE char
+#define YYCURSOR cursor
+#define YYLIMIT limit
+#define YYMARKER marker
+
+/*!re2c
+any     = (.|"\n");
+value	= (":" (.\"$")+)?;
+cvsdat	= "Date";
+cvsid	= "Id";
+cvslog	= "Log";
+cvsrev	= "Revision";
+cvssrc	= "Source";
+*/
+
+#define APPEND(text) \
+	append(output, outsize, text, sizeof(text) - sizeof(YYCTYPE))
+
+inline void append(YYCTYPE *output, size_t & outsize, const YYCTYPE * text, size_t len)
+{
+	memcpy(output + outsize, text, len);
+	outsize += (len / sizeof(YYCTYPE));
+}
+
+void scan(YYCTYPE *pText, size_t *pSize, int *pbChanged)
+{
+	// rule
+	// scan lines
+	// find $ in lines
+	//   compact $<keyword>: .. $ to $<keyword>$
+  
+	YYCTYPE *output;
+	const YYCTYPE *cursor, *limit, *marker;
+
+	cursor = marker = output = *pText;
+
+	size_t insize = *pSize;
+	size_t outsize = 0;
+
+	limit = cursor + insize;
+
+	while(1) {
+loop:
+/*!re2c
+
+"$" cvsdat value "$"	{ APPEND(L"$" L"Date$"); goto loop; }
+"$" cvsid  value "$"	{ APPEND(L"$" L"Id$"); goto loop; }
+"$" cvslog value "$"	{ APPEND(L"$" L"Log$"); goto loop; }
+"$" cvsrev value "$"	{ APPEND(L"$" L"Revision$"); goto loop; }
+"$" cvssrc value "$"	{ APPEND(L"$" L"Source$"); goto loop; }
+any						{ output[outsize++] = cursor[-1]; if (cursor >= limit) break; goto loop; }
+
+*/
+	}
+	output[outsize] = '\0';
+
+	// set the new size
+	*pSize = outsize;
+	
+	*pbChanged = (insize == outsize) ? 0 : 1;
+}

samples/C++/grpc.pb.cc

@@ -0,0 +1,2 @@
+// Generated by the gRPC protobuf plugin.
+// If you make any local change, they will be lost.

samples/C++/hello.grpc.pb.h

@@ -0,0 +1,125 @@
+// Generated by the gRPC C++ plugin.
+// If you make any local change, they will be lost.
+// source: hello.proto
+#ifndef GRPC_hello_2eproto__INCLUDED
+#define GRPC_hello_2eproto__INCLUDED
+
+#include "hello.pb.h"
+
+#include <grpc++/impl/codegen/async_stream.h>
+#include <grpc++/impl/codegen/async_unary_call.h>
+#include <grpc++/impl/codegen/method_handler_impl.h>
+#include <grpc++/impl/codegen/proto_utils.h>
+#include <grpc++/impl/codegen/rpc_method.h>
+#include <grpc++/impl/codegen/service_type.h>
+#include <grpc++/impl/codegen/status.h>
+#include <grpc++/impl/codegen/stub_options.h>
+#include <grpc++/impl/codegen/sync_stream.h>
+
+namespace grpc {
+class CompletionQueue;
+class Channel;
+class RpcService;
+class ServerCompletionQueue;
+class ServerContext;
+}  // namespace grpc
+
+class HelloService final {
+ public:
+  class StubInterface {
+   public:
+    virtual ~StubInterface() {}
+    virtual ::grpc::Status SayHello(::grpc::ClientContext* context, const ::HelloRequest& request, ::HelloResponse* response) = 0;
+    std::unique_ptr< ::grpc::ClientAsyncResponseReaderInterface< ::HelloResponse>> AsyncSayHello(::grpc::ClientContext* context, const ::HelloRequest& request, ::grpc::CompletionQueue* cq) {
+      return std::unique_ptr< ::grpc::ClientAsyncResponseReaderInterface< ::HelloResponse>>(AsyncSayHelloRaw(context, request, cq));
+    }
+  private:
+    virtual ::grpc::ClientAsyncResponseReaderInterface< ::HelloResponse>* AsyncSayHelloRaw(::grpc::ClientContext* context, const ::HelloRequest& request, ::grpc::CompletionQueue* cq) = 0;
+  };
+  class Stub final : public StubInterface {
+   public:
+    Stub(const std::shared_ptr< ::grpc::ChannelInterface>& channel);
+    ::grpc::Status SayHello(::grpc::ClientContext* context, const ::HelloRequest& request, ::HelloResponse* response) override;
+    std::unique_ptr< ::grpc::ClientAsyncResponseReader< ::HelloResponse>> AsyncSayHello(::grpc::ClientContext* context, const ::HelloRequest& request, ::grpc::CompletionQueue* cq) {
+      return std::unique_ptr< ::grpc::ClientAsyncResponseReader< ::HelloResponse>>(AsyncSayHelloRaw(context, request, cq));
+    }
+
+   private:
+    std::shared_ptr< ::grpc::ChannelInterface> channel_;
+    ::grpc::ClientAsyncResponseReader< ::HelloResponse>* AsyncSayHelloRaw(::grpc::ClientContext* context, const ::HelloRequest& request, ::grpc::CompletionQueue* cq) override;
+    const ::grpc::RpcMethod rpcmethod_SayHello_;
+  };
+  static std::unique_ptr<Stub> NewStub(const std::shared_ptr< ::grpc::ChannelInterface>& channel, const ::grpc::StubOptions& options = ::grpc::StubOptions());
+
+  class Service : public ::grpc::Service {
+   public:
+    Service();
+    virtual ~Service();
+    virtual ::grpc::Status SayHello(::grpc::ServerContext* context, const ::HelloRequest* request, ::HelloResponse* response);
+  };
+  template <class BaseClass>
+  class WithAsyncMethod_SayHello : public BaseClass {
+   private:
+    void BaseClassMustBeDerivedFromService(const Service *service) {}
+   public:
+    WithAsyncMethod_SayHello() {
+      ::grpc::Service::MarkMethodAsync(0);
+    }
+    ~WithAsyncMethod_SayHello() override {
+      BaseClassMustBeDerivedFromService(this);
+    }
+    // disable synchronous version of this method
+    ::grpc::Status SayHello(::grpc::ServerContext* context, const ::HelloRequest* request, ::HelloResponse* response) final override {
+      abort();
+      return ::grpc::Status(::grpc::StatusCode::UNIMPLEMENTED, "");
+    }
+    void RequestSayHello(::grpc::ServerContext* context, ::HelloRequest* request, ::grpc::ServerAsyncResponseWriter< ::HelloResponse>* response, ::grpc::CompletionQueue* new_call_cq, ::grpc::ServerCompletionQueue* notification_cq, void *tag) {
+      ::grpc::Service::RequestAsyncUnary(0, context, request, response, new_call_cq, notification_cq, tag);
+    }
+  };
+  typedef WithAsyncMethod_SayHello<Service > AsyncService;
+  template <class BaseClass>
+  class WithGenericMethod_SayHello : public BaseClass {
+   private:
+    void BaseClassMustBeDerivedFromService(const Service *service) {}
+   public:
+    WithGenericMethod_SayHello() {
+      ::grpc::Service::MarkMethodGeneric(0);
+    }
+    ~WithGenericMethod_SayHello() override {
+      BaseClassMustBeDerivedFromService(this);
+    }
+    // disable synchronous version of this method
+    ::grpc::Status SayHello(::grpc::ServerContext* context, const ::HelloRequest* request, ::HelloResponse* response) final override {
+      abort();
+      return ::grpc::Status(::grpc::StatusCode::UNIMPLEMENTED, "");
+    }
+  };
+  template <class BaseClass>
+  class WithStreamedUnaryMethod_SayHello : public BaseClass {
+   private:
+    void BaseClassMustBeDerivedFromService(const Service *service) {}
+   public:
+    WithStreamedUnaryMethod_SayHello() {
+      ::grpc::Service::MarkMethodStreamed(0,
+        new ::grpc::StreamedUnaryHandler< ::HelloRequest, ::HelloResponse>(std::bind(&WithStreamedUnaryMethod_SayHello<BaseClass>::StreamedSayHello, this, std::placeholders::_1, std::placeholders::_2)));
+    }
+    ~WithStreamedUnaryMethod_SayHello() override {
+      BaseClassMustBeDerivedFromService(this);
+    }
+    // disable regular version of this method
+    ::grpc::Status SayHello(::grpc::ServerContext* context, const ::HelloRequest* request, ::HelloResponse* response) final override {
+      abort();
+      return ::grpc::Status(::grpc::StatusCode::UNIMPLEMENTED, "");
+    }
+    // replace default version of method with streamed unary
+    virtual ::grpc::Status StreamedSayHello(::grpc::ServerContext* context, ::grpc::ServerUnaryStreamer< ::HelloRequest,::HelloResponse>* server_unary_streamer) = 0;
+  };
+  typedef WithStreamedUnaryMethod_SayHello<Service > StreamedUnaryService;
+  typedef Service SplitStreamedService;
+  typedef WithStreamedUnaryMethod_SayHello<Service > StreamedService;
+};
+
+
+#endif  // GRPC_hello_2eproto__INCLUDED
+

samples/C++/simple.re

@@ -0,0 +1,13 @@
+#define	NULL		((char*) 0)
+char *scan(char *p){
+char *q;
+#define	YYCTYPE		char
+#define	YYCURSOR	p
+#define	YYLIMIT		p
+#define	YYMARKER	q
+#define	YYFILL(n)
+/*!re2c
+	[0-9]+		{return YYCURSOR;}
+	[\000-\377]	{return NULL;}
+*/
+}

samples/C/asm.h

@@ -0,0 +1,27 @@
+/* CarbonOS System/Kernel
+ * Copyright 2015-2017 David Aylaian
+ * Licensed under Apache 2.0: https://github.com/DavidAylaian/CarbonOS/blob/master/LICENSE.md
+ */
+
+#ifndef ASM_H
+#define ASM_H
+
+#include <stdint.h>
+
+// macros for enabling and disabling interrupts
+#define enable()	asm("sti");
+#define disable()	asm("cli");
+
+// inb instruction
+uint8_t inb (uint16_t port) {
+	uint8_t val;
+	asm volatile ("inb %0, %1" : "=a"(val): "Nd"(port));
+	return val;
+}
+
+// outb instruction
+void outb (uint16_t port, uint8_t val) {
+	asm volatile ("outb %1, %0" : : "a"(val), "Nd"(port));
+}
+
+#endif

samples/C/cpuid.h

@@ -0,0 +1,25 @@
+#ifndef CPUID_H
+#define CPUID_H
+
+#include "misc.h"
+
+static inline void do_cpuid(dword_t *eax, dword_t *ebx, dword_t *ecx, dword_t *edx) {
+    dword_t leaf = *eax;
+    switch (leaf) {
+        case 0:
+            *eax = 0x01; // we support barely anything
+            *ebx = 0x756e6547; // Genu
+            *edx = 0x49656e69; // ineI
+            *ecx = 0x6c65746e; // ntel
+            break;
+        default: // if leaf is too high, use highest supported leaf
+        case 1:
+            *eax = 0x0; // say nothing about cpu model number
+            *ebx = 0x0; // processor number 0, flushes 0 bytes on clflush
+            *ecx = 0b00000000000000000000000000000000; // we support none of the features in ecx
+            *edx = 0b00000000000000000000000000000000; // we also support none of the features in edx
+            break;
+    }
+}
+
+#endif

samples/CSON/base.cson

@@ -0,0 +1,72 @@
+'atom-text-editor':
+  # Platform Bindings
+  'home': 'editor:move-to-first-character-of-line'
+  'end': 'editor:move-to-end-of-screen-line'
+  'shift-home': 'editor:select-to-first-character-of-line'
+  'shift-end': 'editor:select-to-end-of-line'
+
+'atom-text-editor:not([mini])':
+  # Atom Specific
+  'ctrl-C': 'editor:copy-path'
+
+  # Sublime Parity
+  'tab': 'editor:indent'
+  'enter': 'editor:newline'
+  'shift-tab': 'editor:outdent-selected-rows'
+  'ctrl-K': 'editor:delete-line'
+
+'.select-list atom-text-editor[mini]':
+  'enter': 'core:confirm'
+
+'.tool-panel.panel-left, .tool-panel.panel-right':
+  'escape': 'tool-panel:unfocus'
+
+'atom-text-editor !important, atom-text-editor[mini] !important':
+  'escape': 'editor:consolidate-selections'
+
+# allow standard input fields to work correctly
+'body .native-key-bindings':
+  'tab': 'core:focus-next'
+  'shift-tab': 'core:focus-previous'
+  'enter': 'native!'
+  'backspace': 'native!'
+  'shift-backspace': 'native!'
+  'delete': 'native!'
+  'up': 'native!'
+  'down': 'native!'
+  'shift-up': 'native!'
+  'shift-down': 'native!'
+  'alt-up': 'native!'
+  'alt-down': 'native!'
+  'alt-shift-up': 'native!'
+  'alt-shift-down': 'native!'
+  'cmd-up': 'native!'
+  'cmd-down': 'native!'
+  'cmd-shift-up': 'native!'
+  'cmd-shift-down': 'native!'
+  'ctrl-up': 'native!'
+  'ctrl-down': 'native!'
+  'ctrl-shift-up': 'native!'
+  'ctrl-shift-down': 'native!'
+  'left': 'native!'
+  'right': 'native!'
+  'shift-left': 'native!'
+  'shift-right': 'native!'
+  'alt-left': 'native!'
+  'alt-right': 'native!'
+  'alt-shift-left': 'native!'
+  'alt-shift-right': 'native!'
+  'cmd-left': 'native!'
+  'cmd-right': 'native!'
+  'cmd-shift-left': 'native!'
+  'cmd-shift-right': 'native!'
+  'ctrl-left': 'native!'
+  'ctrl-right': 'native!'
+  'ctrl-shift-left': 'native!'
+  'ctrl-shift-right': 'native!'
+  'ctrl-b': 'native!'
+  'ctrl-f': 'native!'
+  'ctrl-F': 'native!'
+  'ctrl-B': 'native!'
+  'ctrl-h': 'native!'
+  'ctrl-d': 'native!'

samples/CSON/config.cson

@@ -0,0 +1,59 @@
+directoryIcons:
+
+	Atom:
+		icon: "atom"
+		match: /^\.atom$/
+		colour: "dark-green"
+
+	Bower:
+		icon: "bower"
+		match: /^bower[-_]components$/
+		colour: "bower"
+
+	Dropbox:
+		icon: "dropbox"
+		match: /^(?:Dropbox|\.dropbox\.cache)$/
+		colour: "medium-blue"
+
+	Git:
+		icon: "git"
+		match: /^\.git$/
+
+	GitHub:
+		icon: "github"
+		match: /^\.github$/
+
+	Meteor:
+		icon: "meteor"
+		match: /^\.meteor$/
+
+	NodeJS:
+		icon: "node"
+		match: /^node_modules$/
+		colour: "medium-green"
+
+	Package:
+		icon: "package"
+		match: /^\.bundle$/i
+
+	TextMate:
+		icon: "textmate"
+		match: ".tmBundle"
+
+
+fileIcons:
+
+	ABAP:
+		icon: "abap"
+		scope: "abp"
+		match: ".abap"
+		colour: "medium-orange"
+
+	ActionScript: # Or Flash-related
+		icon: "as"
+		match: [
+			[".swf", "medium-blue"]
+			[".as", "medium-red", scope: /\.(?:flex-config|actionscript(?:\.\d+)?)$/i, alias: /ActionScript\s?3|as3/i]
+			[".jsfl", "auto-yellow"]
+			[".swc", "dark-red"]
+		]

samples/CSON/ff-sfd.cson

@@ -0,0 +1,108 @@
+name: "Spline Font Database"
+scopeName: "text.sfd"
+fileTypes: ["sfd"]
+firstLineMatch: "^SplineFontDB: [\\d.]+"
+patterns: [include: "#main"]
+
+repository:
+	main:
+		patterns: [
+			{include: "#punctuation"}
+			{include: "#private"}
+			{include: "#image"}
+			{include: "#pickleData"}
+			{include: "#sections"}
+			{include: "#copyright"}
+			{include: "#property"}
+			{include: "#control"}
+			{include: "#address"}
+			{include: "#encoding"}
+			{include: "source.fontforge#shared"}
+			{include: "#colour"}
+		]
+	
+	punctuation:
+		patterns: [
+			{match: "<|>",  name: "punctuation.definition.brackets.angle.sfd"}
+			{match: "[{}]", name: "punctuation.definition.brackets.curly.sfd"}
+		]
+
+	private:
+		name: "meta.section.private.sfd"
+		begin: "^BeginPrivate(?=:)"
+		end:   "^EndPrivate\\b"
+		beginCaptures: 0: name: "keyword.control.begin.private.sfd"
+		endCaptures:   0: name: "keyword.control.end.private.sfd"
+		patterns: [
+			{match: "^\\S+", name: "entity.name.private.property.sfd"}
+			{include: "$self"}
+		]
+	
+	image:
+		name: "meta.image.sfd"
+		begin: "^(Image)(?=:)(.+)$"
+		end:   "^(EndImage)\\b"
+		contentName: "string.unquoted.raw.data.sfd"
+		beginCaptures:
+			1: name: "keyword.control.begin.image.sfd"
+			2: patterns: [include: "$self"]
+		endCaptures:
+			1: name: "keyword.control.end.image.sfd"
+
+	pickleData:
+		name: "meta.pickle-data.sfd"
+		begin: "^(PickledData)(:)\\s*(\")"
+		end:   '"'
+		beginCaptures:
+			1: name: "entity.name.property.sfd"
+			2: name: "punctuation.separator.dictionary.key-value.sfd"
+			3: name: "punctuation.definition.string.begin.sfd"
+		endCaptures:
+			0: name: "punctuation.definition.string.end.sfd"
+		patterns: [match: "\\\\.", name: "constant.character.escape.sfd"]
+
+	sections:
+		name: "meta.section.${2:/downcase}.sfd"
+		begin: "^(Start|Begin)([A-Z]\\w+)(?=:)"
+		end:   "^(End\\2)\\b"
+		beginCaptures: 0: name: "keyword.control.begin.${2:/downcase}.sfd"
+		endCaptures:   0: name: "keyword.control.end.${2:/downcase}.sfd"
+		patterns: [include: "$self"]
+
+	control:
+		name: "keyword.control.${1:/downcase}.sfd"
+		match: "\\b(Fore|Back|SplineSet|^End\\w+)\\b"
+
+	colour:
+		name: "constant.other.hex.colour.sfd"
+		match: "(#)[A-Fa-f0-9]{3,}|(?<=\\s)[A-Fa-f0-9]{6,8}"
+		captures:
+			1: name: "punctuation.definition.colour.sfd"
+
+	encoding:
+		name: "constant.language.encoding.sfd"
+		match: "(?i)\\b(ISO[-\\w]+)(?<=\\d)(?=\\s|$)"
+
+	# Don't highlight numbers in freeform strings (years/version strings)
+	copyright:
+		name: "meta.${1:/downcase}-string.sfd"
+		begin: "^(Copyright|U?Comments?|\\w+Name)(:)"
+		end:   "$"
+		beginCaptures:
+			1: name: "entity.name.property.sfd"
+			2: name: "punctuation.separator.dictionary.key-value.sfd"
+		patterns: [include: "source.fontforge#stringEscapes"]
+	
+	# No idea what this is, but it looks distracting without a fix
+	# Assuming it's referring to a memory register or something.
+	address:
+		match: "\\d+[xX][A-Fa-f0-9]+"
+		name: "constant.numeric.hexadecimal.sfd"
+	
+	property:
+		match: "^([^:]+)(:)"
+		name: "meta.dictionary.key-value.sfd"
+		captures:
+			1: name: "entity.name.property.sfd"
+			2: name: "punctuation.separator.dictionary.key-value.sfd"
+	

samples/CSON/wercker-status.cson

@@ -0,0 +1,11 @@
+'menu': [
+  {
+    'label': 'Packages'
+    'submenu': [
+      'label': 'Wercker Status'
+      'submenu': [
+        { 'label': 'Check now!', 'command': 'wercker-status:checknow' }
+      ]
+    ]
+  }
+]

samples/CWeb/sat-life.w

@@ -0,0 +1,404 @@
+\datethis
+@*Intro. This program generates clauses for the transition relation
+from time $t$ to time $t+1$ in Conway's Game of Life, assuming that
+all of the potentially live cells at time $t$ belong to a pattern
+that's specified in |stdin|. The pattern is defined by one or more
+lines representing rows of cells, where each line has `\..' in a
+cell that's guaranteed to be dead at time~$t$, otherwise it has `\.*'.
+The time is specified separately as a command-line parameter.
+
+The Boolean variable for cell $(x,y)$ at time $t$ is named by its
+so-called ``xty code,'' namely by the decimal value of~$x$, followed
+by a code letter for~$t$, followed by the decimal value of~$y$. For
+example, if $x=10$ and $y=11$ and $t=0$, the variable that indicates
+liveness of the cell is \.{10a11}; and the corresponding variable
+for $t=1$ is \.{10b11}.
+
+Up to 19 auxiliary variables are used together with each xty code,
+in order to construct clauses that define the successor state.
+The names of these variables are obtained by appending one of
+the following two-character combinations to the xty code:
+\.{A2}, \.{A3}, \.{A4},
+\.{B1}, \.{B2}, \.{B3}, \.{B4},
+\.{C1}, \.{C2}, \.{C3}, \.{C4},
+\.{D1}, \.{D2},
+\.{E1}, \.{E2},
+\.{F1}, \.{F2},
+\.{G1}, \.{G2}.
+These variables are derived from the Bailleux--Boufkhad method
+of encoding cardinality constraints:
+The auxiliary variable \.{A$k$} stands for the condition
+``at least $k$ of the eight neighbors are alive.'' Similarly,
+\.{B$k$} stands for ``at least $k$ of the first four neighbors
+are alive,'' and \.{C$k$} accounts for the other four neighbors.
+Codes \.D, \.E, \.F, and~\.G refer to pairs of neighbors.
+Thus, for instance, \.{10a11C2} means that at least two of the
+last four neighbors of cell $(10,11)$ are alive.
+
+Those auxiliary variables receive values by means of up to 77 clauses per cell.
+For example, if $u$ and~$v$ are the neighbors of cell~$z$ that correspond
+to a pairing of type~\.D, there are six clauses
+$$\bar u d_1,\quad
+  \bar v d_1,\quad
+  \bar u\bar v d_2,\quad
+  u v\bar d_1,\quad
+  u\bar d_2,\quad
+  v\bar d_2.$$
+The sixteen clauses
+$$\displaylines{\hfill
+\bar d_1b_1,\quad
+\bar e_1b_1,\quad
+\bar d_2b_2,\quad
+\bar d_1\bar e_1b_2,\quad
+\bar e_2b_2,\quad
+\bar d_2\bar e_1b_3,\quad
+\bar d_1\bar e_2b_3,\quad
+\bar d_2\bar e_2b_4,
+\hfill\cr\hfill
+d_1e_1\bar b_1,\quad
+d_1e_2\bar b_2,\quad
+d_2e_1\bar b_2,\quad
+d_1\bar b_3,\quad
+d_2e_2\bar b_3,\quad
+e_1\bar b_3,\quad
+d_2\bar b_4,\quad
+e_2\bar b_4
+\hfill}$$
+define $b$ variables from $d$'s and $e$'s; and another sixteen
+define $c$'s from $f$'s and $g$'s in the same fashion.
+A similar set of 21 clauses will define the $a$'s from the $b$'s and $c$'s.
+
+Once the $a$'s are defined, thus essentially counting the
+live neighbors of cell $z$, the next
+state~$z'$ is defined by five further clauses
+$$\bar a_4\bar z',\quad
+a_2\bar z',\quad
+a_3z\bar z',\quad
+\bar a_3a_4z',\quad
+\bar a_2a_4\bar zz'.$$
+For example, the last of these states that $z'$ will be true
+(i.e., that cell $z$ will be alive at time $t+1$) if
+$z$ is alive at time~$t$ and has $\ge2$ live neighbors
+but not $\ge4$.
+
+Nearby cells can share auxiliary variables, according to a tricky scheme that
+is worked out below. In consequence, the actual number of auxiliary variables
+and clauses per cell is reduced from 19 and $77+5$ to 13 and $57+5$,
+respectively, except at the boundaries.
+
+@ So here's the overall outline of the program.
+
+@d maxx 50 /* maximum number of lines in the pattern supplied by |stdin| */
+@d maxy 50 /* maximum number of columns per line in |stdin| */
+
+@c
+#include <stdio.h>
+#include <stdlib.h>
+char p[maxx+2][maxy+2]; /* is cell $(x,y)$ potentially alive? */
+char have_b[maxx+2][maxy+2]; /* did we already generate $b(x,y)$? */
+char have_d[maxx+2][maxy+2]; /* did we already generate $d(x,y)$? */
+char have_e[maxx+2][maxy+4]; /* did we already generate $e(x,y)$? */
+char have_f[maxx+4][maxy+2]; /* did we already generate $f(x-2,y)$? */
+int tt; /* time as given on the command line */
+int xmax,ymax; /* the number of rows and columns in the input pattern */
+int xmin=maxx,ymin=maxy; /* limits in the other direction */
+char timecode[]="abcdefghijklmnopqrstuvwxyz"@|
+      "ABCDEFGHIJKLMNOPQRSTUVWXYZ"@|
+      "!\"#$%&'()*+,-./:;<=>?@@[\\]^_`{|}~"; /* codes for $0\le t\le83$ */
+@q$@>
+char buf[maxy+2]; /* input buffer */
+unsigned int clause[4]; /* clauses are assembled here */
+int clauseptr; /* this many literals are in the current clause */
+@<Subroutines@>@;
+main(int argc,char*argv[]) {
+  register int j,k,x,y;
+  @<Process the command line@>;
+  @<Input the pattern@>;
+  for (x=xmin-1;x<=xmax+1;x++) for (y=ymin-1;y<=ymax+1;y++) {
+    @<If cell $(x,y)$ is obviously dead at time $t+1$, |continue|@>;
+    a(x,y);
+    zprime(x,y);
+  }
+}
+
+@ @<Process the command line@>=
+if (argc!=2 || sscanf(argv[1],"%d",&tt)!=1) {
+  fprintf(stderr,"Usage: %s t\n",argv[0]);
+  exit(-1);
+}
+if (tt<0 || tt>82) {
+  fprintf(stderr,"The time should be between 0 and 82 (not %d)!\n",tt);
+  exit(-2);
+}
+
+@ @<Input the pattern@>=
+for (x=1;;x++) {
+  if (!fgets(buf,maxy+2,stdin)) break;
+  if (x>maxx) {
+    fprintf(stderr,"Sorry, the pattern should have at most %d rows!\n",maxx);
+    exit(-3);
+  }
+  for (y=1;buf[y-1]!='\n';y++) {
+    if (y>maxy) {
+      fprintf(stderr,"Sorry, the pattern should have at most %d columns!\n",
+             maxy);
+      exit(-4);
+    }
+    if (buf[y-1]=='*') {
+      p[x][y]=1;
+      if (y>ymax) ymax=y;
+      if (y<ymin) ymin=y;
+      if (x>xmax) xmax=x;
+      if (x<xmin) xmin=x;
+    }@+else if (buf[y-1]!='.') {
+      fprintf(stderr,"Unexpected character `%c' found in the pattern!\n",
+              buf[y-1]);
+      exit(-5);
+    }
+  }
+}
+
+@ @d pp(xx,yy) ((xx)>=0 && (yy)>=0? p[xx][yy]: 0)
+
+@<If cell $(x,y)$ is obviously dead at time $t+1$, |continue|@>=
+if (pp(x-1,y-1)+pp(x-1,y)+pp(x-1,y+1)+
+    pp(x,y-1)+p[x][y]+p[x][y+1]+
+    pp(x+1,y-1)+p[x+1][y]+p[x+1][y+1]<3) continue;
+
+@ Clauses are assembled in the |clause| array (surprise), where we
+put encoded literals.
+
+The code for a literal is an unsigned 32-bit quantity, where the leading
+bit is 1 if the literal should be complemented. The next three bits
+specify the type of the literal (0 thru 7 for plain and \.A--\.G);
+the next three bits specify an integer~$k$; and the next bit is zero.
+That leaves room for two 12-bit fields, which specify $x$ and $y$.
+
+Type 0 literals have $k=0$ for the ordinary xty code. However, the
+value $k=1$ indicates that the time code should be for $t+1$ instead of~$t$.
+And $k=2$ denotes a special ``tautology'' literal, which is always true.
+If the tautology literal is complemented, we omit it from the clause;
+otherwise we omit the entire clause.
+Finally, $k=7$ denotes an auxiliary literal, used to avoid
+clauses of length~4.
+
+Here's a subroutine that outputs the current clause and resets
+the |clause| array.
+
+@d taut (2<<25)
+@d sign (1U<<31)
+
+@<Sub...@>=
+void outclause(void) {
+  register int c,k,x,y,p;
+  for (p=0;p<clauseptr;p++)
+    if (clause[p]==taut) goto done;
+  for (p=0;p<clauseptr;p++) if (clause[p]!=taut+sign) {
+    if (clause[p]>>31) printf(" ~");@+else printf(" ");
+    c=(clause[p]>>28)&0x7;
+    k=(clause[p]>>25)&0x7;
+    x=(clause[p]>>12)&0xfff;
+    y=clause[p]&0xfff;
+    if (c) printf("%d%c%d%c%d",
+             x,timecode[tt],y,c+'@@',k);
+    else if (k==7) printf("%d%c%dx",
+             x,timecode[tt],y);
+    else printf("%d%c%d",
+             x,timecode[tt+k],y);
+  }
+  printf("\n");
+done: clauseptr=0;
+}
+
+@ And here's another, which puts a type-0 literal into |clause|.
+
+@<Sub...@>=
+void applit(int x,int y,int bar,int k) {
+  if (k==0 && (x<xmin || x>xmax || y<ymin || y>ymax || p[x][y]==0))
+    clause[clauseptr++]=(bar? 0: sign)+taut;
+  else clause[clauseptr++]=(bar? sign:0)+(k<<25)+(x<<12)+y;
+}
+
+@ The |d| and |e| subroutines are called for only one-fourth
+of all cell addresses $(x,y)$. Indeed, one can show that
+$x$ is always odd, and that $y\bmod4<2$.
+
+Therefore we remember if we've seen $(x,y)$ before.
+
+Slight trick: If |yy| is not in range, we avoid generating the
+clause $\bar d_k$ twice.
+
+@d newlit(x,y,c,k) clause[clauseptr++]=((c)<<28)+((k)<<25)+((x)<<12)+(y)
+@d newcomplit(x,y,c,k) 
+   clause[clauseptr++]=sign+((c)<<28)+((k)<<25)+((x)<<12)+(y)
+
+@<Sub...@>=
+void d(int x,int y) {
+  register x1=x-1,x2=x,yy=y+1;
+  if (have_d[x][y]!=tt+1) {
+    applit(x1,yy,1,0),newlit(x,y,4,1),outclause();
+    applit(x2,yy,1,0),newlit(x,y,4,1),outclause();
+    applit(x1,yy,1,0),applit(x2,yy,1,0),newlit(x,y,4,2),outclause();
+    applit(x1,yy,0,0),applit(x2,yy,0,0),newcomplit(x,y,4,1),outclause();
+    applit(x1,yy,0,0),newcomplit(x,y,4,2),outclause();
+    if (yy>=ymin && yy<=ymax)
+      applit(x2,yy,0,0),newcomplit(x,y,4,2),outclause();
+    have_d[x][y]=tt+1;
+  }
+}
+@#
+void e(int x,int y) {
+  register x1=x-1,x2=x,yy=y-1;
+  if (have_e[x][y]!=tt+1) {
+    applit(x1,yy,1,0),newlit(x,y,5,1),outclause();
+    applit(x2,yy,1,0),newlit(x,y,5,1),outclause();
+    applit(x1,yy,1,0),applit(x2,yy,1,0),newlit(x,y,5,2),outclause();
+    applit(x1,yy,0,0),applit(x2,yy,0,0),newcomplit(x,y,5,1),outclause();
+    applit(x1,yy,0,0),newcomplit(x,y,5,2),outclause();
+    if (yy>=ymin && yy<=ymax)
+      applit(x2,yy,0,0),newcomplit(x,y,5,2),outclause();
+    have_e[x][y]=tt+1;
+  }
+}
+    
+@ The |f| subroutine can't be shared quite so often. But we
+do save a factor of~2, because $x+y$ is always even.
+
+@<Sub...@>=
+void f(int x,int y) {
+  register xx=x-1,y1=y,y2=y+1;
+  if (have_f[x][y]!=tt+1) {
+    applit(xx,y1,1,0),newlit(x,y,6,1),outclause();
+    applit(xx,y2,1,0),newlit(x,y,6,1),outclause();
+    applit(xx,y1,1,0),applit(xx,y2,1,0),newlit(x,y,6,2),outclause();
+    applit(xx,y1,0,0),applit(xx,y2,0,0),newcomplit(x,y,6,1),outclause();
+    applit(xx,y1,0,0),newcomplit(x,y,6,2),outclause();
+    if (xx>=xmin && xx<=xmax)
+      applit(xx,y2,0,0),newcomplit(x,y,6,2),outclause();
+    have_f[x][y]=tt+1;
+  }
+}
+
+@ The |g| subroutine cleans up the dregs, by somewhat tediously
+locating the two neighbors that weren't handled by |d|, |e|, or~|f|.
+No sharing is possible here.
+
+@<Sub...@>=
+void g(int x,int y) {
+  register x1,x2,y1,y2;
+  if (x&1) x1=x-1,y1=y,x2=x+1,y2=y^1;
+  else x1=x+1,y1=y,x2=x-1,y2=y-1+((y&1)<<1);
+  applit(x1,y1,1,0),newlit(x,y,7,1),outclause();
+  applit(x2,y2,1,0),newlit(x,y,7,1),outclause();
+  applit(x1,y1,1,0),applit(x2,y2,1,0),newlit(x,y,7,2),outclause();
+  applit(x1,y1,0,0),applit(x2,y2,0,0),newcomplit(x,y,7,1),outclause();
+  applit(x1,y1,0,0),newcomplit(x,y,7,2),outclause();
+  applit(x2,y2,0,0),newcomplit(x,y,7,2),outclause();
+}
+
+@ Fortunately the |b| subroutine {\it can\/} be shared (since |x| is always
+odd), thus saving half of the sixteen clauses generated.
+
+@<Sub...@>=
+void b(int x,int y) {
+  register j,k,xx=x,y1=y-(y&2),y2=y+(y&2);
+  if (have_b[x][y]!=tt+1) {
+    d(xx,y1);
+    e(xx,y2);
+    for (j=0;j<3;j++) for (k=0;k<3;k++) if (j+k) {
+      if (j) newcomplit(xx,y1,4,j); /* $\bar d_j$ */
+      if (k) newcomplit(xx,y2,5,k); /* $\bar e_k$ */
+      newlit(x,y,2,j+k); /* $b_{j+k}$ */
+      outclause();
+      if (j) newlit(xx,y1,4,3-j); /* $d_{3-j}$ */
+      if (k) newlit(xx,y2,5,3-k); /* $e_{3-k}$ */
+      newcomplit(x,y,2,5-j-k); /* $\bar b_{5-j-k}$ */
+      outclause();      
+    }
+    have_b[x][y]=tt+1;
+  }
+}
+
+@ The (unshared) |c| subroutine handles the other four neighbors,
+by working with |f| and |g| instead of |d| and~|e|.
+
+If |y=0|, the overlap rules set |y1=-1|, which can be problematic.
+I've decided to avoid this case by omitting |f| when it is
+guaranteed to be zero.
+
+@<Sub...@>=
+void c(int x,int y) {
+  register j,k,x1,y1;
+  if (x&1) x1=x+2,y1=(y-1)|1;
+  else x1=x,y1=y&-2;
+  g(x,y);
+  if (x1-1<xmin || x1-1>xmax || y1+1<ymin || y1>ymax)
+    @<Set |c| equal to |g|@>@;
+  else {
+    f(x1,y1);
+    for (j=0;j<3;j++) for (k=0;k<3;k++) if (j+k) {
+      if (j) newcomplit(x1,y1,6,j); /* $\bar f_j$ */
+      if (k) newcomplit(x,y,7,k); /* $\bar g_k$ */
+      newlit(x,y,3,j+k); /* $c_{j+k}$ */
+      outclause();
+      if (j) newlit(x1,y1,6,3-j); /* $f_{3-j}$ */
+      if (k) newlit(x,y,7,3-k); /* $g_{3-k}$ */
+      newcomplit(x,y,3,5-j-k); /* $\bar c_{5-j-k}$ */
+      outclause();
+    }
+  }
+}
+
+@ @<Set |c| equal to |g|@>=
+{
+  for (k=1;k<3;k++) {
+    newcomplit(x,y,7,k),newlit(x,y,3,k),outclause(); /* $\bar g_k\lor c_k$ */
+    newlit(x,y,7,k),newcomplit(x,y,3,k),outclause(); /* $g_k\lor\bar c_k$ */
+  }
+  newcomplit(x,y,3,3),outclause(); /* $\bar c_3$ */
+  newcomplit(x,y,3,4),outclause(); /* $\bar c_4$ */
+}
+
+@ Totals over all eight neighbors are then deduced by the |a|
+subroutine.
+
+@<Sub...@>=
+void a(int x,int y) {
+  register j,k,xx=x|1;
+  b(xx,y);
+  c(x,y);
+  for (j=0;j<5;j++) for (k=0;k<5;k++) if (j+k>1 && j+k<5) {
+    if (j) newcomplit(xx,y,2,j); /* $\bar b_j$ */
+    if (k) newcomplit(x,y,3,k); /* $\bar c_k$ */
+    newlit(x,y,1,j+k); /* $a_{j+k}$ */
+    outclause();
+  }
+  for (j=0;j<5;j++) for (k=0;k<5;k++) if (j+k>2 && j+k<6 && j*k) {
+    if (j) newlit(xx,y,2,j); /* $b_j$ */
+    if (k) newlit(x,y,3,k); /* $c_k$ */
+    newcomplit(x,y,1,j+k-1); /* $\bar a_{j+k-1}$ */
+    outclause();
+  }
+}
+
+@ Finally, as mentioned at the beginning, $z'$ is determined
+from $z$, $a_2$, $a_3$, and $a_4$.
+
+I actually generate six clauses, not five, in order to stick to
+{\mc 3SAT}.
+
+@<Sub...@>=
+void zprime(int x,int y) {
+  newcomplit(x,y,1,4),applit(x,y,1,1),outclause(); /* $\bar a_4\bar z'$ */
+  newlit(x,y,1,2),applit(x,y,1,1),outclause(); /* $a_2\bar z'$ */
+  newlit(x,y,1,3),applit(x,y,0,0),applit(x,y,1,1),outclause();
+               /* $a_3z\bar z'$ */
+  newcomplit(x,y,1,3),newlit(x,y,1,4),applit(x,y,0,1),outclause();
+               /* $\bar a_3a_4z'$ */
+  applit(x,y,0,7),newcomplit(x,y,1,2),newlit(x,y,1,4),outclause();
+               /* $x\bar a_2a_4$ */
+  applit(x,y,1,7),applit(x,y,1,0),applit(x,y,0,1),outclause();
+               /* $\bar x\bar zz'$ */
+}
+
+@*Index.

samples/Closure Templates/example.soy

@@ -0,0 +1,24 @@
+{namespace Exmaple}
+
+/**
+ * Example
+ */
+{template .foo}
+  {@param count: string}
+  {@param? name: int}
+
+  {if isNonnull($name)}
+    <h1>{$name}</h1>
+  {/if}
+
+  <div class="content">
+    {switch count}
+      {case 0}
+        {call Empty.view}
+          {param count: $count /}
+        {/call}
+      {default}
+        <h2>Wow, so many!</h2>
+    {/switch}
+  </div>
+{/template}

samples/CoNLL-U/CF1.conllu

@@ -0,0 +1,159 @@
+# text = PT no governo
+# source = CETENFolha n=1 cad=Opinião sec=opi sem=94a
+# sent_id = CF1-1
+# id = 1
+1	PT	PT	PROPN	PROP|M|S|@NPHR	Gender=Masc|Number=Sing	0	root	_	_
+2-3	no	_	_	_	_	_	_	_	_
+2	em	em	ADP	<sam->|PRP|@N<	_	4	case	_	_
+3	o	o	DET	<-sam>|<artd>|ART|M|S|@>N	Definite=Def|Gender=Masc|Number=Sing|PronType=Art	4	det	_	_
+4	governo	governo	NOUN	<np-def>|N|M|S|@P<	Gender=Masc|Number=Sing	1	nmod	_	_
+
+# text = BRASÍLIA Pesquisa Datafolha publicada hoje revela um dado supreendente: recusando uma postura radical, a esmagadora maioria (77%) dos eleitores quer o PT participando do Governo Fernando Henrique Cardoso.
+# source = CETENFolha n=1 cad=Opinião sec=opi sem=94a &W
+# sent_id = CF1-3
+# id = 2
+1	BRASÍLIA	Brasília	PROPN	PROP|F|S|@ADVL>	Gender=Fem|Number=Sing	6	dep	_	_
+2	Pesquisa	Pesquisa	PROPN	_	Gender=Fem|Number=Sing	6	nsubj	_	ChangedBy=Issue119|MWE=Pesquisa_Datafolha|MWEPOS=PROPN
+3	Datafolha	Datafolha	PROPN	_	Number=Sing	2	flat:name	_	ChangedBy=Issue119
+4	publicada	publicar	VERB	<mv>|V|PCP|F|S|@ICL-N<	Gender=Fem|Number=Sing|VerbForm=Part	2	acl	_	_
+5	hoje	hoje	ADV	ADV|@<ADVL	_	4	advmod	_	_
+6	revela	revelar	VERB	<mv>|V|PR|3S|IND|@FS-STA	Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin	0	root	_	_
+7	um	um	DET	<arti>|ART|M|S|@>N	Definite=Ind|Gender=Masc|Number=Sing|PronType=Art	8	det	_	_
+8	dado	dado	NOUN	<np-idf>|N|M|S|@<ACC	Gender=Masc|Number=Sing	6	obj	_	_
+9	supreendente	surpreendente	ADJ	ADJ|M|S|@N<	Gender=Masc|Number=Sing	8	amod	_	ChangedBy=Issue165|SpaceAfter=No
+10	:	:	PUNCT	PU|@PU	_	26	punct	_	_
+11	recusando	recusar	VERB	<mv>|V|GER|@ICL-ADVL>	VerbForm=Ger	26	advcl	_	_
+12	uma	um	DET	<arti>|ART|F|S|@>N	Definite=Ind|Gender=Fem|Number=Sing|PronType=Art	13	det	_	_
+13	postura	postura	NOUN	<np-idf>|N|F|S|@<ACC	Gender=Fem|Number=Sing	11	obj	_	_
+14	radical	radical	ADJ	ADJ|F|S|@N<	Gender=Fem|Number=Sing	13	amod	_	ChangedBy=Issue165|SpaceAfter=No
+15	,	,	PUNCT	PU|@PU	_	26	punct	_	_
+16	a	o	DET	<artd>|ART|F|S|@>N	Definite=Def|Gender=Fem|Number=Sing|PronType=Art	18	det	_	_
+17	esmagadora	esmagador	ADJ	ADJ|F|S|@>N	Gender=Fem|Number=Sing	18	amod	_	_
+18	maioria	maioria	NOUN	<np-def>|N|F|S|@SUBJ>	Gender=Fem|Number=Sing	26	nsubj	_	_
+19	(	(	PUNCT	PU|@PU	_	21	punct	_	ChangedBy=Issue165|SpaceAfter=No
+20	77	77	NUM	<card>|NUM|M|P|@>N	NumType=Card	21	nummod	_	ChangedBy=Issue165|ChangedBy=Issue168|SpaceAfter=No
+21	%	%	SYM	<np-def>|N|M|P|@N<PRED	Gender=Masc|Number=Plur	18	appos	_	ChangedBy=Issue165|SpaceAfter=No
+22	)	)	PUNCT	PU|@PU	_	21	punct	_	_
+23-24	dos	_	_	_	_	_	_	_	_
+23	de	de	ADP	<sam->|PRP|@N<	_	25	case	_	_
+24	os	o	DET	<-sam>|<artd>|ART|M|P|@>N	Definite=Def|Gender=Masc|Number=Plur|PronType=Art	25	det	_	_
+25	eleitores	eleitor	NOUN	<np-def>|N|M|P|@P<	Gender=Masc|Number=Plur	18	nmod	_	_
+26	quer	querer	VERB	<mv>|V|PR|3S|IND|@FS-N<PRED	Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin	8	acl:relcl	_	_
+27	o	o	DET	<artd>|ART|M|S|@>N	Definite=Def|Gender=Masc|Number=Sing|PronType=Art	28	det	_	_
+28	PT	PT	PROPN	PROP|M|S|@<ACC	Gender=Masc|Number=Sing	26	obj	_	_
+29	participando	participar	VERB	<mv>|V|GER|@ICL-<OC	VerbForm=Ger	26	xcomp	_	_
+30-31	do	_	_	_	_	_	_	_	_
+30	de	de	ADP	<sam->|PRP|@<PIV	_	32	case	_	_
+31	o	o	DET	<-sam>|<artd>|ART|M|S|@>N	Definite=Def|Gender=Masc|Number=Sing|PronType=Art	32	det	_	_
+32	Governo	governo	NOUN	<prop>|<np-def>|N|M|S|@P<	Gender=Masc|Number=Sing	29	obl	_	_
+33	Fernando	Fernando	PROPN	_	Gender=Masc|Number=Sing	32	nmod	_	ChangedBy=Issue119|MWE=Fernando_Henrique_Cardoso|MWEPOS=PROPN
+34	Henrique	Henrique	PROPN	_	Number=Sing	33	flat:name	_	ChangedBy=Issue119
+35	Cardoso	Cardoso	PROPN	_	Number=Sing	33	flat:name	_	SpaceAfter=No
+36	.	.	PUNCT	PU|@PU	_	6	punct	_	_
+
+# text = Tem sentido -- aliás, muitíssimo sentido.
+# source = CETENFolha n=1 cad=Opinião sec=opi sem=94a &D
+# sent_id = CF1-4
+# id = 3
+1	Tem	ter	VERB	<mv>|V|PR|3S|IND|@FS-STA	Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin	0	root	_	_
+2	sentido	sentido	NOUN	<np-idf>|N|M|S|@<ACC	Gender=Masc|Number=Sing	1	obj	_	_
+3	--	--	PUNCT	PU|@PU	_	1	punct	_	_
+4	aliás	aliás	ADV	<kc>|ADV|@<ADVL	_	1	advmod	_	ChangedBy=Issue165|SpaceAfter=No
+5	,	,	PUNCT	PU|@PU	_	7	punct	_	_
+6	muitíssimo	muitíssimo	DET	<quant>|<SUP>|DET|M|S|@>N	Gender=Masc|Number=Sing|PronType=Ind	7	det	_	_
+7	sentido	sentido	NOUN	<np-idf>|N|M|S|@N<PRED	Gender=Masc|Number=Sing	2	appos	_	ChangedBy=Issue165|SpaceAfter=No
+8	.	.	PUNCT	PU|@PU	_	1	punct	_	_
+
+# text = Muito mais do que nos tempos na ditadura, a solidez do PT está, agora, ameaçada.
+# source = CETENFolha n=1 cad=Opinião sec=opi sem=94a
+# sent_id = CF1-5
+# id = 4
+1	Muito	muito	ADV	<quant>|ADV|@>A	_	2	advmod	_	_
+2	mais	mais	ADV	<quant>|<KOMP>|<COMP>|ADV|@ADVL>	_	22	advmod	_	_
+3-4	do	_	_	_	_	_	_	_	_
+3	de	de	ADP	<sam->|PRP|@COM	_	8	case	_	_
+4	o	o	PRON	<dem>|<-sam>|DET|M|S|@P<	Gender=Masc|Number=Sing|PronType=Dem	3	fixed	_	_
+5	que	que	PRON	<rel>|INDP|M|S|@N<	Gender=Masc|Number=Sing|PronType=Rel	3	fixed	_	_
+6-7	nos	_	_	_	_	_	_	_	_
+6	em	em	ADP	<sam->|<first-cjt>|PRP|@KOMP<	_	8	case	_	_
+7	os	o	DET	<-sam>|<artd>|ART|M|P|@>N	Definite=Def|Gender=Masc|Number=Plur|PronType=Art	8	det	_	_
+8	tempos	tempo	NOUN	<first-cjt>|<np-def>|N|M|P|@P<	Gender=Masc|Number=Plur	2	obl	_	_
+9-10	na	_	_	_	_	_	_	_	_
+9	em	em	ADP	<sam->|PRP|@N<	_	11	case	_	_
+10	a	o	DET	<-sam>|<artd>|ART|F|S|@>N	Definite=Def|Gender=Fem|Number=Sing|PronType=Art	11	det	_	_
+11	ditadura	ditadura	NOUN	<np-def>|N|F|S|@P<	Gender=Fem|Number=Sing	8	nmod	_	ChangedBy=Issue165|SpaceAfter=No
+12	,	,	PUNCT	PU|@PU	_	2	punct	_	_
+13	a	o	DET	<artd>|ART|F|S|@>N	Definite=Def|Gender=Fem|Number=Sing|PronType=Art	14	det	_	_
+14	solidez	solidez	NOUN	<np-def>|N|F|S|@SUBJ>	Gender=Fem|Number=Sing	22	nsubj	_	_
+15-16	do	_	_	_	_	_	_	_	_
+15	de	de	ADP	<sam->|PRP|@N<	_	17	case	_	_
+16	o	o	DET	<-sam>|<artd>|ART|M|S|@>N	Definite=Def|Gender=Masc|Number=Sing|PronType=Art	17	det	_	_
+17	PT	PT	PROPN	PROP|M|S|@P<	Gender=Masc|Number=Sing	14	nmod	_	_
+18	está	estar	AUX	<mv>|V|PR|3S|IND|@FS-STA	Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin	22	cop	_	ChangedBy=Issue165|ChangedBy=Issue167|SpaceAfter=No
+19	,	,	PUNCT	PU|@PU	_	20	punct	_	_
+20	agora	agora	ADV	<kc>|ADV|@<ADVL	_	22	advmod	_	ChangedBy=Issue165|SpaceAfter=No
+21	,	,	PUNCT	PU|@PU	_	20	punct	_	_
+22	ameaçada	ameaçar	VERB	<mv>|V|PCP|F|S|@ICL-<SC	Gender=Fem|Number=Sing|VerbForm=Part	0	root	_	ChangedBy=Issue165|SpaceAfter=No
+23	.	.	PUNCT	PU|@PU	_	22	punct	_	_
+
+# text = Nem Lula nem o partido ainda encontraram um discurso para se diferenciar.
+# source = CETENFolha n=1 cad=Opinião sec=opi sem=94a
+# sent_id = CF1-6
+# id = 5
+1	Nem	nem	CCONJ	<parkc-1>|KC|@CO	_	2	cc	_	_
+2	Lula	Lula	PROPN	<first-cjt>|PROP|M|S|@SUBJ>	Gender=Masc|Number=Sing	7	nsubj	_	_
+3	nem	nem	CCONJ	<co-subj>|<parkc-2>|KC|@CO	_	5	cc	_	_
+4	o	o	DET	<artd>|ART|M|S|@>N	Definite=Def|Gender=Masc|Number=Sing|PronType=Art	5	det	_	_
+5	partido	partido	NOUN	<cjt>|<np-def>|N|M|S|@SUBJ>	Gender=Masc|Number=Sing	2	conj	_	_
+6	ainda	ainda	ADV	ADV|@ADVL>	_	7	advmod	_	_
+7	encontraram	encontrar	VERB	<mv>|V|PS/MQP|3P|IND|@FS-STA	Mood=Ind|Number=Plur|Person=3|VerbForm=Fin	0	root	_	_
+8	um	um	DET	_	Definite=Ind|Gender=Masc|Number=Sing|PronType=Art	9	det	_	_
+9	discurso	discurso	NOUN	<np-idf>|N|M|S|@<ACC	Gender=Masc|Number=Sing	7	obj	_	_
+10	para	para	ADP	_	_	12	case	_	_
+11	se	se	PRON	PERS|M|3S|ACC|@ACC>-PASS	Case=Acc|Gender=Masc|Number=Sing|Person=3|PronType=Prs	12	expl	_	ChangedBy=Issue135
+12	diferenciar	diferenciar	VERB	_	VerbForm=Inf	9	acl	_	ChangedBy=Issue165|SpaceAfter=No
+13	.	.	PUNCT	PU|@PU	_	7	punct	_	_
+
+# text = Eles se dizem oposição, mas ainda não informaram o que vão combater.
+# source = CETENFolha n=1 cad=Opinião sec=opi sem=94a
+# sent_id = CF1-7
+# id = 6
+1	Eles	eles	PRON	PERS|M|3P|NOM|@SUBJ>	Case=Nom|Gender=Masc|Number=Plur|Person=3|PronType=Prs	3	nsubj	_	_
+2	se	se	PRON	PERS|M|3P|ACC|@ACC>-PASS	Case=Acc|Gender=Masc|Number=Plur|Person=3|PronType=Prs	3	expl	_	ChangedBy=Issue135
+3	dizem	dizer	VERB	<first-cjt>|<mv>|<se-passive>|V|PR|3P|IND|@FS-STA	Mood=Ind|Number=Plur|Person=3|Tense=Pres|VerbForm=Fin	0	root	_	_
+4	oposição	oposição	NOUN	<np-idf>|N|F|S|@<OC	Gender=Fem|Number=Sing	3	xcomp	_	ChangedBy=Issue165|SpaceAfter=No
+5	,	,	PUNCT	PU|@PU	_	9	punct	_	_
+6	mas	mas	CCONJ	<co-fcl>|KC|@CO	_	9	cc	_	_
+7	ainda	ainda	ADV	ADV|@>A	_	8	advmod	_	_
+8	não	não	ADV	_	Polarity=Neg	9	advmod	_	_
+9	informaram	informar	VERB	<cjt>|<mv>|V|PS/MQP|3P|IND|@FS-STA	Mood=Ind|Number=Plur|Person=3|VerbForm=Fin	3	conj	_	_
+10	o	o	PRON	_	Gender=Masc|Number=Sing|PronType=Dem	11	det	_	_
+11	que	que	PRON	<interr>|INDP|M|S|@ACC>	Gender=Masc|Number=Sing|PronType=Int	13	obj	_	_
+12	vão	ir	AUX	<aux>|V|PR|3P|IND|@FS-<ACC	Mood=Ind|Number=Plur|Person=3|Tense=Pres|VerbForm=Fin	13	aux	_	_
+13	combater	combater	VERB	<mv>|V|INF|@ICL-AUX<	VerbForm=Inf	9	ccomp	_	ChangedBy=Issue165|SpaceAfter=No
+14	.	.	PUNCT	PU|@PU	_	3	punct	_	_
+
+# text = Muitas das prioridades do novo governo coincidem com as prioridades do PT.
+# source = CETENFolha n=1 cad=Opinião sec=opi sem=94a
+# sent_id = CF1-8
+# id = 7
+1	Muitas	muito	PRON	<quant>|DET|F|P|@SUBJ>	Gender=Fem|Number=Plur|PronType=Ind	9	nsubj	_	_
+2-3	das	_	_	_	_	_	_	_	_
+2	de	de	ADP	<sam->|PRP|@N<	_	4	case	_	_
+3	as	o	DET	<-sam>|<artd>|ART|F|P|@>N	Definite=Def|Gender=Fem|Number=Plur|PronType=Art	4	det	_	_
+4	prioridades	prioridade	NOUN	<np-def>|N|F|P|@P<	Gender=Fem|Number=Plur	1	nmod	_	_
+5-6	do	_	_	_	_	_	_	_	_
+5	de	de	ADP	<sam->|PRP|@N<	_	8	case	_	_
+6	o	o	DET	<-sam>|<artd>|ART|M|S|@>N	Definite=Def|Gender=Masc|Number=Sing|PronType=Art	8	det	_	_
+7	novo	novo	ADJ	ADJ|M|S|@>N	Gender=Masc|Number=Sing	8	amod	_	_
+8	governo	governo	NOUN	<np-def>|N|M|S|@P<	Gender=Masc|Number=Sing	4	nmod	_	_
+9	coincidem	coincidir	VERB	<mv>|V|PR|3P|IND|@FS-STA	Mood=Ind|Number=Plur|Person=3|Tense=Pres|VerbForm=Fin	0	root	_	_
+10	com	com	ADP	PRP|@<PIV	_	12	case	_	_
+11	as	o	DET	<artd>|ART|F|P|@>N	Definite=Def|Gender=Fem|Number=Plur|PronType=Art	12	det	_	_
+12	prioridades	prioridade	NOUN	<np-def>|N|F|P|@P<	Gender=Fem|Number=Plur	9	obj	_	_
+13-14	do	_	_	_	_	_	_	_	_
+13	de	de	ADP	<sam->|PRP|@N<	_	15	case	_	_
+14	o	o	DET	<-sam>|<artd>|ART|M|S|@>N	Definite=Def|Gender=Masc|Number=Sing|PronType=Art	15	det	_	_
+15	PT	PT	PROPN	PROP|M|S|@P<	Gender=Masc|Number=Sing	12	nmod	_	ChangedBy=Issue165|SpaceAfter=No
+16	.	.	PUNCT	PU|@PU	_	9	punct	_	_
+

samples/CoNLL-U/en-ud-test-abridged.conllu

@@ -0,0 +1,122 @@
+# newdoc id = weblog-blogspot.com_zentelligence_20040423000200_ENG_20040423_000200
+# sent_id = weblog-blogspot.com_zentelligence_20040423000200_ENG_20040423_000200-0001
+# text = What if Google Morphed Into GoogleOS?
+1	What	what	PRON	WP	PronType=Int	0	root	0:root	_
+2	if	if	SCONJ	IN	_	4	mark	4:mark	_
+3	Google	Google	PROPN	NNP	Number=Sing	4	nsubj	4:nsubj	_
+4	Morphed	morph	VERB	VBD	Mood=Ind|Tense=Past|VerbForm=Fin	1	advcl	1:advcl	_
+5	Into	into	ADP	IN	_	6	case	6:case	_
+6	GoogleOS	GoogleOS	PROPN	NNP	Number=Sing	4	obl	4:obl	SpaceAfter=No
+7	?	?	PUNCT	.	_	4	punct	4:punct	_
+
+# sent_id = weblog-blogspot.com_zentelligence_20040423000200_ENG_20040423_000200-0002
+# text = What if Google expanded on its search-engine (and now e-mail) wares into a full-fledged operating system?
+1	What	what	PRON	WP	PronType=Int	0	root	0:root	_
+2	if	if	SCONJ	IN	_	4	mark	4:mark	_
+3	Google	Google	PROPN	NNP	Number=Sing	4	nsubj	4:nsubj	_
+4	expanded	expand	VERB	VBD	Mood=Ind|Tense=Past|VerbForm=Fin	1	advcl	1:advcl	_
+5	on	on	ADP	IN	_	15	case	15:case	_
+6	its	its	PRON	PRP$	Gender=Neut|Number=Sing|Person=3|Poss=Yes|PronType=Prs	15	nmod:poss	15:nmod:poss	_
+7	search	search	NOUN	NN	Number=Sing	9	compound	9:compound	SpaceAfter=No
+8	-	-	PUNCT	HYPH	_	9	punct	9:punct	SpaceAfter=No
+9	engine	engine	NOUN	NN	Number=Sing	15	compound	15:compound	_
+10	(	(	PUNCT	-LRB-	_	9	punct	9:punct	SpaceAfter=No
+11	and	and	CCONJ	CC	_	13	cc	13:cc	_
+12	now	now	ADV	RB	_	13	advmod	13:advmod	_
+13	e-mail	e-mail	NOUN	NN	Number=Sing	9	conj	9:conj	SpaceAfter=No
+14	)	)	PUNCT	-RRB-	_	15	punct	15:punct	_
+15	wares	wares	NOUN	NNS	Number=Plur	4	obl	4:obl	_
+16	into	into	ADP	IN	_	22	case	22:case	_
+17	a	a	DET	DT	Definite=Ind|PronType=Art	22	det	22:det	_
+18	full	full	ADV	RB	_	20	advmod	20:advmod	SpaceAfter=No
+19	-	-	PUNCT	HYPH	_	20	punct	20:punct	SpaceAfter=No
+20	fledged	fledged	ADJ	JJ	Degree=Pos	22	amod	22:amod	_
+21	operating	operating	NOUN	NN	Number=Sing	22	compound	22:compound	_
+22	system	system	NOUN	NN	Number=Sing	4	obl	4:obl	SpaceAfter=No
+23	?	?	PUNCT	.	_	4	punct	4:punct	_
+
+# sent_id = weblog-blogspot.com_zentelligence_20040423000200_ENG_20040423_000200-0003
+# text = [via Microsoft Watch from Mary Jo Foley ]
+1	[	[	PUNCT	-LRB-	_	4	punct	4:punct	SpaceAfter=No
+2	via	via	ADP	IN	_	4	case	4:case	_
+3	Microsoft	Microsoft	PROPN	NNP	Number=Sing	4	compound	4:compound	_
+4	Watch	Watch	PROPN	NNP	Number=Sing	0	root	0:root	_
+5	from	from	ADP	IN	_	6	case	6:case	_
+6	Mary	Mary	PROPN	NNP	Number=Sing	4	nmod	4:nmod	_
+7	Jo	Jo	PROPN	NNP	Number=Sing	6	flat	6:flat	_
+8	Foley	Foley	PROPN	NNP	Number=Sing	6	flat	6:flat	_
+9	]	]	PUNCT	-RRB-	_	4	punct	4:punct	_
+
+# newdoc id = weblog-blogspot.com_marketview_20050511222700_ENG_20050511_222700
+# sent_id = weblog-blogspot.com_marketview_20050511222700_ENG_20050511_222700-0001
+# text = (And, by the way, is anybody else just a little nostalgic for the days when that was a good thing?)
+1	(	(	PUNCT	-LRB-	_	14	punct	14:punct	SpaceAfter=No
+2	And	and	CCONJ	CC	_	14	cc	14:cc	SpaceAfter=No
+3	,	,	PUNCT	,	_	14	punct	14:punct	_
+4	by	by	ADP	IN	_	6	case	6:case	_
+5	the	the	DET	DT	Definite=Def|PronType=Art	6	det	6:det	_
+6	way	way	NOUN	NN	Number=Sing	14	obl	14:obl	SpaceAfter=No
+7	,	,	PUNCT	,	_	14	punct	14:punct	_
+8	is	be	AUX	VBZ	Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin	14	cop	14:cop	_
+9	anybody	anybody	PRON	NN	Number=Sing	14	nsubj	14:nsubj	_
+10	else	else	ADJ	JJ	Degree=Pos	9	amod	9:amod	_
+11	just	just	ADV	RB	_	13	advmod	13:advmod	_
+12	a	a	DET	DT	Definite=Ind|PronType=Art	13	det	13:det	_
+13	little	little	ADJ	JJ	Degree=Pos	14	obl:npmod	14:obl:npmod	_
+14	nostalgic	nostalgic	NOUN	NN	Number=Sing	0	root	0:root	_
+15	for	for	ADP	IN	_	17	case	17:case	_
+16	the	the	DET	DT	Definite=Def|PronType=Art	17	det	17:det	_
+17	days	day	NOUN	NNS	Number=Plur	14	nmod	14:nmod	_
+18	when	when	ADV	WRB	PronType=Rel	23	advmod	23:advmod	_
+19	that	that	PRON	DT	Number=Sing|PronType=Dem	23	nsubj	23:nsubj	_
+20	was	be	AUX	VBD	Mood=Ind|Number=Sing|Person=3|Tense=Past|VerbForm=Fin	23	cop	23:cop	_
+21	a	a	DET	DT	Definite=Ind|PronType=Art	23	det	23:det	_
+22	good	good	ADJ	JJ	Degree=Pos	23	amod	23:amod	_
+23	thing	thing	NOUN	NN	Number=Sing	17	acl:relcl	17:acl:relcl	SpaceAfter=No
+24	?	?	PUNCT	.	_	14	punct	14:punct	SpaceAfter=No
+25	)	)	PUNCT	-RRB-	_	14	punct	14:punct	_
+
+# sent_id = weblog-blogspot.com_marketview_20050511222700_ENG_20050511_222700-0002
+# text = This BuzzMachine post argues that Google's rush toward ubiquity might backfire -- which we've all heard before, but it's particularly well-put in this post.
+1	This	this	DET	DT	Number=Sing|PronType=Dem	3	det	3:det	_
+2	BuzzMachine	BuzzMachine	PROPN	NNP	Number=Sing	3	compound	3:compound	_
+3	post	post	NOUN	NN	Number=Sing	4	nsubj	4:nsubj	_
+4	argues	argue	VERB	VBZ	Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin	0	root	0:root	_
+5	that	that	SCONJ	IN	_	12	mark	12:mark	_
+6	Google	Google	PROPN	NNP	Number=Sing	8	nmod:poss	8:nmod:poss	SpaceAfter=No
+7	's	's	PART	POS	_	6	case	6:case	_
+8	rush	rush	NOUN	NN	Number=Sing	12	nsubj	12:nsubj	_
+9	toward	toward	ADP	IN	_	10	case	10:case	_
+10	ubiquity	ubiquity	NOUN	NN	Number=Sing	8	nmod	8:nmod	_
+11	might	might	AUX	MD	VerbForm=Fin	12	aux	12:aux	_
+12	backfire	backfire	VERB	VB	VerbForm=Inf	4	ccomp	4:ccomp	_
+13	--	--	PUNCT	,	_	12	punct	12:punct	_
+14	which	which	PRON	WDT	PronType=Rel	18	obj	18:obj	_
+15	we	we	PRON	PRP	Case=Nom|Number=Plur|Person=1|PronType=Prs	18	nsubj	18:nsubj	SpaceAfter=No
+16	've	have	AUX	VBP	Mood=Ind|Tense=Pres|VerbForm=Fin	18	aux	18:aux	_
+17	all	all	ADV	RB	_	18	advmod	18:advmod	_
+18	heard	hear	VERB	VBN	Tense=Past|VerbForm=Part	12	acl:relcl	12:acl:relcl	_
+19	before	before	ADV	RB	_	18	advmod	18:advmod	SpaceAfter=No
+20	,	,	PUNCT	,	_	27	punct	27:punct	_
+21	but	but	CCONJ	CC	_	27	cc	27:cc	_
+22	it	it	PRON	PRP	Case=Nom|Gender=Neut|Number=Sing|Person=3|PronType=Prs	27	nsubj:pass	27:nsubj:pass	SpaceAfter=No
+23	's	be	VERB	VBZ	Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin	27	aux:pass	27:aux:pass	_
+24	particularly	particularly	ADV	RB	_	27	advmod	27:advmod	_
+25	well	well	ADV	RB	Degree=Pos	27	advmod	27:advmod	SpaceAfter=No
+26	-	-	PUNCT	HYPH	_	27	punct	27:punct	SpaceAfter=No
+27	put	put	VERB	VBN	Tense=Past|VerbForm=Part	4	conj	4:conj	_
+28	in	in	ADP	IN	_	30	case	30:case	_
+29	this	this	DET	DT	Number=Sing|PronType=Dem	30	det	30:det	_
+30	post	post	NOUN	NN	Number=Sing	27	obl	27:obl	SpaceAfter=No
+31	.	.	PUNCT	.	_	4	punct	4:punct	_
+
+# sent_id = weblog-blogspot.com_marketview_20050511222700_ENG_20050511_222700-0003
+# text = Google is a nice search engine.
+1	Google	Google	PROPN	NNP	Number=Sing	6	nsubj	6:nsubj	_
+2	is	be	AUX	VBZ	Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin	6	cop	6:cop	_
+3	a	a	DET	DT	Definite=Ind|PronType=Art	6	det	6:det	_
+4	nice	nice	ADJ	JJ	Degree=Pos	6	amod	6:amod	_
+5	search	search	NOUN	NN	Number=Sing	6	compound	6:compound	_
+6	engine	engine	NOUN	NN	Number=Sing	0	root	0:root	SpaceAfter=No
+7	.	.	PUNCT	.	_	6	punct	6:punct	_
+

samples/CoNLL-U/ug-ud-test-abridged.conllu

@@ -0,0 +1,121 @@
+# sent_id = s1
+# text = ئاسماننى كۆپكۈك، دەريا، كۆل سۇلىرىنى سۈپسۈزۈك تۇرۇشقا، دەل - دەرەخلەرنى بۈك - باراقسان بولۇشقا، ھايۋانلارنى ئەركىن ئازادە ياشاشقا ئىگە قىلىش... بىزنىڭ ئورتاق ئارزۇيىمىز.
+1	ئاسماننى	_	NOUN	N	_	30	csubj	_	Translit=asmanni
+2	كۆپكۈك	_	VERB	V	_	1	orphan	_	SpaceAfter=No|Translit=köpkük
+3	،	_	PUNCT	Y	_	2	punct	_	Translit=,
+4	دەريا	_	NOUN	N	_	7	nmod:poss	_	SpaceAfter=No|Translit=derya
+5	،	_	PUNCT	Y	_	4	punct	_	Translit=,
+6	كۆل	_	NOUN	N	_	4	conj	_	Translit=köl
+7	سۇلىرىنى	_	NOUN	N	_	9	obj	_	Translit=sulirini
+8	سۈپسۈزۈك	_	ADJ	A	_	9	advmod	_	Translit=süpsüzük
+9	تۇرۇشقا	_	VERB	V	_	1	conj	_	SpaceAfter=No|Translit=turushqa
+10	،	_	PUNCT	Y	_	1	punct	_	Translit=,
+11	دەل	_	ADV	D	_	13	compound:redup	_	Translit=del
+12	-	_	PUNCT	Y	_	11	punct	_	Translit=-
+13	دەرەخلەرنى	_	NOUN	N	_	17	obj	_	Translit=derexlerni
+14	بۈك	_	ADJ	A	_	16	compound	_	Translit=bük
+15	-	_	PUNCT	Y	_	14	punct	_	Translit=-
+16	باراقسان	_	ADJ	A	_	17	advmod	_	Translit=baraqsan
+17	بولۇشقا	_	VERB	V	_	9	orphan	_	SpaceAfter=No|Translit=bolushqa
+18	،	_	PUNCT	Y	_	17	punct	_	Translit=,
+19	ھايۋانلارنى	_	NOUN	N	_	24	obj	_	Translit=haywanlarni
+20	ئەركىن	_	ADJ	A	_	21	compound:redup	_	Translit=erkin
+21	ئازادە	_	ADJ	A	_	22	advmod	_	Translit=azade
+22	ياشاشقا	_	NOUN	N	_	24	advcl	_	Translit=yashashqa
+23	ئىگە	_	NOUN	N	_	24	compound	_	Translit=ige
+24	قىلىش	_	VERB	V	_	1	conj	_	SpaceAfter=No|Translit=qilish
+25	.	_	PUNCT	Y	_	1	punct	_	SpaceAfter=No|Translit=.
+26	.	_	PUNCT	Y	_	1	punct	_	SpaceAfter=No|Translit=.
+27	.	_	PUNCT	Y	_	1	punct	_	Translit=.
+28	بىزنىڭ	_	PRON	P	_	30	nmod:poss	_	Translit=bizning
+29	ئورتاق	_	ADJ	A	_	30	amod	_	Translit=ortaq
+30	ئارزۇيىمىز	_	NOUN	N	_	0	root	_	SpaceAfter=No|Translit=arzuyimiz
+31	.	_	PUNCT	Y	_	30	punct	_	Translit=.
+
+# sent_id = s2
+# text = بۇ بۆلەكتىكى تېكىستلەرنى ئوقۇش ئارقىلىق، كىشىلەرنىڭ ھايۋانلار ۋە ئۆسۈملۈكلەرگە قانداق مۇئامىلە قىلغانلىقى، ئاقىۋىتىنىڭ قانداق بولغانلىقىنى كۆرۈپ باقايلى،
+1	بۇ	_	PRON	P	_	2	det	_	Translit=bu
+2	بۆلەكتىكى	_	NOUN	N	_	3	nmod	_	Translit=bölektiki
+3	تېكىستلەرنى	_	NOUN	N	_	4	obj	_	Translit=tëkistlerni
+4	ئوقۇش	_	VERB	V	_	18	advcl	_	Translit=oqush
+5	ئارقىلىق	_	ADP	R	_	4	case	_	SpaceAfter=No|Translit=arqiliq
+6	،	_	PUNCT	Y	_	5	punct	_	Translit=,
+7	كىشىلەرنىڭ	_	NOUN	N	_	13	nsubj	_	Translit=kishilerning
+8	ھايۋانلار	_	NOUN	N	_	13	obl	_	Translit=haywanlar
+9	ۋە	_	CCONJ	C	_	10	cc	_	Translit=we
+10	ئۆسۈملۈكلەرگە	_	NOUN	N	_	8	conj	_	Translit=ösümlüklerge
+11	قانداق	_	PRON	P	_	13	advmod	_	Translit=qandaq
+12	مۇئامىلە	_	NOUN	N	_	13	compound	_	Translit=muamile
+13	قىلغانلىقى	_	VERB	V	_	18	conj	_	SpaceAfter=No|Translit=qilghanliqi
+14	،	_	PUNCT	Y	_	13	punct	_	Translit=,
+15	ئاقىۋىتىنىڭ	_	NOUN	N	_	17	nsubj	_	Translit=aqiwitining
+16	قانداق	_	PRON	P	_	17	advmod	_	Translit=qandaq
+17	بولغانلىقىنى	_	VERB	V	_	18	obj	_	Translit=bolghanliqini
+18	كۆرۈپ	_	VERB	V	_	0	root	_	Translit=körüp
+19	باقايلى	_	VERB	V	_	18	aux	_	SpaceAfter=No|Translit=baqayli
+20	،	_	PUNCT	Y	_	19	punct	_	Translit=,
+
+# sent_id = s3
+# text = يەنە ئەتراپىمىزدىكى مۇھىتنى ياخشى كۆزىتىپ، مۇھىتنى قوغداش ئۈچۈن نېمىلەرنى قىلالايدىغانلىقىمىز توغرۇلۇق ئويلىنىپ باقايلى.
+1	يەنە	_	ADV	D	_	13	cc	_	Translit=yene
+2	ئەتراپىمىزدىكى	_	NOUN	N	_	3	amod	_	Translit=etrapimizdiki
+3	مۇھىتنى	_	NOUN	N	_	5	obj	_	Translit=muhitni
+4	ياخشى	_	ADJ	A	_	5	advmod	_	Translit=yaxshi
+5	كۆزىتىپ	_	VERB	V	_	13	advcl	_	SpaceAfter=No|Translit=közitip
+6	،	_	PUNCT	Y	_	5	punct	_	Translit=,
+7	مۇھىتنى	_	NOUN	N	_	8	obj	_	Translit=muhitni
+8	قوغداش	_	VERB	V	_	11	advcl	_	Translit=qoghdash
+9	ئۈچۈن	_	CCONJ	C	_	8	case	_	Translit=üchün
+10	نېمىلەرنى	_	PRON	P	_	11	obj	_	Translit=nëmilerni
+11	قىلالايدىغانلىقىمىز	_	VERB	V	_	13	obj	_	Translit=qilalaydighanliqimiz
+12	توغرۇلۇق	_	ADP	R	_	11	case	_	Translit=toghruluq
+13	ئويلىنىپ	_	VERB	V	_	0	root	_	Translit=oylinip
+14	باقايلى	_	VERB	V	_	13	aux	_	SpaceAfter=No|Translit=baqayli
+15	.	_	PUNCT	Y	_	14	punct	_	Translit=.
+
+# sent_id = s4
+# text = بىر يىلى باھار كۈنلىرىنىڭ بىرىدە، شىۋېتسارىيىنىڭ بىر ۋوگزالىدا ھاۋا تەڭشىگۈچ ئورنىتىلغان چىرايلىق، ئازادە بىر پويىز قوزغىلىش ئالدىدا تۇراتتى.
+1	بىر	_	NUM	M	_	2	nummod	_	Translit=bir
+2	يىلى	_	NOUN	N	_	20	nmod:tmod	_	Translit=yili
+3	باھار	_	NOUN	N	_	4	nmod:poss	_	Translit=bahar
+4	كۈنلىرىنىڭ	_	NOUN	N	_	5	nmod:part	_	Translit=künlirining
+5	بىرىدە	_	NUM	M	_	20	nmod:tmod	_	SpaceAfter=No|Translit=biride
+6	،	_	PUNCT	Y	_	5	punct	_	Translit=,
+7	شىۋېتسارىيىنىڭ	_	NOUN	N	_	9	nmod:poss	_	Translit=shiwëtsariyining
+8	بىر	_	NUM	M	_	9	det	_	Translit=bir
+9	ۋوگزالىدا	_	NOUN	N	_	20	obl	_	Translit=wogzalida
+10	ھاۋا	_	NOUN	N	_	11	compound	_	Translit=hawa
+11	تەڭشىگۈچ	_	NOUN	N	_	12	nsubj	_	Translit=tengshigüch
+12	ئورنىتىلغان	_	NOUN	N	_	17	amod	_	Translit=ornitilghan
+13	چىرايلىق	_	ADJ	A	_	17	amod	_	SpaceAfter=No|Translit=chirayliq
+14	،	_	PUNCT	Y	_	13	punct	_	Translit=,
+15	ئازادە	_	ADJ	A	_	13	conj	_	Translit=azade
+16	بىر	_	NUM	M	_	17	det	_	Translit=bir
+17	پويىز	_	NOUN	N	_	20	nsubj	_	Translit=poyiz
+18	قوزغىلىش	_	VERB	V	_	19	nmod:poss	_	Translit=qozghilish
+19	ئالدىدا	_	NOUN	N	_	20	obl	_	Translit=aldida
+20	تۇراتتى	_	VERB	V	_	0	root	_	SpaceAfter=No|Translit=turatti
+21	.	_	PUNCT	Y	_	20	punct	_	Translit=.
+
+# sent_id = s5
+# text = ۋوگزال سۇپىسى ئۇزاتقۇچىلار بىلەن تولۇپ كەتكەنىدى.
+1	ۋوگزال	_	NOUN	N	_	2	nmod:poss	_	Translit=wogzal
+2	سۇپىسى	_	NOUN	N	_	5	nsubj	_	Translit=supisi
+3	ئۇزاتقۇچىلار	_	NOUN	N	_	5	obl	_	Translit=uzatquchilar
+4	بىلەن	_	ADP	R	_	3	case	_	Translit=bilen
+5	تولۇپ	_	VERB	V	_	0	root	_	Translit=tolup
+6	كەتكەنىدى	_	VERB	V	_	5	aux	_	SpaceAfter=No|Translit=ketkenidi
+7	.	_	PUNCT	Y	_	6	punct	_	Translit=.
+
+# sent_id = s6
+# text = ئۇلارنىڭ ئۇزاتماقچى بولغىنى ئۆزگىچە مىھمان - قارلىغاچلار ئىدى.
+1	ئۇلارنىڭ	_	PRON	P	_	2	nsubj	_	Translit=ularning
+2	ئۇزاتماقچى	_	NOUN	N	_	5	acl	_	Translit=uzatmaqchi
+3	بولغىنى	_	AUX	V	_	2	cop	_	Translit=bolghini
+4	ئۆزگىچە	_	ADJ	A	_	5	amod	_	Translit=özgiche
+5	مىھمان	_	NOUN	N	_	7	appos	_	Translit=mihman
+6	-	_	PUNCT	Y	_	5	punct	_	Translit=-
+7	قارلىغاچلار	_	NOUN	N	_	0	root	_	Translit=qarlighachlar
+8	ئىدى	_	AUX	V	_	7	cop	_	SpaceAfter=No|Translit=idi
+9	.	_	PUNCT	Y	_	8	punct	_	Translit=.
+

samples/Common Workflow Language/trunk-peak-score.cwl

@@ -0,0 +1,36 @@
+#!/usr/bin/env cwl-runner
+# Originally from
+# https://github.com/Duke-GCB/GGR-cwl/blob/54e897263a702ff1074c8ac814b4bf7205d140dd/utils/trunk-peak-score.cwl
+# Released under the MIT License:
+# https://github.com/Duke-GCB/GGR-cwl/blob/54e897263a702ff1074c8ac814b4bf7205d140dd/LICENSE
+# Converted to CWL v1.0 syntax using
+# https://github.com/common-workflow-language/cwl-upgrader
+# and polished by Michael R. Crusoe <mrc@commonwl.org>
+# All modifications also released under the MIT License
+cwlVersion: v1.0
+class: CommandLineTool
+doc: Trunk scores in ENCODE bed6+4 files
+
+hints:
+  DockerRequirement:
+    dockerPull: dukegcb/workflow-utils
+
+inputs:
+  peaks:
+    type: File
+  sep:
+    type: string
+    default: \t
+
+outputs:
+  trunked_scores_peaks:
+    type: stdout
+
+baseCommand: awk
+
+arguments:
+- -F $(inputs.sep)
+- BEGIN{OFS=FS}$5>1000{$5=1000}{print}
+- $(inputs.peaks.path)
+
+stdout: $(inputs.peaks.nameroot).trunked_scores$(inputs.peaks.nameext)

samples/Coq/Basics.v

@@ -1,707 +0,0 @@
-Inductive day : Type :=
-| monday : day
-| tuesday : day
-| wednesday : day
-| thursday : day
-| friday : day
-| saturday : day
-| sunday : day.
-
-Definition next_weekday (d:day) : day :=
-  match d with
-  | monday => tuesday
-  | tuesday => wednesday
-  | wednesday => thursday
-  | thursday => friday
-  | friday => monday
-  | saturday => monday
-  | sunday => monday
-  end.
-
-Example test_next_weekday:
-(next_weekday (next_weekday saturday)) = tuesday.
-
-Proof. simpl. reflexivity. Qed.
-
-Inductive bool : Type :=
-	| true : bool
-	| false : bool.
-
-Definition negb (b:bool) : bool :=
-													 match b with
-																			 | true => false
-																			 | false => true
-													 end.
-
-Definition andb (b1:bool) (b2:bool) : bool :=
-		match b1 with
-		 | true => b2
-		 | false => false
-	  end.
-
-Definition orb (b1:bool) (b2:bool) : bool :=
-		match b1 with
-		  | true => true
-		  | false => b2
-		end.
-
-Example test_orb1: (orb true false) = true.
-Proof. simpl. reflexivity. Qed.
-
-Example test_orb2: (orb false false) = false.
-Proof. simpl. reflexivity. Qed.
-
-Example test_orb3: (orb false true) = true.
-Proof. simpl. reflexivity. Qed.
-
-Example test_orb4: (orb true true) = true.
-Proof. simpl. reflexivity. Qed.
-
-Definition nandb (b1: bool) (b2:bool) : bool :=
-	match b1 with
-		| true => match b2 with
-										| false => true
-										| true => false
-							end
-		| false => true
-	end.
-
-Example test_nandb1: (nandb true false) = true.
-Proof. simpl. reflexivity. Qed.
-Example test_nandb2: (nandb false false) = true.
-Proof. simpl. reflexivity. Qed.
-Example test_nandb3: (nandb false true) = true.
-Proof. simpl. reflexivity. Qed.
-Example test_nandb4: (nandb true true) = false.
-Proof. simpl. reflexivity. Qed.
-
-Definition andb3 (b1: bool) (b2:bool) (b3:bool) : bool :=
-	match b1 with
-    | false => false
-		| true => match b2 with
-								| false => false
-								| true => b3
-							end
-	end.
-
-Example test_andb31: (andb3 true true true) = true.
-Proof. simpl. reflexivity. Qed.
-Example test_andb32: (andb3 false true true) = false.
-Proof. simpl. reflexivity. Qed.
-Example test_andb33: (andb3 true false true) = false.
-Proof. simpl. reflexivity. Qed.
-Example test_andb34: (andb3 true true false) = false.
-Proof. simpl. reflexivity. Qed.
-
-Module Playground1.
-
-Inductive nat : Type :=
-	| O : nat
-	| S : nat -> nat.
-
-Definition pred (n : nat) : nat :=
-	match n with
-		| O => O
-		| S n' => n'
-	end.
-
-Definition minustwo (n : nat) : nat :=
-	match n with
-		| O => O
-		| S O => O
-		| S (S n') => n'
-	end.
-
-Fixpoint evenb (n : nat) : bool :=
-	match n with
-		| O => true
-		| S O => false
-		| S (S n') => evenb n'
-	end.
-
-Definition oddb (n : nat) : bool := negb (evenb n).
-
-Example test_oddb1: (oddb (S O)) = true.
-Proof. reflexivity. Qed.
-Example test_oddb2: (oddb (S (S (S (S O))))) = false.
-Proof. reflexivity. Qed.
-
-Fixpoint plus (n : nat) (m : nat) : nat :=
-	match n with
-		| O => m
-		| S n' => S (plus n' m)
-	end.
-
-Fixpoint mult (n m : nat) : nat :=
-	match n with
-		| O => O
-		| S n' => plus m (mult n' m)
-	end.
-
-Fixpoint minus (n m : nat) : nat :=
-	match n, m with
-		| O, _ => n
-		| S n', O => S n'
-		| S n', S m' => minus n' m'
-	end.
-
-Fixpoint exp (base power : nat) : nat :=
-	match power with
-		| O => S O
-		| S p => mult base (exp base p)
-	end.
-
-Fixpoint factorial (n : nat) : nat :=
-	match n with
-		| O => S O
-		| S n' => mult n (factorial n')
-	end.
-
-Example test_factorial1: (factorial (S (S (S O)))) = (S (S (S (S (S (S O)))))).
-Proof. simpl. reflexivity. Qed.
-
-Notation "x + y" := (plus x y) (at level 50, left associativity) : nat_scope.
-Notation "x - y" := (minus x y) (at level 50, left associativity) : nat_scope.
-Notation "x * y" := (mult x y) (at level 40, left associativity) : nat_scope.
-
-Fixpoint beq_nat (n m : nat) : bool :=
-	match n with
-		| O => match m with
-						| O => true
-						| S m' => false
-					 end
-		| S n' => match m with
-							| O => false
-							| S m' => beq_nat n' m'
-							end
-	end.
-
-Fixpoint ble_nat (n m : nat) : bool :=
-	match n with
-		| O => true
-		| S n' => 
-				match m with
-					| O => false
-					| S m' => ble_nat n' m'
-				end
-	end.
-
-Example test_ble_nat1: (ble_nat (S (S O)) (S (S O))) = true.
-Proof. simpl. reflexivity. Qed.
-Example test_ble_nat2: (ble_nat (S (S O)) (S (S (S (S O))))) = true.
-Proof. simpl. reflexivity. Qed.
-Example test_ble_nat3: (ble_nat (S (S (S (S O)))) (S (S O))) = false.
-Proof. simpl. reflexivity. Qed.
-
-Definition blt_nat (n m : nat) : bool :=
-		(andb (negb (beq_nat n m)) (ble_nat n m)).
-
-Example test_blt_nat1: (blt_nat (S (S O)) (S (S O))) = false.
-Proof. simpl. reflexivity. Qed.
-Example test_blt_nat3: (blt_nat (S (S (S (S O)))) (S (S O))) = false.
-Proof. simpl. reflexivity. Qed.
-Example test_blt_nat2 : (blt_nat (S (S O)) (S (S (S (S O))))) = true.
-Proof. simpl. reflexivity. Qed.
-
-Theorem plus_O_n : forall n : nat, O + n = n.
-Proof.
-	simpl. reflexivity. Qed.
-
-Theorem plus_O_n' : forall n : nat, O + n = n.
-Proof.
-	reflexivity. Qed.
-
-Theorem plus_O_n'' : forall n : nat, O + n = n.
-Proof.
-	intros n. reflexivity. Qed.
-
-Theorem plus_1_1 : forall n : nat, (S O) + n = S n.
-Proof.
-	intros n. reflexivity. Qed.
-
-Theorem mult_0_1: forall n : nat, O * n = O.
-Proof.
-	intros n. reflexivity. Qed.
-
-Theorem plus_id_example : forall n m:nat,
-	n = m -> n + n = m + m.
-Proof.
-	intros n m.
-	intros H.
-	rewrite -> H.
-	reflexivity. Qed.
-
-Theorem plus_id_exercise : forall n m o: nat,
-	n = m -> m = o -> n + m = m + o.
-Proof.
-	intros n m o.
-	intros H.
-	intros H'.
-	rewrite -> H.
-	rewrite <- H'.
-	reflexivity.
-	Qed.
-
-Theorem mult_0_plus : forall n m : nat,
-				(O + n) * m = n * m.
-Proof.
-	intros n m.
-	rewrite -> plus_O_n.
-	reflexivity. Qed.
-
-Theorem mult_1_plus : forall n m: nat,
-	((S O) + n) * m = m + (n * m).
-Proof.
-	intros n m.
-	rewrite -> plus_1_1.
-	reflexivity.
-	Qed.
-
-Theorem mult_1 : forall n : nat,
-				n * (S O) = n.
-Proof.
-	intros n.
-	induction n as [| n'].
-	reflexivity.
-	simpl.
-	rewrite -> IHn'.
-	reflexivity.
-	Qed.
-
-Theorem plus_1_neq_0 : forall n : nat,
-				beq_nat (n + (S O)) O = false.
-Proof.
-	intros n.
-	destruct n as [| n'].
-	reflexivity.
-	reflexivity.
-	Qed.
-
-Theorem zero_nbeq_plus_1 : forall n : nat,
-				beq_nat O (n + (S O)) = false.
-Proof.
-	intros n.
-	destruct n.
-	reflexivity.
-	reflexivity.
-Qed.
-
-Require String. Open Scope string_scope.
-
-Ltac move_to_top x :=
-match reverse goal with
-| H : _ |- _ => try move x after H
-end.
-
-Tactic Notation "assert_eq" ident(x) constr(v) :=
-	let H := fresh in
-	assert (x = v) as H by reflexivity;
-	clear H.
-
-	Tactic Notation "Case_aux" ident(x) constr(name) :=
-		first [
-		set (x := name); move_to_top x
-		| assert_eq x name; move_to_top x
-		| fail 1 "because we are working on a different case" ].
-
-		Ltac Case name := Case_aux Case name.
-		Ltac SCase name := Case_aux SCase name.
-		Ltac SSCase name := Case_aux SSCase name.
-		Ltac SSSCase name := Case_aux SSSCase name.
-		Ltac SSSSCase name := Case_aux SSSSCase name.
-		Ltac SSSSSCase name := Case_aux SSSSSCase name.
-		Ltac SSSSSSCase name := Case_aux SSSSSSCase name.
-		Ltac SSSSSSSCase name := Case_aux SSSSSSSCase name.
-
-Theorem andb_true_elim1 : forall b c : bool,
-				andb b c = true -> b = true.
-Proof.
-	intros b c H.
-	destruct b.
-	Case "b = true".
-		reflexivity.
-	Case "b = false".
-		rewrite <- H. reflexivity. Qed.
-
-Theorem plus_0_r : forall n : nat, n + O = n.
-Proof.
-	intros n. induction n as [| n'].
-	Case "n = 0". reflexivity.
-	Case "n = S n'". simpl. rewrite -> IHn'. reflexivity. Qed.
-
-Theorem minus_diag : forall n,
-				minus n n = O.
-Proof.
-	intros n. induction n as [| n'].
-	Case "n = 0".
-		simpl. reflexivity.
-	Case "n = S n'".
-		simpl. rewrite -> IHn'. reflexivity. Qed.
-
-
-Theorem mult_0_r : forall n:nat,
-				n * O = O.
-Proof.
-	intros n. induction n as [| n'].
-	Case "n = 0".
-		reflexivity.
-	Case "n = S n'".
-		simpl. rewrite -> IHn'. reflexivity. Qed.
-
-Theorem plus_n_Sm : forall n m : nat,
-				S (n + m) = n + (S m).
-Proof.
-	intros n m. induction n as [| n'].
-	Case "n = 0".
-		reflexivity.
-	Case "n = S n'".
-		simpl. rewrite -> IHn'. reflexivity. Qed.
-
-Theorem plus_assoc : forall n m p : nat,
-					n + (m + p) = (n + m) + p.
-Proof.
-	intros n m p.
-	induction n as [| n'].
-	reflexivity.
-	simpl.
-	rewrite -> IHn'.
-	reflexivity. Qed.
-
-Theorem plus_distr : forall n m: nat, S (n + m) = n + (S m).
-Proof.
-	intros n m.  induction n as [| n'].
-	Case "n = 0".
-		reflexivity.
-	Case "n = S n'".
-		simpl. rewrite -> IHn'. reflexivity. Qed.
-
-Theorem mult_distr : forall n m: nat, n * ((S O) + m) = n * (S m).
-Proof.
-	intros n m.
-	induction n as [| n'].
-	reflexivity.
-	reflexivity.
-	Qed.
-
-Theorem plus_comm : forall n m : nat,
-	n + m = m + n.
-Proof.
-	intros n m.
-	induction n as [| n'].
-	Case "n = 0".
-		simpl.
-		rewrite -> plus_0_r.
-		reflexivity.
-	Case "n = S n'".
-		simpl.
-		rewrite -> IHn'.
-		rewrite -> plus_distr.
-		reflexivity. Qed.
-
-Fixpoint double (n:nat) :=
-	match n with
-		| O => O
-		| S n' => S (S (double n'))
-	end.
-
-Lemma double_plus : forall n, double n = n + n.
-Proof.
-	intros n. induction n as [| n'].
-	Case "n = 0".
-		reflexivity.
-	Case "n = S n'".
-		simpl. rewrite -> IHn'.
-		rewrite -> plus_distr. reflexivity.
-		Qed.
-
-Theorem beq_nat_refl : forall n : nat,
-	true = beq_nat n n.
-Proof.
-	intros n. induction n as [| n'].
-	Case "n = 0".
-		reflexivity.
-	Case "n = S n".
-		simpl. rewrite <- IHn'.
-		reflexivity. Qed.
-
-Theorem plus_rearrange: forall n m p q : nat,
-				(n + m) + (p + q) = (m + n) + (p + q).
-Proof.
-	intros n m p q.
-	assert(H: n + m = m + n).
-		Case "Proof by assertion".
-		rewrite -> plus_comm. reflexivity.
-	rewrite -> H. reflexivity. Qed.
-
-Theorem plus_swap : forall n m p: nat,
-				n + (m + p) = m + (n + p).
-Proof.
-	intros n m p.
-	rewrite -> plus_assoc.
-	assert(H: m + (n + p) = (m + n) + p).
-	rewrite -> plus_assoc.
-	reflexivity.
-	rewrite -> H.
-	assert(H2: m + n = n + m).
-	rewrite -> plus_comm.
-	reflexivity.
-	rewrite -> H2.
-	reflexivity.
-	Qed.
-
-Theorem plus_swap' : forall n m p: nat,
-				n + (m + p) = m + (n + p).
-Proof.
-	intros n m p.
-	rewrite -> plus_assoc.
-	assert(H: m + (n + p) = (m + n) + p).
-	rewrite -> plus_assoc.
-	reflexivity.
-	rewrite -> H.
-	replace (m + n) with (n + m).
-	rewrite -> plus_comm.
-	reflexivity.
-	rewrite -> plus_comm.
-	reflexivity.
-	Qed.
-
-Theorem mult_1_distr: forall m n: nat,
-				n * ((S O) + m) = n * (S O) + n * m.
-Proof.
-	intros n m.
-	rewrite -> mult_1.
-	rewrite -> plus_1_1.
-	simpl.
-	induction m as [|m'].
-	simpl.
-	reflexivity.
-	simpl.
-	rewrite -> plus_swap.
-	rewrite <- IHm'.
-	reflexivity.
-	Qed.
-
-Theorem mult_comm: forall m n : nat,
-				m * n = n * m.
-Proof.
-	intros m n.
-	induction n as [| n'].
-	Case "n = 0".
-		simpl.
-		rewrite -> mult_0_r.
-		reflexivity.
-	Case "n = S n'".
-		simpl.
-		rewrite <- mult_distr.
-		rewrite -> mult_1_distr.
-		rewrite -> mult_1.
-		rewrite -> IHn'.
-		reflexivity.
-		Qed.
-
-Theorem evenb_next : forall n : nat,
-				evenb n = evenb (S (S n)).
-Proof.
-	intros n.
-Admitted.
-
-Theorem negb_negb : forall n : bool,
-				n = negb (negb n).
-Proof.
-	intros n.
-	destruct n.
-	reflexivity.
-	reflexivity.
-	Qed.
-
-Theorem evenb_n_oddb_Sn : forall n : nat,
-				evenb n = negb (evenb (S n)).
-Proof.
-	intros n.
-	induction n as [|n'].
-	reflexivity.
-	assert(H: evenb n' = evenb (S (S n'))).
-	reflexivity.
-	rewrite <- H.
-	rewrite -> IHn'.
-	rewrite <- negb_negb.
-	reflexivity.
-	Qed.
-
-(*Fixpoint bad (n : nat) : bool :=
-	match n with
-		| O => true
-		| S O => bad (S n)
-		| S (S n') => bad n'
-	end.*)
-
-Theorem ble_nat_refl : forall n:nat,
-				true = ble_nat n n.
-Proof.
-	intros n.
-	induction n as [|n'].
-	Case "n = 0".
-		reflexivity.
-	Case "n = S n".
-		simpl.
-		rewrite <- IHn'.
-		reflexivity.
-	Qed.
-
-Theorem zero_nbeq_S : forall n: nat,
-				beq_nat O (S n) = false.
-Proof.
-	intros n.
-	reflexivity.
-	Qed.
-
-Theorem andb_false_r : forall b : bool,
-				andb b false = false.
-Proof.
-	intros b.
-	destruct b.
-	reflexivity.
-	reflexivity.
-	Qed.
-
-Theorem plus_ble_compat_1 : forall n m p : nat,
-				ble_nat n m = true -> ble_nat (p + n) (p + m) = true.
-Proof.
-	intros n m p.
-	intros H.
-	induction p.
-	Case "p = 0".
-		simpl.
-		rewrite -> H.
-		reflexivity.
-	Case "p = S p'".
-		simpl.
-		rewrite -> IHp.
-		reflexivity.
-		Qed.
-
-Theorem S_nbeq_0 : forall n:nat,
-				beq_nat (S n) O = false.
-Proof.
-	intros n.
-	reflexivity.
-	Qed.
-
-Theorem mult_1_1 : forall n:nat, (S O) * n = n.
-Proof.
-	intros n.
-	simpl.
-	rewrite -> plus_0_r.
-	reflexivity. Qed.
-
-Theorem all3_spec : forall b c : bool,
-	orb (andb b c)
-			(orb (negb b)
-			 		 (negb c))
-	= true.
-Proof.
-	intros b c.
-	destruct b.
-	destruct c.
-	reflexivity.
-	reflexivity.
-	reflexivity.
-	Qed.
-
-Lemma mult_plus_1 : forall n m : nat,
-			S(m + n) = m + (S n).
-Proof.
-	intros n m.
-	induction m.
-	reflexivity.
-	simpl.
-	rewrite -> IHm.
-	reflexivity.
-	Qed.
-
-Theorem mult_mult : forall n m : nat,
-	n * (S m) = n * m + n.
-Proof.
-	intros n m.
-	induction n.
-	reflexivity.
-	simpl.
-	rewrite -> IHn.
-	rewrite -> plus_assoc.
-	rewrite -> mult_plus_1.
-	reflexivity.
-	Qed.
-
-Theorem mult_plus_distr_r : forall n m p:nat,
-				(n + m) * p = (n * p) + (m * p).
-Proof.
-	intros n m p.
-	induction p.
-	rewrite -> mult_0_r.
-	rewrite -> mult_0_r.
-	rewrite -> mult_0_r.
-	reflexivity.
-	rewrite -> mult_mult.
-	rewrite -> mult_mult.
-	rewrite -> mult_mult.
-	rewrite -> IHp.
-	assert(H1: ((n * p) + n) + (m * p + m) = (n * p) + (n + (m * p + m))).
-	rewrite <- plus_assoc.
-	reflexivity.
-	rewrite -> H1.
-	assert(H2: (n + (m * p + m)) = (m * p + (n + m))).
-	rewrite -> plus_swap.
-	reflexivity.
-	rewrite -> H2.
-	assert(H3: (n * p) + (m * p + (n + m)) = ((n * p ) + (m * p)) + (n + m)).
-	rewrite -> plus_assoc.
-	reflexivity.
-	rewrite -> H3.
-	reflexivity.
-	Qed.
-
-Theorem mult_assoc : forall n m p : nat,
-				n * (m * p) = (n * m) * p.
-Proof.
-	intros n m p.
-	induction n.
-	simpl.
-	reflexivity.
-	simpl.
-	rewrite -> mult_plus_distr_r.
-	rewrite -> IHn.
-	reflexivity.
-	Qed.
-
-Inductive bin : Type :=
-	| BO : bin
-	| D : bin -> bin
-	| M : bin -> bin.
-
-Fixpoint incbin (n : bin) : bin :=
-	match n with
-		| BO => M (BO)
-		| D n' => M n'
-		| M n' => D (incbin n')
-	end.
-
-Fixpoint bin2un (n : bin) : nat :=
-	match n with
-		| BO => O
-		| D n' => double (bin2un n')
-		| M n' => S (double (bin2un n'))
-	end.
-
-Theorem bin_comm : forall n : bin,
-				bin2un(incbin n) = S (bin2un n).
-Proof.
-	intros n.
-	induction n.
-		reflexivity.
-		reflexivity.
-		simpl.
-		rewrite -> IHn.
-		reflexivity.
-	Qed.
-
-End Playground1.

samples/Coq/Computation.v

@@ -0,0 +1,85 @@
+(** The definition of computations, used to represent interactive programs. *)
+Require Import Coq.NArith.NArith.
+Require Import ListString.All.
+
+Local Open Scope type.
+
+(** System calls. *)
+Module Command.
+  Inductive t :=
+  | AskCard
+  | AskPIN
+  | CheckPIN (pin : N)
+  | AskAmount
+  | CheckAmount (amount : N)
+  | GiveCard
+  | GiveAmount (amount : N)
+  | ShowError (message : LString.t).
+
+  (** The type of an answer for a command depends on the value of the command. *)
+  Definition answer (command : t) : Type :=
+    match command with
+    | AskCard => bool (* If the given card seems valid. *)
+    | AskPIN => option N (* A number or cancellation. *)
+    | CheckPIN _ => bool (* If the PIN number is valid. *)
+    | AskAmount => option N (* A number or cancellation. *)
+    | CheckAmount _ => bool (* If the amount can be withdrawn. *)
+    | GiveCard => bool (* If the card was given. *)
+    | GiveAmount _ => bool (* If the money was given. *)
+    | ShowError _ => unit (* Show an error message. *)
+    end.
+End Command.
+
+(** Computations with I/Os. *)
+Module C.
+  (** A computation can either does nothing, or do a system call and wait
+      for the answer to run another computation. *)
+  Inductive t : Type :=
+  | Ret : t
+  | Call : forall (command : Command.t), (Command.answer command -> t) -> t.
+  Arguments Ret.
+  Arguments Call _ _.
+
+  (** Some optional notations. *)
+  Module Notations.
+    (** A nicer notation for `Ret`. *)
+    Definition ret : t :=
+      Ret.
+
+    (** We define an explicit apply function so that Coq does not try to expand
+        the notations everywhere. *)
+    Definition apply {A B} (f : A -> B) (x : A) := f x.
+
+    (** System call. *)
+    Notation "'call!' answer ':=' command 'in' X" :=
+      (Call command (fun answer => X))
+      (at level 200, answer ident, command at level 100, X at level 200).
+
+    (** System call with typed answer. *)
+    Notation "'call!' answer : A ':=' command 'in' X" :=
+      (Call command (fun (answer : A) => X))
+      (at level 200, answer ident, command at level 100, A at level 200, X at level 200).
+
+    (** System call ignoring the answer. *)
+    Notation "'do_call!' command 'in' X" :=
+      (Call command (fun _ => X))
+      (at level 200, command at level 100, X at level 200).
+
+    (** This notation is useful to compose computations which wait for a
+        continuation. We do not have an explicit bind operator to simplify the
+        language and the proofs. *)
+    Notation "'let!' x ':=' X 'in' Y" :=
+      (apply X (fun x => Y))
+      (at level 200, x ident, X at level 100, Y at level 200).
+
+    (** Let with a typed answer. *)
+    Notation "'let!' x : A ':=' X 'in' Y" :=
+      (apply X (fun (x : A) => Y))
+      (at level 200, x ident, X at level 100, A at level 200, Y at level 200).
+
+    (** Let ignoring the answer. *)
+    Notation "'do!' X 'in' Y" :=
+      (apply X (fun _ => Y))
+      (at level 200, X at level 100, Y at level 200).
+  End Notations.
+End C.

samples/Coq/Heap.v

@@ -1,290 +0,0 @@
-(** A development of Treesort on Heap trees. It has an average
-    complexity of O(n.log n) but of O(n²) in the worst case (e.g. if
-    the list is already sorted) *)
-
-(* G. Huet 1-9-95 uses Multiset *)
-
-Require Import List Multiset PermutSetoid Relations Sorting.
-
-Section defs.
-
-  (** * Trees and heap trees *)
-
-  (** ** Definition of trees over an ordered set *)
-
-  Variable A : Type.
-  Variable leA : relation A.
-  Variable eqA : relation A.
-
-  Let gtA (x y:A) := ~ leA x y.
-
-  Hypothesis leA_dec : forall x y:A, {leA x y} + {leA y x}.
-  Hypothesis eqA_dec : forall x y:A, {eqA x y} + {~ eqA x y}.
-  Hypothesis leA_refl : forall x y:A, eqA x y -> leA x y.
-  Hypothesis leA_trans : forall x y z:A, leA x y -> leA y z -> leA x z.
-  Hypothesis leA_antisym : forall x y:A, leA x y -> leA y x -> eqA x y.
-
-  Hint Resolve leA_refl.
-  Hint Immediate eqA_dec leA_dec leA_antisym.
-
-  Let emptyBag := EmptyBag A.
-  Let singletonBag := SingletonBag _ eqA_dec.
-
-  Inductive Tree :=
-    | Tree_Leaf : Tree
-    | Tree_Node : A -> Tree -> Tree -> Tree.
-
-  (** [a] is lower than a Tree [T] if [T] is a Leaf
-      or [T] is a Node holding [b>a] *)
-
-  Definition leA_Tree (a:A) (t:Tree) :=
-    match t with
-      | Tree_Leaf => True
-      | Tree_Node b T1 T2 => leA a b
-    end.
-
-  Lemma leA_Tree_Leaf : forall a:A, leA_Tree a Tree_Leaf.
-  Proof.
-    simpl; auto with datatypes.
-  Qed.
-
-  Lemma leA_Tree_Node :
-    forall (a b:A) (G D:Tree), leA a b -> leA_Tree a (Tree_Node b G D).
-  Proof.
-    simpl; auto with datatypes.
-  Qed.
-
-
-  (** ** The heap property *)
-
-  Inductive is_heap : Tree -> Prop :=
-    | nil_is_heap : is_heap Tree_Leaf
-    | node_is_heap :
-      forall (a:A) (T1 T2:Tree),
-        leA_Tree a T1 ->
-        leA_Tree a T2 ->
-        is_heap T1 -> is_heap T2 -> is_heap (Tree_Node a T1 T2).
-
-  Lemma invert_heap :
-    forall (a:A) (T1 T2:Tree),
-      is_heap (Tree_Node a T1 T2) ->
-      leA_Tree a T1 /\ leA_Tree a T2 /\ is_heap T1 /\ is_heap T2.
-  Proof.
-    intros; inversion H; auto with datatypes.
-  Qed.
-
-  (* This lemma ought to be generated automatically by the Inversion tools *)
-  Lemma is_heap_rect :
-    forall P:Tree -> Type,
-      P Tree_Leaf ->
-      (forall (a:A) (T1 T2:Tree),
-        leA_Tree a T1 ->
-        leA_Tree a T2 ->
-        is_heap T1 -> P T1 -> is_heap T2 -> P T2 -> P (Tree_Node a T1 T2)) ->
-      forall T:Tree, is_heap T -> P T.
-  Proof.
-    simple induction T; auto with datatypes.
-    intros a G PG D PD PN.
-    elim (invert_heap a G D); auto with datatypes.
-    intros H1 H2; elim H2; intros H3 H4; elim H4; intros.
-    apply X0; auto with datatypes.
-  Qed.
-
-  (* This lemma ought to be generated automatically by the Inversion tools *)
-  Lemma is_heap_rec :
-    forall P:Tree -> Set,
-      P Tree_Leaf ->
-      (forall (a:A) (T1 T2:Tree),
-        leA_Tree a T1 ->
-        leA_Tree a T2 ->
-        is_heap T1 -> P T1 -> is_heap T2 -> P T2 -> P (Tree_Node a T1 T2)) ->
-      forall T:Tree, is_heap T -> P T.
-  Proof.
-    simple induction T; auto with datatypes.
-    intros a G PG D PD PN.
-    elim (invert_heap a G D); auto with datatypes.
-    intros H1 H2; elim H2; intros H3 H4; elim H4; intros.
-    apply X; auto with datatypes.
-  Qed.
-
-  Lemma low_trans :
-    forall (T:Tree) (a b:A), leA a b -> leA_Tree b T -> leA_Tree a T.
-  Proof.
-    simple induction T; auto with datatypes.
-    intros; simpl; apply leA_trans with b; auto with datatypes.
-  Qed.
-
-  (** ** Merging two sorted lists *)
-
-  Inductive merge_lem (l1 l2:list A) : Type :=
-    merge_exist :
-    forall l:list A,
-      Sorted leA l ->
-      meq (list_contents _ eqA_dec l)
-      (munion (list_contents _ eqA_dec l1) (list_contents _ eqA_dec l2)) ->
-      (forall a, HdRel leA a l1 -> HdRel leA a l2 -> HdRel leA a l) ->
-      merge_lem l1 l2.
-  Require Import Morphisms.
-
-  Instance: Equivalence (@meq A).
-  Proof. constructor; auto with datatypes. red. apply meq_trans. Defined.
-
-  Instance: Proper (@meq A ++> @meq _ ++> @meq _) (@munion A).
-  Proof. intros x y H x' y' H'. now apply meq_congr. Qed.
-
-  Lemma merge :
-    forall l1:list A, Sorted leA l1 ->
-    forall l2:list A, Sorted leA l2 -> merge_lem l1 l2.
-  Proof.
-    fix 1; intros; destruct l1.
-    apply merge_exist with l2; auto with datatypes.
-    rename l1 into l.
-    revert l2 H0. fix 1. intros.
-    destruct l2 as [|a0 l0].
-    apply merge_exist with (a :: l); simpl; auto with datatypes.
-    elim (leA_dec a a0); intros.
-
-    (* 1 (leA a a0) *)
-    apply Sorted_inv in H. destruct H.
-    destruct (merge l H (a0 :: l0) H0).
-    apply merge_exist with (a :: l1). clear merge merge0.
-      auto using cons_sort, cons_leA with datatypes.
-    simpl. rewrite m. now rewrite munion_ass.
-    intros. apply cons_leA.
-    apply (@HdRel_inv _ leA) with l; trivial with datatypes.
-
-    (* 2 (leA a0 a) *)
-    apply Sorted_inv in H0. destruct H0.
-    destruct (merge0 l0 H0). clear merge merge0.
-    apply merge_exist with (a0 :: l1);
-      auto using cons_sort, cons_leA with datatypes.
-    simpl; rewrite m. simpl. setoid_rewrite munion_ass at 1. rewrite munion_comm.
-    repeat rewrite munion_ass. setoid_rewrite munion_comm at 3. reflexivity.
-    intros. apply cons_leA.
-    apply (@HdRel_inv _ leA) with l0; trivial with datatypes.
-  Qed.
-
-  (** ** From trees to multisets *)
-
-  (** contents of a tree as a multiset *)
-
-  (** Nota Bene : In what follows the definition of SingletonBag
-      in not used. Actually, we could just take as postulate:
-      [Parameter SingletonBag : A->multiset].  *)
-
-  Fixpoint contents (t:Tree) : multiset A :=
-    match t with
-      | Tree_Leaf => emptyBag
-      | Tree_Node a t1 t2 =>
-        munion (contents t1) (munion (contents t2) (singletonBag a))
-    end.
-
-
-  (** equivalence of two trees is equality of corresponding multisets *)
-  Definition equiv_Tree (t1 t2:Tree) := meq (contents t1) (contents t2).
-
-
-
-  (** * From lists to sorted lists *)
-
-  (** ** Specification of heap insertion *)
-
-  Inductive insert_spec (a:A) (T:Tree) : Type :=
-    insert_exist :
-    forall T1:Tree,
-      is_heap T1 ->
-      meq (contents T1) (munion (contents T) (singletonBag a)) ->
-      (forall b:A, leA b a -> leA_Tree b T -> leA_Tree b T1) ->
-      insert_spec a T.
-
-
-  Lemma insert : forall T:Tree, is_heap T -> forall a:A, insert_spec a T.
-  Proof.
-    simple induction 1; intros.
-    apply insert_exist with (Tree_Node a Tree_Leaf Tree_Leaf);
-      auto using node_is_heap, nil_is_heap, leA_Tree_Leaf with datatypes.
-    simpl; unfold meq, munion; auto using node_is_heap with datatypes.
-    elim (leA_dec a a0); intros.
-    elim (X a0); intros.
-    apply insert_exist with (Tree_Node a T2 T0);
-      auto using node_is_heap, nil_is_heap, leA_Tree_Leaf with datatypes.
-    simpl; apply treesort_twist1; trivial with datatypes.
-    elim (X a); intros T3 HeapT3 ConT3 LeA.
-    apply insert_exist with (Tree_Node a0 T2 T3);
-      auto using node_is_heap, nil_is_heap, leA_Tree_Leaf with datatypes.
-    apply node_is_heap; auto using node_is_heap, nil_is_heap, leA_Tree_Leaf with datatypes.
-    apply low_trans with a; auto with datatypes.
-    apply LeA; auto with datatypes.
-    apply low_trans with a; auto with datatypes.
-    simpl; apply treesort_twist2; trivial with datatypes.
-  Qed.
-
-
-  (** ** Building a heap from a list *)
-
-  Inductive build_heap (l:list A) : Type :=
-    heap_exist :
-    forall T:Tree,
-      is_heap T ->
-      meq (list_contents _ eqA_dec l) (contents T) -> build_heap l.
-
-  Lemma list_to_heap : forall l:list A, build_heap l.
-  Proof.
-    simple induction l.
-    apply (heap_exist nil Tree_Leaf); auto with datatypes.
-    simpl; unfold meq; exact nil_is_heap.
-    simple induction 1.
-    intros T i m; elim (insert T i a).
-    intros; apply heap_exist with T1; simpl; auto with datatypes.
-    apply meq_trans with (munion (contents T) (singletonBag a)).
-    apply meq_trans with (munion (singletonBag a) (contents T)).
-    apply meq_right; trivial with datatypes.
-    apply munion_comm.
-    apply meq_sym; trivial with datatypes.
-  Qed.
-
-
-  (** ** Building the sorted list *)
-
-  Inductive flat_spec (T:Tree) : Type :=
-    flat_exist :
-    forall l:list A,
-      Sorted leA l ->
-      (forall a:A, leA_Tree a T -> HdRel leA a l) ->
-      meq (contents T) (list_contents _ eqA_dec l) -> flat_spec T.
-
-  Lemma heap_to_list : forall T:Tree, is_heap T -> flat_spec T.
-  Proof.
-    intros T h; elim h; intros.
-    apply flat_exist with (nil (A:=A)); auto with datatypes.
-    elim X; intros l1 s1 i1 m1; elim X0; intros l2 s2 i2 m2.
-    elim (merge _ s1 _ s2); intros.
-    apply flat_exist with (a :: l); simpl; auto with datatypes.
-    apply meq_trans with
-      (munion (list_contents _ eqA_dec l1)
-        (munion (list_contents _ eqA_dec l2) (singletonBag a))).
-    apply meq_congr; auto with datatypes.
-    apply meq_trans with
-      (munion (singletonBag a)
-        (munion (list_contents _ eqA_dec l1) (list_contents _ eqA_dec l2))).
-    apply munion_rotate.
-    apply meq_right; apply meq_sym; trivial with datatypes.
-  Qed.
-
-
-  (** * Specification of treesort *)
-
-  Theorem treesort :
-    forall l:list A,
-    {m : list A | Sorted leA m & permutation _ eqA_dec l m}.
-  Proof.
-    intro l; unfold permutation.
-    elim (list_to_heap l).
-    intros.
-    elim (heap_to_list T); auto with datatypes.
-    intros.
-    exists l0; auto with datatypes.
-    apply meq_trans with (contents T); trivial with datatypes.
-  Qed.
-
-End defs.

samples/Coq/JsInterpreterExtraction.v

@@ -0,0 +1,249 @@
+Set Implicit Arguments.
+Require Import JsSyntax JsInterpreterMonads JsInterpreter JsInit.
+Require Import LibFix LibList.
+
+Require Export Shared.
+Require Export LibTactics LibLogic LibReflect LibList
+  LibOperation LibStruct LibNat LibEpsilon LibFunc LibHeap.
+Require Flocq.Appli.Fappli_IEEE Flocq.Appli.Fappli_IEEE_bits.
+
+
+
+(* Here stands some commands to extract relatively correctly the interpreter to Ocaml. *)
+Extraction Language Ocaml.
+
+Require Import ExtrOcamlBasic.
+Require Import ExtrOcamlNatInt.
+Require Import ExtrOcamlString.
+
+(* Optimal fixpoint. *)
+Extraction Inline FixFun3 FixFun3Mod FixFun4 FixFun4Mod FixFunMod curry3 uncurry3 curry4 uncurry4.
+(* As classical logic statements are now unused, they should not be extracted
+   (otherwise, useless errors will be launched). *)
+Extraction Inline epsilon epsilon_def classicT arbitrary indefinite_description Inhab_witness Fix isTrue.
+
+(**************************************************************)
+(** ** Numerical values *)
+
+(* number *)
+
+Extract Inductive positive => float
+[ "(fun p -> 1. +. (2. *. p))"
+  "(fun p -> 2. *. p)"
+  "1." ]
+"(fun f2p1 f2p f1 p ->
+if p <= 1. then f1 () else if mod_float p 2. = 0. then f2p (floor (p /. 2.)) else f2p1 (floor (p /. 2.)))".
+
+Extract Inductive Z => float [ "0." "" "(~-.)" ]
+"(fun f0 fp fn z -> if z=0. then f0 () else if z>0. then fp z else fn (~-. z))".
+
+Extract Inductive N => float [ "0." "" ]
+"(fun f0 fp n -> if n=0. then f0 () else fp n)".
+
+Extract Constant Z.add => "(+.)".
+Extract Constant Z.succ => "(+.) 1.".
+Extract Constant Z.pred => "(fun x -> x -. 1.)".
+Extract Constant Z.sub => "(-.)".
+Extract Constant Z.mul => "( *. )".
+Extract Constant Z.opp => "(~-.)".
+Extract Constant Z.abs => "abs_float".
+Extract Constant Z.min => "min".
+Extract Constant Z.max => "max".
+Extract Constant Z.compare =>
+ "fun x y -> if x=y then Eq else if x<y then Lt else Gt".
+
+Extract Constant Pos.add => "(+.)".
+Extract Constant Pos.succ => "(+.) 1.".
+Extract Constant Pos.pred => "(fun x -> x -. 1.)".
+Extract Constant Pos.sub => "(-.)".
+Extract Constant Pos.mul => "( *. )".
+Extract Constant Pos.min => "min".
+Extract Constant Pos.max => "max".
+Extract Constant Pos.compare =>
+ "fun x y -> if x=y then Eq else if x<y then Lt else Gt".
+Extract Constant Pos.compare_cont =>
+ "fun x y c -> if x=y then c else if x<y then Lt else Gt".
+
+Extract Constant N.add => "(+.)".
+Extract Constant N.succ => "(+.) 1.".
+Extract Constant N.pred => "(fun x -> x -. 1.)".
+Extract Constant N.sub => "(-.)".
+Extract Constant N.mul => "( *. )".
+Extract Constant N.min => "min".
+Extract Constant N.max => "max".
+Extract Constant N.div => "(fun x y -> if x = 0. then 0. else floor (x /. y))".
+Extract Constant N.modulo => "mod_float".
+Extract Constant N.compare =>
+ "fun x y -> if x=y then Eq else if x<y then Lt else Gt".
+
+Extract Inductive Fappli_IEEE.binary_float => float [
+  "(fun s -> if s then (0.) else (-0.))"
+  "(fun s -> if s then infinity else neg_infinity)"
+  "nan"
+  "(fun (s, m, e) -> failwith ""FIXME: No extraction from binary float allowed yet."")"
+]. 
+
+Extract Constant JsNumber.of_int => "fun x -> x".
+
+Extract Constant JsNumber.nan => "nan".
+Extract Constant JsNumber.zero => "0.".
+Extract Constant JsNumber.neg_zero => "(-0.)".
+Extract Constant JsNumber.one => "1.".
+Extract Constant JsNumber.infinity => "infinity".
+Extract Constant JsNumber.neg_infinity => "neg_infinity".
+Extract Constant JsNumber.max_value => "max_float".
+Extract Constant JsNumber.min_value => "(Int64.float_of_bits Int64.one)".
+Extract Constant JsNumber.pi => "(4. *. atan 1.)".
+Extract Constant JsNumber.e => "(exp 1.)".
+Extract Constant JsNumber.ln2 => "(log 2.)".
+Extract Constant JsNumber.floor => "floor".
+Extract Constant JsNumber.absolute => "abs_float".
+
+Extract Constant JsNumber.from_string =>
+  "(fun s ->
+    try
+      let s = (String.concat """" (List.map (String.make 1) s)) in
+      if s = """" then 0. else float_of_string s
+    with Failure ""float_of_string"" -> nan)
+   (* Note that we're using `float_of_string' there, which does not have the same
+      behavior than JavaScript.  For instance it will read ""022"" as 22 instead of
+      18, which should be the JavaScript result for it. *)".
+
+Extract Constant JsNumber.to_string =>
+  "(fun f -> 
+    prerr_string (""Warning:  JsNumber.to_string called.  This might be responsible for errors.  Argument value:  "" ^ string_of_float f ^ ""."");
+    prerr_newline();
+    let string_of_number n =
+      let sfn = string_of_float n in
+      (if (sfn = ""inf"") then ""Infinity"" else
+       if (sfn = ""-inf"") then ""-Infinity"" else
+       if (sfn = ""nan"") then ""NaN"" else
+       let inum = int_of_float n in
+       if (float_of_int inum = n) then (string_of_int inum) else (string_of_float n)) in
+    let ret = ref [] in (* Ugly, but the API for OCaml string is not very functional... *)
+    String.iter (fun c -> ret := c :: !ret) (string_of_number f);
+    List.rev !ret)
+   (* Note that this is ugly, we should use the spec of JsNumber.to_string here (9.8.1). *)".
+
+Extract Constant JsNumber.add => "(+.)".
+Extract Constant JsNumber.sub => "(-.)".
+Extract Constant JsNumber.mult => "( *. )".
+Extract Constant JsNumber.div => "(/.)".
+Extract Constant JsNumber.fmod => "mod_float".
+Extract Constant JsNumber.neg => "(~-.)".
+Extract Constant JsNumber.sign => "(fun f -> float_of_int (compare f 0.))".
+Extract Constant JsNumber.number_comparable => "(fun n1 n2 -> 0 = compare n1 n2)".
+Extract Constant JsNumber.lt_bool => "(<)".
+
+Extract Constant JsNumber.to_int32 => 
+"fun n ->
+  match classify_float n with
+  | FP_normal | FP_subnormal ->
+    let i32 = 2. ** 32. in
+    let i31 = 2. ** 31. in
+    let posint = (if n < 0. then (-1.) else 1.) *. (floor (abs_float n)) in
+    let int32bit =
+      let smod = mod_float posint i32 in
+      if smod < 0. then smod +. i32 else smod
+    in
+    (if int32bit >= i31 then int32bit -. i32 else int32bit)
+  | _ -> 0.". (* LATER:  do in Coq.  Spec is 9.5, p. 47.*)
+
+Extract Constant JsNumber.to_uint32 =>
+"fun n ->
+  match classify_float n with
+  | FP_normal | FP_subnormal ->
+    let i32 = 2. ** 32. in
+    let posint = (if n < 0. then (-1.) else 1.) *. (floor (abs_float n)) in
+    let int32bit =
+      let smod = mod_float posint i32 in
+      if smod < 0. then smod +. i32 else smod
+    in
+    int32bit
+  | _ -> 0.". (* LAER:  do in Coq.  Spec is 9.6, p47.*)
+
+Extract Constant JsNumber.modulo_32 => "(fun x -> let r = mod_float x 32. in if x < 0. then r +. 32. else r)".
+Extract Constant JsNumber.int32_bitwise_not => "fun x -> Int32.to_float (Int32.lognot (Int32.of_float x))".
+Extract Constant JsNumber.int32_bitwise_and => "fun x y -> Int32.to_float (Int32.logand (Int32.of_float x) (Int32.of_float y))".
+Extract Constant JsNumber.int32_bitwise_or => "fun x y -> Int32.to_float (Int32.logor (Int32.of_float x) (Int32.of_float y))".
+Extract Constant JsNumber.int32_bitwise_xor => "fun x y -> Int32.to_float (Int32.logxor (Int32.of_float x) (Int32.of_float y))".
+Extract Constant JsNumber.int32_left_shift => "(fun x y -> Int32.to_float (Int32.shift_left (Int32.of_float x) (int_of_float y)))".
+Extract Constant JsNumber.int32_right_shift => "(fun x y -> Int32.to_float (Int32.shift_right (Int32.of_float x) (int_of_float y)))".
+Extract Constant JsNumber.uint32_right_shift => 
+"(fun x y ->
+  let i31 = 2. ** 31. in
+  let i32 = 2. ** 32. in
+  let newx = if x >= i31 then x -. i32 else x in
+  let r = Int32.to_float (Int32.shift_right_logical (Int32.of_float newx) (int_of_float y)) in
+  if r < 0. then r +. i32 else r)".
+
+Extract Constant int_of_char => "(fun c -> float_of_int (int_of_char c))".
+
+Extract Constant ascii_comparable => "(=)".
+Extract Constant lt_int_decidable => "(<)".
+Extract Constant le_int_decidable => "(<=)".
+Extract Constant ge_nat_decidable => "(>=)".
+
+(* TODO ARTHUR:  This TLC lemma does not extract to something computable... whereas it should! *)
+Extract Constant prop_eq_decidable => "(=)".
+
+Extract Constant env_loc_global_env_record => "0".
+
+(* The following functions make pattern matches with floats and shall thus be removed. *)
+Extraction Inline Fappli_IEEE.Bplus Fappli_IEEE.binary_normalize Fappli_IEEE_bits.b64_plus.
+Extraction Inline Fappli_IEEE.Bmult Fappli_IEEE.Bmult_FF Fappli_IEEE_bits.b64_mult.
+Extraction Inline Fappli_IEEE.Bdiv Fappli_IEEE_bits.b64_div.
+
+(* New options for the interpreter to work in Coq 8.4 *)
+Set Extraction AccessOpaque.
+
+(* These parameters are implementation-dependant according to the spec.
+   I've chosed some very simple values, but we could choose another thing for them. *)
+Extract Constant object_prealloc_global_proto => "(Coq_value_prim Coq_prim_null)".
+Extract Constant object_prealloc_global_class => "(
+  let rec aux s = function
+  | 0 -> []
+  | n -> let n' = n - 1 in
+    s.[n'] :: aux s n'
+  in let aux2 s =
+    List.rev (aux s (String.length s))
+  in aux2 ""GlobalClass"")".
+
+
+(* Parsing *)
+Extract Constant parse_pickable => "(fun s strict ->
+    let str = String.concat """" (List.map (String.make 1) s) in
+    try
+      let parserExp = Parser_main.exp_from_string ~force_strict:strict str in
+      Some (JsSyntaxInfos.add_infos_prog strict
+        (Translate_syntax.exp_to_prog parserExp))
+    with
+    (* | Translate_syntax.CoqSyntaxDoesNotSupport _ -> assert false (* Temporary *) *)
+    | Parser.ParserFailure _
+    | Parser.InvalidArgument ->
+      prerr_string (""Warning:  Parser error on eval.  Input string:  \"""" ^ str ^ ""\""\n"");
+      None
+  )".
+
+
+(* Debugging *)
+Extract Inlined Constant not_yet_implemented_because => "(fun s ->
+  print_endline (__LOC__ ^ "": Not implemented because: "" ^ Prheap.string_of_char_list s) ;
+  Coq_result_not_yet_implemented)".
+Extract Inlined Constant impossible_because => "(fun s ->
+  print_endline (__LOC__ ^ "": Stuck because: "" ^ Prheap.string_of_char_list s) ;
+  Coq_result_impossible)".
+Extract Inlined Constant impossible_with_heap_because => "(fun s message ->
+  print_endline (__LOC__ ^ "": Stuck!\nState:  "" ^ Prheap.prstate true s
+    ^ ""\nMessage:\t"" ^ Prheap.string_of_char_list message) ;
+  Coq_result_impossible)".
+
+
+(* Final Extraction *)
+Extraction Blacklist string list bool.
+Separate Extraction runs run_javascript.
+
+
+
+(* -- LATER: extract inequality_test_string in more efficient way*)
+

samples/Coq/JsNumber.v

@@ -0,0 +1,107 @@
+Set Implicit Arguments.
+Require Export Shared.
+Require Flocq.Appli.Fappli_IEEE Flocq.Appli.Fappli_IEEE_bits.
+
+
+(**************************************************************)
+(** ** Type for number (IEEE floats) *)
+
+Definition number : Type :=
+  Fappli_IEEE_bits.binary64.
+
+
+(**************************************************************)
+(** ** Particular values of numbers *)
+
+(* LATER: find definitions in Flocq *)
+Parameter nan : number.
+Parameter zero : number.
+Parameter neg_zero : number.
+Definition one := Fappli_IEEE.binary_normalize 53 1024 eq_refl eq_refl Fappli_IEEE.mode_NE 1 0 false.
+Parameter infinity : number.
+Parameter neg_infinity : number.
+Parameter max_value : number.
+Parameter min_value : number.
+Parameter pi : number.
+Parameter e : number.
+Parameter ln2 : number.
+
+(**************************************************************)
+(** ** Conversions on numbers *)
+
+(* LATER: implement definitions *)
+Parameter from_string : string -> number.
+Parameter to_string : number -> string.
+
+(**************************************************************)
+(** ** Unary operations on numbers *)
+
+(* LATER: find definitions in Flocq *)
+
+Parameter neg : number -> number.
+Parameter floor : number -> number.
+Parameter absolute : number -> number.
+Parameter sign : number -> number. (* returns arbitrary when x is zero or nan *)
+Parameter lt_bool : number -> number -> bool.
+
+
+(**************************************************************)
+(** ** Binary operations on numbers *)
+
+Definition add : number -> number -> number :=
+  Fappli_IEEE_bits.b64_plus Fappli_IEEE.mode_NE.
+
+Parameter sub : number -> number -> number. (* todo: bind *)
+
+Parameter fmod : number -> number -> number. (* todo: bind *)
+
+Definition mult : number -> number -> number :=
+  Fappli_IEEE_bits.b64_mult Fappli_IEEE.mode_NE.
+
+Definition div : number -> number -> number :=
+  Fappli_IEEE_bits.b64_div Fappli_IEEE.mode_NE.
+
+(* Todo: find comparison operator *)
+Global Instance number_comparable : Comparable number.
+Proof. Admitted.
+
+
+
+(**************************************************************)
+(** ** Conversions with Int32 *)
+
+Parameter of_int : int -> number. (* LATER: this is quite complex. Should we make it precise? *)
+
+Parameter to_int32 : number -> int. (* Remark: extracted code could, for efficiency reasons, use Ocaml Int32 *) 
+
+Parameter to_uint32 : number -> int.
+
+Parameter to_int16 : number -> int. (* currently not used *)
+
+(* LATER: Check that the OCaml extraction is correct. *)
+
+
+(**************************************************************)
+
+(** Implements the operation that masks all but the 5 least significant bits
+   of a non-negative number (obtained as the result of to_uint32 *)
+
+Parameter modulo_32 : int -> int.
+
+(** Implements int32 operation *)
+
+Parameter int32_bitwise_not : int -> int.
+
+Parameter int32_bitwise_and : int -> int -> int.
+Parameter int32_bitwise_or : int -> int -> int.
+Parameter int32_bitwise_xor : int -> int -> int.
+
+Parameter int32_left_shift : int -> int -> int.
+Parameter int32_right_shift : int -> int -> int.
+Parameter uint32_right_shift : int -> int -> int.
+
+
+
+
+(**************************************************************)
+(** ** Int32 related conversion *)

samples/Coq/Main.v

@@ -0,0 +1,42 @@
+Require Import FunctionNinjas.All.
+Require Import ListString.All.
+Require Import Computation.
+
+Import C.Notations.
+
+Definition error (message : LString.t) : C.t :=
+  do_call! Command.ShowError message in
+  ret.
+
+Definition main : C.t :=
+  call! card_is_valid := Command.AskCard in
+  if card_is_valid then
+    call! pin := Command.AskPIN in
+    match pin with
+    | None => error @@ LString.s "No PIN given."
+    | Some pin =>
+      call! pin_is_valid := Command.CheckPIN pin in
+      if pin_is_valid then
+        call! ask_amount := Command.AskAmount in
+        match ask_amount with
+        | None => error @@ LString.s "No amount given."
+        | Some amount =>
+          call! amount_is_valid := Command.CheckAmount amount in
+          if amount_is_valid then
+            call! card_is_given := Command.GiveCard in
+            if card_is_given then
+              call! amount_is_given := Command.GiveAmount amount in
+              if amount_is_given then
+                ret
+              else
+                error @@ LString.s "Cannot give you the amount. Please contact your bank."
+            else
+              error @@ LString.s "Cannot give you back the card. Please contact your bank."
+          else
+            error @@ LString.s "Invalid amount."
+        end
+      else
+        error @@ LString.s "Invalid PIN."
+    end
+  else
+    error @@ LString.s "Invalid card.".

samples/Coq/PermutSetoid.v

@@ -1,539 +0,0 @@
-Require Import Omega Relations Multiset SetoidList.
-
-(** This file is deprecated, use [Permutation.v] instead.
-
-    Indeed, this file defines a notion of permutation based on
-    multisets (there exists a permutation between two lists iff every
-    elements have the same multiplicity in the two lists) which
-    requires a more complex apparatus (the equipment of the domain
-    with a decidable equality) than [Permutation] in [Permutation.v].
-
-    The relation between the two relations are in lemma
-    [permutation_Permutation].
-
-    File [Permutation] concerns Leibniz equality : it shows in particular
-    that [List.Permutation] and [permutation] are equivalent in this context.
-*)
-
-Set Implicit Arguments.
-
-Local Notation "[ ]" := nil.
-Local Notation "[ a ; .. ; b ]" := (a :: .. (b :: []) ..).
-
-Section Permut.
-
-(** * From lists to multisets *)
-
-Variable A : Type.
-Variable eqA : relation A.
-Hypothesis eqA_equiv : Equivalence eqA.
-Hypothesis eqA_dec : forall x y:A, {eqA x y} + {~ eqA x y}.
-
-Let emptyBag := EmptyBag A.
-Let singletonBag := SingletonBag _ eqA_dec.
-
-(** contents of a list *)
-
-Fixpoint list_contents (l:list A) : multiset A :=
-  match l with
-  | [] => emptyBag
-  | a :: l => munion (singletonBag a) (list_contents l)
-  end.
-
-Lemma list_contents_app :
-  forall l m:list A,
-    meq (list_contents (l ++ m)) (munion (list_contents l) (list_contents m)).
-Proof.
-  simple induction l; simpl; auto with datatypes.
-  intros.
-  apply meq_trans with
-    (munion (singletonBag a) (munion (list_contents l0) (list_contents m)));
-    auto with datatypes.
-Qed.
-
-(** * [permutation]: definition and basic properties *)
-
-Definition permutation (l m:list A) := meq (list_contents l) (list_contents m).
-
-Lemma permut_refl : forall l:list A, permutation l l.
-Proof.
-  unfold permutation; auto with datatypes.
-Qed.
-
-Lemma permut_sym :
-  forall l1 l2 : list A, permutation l1 l2 -> permutation l2 l1.
-Proof.
-  unfold permutation, meq; intros; symmetry; trivial.
-Qed.
-
-Lemma permut_trans :
-  forall l m n:list A, permutation l m -> permutation m n -> permutation l n.
-Proof.
-  unfold permutation; intros.
-  apply meq_trans with (list_contents m); auto with datatypes.
-Qed.
-
-Lemma permut_cons_eq :
-  forall l m:list A,
-    permutation l m -> forall a a', eqA a a' -> permutation (a :: l) (a' :: m).
-Proof.
-  unfold permutation; simpl; intros.
-  apply meq_trans with (munion (singletonBag a') (list_contents l)).
-  apply meq_left, meq_singleton; auto.
-  auto with datatypes.
-Qed.
-
-Lemma permut_cons :
-  forall l m:list A,
-    permutation l m -> forall a:A, permutation (a :: l) (a :: m).
-Proof.
-  unfold permutation; simpl; auto with datatypes.
-Qed.
-
-Lemma permut_app :
-  forall l l' m m':list A,
-    permutation l l' -> permutation m m' -> permutation (l ++ m) (l' ++ m').
-Proof.
-  unfold permutation; intros.
-  apply meq_trans with (munion (list_contents l) (list_contents m));
-    auto using permut_cons, list_contents_app with datatypes.
-  apply meq_trans with (munion (list_contents l') (list_contents m'));
-    auto using permut_cons, list_contents_app with datatypes.
-  apply meq_trans with (munion (list_contents l') (list_contents m));
-    auto using permut_cons, list_contents_app with datatypes.
-Qed.
-
-Lemma permut_add_inside_eq :
-  forall a a' l1 l2 l3 l4, eqA a a' ->
-    permutation (l1 ++ l2) (l3 ++ l4) ->
-    permutation (l1 ++ a :: l2) (l3 ++ a' :: l4).
-Proof.
-  unfold permutation, meq in *; intros.
-  specialize H0 with a0.
-  repeat rewrite list_contents_app in *; simpl in *.
-  destruct (eqA_dec a a0) as [Ha|Ha]; rewrite H in Ha;
-    decide (eqA_dec a' a0) with Ha; simpl; auto with arith.
-  do 2 rewrite <- plus_n_Sm; f_equal; auto.
-Qed.
-
-Lemma permut_add_inside :
-  forall a l1 l2 l3 l4,
-    permutation (l1 ++ l2) (l3 ++ l4) ->
-    permutation (l1 ++ a :: l2) (l3 ++ a :: l4).
-Proof.
-  unfold permutation, meq in *; intros.
-  generalize (H a0); clear H.
-  do 4 rewrite list_contents_app.
-  simpl.
-  destruct (eqA_dec a a0); simpl; auto with arith.
-  do 2 rewrite <- plus_n_Sm; f_equal; auto.
-Qed.
-
-Lemma permut_add_cons_inside_eq :
-  forall a a' l l1 l2, eqA a a' ->
-    permutation l (l1 ++ l2) ->
-    permutation (a :: l) (l1 ++ a' :: l2).
-Proof.
-  intros;
-  replace (a :: l) with ([] ++ a :: l); trivial;
-    apply permut_add_inside_eq; trivial.
-Qed.
-
-Lemma permut_add_cons_inside :
-  forall a l l1 l2,
-    permutation l (l1 ++ l2) ->
-    permutation (a :: l) (l1 ++ a :: l2).
-Proof.
-  intros;
-    replace (a :: l) with ([] ++ a :: l); trivial;
-        apply permut_add_inside; trivial.
-Qed.
-
-Lemma permut_middle :
-  forall (l m:list A) (a:A), permutation (a :: l ++ m) (l ++ a :: m).
-Proof.
-  intros; apply permut_add_cons_inside; auto using permut_sym, permut_refl.
-Qed.
-
-Lemma permut_sym_app :
-  forall l1 l2, permutation (l1 ++ l2) (l2 ++ l1).
-Proof.
-  intros l1 l2;
-    unfold permutation, meq;
-        intro a; do 2 rewrite list_contents_app; simpl;
-          auto with arith.
-Qed.
-
-Lemma permut_rev :
-  forall l, permutation l (rev l).
-Proof.
-  induction l.
-  simpl; trivial using permut_refl.
-  simpl.
-  apply permut_add_cons_inside.
-  rewrite <- app_nil_end. trivial.
-Qed.
-
-(** * Some inversion results. *)
-Lemma permut_conv_inv :
-  forall e l1 l2, permutation (e :: l1) (e :: l2) -> permutation l1 l2.
-Proof.
-  intros e l1 l2; unfold permutation, meq; simpl; intros H a;
-    generalize (H a); apply plus_reg_l.
-Qed.
-
-Lemma permut_app_inv1 :
-  forall l l1 l2, permutation (l1 ++ l) (l2 ++ l) -> permutation l1 l2.
-Proof.
-  intros l l1 l2; unfold permutation, meq; simpl;
-    intros H a; generalize (H a); clear H.
-  do 2 rewrite list_contents_app.
-  simpl.
-  intros; apply plus_reg_l with (multiplicity (list_contents l) a).
-  rewrite plus_comm; rewrite H; rewrite plus_comm.
-  trivial.
-Qed.
-
-(** we can use [multiplicity] to define [InA] and [NoDupA]. *)
-
-Fact if_eqA_then : forall a a' (B:Type)(b b':B),
- eqA a a' -> (if eqA_dec a a' then b else b') = b.
-Proof.
-  intros. destruct eqA_dec as [_|NEQ]; auto.
-  contradict NEQ; auto.
-Qed.
-
-Lemma permut_app_inv2 :
-  forall l l1 l2, permutation (l ++ l1) (l ++ l2) -> permutation l1 l2.
-Proof.
-  intros l l1 l2; unfold permutation, meq; simpl;
-    intros H a; generalize (H a); clear H.
-  do 2 rewrite list_contents_app.
-  simpl.
-  intros; apply plus_reg_l with (multiplicity (list_contents l) a).
-  trivial.
-Qed.
-
-Lemma permut_remove_hd_eq :
-  forall l l1 l2 a b, eqA a b ->
-    permutation (a :: l) (l1 ++ b :: l2) -> permutation l (l1 ++ l2).
-Proof.
-  unfold permutation, meq; simpl; intros l l1 l2 a b Heq H a0.
-  specialize H with a0.
-  rewrite list_contents_app in *; simpl in *.
-  apply plus_reg_l with (if eqA_dec a a0 then 1 else 0).
-  rewrite H; clear H.
-  symmetry; rewrite plus_comm, <- ! plus_assoc; f_equal.
-  rewrite plus_comm.
-  destruct (eqA_dec a a0) as [Ha|Ha]; rewrite Heq in Ha;
-    decide (eqA_dec b a0) with Ha; reflexivity.
-Qed.
-
-Lemma permut_remove_hd :
-  forall l l1 l2 a,
-    permutation (a :: l) (l1 ++ a :: l2) -> permutation l (l1 ++ l2).
-Proof.
-  eauto using permut_remove_hd_eq, Equivalence_Reflexive.
-Qed.
-
-Fact if_eqA_else : forall a a' (B:Type)(b b':B),
- ~eqA a a' -> (if eqA_dec a a' then b else b') = b'.
-Proof.
-  intros. decide (eqA_dec a a') with H; auto.
-Qed.
-
-Fact if_eqA_refl : forall a (B:Type)(b b':B),
- (if eqA_dec a a then b else b') = b.
-Proof.
-  intros; apply (decide_left (eqA_dec a a)); auto with *.
-Qed.
-
-(** PL: Inutilisable dans un rewrite sans un change prealable. *)
-
-Global Instance if_eqA (B:Type)(b b':B) :
- Proper (eqA==>eqA==>@eq _) (fun x y => if eqA_dec x y then b else b').
-Proof.
- intros x x' Hxx' y y' Hyy'.
- intros; destruct (eqA_dec x y) as [H|H];
-  destruct (eqA_dec x' y') as [H'|H']; auto.
- contradict H'; transitivity x; auto with *; transitivity y; auto with *.
- contradict H; transitivity x'; auto with *; transitivity y'; auto with *.
-Qed.
-
-Fact if_eqA_rewrite_l : forall a1 a1' a2 (B:Type)(b b':B),
- eqA a1 a1' -> (if eqA_dec a1 a2 then b else b') =
-               (if eqA_dec a1' a2 then b else b').
-Proof.
- intros; destruct (eqA_dec a1 a2) as [A1|A1];
-  destruct (eqA_dec a1' a2) as [A1'|A1']; auto.
- contradict A1'; transitivity a1; eauto with *.
- contradict A1; transitivity a1'; eauto with *.
-Qed.
-
-Fact if_eqA_rewrite_r : forall a1 a2 a2' (B:Type)(b b':B),
- eqA a2 a2' -> (if eqA_dec a1 a2 then b else b') =
-               (if eqA_dec a1 a2' then b else b').
-Proof.
- intros; destruct (eqA_dec a1 a2) as [A2|A2];
-  destruct (eqA_dec a1 a2') as [A2'|A2']; auto.
- contradict A2'; transitivity a2; eauto with *.
- contradict A2; transitivity a2'; eauto with *.
-Qed.
-
-
-Global Instance multiplicity_eqA (l:list A) :
- Proper (eqA==>@eq _) (multiplicity (list_contents l)).
-Proof.
-  intros x x' Hxx'.
-  induction l as [|y l Hl]; simpl; auto.
-  rewrite (@if_eqA_rewrite_r y x x'); auto.
-Qed.
-
-Lemma multiplicity_InA :
-  forall l a, InA eqA a l <-> 0 < multiplicity (list_contents l) a.
-Proof.
-  induction l.
-  simpl.
-  split; inversion 1.
-  simpl.
-  intros a'; split; intros H. inversion_clear H.
-  apply (decide_left (eqA_dec a a')); auto with *.
-  destruct (eqA_dec a a'); auto with *. simpl; rewrite <- IHl; auto.
-  destruct (eqA_dec a a'); auto with *. right. rewrite IHl; auto.
-Qed.
-
-Lemma multiplicity_InA_O :
-  forall l a, ~ InA eqA a l -> multiplicity (list_contents l) a = 0.
-Proof.
-  intros l a; rewrite multiplicity_InA;
-    destruct (multiplicity (list_contents l) a); auto with arith.
-  destruct 1; auto with arith.
-Qed.
-
-Lemma multiplicity_InA_S :
-  forall l a, InA eqA a l -> multiplicity (list_contents l) a >= 1.
-Proof.
-  intros l a; rewrite multiplicity_InA; auto with arith.
-Qed.
-
-Lemma multiplicity_NoDupA : forall l,
-  NoDupA eqA l <-> (forall a, multiplicity (list_contents l) a <= 1).
-Proof.
-  induction l.
-  simpl.
-  split; auto with arith.
-  split; simpl.
-  inversion_clear 1.
-  rewrite IHl in H1.
-  intros; destruct (eqA_dec a a0) as [EQ|NEQ]; simpl; auto with *.
-  rewrite <- EQ.
-  rewrite multiplicity_InA_O; auto.
-  intros; constructor.
-  rewrite multiplicity_InA.
-  specialize (H a).
-  rewrite if_eqA_refl in H.
-  clear IHl; omega.
-  rewrite IHl; intros.
-  specialize (H a0). omega.
-Qed.
-
-(** Permutation is compatible with InA. *)
-Lemma permut_InA_InA :
-  forall l1 l2 e, permutation l1 l2 -> InA eqA e l1 -> InA eqA e l2.
-Proof.
-  intros l1 l2 e.
-  do 2 rewrite multiplicity_InA.
-  unfold permutation, meq.
-  intros H;rewrite H; auto.
-Qed.
-
-Lemma permut_cons_InA :
-  forall l1 l2 e, permutation (e :: l1) l2 -> InA eqA e l2.
-Proof.
-  intros; apply (permut_InA_InA (e:=e) H); auto with *.
-Qed.
-
-(** Permutation of an empty list. *)
-Lemma permut_nil :
-  forall l, permutation l [] -> l = [].
-Proof.
-  intro l; destruct l as [ | e l ]; trivial.
-  assert (InA eqA e (e::l)) by (auto with *).
-  intro Abs; generalize (permut_InA_InA Abs H).
-  inversion 1.
-Qed.
-
-(** Permutation for short lists. *)
-
-Lemma permut_length_1:
-  forall a b, permutation [a] [b] -> eqA a b.
-Proof.
-  intros a b; unfold permutation, meq.
-  intro P; specialize (P b); simpl in *.
-  rewrite if_eqA_refl in *.
-  destruct (eqA_dec a b); simpl; auto; discriminate.
-Qed.
-
-Lemma permut_length_2 :
-  forall a1 b1 a2 b2, permutation [a1; b1] [a2; b2] ->
-    (eqA a1 a2) /\ (eqA b1 b2) \/ (eqA a1 b2) /\ (eqA a2 b1).
-Proof.
-  intros a1 b1 a2 b2 P.
-  assert (H:=permut_cons_InA P).
-  inversion_clear H.
-  left; split; auto.
-  apply permut_length_1.
-  red; red; intros.
-  specialize (P a). simpl in *.
-  rewrite (@if_eqA_rewrite_l a1 a2 a) in P by auto. omega.
-  right.
-  inversion_clear H0; [|inversion H].
-  split; auto.
-  apply permut_length_1.
-  red; red; intros.
-  specialize (P a); simpl in *.
-  rewrite (@if_eqA_rewrite_l a1 b2 a) in P by auto. omega.
-Qed.
-
-(** Permutation is compatible with length. *)
-Lemma permut_length :
-  forall l1 l2, permutation l1 l2 -> length l1 = length l2.
-Proof.
-  induction l1; intros l2 H.
-  rewrite (permut_nil (permut_sym H)); auto.
-  assert (H0:=permut_cons_InA H).
-  destruct (InA_split H0) as (h2,(b,(t2,(H1,H2)))).
-  subst l2.
-  rewrite app_length.
-  simpl; rewrite <- plus_n_Sm; f_equal.
-  rewrite <- app_length.
-  apply IHl1.
-  apply permut_remove_hd with b.
-  apply permut_trans with (a::l1); auto.
-  revert H1; unfold permutation, meq; simpl.
-  intros; f_equal; auto.
-  rewrite (@if_eqA_rewrite_l a b a0); auto.
-Qed.
-
-Lemma NoDupA_equivlistA_permut :
-  forall l l', NoDupA eqA l -> NoDupA eqA l' ->
-    equivlistA eqA l l' -> permutation l l'.
-Proof.
-  intros.
-  red; unfold meq; intros.
-  rewrite multiplicity_NoDupA in H, H0.
-  generalize (H a) (H0 a) (H1 a); clear H H0 H1.
-  do 2 rewrite multiplicity_InA.
-  destruct 3; omega.
-Qed.
-
-End Permut.
-
-Section Permut_map.
-
-Variables A B : Type.
-
-Variable eqA : relation A.
-Hypothesis eqA_dec : forall x y:A, {eqA x y} + {~ eqA x y}.
-Hypothesis eqA_equiv : Equivalence eqA.
-
-Variable eqB : B->B->Prop.
-Hypothesis eqB_dec : forall x y:B, { eqB x y }+{ ~eqB x y }.
-Hypothesis eqB_trans : Transitive eqB.
-
-(** Permutation is compatible with map. *)
-
-Lemma permut_map :
-  forall f,
-    (Proper (eqA==>eqB) f) ->
-    forall l1 l2, permutation _ eqA_dec l1 l2 ->
-      permutation _ eqB_dec (map f l1) (map f l2).
-Proof.
-  intros f; induction l1.
-  intros l2 P; rewrite (permut_nil eqA_equiv (permut_sym P)); apply permut_refl.
-  intros l2 P.
-  simpl.
-  assert (H0:=permut_cons_InA eqA_equiv P).
-  destruct (InA_split H0) as (h2,(b,(t2,(H1,H2)))).
-  subst l2.
-  rewrite map_app.
-  simpl.
-  apply permut_trans with (f b :: map f l1).
-  revert H1; unfold permutation, meq; simpl.
-  intros; f_equal; auto.
-  destruct (eqB_dec (f b) a0) as [H2|H2];
-    destruct (eqB_dec (f a) a0) as [H3|H3]; auto.
-  destruct H3; transitivity (f b); auto with *.
-  destruct H2; transitivity (f a); auto with *.
-  apply permut_add_cons_inside.
-  rewrite <- map_app.
-  apply IHl1; auto.
-  apply permut_remove_hd with b; trivial.
-  apply permut_trans with (a::l1); auto.
-  revert H1; unfold permutation, meq; simpl.
-  intros; f_equal; auto.
-  rewrite (@if_eqA_rewrite_l _ _ eqA_equiv eqA_dec a b a0); auto.
-Qed.
-
-End Permut_map.
-
-Require Import Permutation.
-
-Section Permut_permut.
-
-Variable A : Type.
-
-Variable eqA : relation A.
-Hypothesis eqA_dec : forall x y:A, {eqA x y} + {~ eqA x y}.
-Hypothesis eqA_equiv : Equivalence eqA.
-
-Lemma Permutation_impl_permutation : forall l l',
-  Permutation l l' -> permutation _ eqA_dec l l'.
-Proof.
-  induction 1.
-    apply permut_refl.
-    apply permut_cons; auto using Equivalence_Reflexive.
-    change (x :: y :: l) with ([x] ++ y :: l);
-      apply permut_add_cons_inside; simpl;
-      apply permut_cons_eq; auto using Equivalence_Reflexive, permut_refl.
-    apply permut_trans with l'; trivial.
-Qed.
-
-Lemma permut_eqA : forall l l', Forall2 eqA l l' -> permutation _ eqA_dec l l'.
-Proof.
-  induction 1.
-    apply permut_refl.
-    apply permut_cons_eq; trivial.
-Qed.
-
-Lemma permutation_Permutation : forall l l',
-  permutation _ eqA_dec l l' <->
-  exists l'', Permutation l l'' /\ Forall2 eqA l'' l'.
-Proof.
-  split; intro H.
-  (* -> *)
-  induction l in l', H |- *.
-    exists []; apply permut_sym, permut_nil in H as ->; auto using Forall2.
-    pose proof H as H'.
-    apply permut_cons_InA, InA_split in H
-      as (l1 & y & l2 & Heq & ->); trivial.
-    apply permut_remove_hd_eq, IHl in H'
-      as (l'' & IHP & IHA); clear IHl; trivial.
-    apply Forall2_app_inv_r in IHA as (l1'' & l2'' & Hl1 & Hl2 & ->).
-    exists (l1'' ++ a :: l2''); split.
-      apply Permutation_cons_app; trivial.
-      apply Forall2_app, Forall2_cons; trivial.
-  (* <- *)
-  destruct H as (l'' & H & Heq).
-  apply permut_trans with l''.
-    apply Permutation_impl_permutation; trivial.
-    apply permut_eqA; trivial.
-Qed.
-
-End Permut_permut.
-
-(* begin hide *)
-(** For compatibilty *)
-Notation permut_right := permut_cons (only parsing).
-Notation permut_tran := permut_trans (only parsing).
-(* end hide *)

samples/Coq/Permutation.v

@@ -1,632 +0,0 @@
-(* Adapted in May 2006 by Jean-Marc Notin from initial contents by
-   Laurent Thery (Huffmann contribution, October 2003) *)
-
-Require Import List Setoid Compare_dec Morphisms.
-Import ListNotations. (* For notations [] and [a;b;c] *)
-Set Implicit Arguments.
-
-Section Permutation.
-
-Variable A:Type.
-
-Inductive Permutation : list A -> list A -> Prop :=
-| perm_nil: Permutation [] []
-| perm_skip x l l' : Permutation l l' -> Permutation (x::l) (x::l')
-| perm_swap x y l : Permutation (y::x::l) (x::y::l)
-| perm_trans l l' l'' :
-    Permutation l l' -> Permutation l' l'' -> Permutation l l''.
-
-Local Hint Constructors Permutation.
-
-(** Some facts about [Permutation] *)
-
-Theorem Permutation_nil : forall (l : list A), Permutation [] l -> l = [].
-Proof.
-  intros l HF.
-  remember (@nil A) as m in HF.
-  induction HF; discriminate || auto.
-Qed.
-
-Theorem Permutation_nil_cons : forall (l : list A) (x : A),
- ~ Permutation nil (x::l).
-Proof.
-  intros l x HF.
-  apply Permutation_nil in HF; discriminate.
-Qed.
-
-(** Permutation over lists is a equivalence relation *)
-
-Theorem Permutation_refl : forall l : list A, Permutation l l.
-Proof.
-  induction l; constructor. exact IHl.
-Qed.
-
-Theorem Permutation_sym : forall l l' : list A,
- Permutation l l' -> Permutation l' l.
-Proof.
-  intros l l' Hperm; induction Hperm; auto.
-  apply perm_trans with (l':=l'); assumption.
-Qed.
-
-Theorem Permutation_trans : forall l l' l'' : list A,
- Permutation l l' -> Permutation l' l'' -> Permutation l l''.
-Proof.
-  exact perm_trans.
-Qed.
-
-End Permutation.
-
-Hint Resolve Permutation_refl perm_nil perm_skip.
-
-(* These hints do not reduce the size of the problem to solve and they
-   must be used with care to avoid combinatoric explosions *)
-
-Local Hint Resolve perm_swap perm_trans.
-Local Hint Resolve Permutation_sym Permutation_trans.
-
-(* This provides reflexivity, symmetry and transitivity and rewriting
-   on morphims to come *)
-
-Instance Permutation_Equivalence A : Equivalence (@Permutation A) | 10 := {
-  Equivalence_Reflexive := @Permutation_refl A ;
-  Equivalence_Symmetric := @Permutation_sym A ;
-  Equivalence_Transitive := @Permutation_trans A }.
-
-Instance Permutation_cons A :
- Proper (Logic.eq ==> @Permutation A ==> @Permutation A) (@cons A) | 10.
-Proof.
-  repeat intro; subst; auto using perm_skip.
-Qed.
-
-Section Permutation_properties.
-
-Variable A:Type.
-
-Implicit Types a b : A.
-Implicit Types l m : list A.
-
-(** Compatibility with others operations on lists *)
-
-Theorem Permutation_in : forall (l l' : list A) (x : A),
- Permutation l l' -> In x l -> In x l'.
-Proof.
-  intros l l' x Hperm; induction Hperm; simpl; tauto.
-Qed.
-
-Global Instance Permutation_in' :
- Proper (Logic.eq ==> @Permutation A ==> iff) (@In A) | 10.
-Proof.
-  repeat red; intros; subst; eauto using Permutation_in.
-Qed.
-
-Lemma Permutation_app_tail : forall (l l' tl : list A),
- Permutation l l' -> Permutation (l++tl) (l'++tl).
-Proof.
-  intros l l' tl Hperm; induction Hperm as [|x l l'|x y l|l l' l'']; simpl; auto.
-  eapply Permutation_trans with (l':=l'++tl); trivial.
-Qed.
-
-Lemma Permutation_app_head : forall (l tl tl' : list A),
- Permutation tl tl' -> Permutation (l++tl) (l++tl').
-Proof.
-  intros l tl tl' Hperm; induction l;
-   [trivial | repeat rewrite <- app_comm_cons; constructor; assumption].
-Qed.
-
-Theorem Permutation_app : forall (l m l' m' : list A),
- Permutation l l' -> Permutation m m' -> Permutation (l++m) (l'++m').
-Proof.
-  intros l m l' m' Hpermll' Hpermmm';
-   induction Hpermll' as [|x l l'|x y l|l l' l''];
-    repeat rewrite <- app_comm_cons; auto.
-  apply Permutation_trans with (l' := (x :: y :: l ++ m));
-   [idtac | repeat rewrite app_comm_cons; apply Permutation_app_head]; trivial.
-  apply Permutation_trans with (l' := (l' ++ m')); try assumption.
-  apply Permutation_app_tail; assumption.
-Qed.
-
-Global Instance Permutation_app' :
- Proper (@Permutation A ==> @Permutation A ==> @Permutation A) (@app A) | 10.
-Proof.
-  repeat intro; now apply Permutation_app.
-Qed.
-
-Lemma Permutation_add_inside : forall a (l l' tl tl' : list A),
-  Permutation l l' -> Permutation tl tl' ->
-  Permutation (l ++ a :: tl) (l' ++ a :: tl').
-Proof.
-  intros; apply Permutation_app; auto.
-Qed.
-
-Lemma Permutation_cons_append : forall (l : list A) x,
-  Permutation (x :: l) (l ++ x :: nil).
-Proof. induction l; intros; auto. simpl. rewrite <- IHl; auto. Qed.
-Local Hint Resolve Permutation_cons_append.
-
-Theorem Permutation_app_comm : forall (l l' : list A),
-  Permutation (l ++ l') (l' ++ l).
-Proof.
-  induction l as [|x l]; simpl; intro l'.
-  rewrite app_nil_r; trivial. rewrite IHl.
-  rewrite app_comm_cons, Permutation_cons_append.
-  now rewrite <- app_assoc.
-Qed.
-Local Hint Resolve Permutation_app_comm.
-
-Theorem Permutation_cons_app : forall (l l1 l2:list A) a,
-  Permutation l (l1 ++ l2) -> Permutation (a :: l) (l1 ++ a :: l2).
-Proof.
-  intros l l1 l2 a H. rewrite H.
-  rewrite app_comm_cons, Permutation_cons_append.
-  now rewrite <- app_assoc.
-Qed.
-Local Hint Resolve Permutation_cons_app.
-
-Theorem Permutation_middle : forall (l1 l2:list A) a,
-  Permutation (a :: l1 ++ l2) (l1 ++ a :: l2).
-Proof.
-  auto.
-Qed.
-Local Hint Resolve Permutation_middle.
-
-Theorem Permutation_rev : forall (l : list A), Permutation l (rev l).
-Proof.
-  induction l as [| x l]; simpl; trivial. now rewrite IHl at 1.
-Qed.
-
-Global Instance Permutation_rev' :
- Proper (@Permutation A ==> @Permutation A) (@rev A) | 10.
-Proof.
-  repeat intro; now rewrite <- 2 Permutation_rev.
-Qed.
-
-Theorem Permutation_length : forall (l l' : list A),
- Permutation l l' -> length l = length l'.
-Proof.
-  intros l l' Hperm; induction Hperm; simpl; auto. now transitivity (length l').
-Qed.
-
-Global Instance Permutation_length' :
- Proper (@Permutation A ==> Logic.eq) (@length A) | 10.
-Proof.
-  exact Permutation_length.
-Qed.
-
-Theorem Permutation_ind_bis :
- forall P : list A -> list A -> Prop,
-   P [] [] ->
-   (forall x l l', Permutation l l' -> P l l' -> P (x :: l) (x :: l')) ->
-   (forall x y l l', Permutation l l' -> P l l' -> P (y :: x :: l) (x :: y :: l')) ->
-   (forall l l' l'', Permutation l l' -> P l l' -> Permutation l' l'' -> P l' l'' -> P l l'') ->
-   forall l l', Permutation l l' -> P l l'.
-Proof.
-  intros P Hnil Hskip Hswap Htrans.
-  induction 1; auto.
-  apply Htrans with (x::y::l); auto.
-  apply Hswap; auto.
-  induction l; auto.
-  apply Hskip; auto.
-  apply Hskip; auto.
-  induction l; auto.
-  eauto.
-Qed.
-
-Ltac break_list l x l' H :=
-  destruct l as [|x l']; simpl in *;
-  injection H; intros; subst; clear H.
-
-Theorem Permutation_nil_app_cons : forall (l l' : list A) (x : A),
- ~ Permutation nil (l++x::l').
-Proof.
-  intros l l' x HF.
-  apply Permutation_nil in HF. destruct l; discriminate.
-Qed.
-
-Theorem Permutation_app_inv : forall (l1 l2 l3 l4:list A) a,
-  Permutation (l1++a::l2) (l3++a::l4) -> Permutation (l1++l2) (l3 ++ l4).
-Proof.
-  intros l1 l2 l3 l4 a; revert l1 l2 l3 l4.
-  set (P l l' :=
-       forall l1 l2 l3 l4, l=l1++a::l2 -> l'=l3++a::l4 ->
-       Permutation (l1++l2) (l3++l4)).
-  cut (forall l l', Permutation l l' -> P l l').
-  intros H; intros; eapply H; eauto.
-  apply (Permutation_ind_bis P); unfold P; clear P.
-  - (* nil *)
-    intros; now destruct l1.
-  - (* skip *)
-    intros x l l' H IH; intros.
-    break_list l1 b l1' H0; break_list l3 c l3' H1.
-    auto.
-    now rewrite H.
-    now rewrite <- H.
-    now rewrite (IH _ _ _ _ eq_refl eq_refl).
-  - (* swap *)
-    intros x y l l' Hp IH; intros.
-    break_list l1 b l1' H; break_list l3 c l3' H0.
-    auto.
-    break_list l3' b l3'' H.
-    auto.
-    constructor. now rewrite Permutation_middle.
-    break_list l1' c l1'' H1.
-    auto.
-    constructor. now rewrite Permutation_middle.
-    break_list l3' d l3'' H; break_list l1' e l1'' H1.
-    auto.
-    rewrite perm_swap. constructor. now rewrite Permutation_middle.
-    rewrite perm_swap. constructor. now rewrite Permutation_middle.
-    now rewrite perm_swap, (IH _ _ _ _ eq_refl eq_refl).
-  - (*trans*)
-    intros.
-    destruct (In_split a l') as (l'1,(l'2,H6)).
-    rewrite <- H.
-    subst l.
-    apply in_or_app; right; red; auto.
-    apply perm_trans with (l'1++l'2).
-    apply (H0 _ _ _ _ H3 H6).
-    apply (H2 _ _ _ _ H6 H4).
-Qed.
-
-Theorem Permutation_cons_inv l l' a :
- Permutation (a::l) (a::l') -> Permutation l l'.
-Proof.
-  intro H; exact (Permutation_app_inv [] l [] l' a H).
-Qed.
-
-Theorem Permutation_cons_app_inv l l1 l2 a :
- Permutation (a :: l) (l1 ++ a :: l2) -> Permutation l (l1 ++ l2).
-Proof.
-  intro H; exact (Permutation_app_inv [] l l1 l2 a H).
-Qed.
-
-Theorem Permutation_app_inv_l : forall l l1 l2,
- Permutation (l ++ l1) (l ++ l2) -> Permutation l1 l2.
-Proof.
-  induction l; simpl; auto.
-  intros.
-  apply IHl.
-  apply Permutation_cons_inv with a; auto.
-Qed.
-
-Theorem Permutation_app_inv_r : forall l l1 l2,
- Permutation (l1 ++ l) (l2 ++ l) -> Permutation l1 l2.
-Proof.
-  induction l.
-  intros l1 l2; do 2 rewrite app_nil_r; auto.
-  intros.
-  apply IHl.
-  apply Permutation_app_inv with a; auto.
-Qed.
-
-Lemma Permutation_length_1_inv: forall a l, Permutation [a] l -> l = [a].
-Proof.
-  intros a l H; remember [a] as m in H.
-  induction H; try (injection Heqm as -> ->; clear Heqm);
-    discriminate || auto.
-  apply Permutation_nil in H as ->; trivial.
-Qed.
-
-Lemma Permutation_length_1: forall a b, Permutation [a] [b] -> a = b.
-Proof.
-  intros a b H.
-  apply Permutation_length_1_inv in H; injection H as ->; trivial.
-Qed.
-
-Lemma Permutation_length_2_inv :
-  forall a1 a2 l, Permutation [a1;a2] l -> l = [a1;a2] \/ l = [a2;a1].
-Proof.
-  intros a1 a2 l H; remember [a1;a2] as m in H.
-  revert a1 a2 Heqm.
-  induction H; intros; try (injection Heqm; intros; subst; clear Heqm);
-    discriminate || (try tauto).
-  apply Permutation_length_1_inv in H as ->; left; auto.
-  apply IHPermutation1 in Heqm as [H1|H1]; apply IHPermutation2 in H1 as ();
-    auto.
-Qed.
-
-Lemma Permutation_length_2 :
-  forall a1 a2 b1 b2, Permutation [a1;a2] [b1;b2] ->
-    a1 = b1 /\ a2 = b2 \/ a1 = b2 /\ a2 = b1.
-Proof.
-  intros a1 b1 a2 b2 H.
-  apply Permutation_length_2_inv in H as [H|H]; injection H as -> ->; auto.
-Qed.
-
-Let in_middle l l1 l2 (a:A) : l = l1 ++ a :: l2 ->
- forall x, In x l <-> a = x \/ In x (l1++l2).
-Proof.
- intros; subst; rewrite !in_app_iff; simpl. tauto.
-Qed.
-
-Lemma NoDup_cardinal_incl (l l' : list A) : NoDup l -> NoDup l' ->
-  length l = length l' -> incl l l' -> incl l' l.
-Proof.
- intros N. revert l'. induction N as [|a l Hal Hl IH].
- - destruct l'; now auto.
- - intros l' Hl' E H x Hx.
-   assert (Ha : In a l') by (apply H; simpl; auto).
-   destruct (in_split _ _ Ha) as (l1 & l2 & H12). clear Ha.
-   rewrite in_middle in Hx; eauto.
-   destruct Hx as [Hx|Hx]; [left|right]; auto.
-   apply (IH (l1++l2)); auto.
-   * apply NoDup_remove_1 with a; rewrite <- H12; auto.
-   * apply eq_add_S.
-     simpl in E; rewrite E, H12, !app_length; simpl; auto with arith.
-   * intros y Hy. assert (Hy' : In y l') by (apply H; simpl; auto).
-     rewrite in_middle in Hy'; eauto.
-     destruct Hy'; auto. subst y; intuition.
-Qed.
-
-Lemma NoDup_Permutation l l' : NoDup l -> NoDup l' ->
-  (forall x:A, In x l <-> In x l') -> Permutation l l'.
-Proof.
- intros N. revert l'. induction N as [|a l Hal Hl IH].
- - destruct l'; simpl; auto.
-   intros Hl' H. exfalso. rewrite (H a); auto.
- - intros l' Hl' H.
-   assert (Ha : In a l') by (apply H; simpl; auto).
-   destruct (In_split _ _ Ha) as (l1 & l2 & H12).
-   rewrite H12.
-   apply Permutation_cons_app.
-   apply IH; auto.
-   * apply NoDup_remove_1 with a; rewrite <- H12; auto.
-   * intro x. split; intros Hx.
-     + assert (Hx' : In x l') by (apply H; simpl; auto).
-       rewrite in_middle in Hx'; eauto.
-       destruct Hx'; auto. subst; intuition.
-     + assert (Hx' : In x l') by (rewrite (in_middle l1 l2 a); eauto).
-       rewrite <- H in Hx'. destruct Hx'; auto.
-       subst. destruct (NoDup_remove_2 _ _ _ Hl' Hx).
-Qed.
-
-Lemma NoDup_Permutation_bis l l' : NoDup l -> NoDup l' ->
-  length l = length l' -> incl l l' -> Permutation l l'.
-Proof.
- intros. apply NoDup_Permutation; auto.
- split; auto. apply NoDup_cardinal_incl; auto.
-Qed.
-
-Lemma Permutation_NoDup l l' : Permutation l l' -> NoDup l -> NoDup l'.
-Proof.
- induction 1; auto.
- * inversion_clear 1; constructor; eauto using Permutation_in.
- * inversion_clear 1 as [|? ? H1 H2]. inversion_clear H2; simpl in *.
-   constructor. simpl; intuition. constructor; intuition.
-Qed.
-
-Global Instance Permutation_NoDup' :
- Proper (@Permutation A ==> iff) (@NoDup A) | 10.
-Proof.
-  repeat red; eauto using Permutation_NoDup.
-Qed.
-
-End Permutation_properties.
-
-Section Permutation_map.
-
-Variable A B : Type.
-Variable f : A -> B.
-
-Lemma Permutation_map l l' :
-  Permutation l l' -> Permutation (map f l) (map f l').
-Proof.
- induction 1; simpl; eauto.
-Qed.
-
-Global Instance Permutation_map' :
-  Proper (@Permutation A ==> @Permutation B) (map f) | 10.
-Proof.
-  exact Permutation_map.
-Qed.
-
-End Permutation_map.
-
-Section Injection.
-
-Definition injective {A B} (f : A->B) :=
- forall x y, f x = f y -> x = y.
-
-Lemma injective_map_NoDup {A B} (f:A->B) (l:list A) :
- injective f -> NoDup l -> NoDup (map f l).
-Proof.
- intros Hf. induction 1 as [|x l Hx Hl IH]; simpl; constructor; trivial.
- rewrite in_map_iff. intros (y & Hy & Hy'). apply Hf in Hy. now subst.
-Qed.
-
-Lemma injective_bounded_surjective n f :
- injective f ->
- (forall x, x < n -> f x < n) ->
- (forall y, y < n -> exists x, x < n /\ f x = y).
-Proof.
- intros Hf H.
- set (l := seq 0 n).
- assert (P : incl (map f l) l).
- { intros x. rewrite in_map_iff. intros (y & <- & Hy').
-   unfold l in *. rewrite in_seq in *. simpl in *.
-   destruct Hy' as (_,Hy'). auto with arith. }
- assert (P' : incl l (map f l)).
- { unfold l.
-   apply NoDup_cardinal_incl; auto using injective_map_NoDup, seq_NoDup.
-   now rewrite map_length. }
- intros x Hx.
- assert (Hx' : In x l) by (unfold l; rewrite in_seq; auto with arith).
- apply P' in Hx'.
- rewrite in_map_iff in Hx'. destruct Hx' as (y & Hy & Hy').
- exists y; split; auto. unfold l in *; rewrite in_seq in Hy'.
- destruct Hy'; auto with arith.
-Qed.
-
-Lemma nat_bijection_Permutation n f :
- injective f -> (forall x, x < n -> f x < n) ->
- let l := seq 0 n in Permutation (map f l) l.
-Proof.
- intros Hf BD.
- apply NoDup_Permutation_bis; auto using injective_map_NoDup, seq_NoDup.
- * now rewrite map_length.
- * intros x. rewrite in_map_iff. intros (y & <- & Hy').
-   rewrite in_seq in *. simpl in *.
-   destruct Hy' as (_,Hy'). auto with arith.
-Qed.
-
-End Injection.
-
-Section Permutation_alt.
-Variable A:Type.
-Implicit Type a : A.
-Implicit Type l : list A.
-
-(** Alternative characterization of permutation
-    via [nth_error] and [nth] *)
-
-Let adapt f n :=
- let m := f (S n) in if le_lt_dec m (f 0) then m else pred m.
-
-Let adapt_injective f : injective f -> injective (adapt f).
-Proof.
- unfold adapt. intros Hf x y EQ.
- destruct le_lt_dec as [LE|LT]; destruct le_lt_dec as [LE'|LT'].
- - now apply eq_add_S, Hf.
- - apply Lt.le_lt_or_eq in LE.
-   destruct LE as [LT|EQ']; [|now apply Hf in EQ'].
-   unfold lt in LT. rewrite EQ in LT.
-   rewrite <- (Lt.S_pred _ _ LT') in LT.
-   elim (Lt.lt_not_le _ _ LT' LT).
- - apply Lt.le_lt_or_eq in LE'.
-   destruct LE' as [LT'|EQ']; [|now apply Hf in EQ'].
-   unfold lt in LT'. rewrite <- EQ in LT'.
-   rewrite <- (Lt.S_pred _ _ LT) in LT'.
-   elim (Lt.lt_not_le _ _ LT LT').
- - apply eq_add_S, Hf.
-   now rewrite (Lt.S_pred _ _ LT), (Lt.S_pred _ _ LT'), EQ.
-Qed.
-
-Let adapt_ok a l1 l2 f : injective f -> length l1 = f 0 ->
- forall n, nth_error (l1++a::l2) (f (S n)) = nth_error (l1++l2) (adapt f n).
-Proof.
- unfold adapt. intros Hf E n.
- destruct le_lt_dec as [LE|LT].
- - apply Lt.le_lt_or_eq in LE.
-   destruct LE as [LT|EQ]; [|now apply Hf in EQ].
-   rewrite <- E in LT.
-   rewrite 2 nth_error_app1; auto.
- - rewrite (Lt.S_pred _ _ LT) at 1.
-   rewrite <- E, (Lt.S_pred _ _ LT) in LT.
-   rewrite 2 nth_error_app2; auto with arith.
-   rewrite <- Minus.minus_Sn_m; auto with arith.
-Qed.
-
-Lemma Permutation_nth_error l l' :
- Permutation l l' <->
-  (length l = length l' /\
-   exists f:nat->nat,
-    injective f /\ forall n, nth_error l' n = nth_error l (f n)).
-Proof.
- split.
- { intros P.
-   split; [now apply Permutation_length|].
-   induction P.
-   - exists (fun n => n).
-     split; try red; auto.
-   - destruct IHP as (f & Hf & Hf').
-     exists (fun n => match n with O => O | S n => S (f n) end).
-     split; try red.
-     * intros [|y] [|z]; simpl; now auto.
-     * intros [|n]; simpl; auto.
-   - exists (fun n => match n with 0 => 1 | 1 => 0 | n => n end).
-     split; try red.
-     * intros [|[|z]] [|[|t]]; simpl; now auto.
-     * intros [|[|n]]; simpl; auto.
-   - destruct IHP1 as (f & Hf & Hf').
-     destruct IHP2 as (g & Hg & Hg').
-     exists (fun n => f (g n)).
-     split; try red.
-     * auto.
-     * intros n. rewrite <- Hf'; auto. }
- { revert l. induction l'.
-   - intros [|l] (E & _); now auto.
-   - intros l (E & f & Hf & Hf').
-     simpl in E.
-     assert (Ha : nth_error l (f 0) = Some a)
-      by (symmetry; apply (Hf' 0)).
-     destruct (nth_error_split l (f 0) Ha) as (l1 & l2 & L12 & L1).
-     rewrite L12. rewrite <- Permutation_middle. constructor.
-     apply IHl'; split; [|exists (adapt f); split].
-     * revert E. rewrite L12, !app_length. simpl.
-       rewrite <- plus_n_Sm. now injection 1.
-     * now apply adapt_injective.
-     * intro n. rewrite <- (adapt_ok a), <- L12; auto.
-       apply (Hf' (S n)). }
-Qed.
-
-Lemma Permutation_nth_error_bis l l' :
- Permutation l l' <->
-  exists f:nat->nat,
-    injective f /\
-    (forall n, n < length l -> f n < length l) /\
-    (forall n, nth_error l' n = nth_error l (f n)).
-Proof.
- rewrite Permutation_nth_error; split.
- - intros (E & f & Hf & Hf').
-   exists f. do 2 (split; trivial).
-   intros n Hn.
-   destruct (Lt.le_or_lt (length l) (f n)) as [LE|LT]; trivial.
-   rewrite <- nth_error_None, <- Hf', nth_error_None, <- E in LE.
-   elim (Lt.lt_not_le _ _ Hn LE).
- - intros (f & Hf & Hf2 & Hf3); split; [|exists f; auto].
-   assert (H : length l' <= length l') by auto with arith.
-   rewrite <- nth_error_None, Hf3, nth_error_None in H.
-   destruct (Lt.le_or_lt (length l) (length l')) as [LE|LT];
-    [|apply Hf2 in LT; elim (Lt.lt_not_le _ _ LT H)].
-   apply Lt.le_lt_or_eq in LE. destruct LE as [LT|EQ]; trivial.
-   rewrite <- nth_error_Some, Hf3, nth_error_Some in LT.
-   destruct (injective_bounded_surjective Hf Hf2 LT) as (y & Hy & Hy').
-   apply Hf in Hy'. subst y. elim (Lt.lt_irrefl _ Hy).
-Qed.
-
-Lemma Permutation_nth l l' d :
- Permutation l l' <->
-  (let n := length l in
-   length l' = n /\
-   exists f:nat->nat,
-    (forall x, x < n -> f x < n) /\
-    (forall x y, x < n -> y < n -> f x = f y -> x = y) /\
-    (forall x, x < n -> nth x l' d = nth (f x) l d)).
-Proof.
- split.
- - intros H.
-   assert (E := Permutation_length H).
-   split; auto.
-   apply Permutation_nth_error_bis in H.
-   destruct H as (f & Hf & Hf2 & Hf3).
-   exists f. split; [|split]; auto.
-   intros n Hn. rewrite <- 2 nth_default_eq. unfold nth_default.
-    now rewrite Hf3.
- - intros (E & f & Hf1 & Hf2 & Hf3).
-   rewrite Permutation_nth_error.
-   split; auto.
-   exists (fun n => if le_lt_dec (length l) n then n else f n).
-   split.
-   * intros x y.
-     destruct le_lt_dec as [LE|LT];
-      destruct le_lt_dec as [LE'|LT']; auto.
-     + apply Hf1 in LT'. intros ->.
-       elim (Lt.lt_irrefl (f y)). eapply Lt.lt_le_trans; eauto.
-     + apply Hf1 in LT. intros <-.
-       elim (Lt.lt_irrefl (f x)). eapply Lt.lt_le_trans; eauto.
-   * intros n.
-     destruct le_lt_dec as [LE|LT].
-     + assert (LE' : length l' <= n) by (now rewrite E).
-       rewrite <- nth_error_None in LE, LE'. congruence.
-     + assert (LT' : n < length l') by (now rewrite E).
-       specialize (Hf3 n LT). rewrite <- 2 nth_default_eq in Hf3.
-       unfold nth_default in Hf3.
-       apply Hf1 in LT.
-       rewrite <- nth_error_Some in LT, LT'.
-       do 2 destruct nth_error; congruence.
-Qed.
-
-End Permutation_alt.
-
-(* begin hide *)
-Notation Permutation_app_swap := Permutation_app_comm (only parsing).
-(* end hide *)

samples/Coq/Sorting.v

@@ -1,2 +0,0 @@
-Require Export Sorted.
-Require Export Mergesort.

samples/Coq/Spec.v

@@ -0,0 +1,62 @@
+(** Specifications. *)
+Require Import Coq.Lists.List.
+Require Import Coq.Strings.Ascii.
+Require Import FunctionNinjas.All.
+Require Import ListString.All.
+Require Import Computation.
+
+Import ListNotations.
+Local Open Scope char.
+
+(** A run is an execution of the program with explicit answers for the
+    system calls. *)
+Module Run.
+  (** We define a run by induction on the structure of a computation. *)
+  Inductive t : C.t -> Type :=
+  | Ret : t C.Ret
+  | Call : forall (command : Command.t) (answer : Command.answer command)
+    {handler : Command.answer command -> C.t}, t (handler answer) ->
+    t (C.Call command handler).
+
+  (** The trace of a run. *)
+  Fixpoint trace {x : C.t} (run : t x)
+    : list {command : Command.t & Command.answer command} :=
+    match run with
+    | Ret => []
+    | Call command answer _ run => existT _ command answer :: trace run
+    end.
+End Run.
+
+Module Temporal.
+  Module All.
+    Inductive t (P : Command.t -> Prop) : C.t -> Prop :=
+    | Ret : t P C.Ret
+    | Call : forall (c : Command.t) (h : Command.answer c -> C.t),
+      P c -> (forall a, t P (h a)) ->
+      t P (C.Call c h).
+  End All.
+
+  Module One.
+    Inductive t (P : Command.t -> Prop) : C.t -> Prop :=
+    | CallThis : forall (c : Command.t) (h : Command.answer c -> C.t),
+      P c ->
+      t P (C.Call c h)
+    | CallOther : forall (c : Command.t) (h : Command.answer c -> C.t),
+      (forall a, t P (h a)) ->
+      t P (C.Call c h).
+  End One.
+
+  Module Then.
+    Inductive t (P1 P2 : Command.t -> Prop) : C.t -> Prop :=
+    | Ret : t P1 P2 C.Ret
+    | Call : forall (c : Command.t) (h : Command.answer c -> C.t),
+      (forall a, t P1 P2 (h a)) ->
+      t P1 P2 (C.Call c h)
+    | CallThen : forall (c : Command.t) (h : Command.answer c -> C.t),
+      P1 c -> (forall a, One.t P2 (h a)) ->
+      t P1 P2 (C.Call c h).
+  End Then.
+End Temporal.
+
+Module CardBeforeMoney.
+End CardBeforeMoney.

samples/Coq/interval_discr.v

@@ -1,419 +0,0 @@
-(** Sketch of the proof of {p:nat|p<=n} = {p:nat|p<=m} -> n=m
-
-  - preliminary results on the irrelevance of boundedness proofs
-  - introduce the notion of finite cardinal |A|
-  - prove that |{p:nat|p<=n}| = n
-  - prove that |A| = n /\ |A| = m -> n = m if equality is decidable on A
-  - prove that equality is decidable on A
-  - conclude
-*)
-
-(** * Preliminary results on [nat] and [le] *)
-
-(** Proving axiom K on [nat] *)
-
-Require Import Eqdep_dec.
-Require Import Arith.
-
-Theorem eq_rect_eq_nat :
-  forall (p:nat) (Q:nat->Type) (x:Q p) (h:p=p), x = eq_rect p Q x p h.
-Proof.
-intros.
-apply K_dec_set with (p := h).
-apply eq_nat_dec.
-reflexivity.
-Qed.
-
-(** Proving unicity of proofs of [(n<=m)%nat] *)
-
-Scheme le_ind' := Induction for le Sort Prop.
-
-Theorem le_uniqueness_proof : forall (n m : nat) (p q : n <= m), p = q.
-Proof.
-induction p using le_ind'; intro q.
- replace (le_n n) with
-  (eq_rect _ (fun n0 => n <= n0) (le_n n) _ (refl_equal n)).
- 2:reflexivity.
-  generalize (refl_equal n).
-    pattern n at 2 4 6 10, q; case q; [intro | intros m l e].
-     rewrite <- eq_rect_eq_nat; trivial.
-     contradiction (le_Sn_n m); rewrite <- e; assumption.
- replace (le_S n m p) with
-  (eq_rect _ (fun n0 => n <= n0) (le_S n m p) _ (refl_equal (S m))).
- 2:reflexivity.
-  generalize (refl_equal (S m)).
-    pattern (S m) at 1 3 4 6, q; case q; [intro Heq | intros m0 l HeqS].
-     contradiction (le_Sn_n m); rewrite Heq; assumption.
-     injection HeqS; intro Heq; generalize l HeqS.
-      rewrite <- Heq; intros; rewrite <- eq_rect_eq_nat.
-      rewrite (IHp l0); reflexivity.
-Qed.
-
-(** Proving irrelevance of boundedness proofs while building
-    elements of interval *)
-
-Lemma dep_pair_intro :
-  forall (n x y:nat) (Hx : x<=n) (Hy : y<=n), x=y ->
-    exist (fun x => x <= n) x Hx = exist (fun x => x <= n) y Hy.
-Proof.
-intros n x y Hx Hy Heq.
-generalize Hy.
-rewrite <- Heq.
-intros.
-rewrite (le_uniqueness_proof x n Hx Hy0).
-reflexivity.
-Qed.
-
-(** * Proving that {p:nat|p<=n} = {p:nat|p<=m} -> n=m *)
-
-(** Definition of having finite cardinality [n+1] for a set [A] *)
-
-Definition card (A:Set) n :=
-  exists f,
-    (forall x:A, f x <= n) /\
-    (forall x y:A, f x = f y -> x = y) /\
-    (forall m, m <= n -> exists x:A, f x = m).
-
-Require Import Arith.
-
-(** Showing that the interval [0;n] has cardinality [n+1] *)
-
-Theorem card_interval : forall n, card {x:nat|x<=n} n.
-Proof.
-intro n.
-exists (fun x:{x:nat|x<=n} => proj1_sig x).
-split.
-(* bounded *)
-intro x; apply (proj2_sig x).
-split.
-(* injectivity *)
-intros (p,Hp) (q,Hq).
-simpl.
-intro Hpq.
-apply dep_pair_intro; assumption.
-(* surjectivity *)
-intros m Hmn.
-exists (exist (fun x : nat => x <= n) m Hmn).
-reflexivity.
-Qed.
-
-(** Showing that equality on the interval [0;n] is decidable *)
-
-Lemma interval_dec :
-  forall n (x y : {m:nat|m<=n}), {x=y}+{x<>y}.
-Proof.
-intros n (p,Hp).
-induction p; intros ([|q],Hq).
-left.
-  apply dep_pair_intro.
-  reflexivity.
-right.
-  intro H; discriminate H.
-right.
-  intro H; discriminate H.
-assert (Hp' : p <= n).
-  apply le_Sn_le; assumption.
-assert (Hq' : q <= n).
-  apply le_Sn_le; assumption.
-destruct (IHp Hp' (exist (fun m => m <= n) q Hq'))
-  as [Heq|Hneq].
-left.
-  injection Heq; intro Heq'.
-  apply dep_pair_intro.
-  apply eq_S.
-  assumption.
-right.
-  intro HeqS.
-  injection HeqS; intro Heq.
-  apply Hneq.
-  apply dep_pair_intro.
-  assumption.
-Qed.
-
-(** Showing that the cardinality relation is functional on decidable sets *)
-
-Lemma card_inj_aux :
-  forall (A:Type) f g n,
-    (forall x:A, f x <= 0) ->
-    (forall x y:A, f x = f y -> x = y) ->
-    (forall m, m <= S n -> exists x:A, g x = m)
-     -> False.
-Proof.
-intros A f g n Hfbound Hfinj Hgsurj.
-destruct (Hgsurj (S n) (le_n _)) as (x,Hx).
-destruct (Hgsurj n (le_S _ _ (le_n _))) as (x',Hx').
-assert (Hfx : 0 = f x).
-apply le_n_O_eq.
-apply Hfbound.
-assert (Hfx' : 0 = f x').
-apply le_n_O_eq.
-apply Hfbound.
-assert (x=x').
-apply Hfinj.
-rewrite <- Hfx.
-rewrite <- Hfx'.
-reflexivity.
-rewrite H in Hx.
-rewrite Hx' in Hx.
-apply (n_Sn _ Hx).
-Qed.
-
-(** For [dec_restrict], we use a lemma on the negation of equality
-that requires proof-irrelevance. It should be possible to avoid this
-lemma by generalizing over a first-order definition of [x<>y], say
-[neq] such that [{x=y}+{neq x y}] and [~(x=y /\ neq x y)]; for such
-[neq], unicity of proofs could be proven *)
-
-  Require Import Classical.
-  Lemma neq_dep_intro :
-   forall (A:Set) (z x y:A) (p:x<>z) (q:y<>z), x=y ->
-      exist (fun x => x <> z) x p = exist (fun x => x <> z) y q.
-  Proof.
-  intros A z x y p q Heq.
-   generalize q; clear q; rewrite <- Heq; intro q.
-   rewrite (proof_irrelevance _ p q); reflexivity.
-  Qed.
-
-Lemma dec_restrict :
-  forall (A:Set),
-    (forall x y :A, {x=y}+{x<>y}) ->
-     forall z (x y :{a:A|a<>z}), {x=y}+{x<>y}.
-Proof.
-intros A Hdec z (x,Hx) (y,Hy).
-destruct (Hdec x y) as [Heq|Hneq].
-left; apply neq_dep_intro; assumption.
-right; intro Heq; injection Heq; exact Hneq.
-Qed.
-
-Lemma pred_inj : forall n m,
-  0 <> n -> 0 <> m -> pred m = pred n -> m = n.
-Proof.
-destruct n.
-intros m H; destruct H; reflexivity.
-destruct m.
-intros _ H; destruct H; reflexivity.
-simpl; intros _ _ H.
-rewrite H.
-reflexivity.
-Qed.
-
-Lemma le_neq_lt : forall n m, n <= m -> n<>m -> n < m.
-Proof.
-intros n m Hle Hneq.
-destruct (le_lt_eq_dec n m Hle).
-assumption.
-contradiction.
-Qed.
-
-Lemma inj_restrict :
-  forall (A:Set) (f:A->nat) x y z,
-    (forall x y : A, f x = f y -> x = y)
-    -> x <> z -> f y < f z -> f z <= f x
-    -> pred (f x) = f y
-    -> False.
-
-(* Search error sans le type de f !! *)
-Proof.
-intros A f x y z Hfinj Hneqx Hfy Hfx Heq.
-assert (f z <> f x).
-  apply sym_not_eq.
-  intro Heqf.
-  apply Hneqx.
-  apply Hfinj.
-  assumption.
-assert (f x = S (f y)).
-  assert (0 < f x).
-    apply le_lt_trans with (f z).
-    apply le_O_n.
-    apply le_neq_lt; assumption.
-  apply pred_inj.
-  apply O_S.
-  apply lt_O_neq; assumption.
-  exact Heq.
-assert (f z <= f y).
-destruct (le_lt_or_eq _ _ Hfx).
-  apply lt_n_Sm_le.
-  rewrite <- H0.
-  assumption.
-  contradiction Hneqx.
-  symmetry.
-  apply Hfinj.
-  assumption.
-contradiction (lt_not_le (f y) (f z)).
-Qed.
-
-Theorem card_inj : forall m n (A:Set),
-  (forall x y :A, {x=y}+{x<>y}) ->
-  card A m -> card A n -> m = n.
-Proof.
-induction m; destruct n;
-intros A Hdec
- (f,(Hfbound,(Hfinj,Hfsurj)))
- (g,(Hgbound,(Hginj,Hgsurj))).
-(* 0/0 *)
-reflexivity.
-(* 0/Sm *)
-destruct (card_inj_aux _ _ _ _ Hfbound Hfinj Hgsurj).
-(* Sn/0 *)
-destruct (card_inj_aux _ _ _ _ Hgbound Hginj Hfsurj).
-(* Sn/Sm *)
-destruct (Hgsurj (S n) (le_n _)) as (xSn,HSnx).
-rewrite IHm with (n:=n) (A := {x:A|x<>xSn}).
-reflexivity.
-(* decidability of eq on {x:A|x<>xSm} *)
-apply dec_restrict.
-assumption.
-(* cardinality of {x:A|x<>xSn} is m *)
-pose (f' := fun x' : {x:A|x<>xSn} =>
-    let (x,Hneq) := x' in
-    if le_lt_dec (f xSn) (f x)
-    then pred (f x)
-    else f x).
-exists f'.
-split.
-(* f' is bounded *)
-unfold f'.
-intros (x,_).
-destruct (le_lt_dec (f xSn) (f x)) as [Hle|Hge].
-change m with (pred (S m)).
-apply le_pred.
-apply Hfbound.
-apply le_S_n.
-apply le_trans with (f xSn).
-exact Hge.
-apply Hfbound.
-split.
-(* f' is injective *)
-unfold f'.
-intros (x,Hneqx) (y,Hneqy) Heqf'.
-destruct (le_lt_dec (f xSn) (f x)) as [Hlefx|Hgefx];
-destruct (le_lt_dec (f xSn) (f y)) as [Hlefy|Hgefy].
-(* f xSn <= f x et f xSn <= f y *)
-assert (Heq : x = y).
-  apply Hfinj.
-  assert (f xSn <> f y).
-    apply sym_not_eq.
-    intro Heqf.
-    apply Hneqy.
-    apply Hfinj.
-    assumption.
-  assert (0 < f y).
-    apply le_lt_trans with (f xSn).
-    apply le_O_n.
-    apply le_neq_lt; assumption.
-  assert (f xSn <> f x).
-    apply sym_not_eq.
-    intro Heqf.
-    apply Hneqx.
-    apply Hfinj.
-    assumption.
-  assert (0 < f x).
-    apply le_lt_trans with (f xSn).
-    apply le_O_n.
-    apply le_neq_lt; assumption.
-  apply pred_inj.
-  apply lt_O_neq; assumption.
-  apply lt_O_neq; assumption.
-  assumption.
-apply neq_dep_intro; assumption.
-(* f y < f xSn <= f x *)
-destruct (inj_restrict A f x y xSn); assumption.
-(* f x < f xSn <= f y *)
-symmetry in Heqf'.
-destruct (inj_restrict A f y x xSn); assumption.
-(* f x < f xSn et f y < f xSn *)
-assert (Heq : x=y).
-  apply Hfinj; assumption.
-apply neq_dep_intro; assumption.
-(* f' is surjective *)
-intros p Hlep.
-destruct (le_lt_dec (f xSn) p) as [Hle|Hlt].
-(* case f xSn <= p *)
-destruct (Hfsurj (S p) (le_n_S _ _ Hlep)) as (x,Hx).
-assert (Hneq : x <> xSn).
-  intro Heqx.
-  rewrite Heqx in Hx.
-  rewrite Hx in Hle.
-  apply le_Sn_n with p; assumption.
-exists (exist (fun a => a<>xSn) x Hneq).
-unfold f'.
-destruct (le_lt_dec (f xSn) (f x)) as [Hle'|Hlt'].
-rewrite Hx; reflexivity.
-rewrite Hx in Hlt'.
-contradiction (le_not_lt (f xSn) p).
-apply lt_trans with (S p).
-apply lt_n_Sn.
-assumption.
-(* case p < f xSn *)
-destruct (Hfsurj p (le_S _ _ Hlep)) as (x,Hx).
-assert (Hneq : x <> xSn).
-  intro Heqx.
-  rewrite Heqx in Hx.
-  rewrite Hx in Hlt.
-  apply (lt_irrefl p).
-  assumption.
-exists (exist (fun a => a<>xSn) x Hneq).
-unfold f'.
-destruct (le_lt_dec (f xSn) (f x)) as [Hle'|Hlt'].
-  rewrite Hx in Hle'.
-  contradiction (lt_irrefl p).
-  apply lt_le_trans with (f xSn); assumption.
-  assumption.
-(* cardinality of {x:A|x<>xSn} is n *)
-pose (g' := fun x' : {x:A|x<>xSn} =>
-   let (x,Hneq) := x' in
-   if Hdec x xSn then 0 else g x).
-exists g'.
-split.
-(* g is bounded *)
-unfold g'.
-intros (x,_).
-destruct (Hdec x xSn) as [_|Hneq].
-apply le_O_n.
-assert (Hle_gx:=Hgbound x).
-destruct (le_lt_or_eq _ _ Hle_gx).
-apply lt_n_Sm_le.
-assumption.
-contradiction Hneq.
-apply Hginj.
-rewrite HSnx.
-assumption.
-split.
-(* g is injective *)
-unfold g'.
-intros (x,Hneqx) (y,Hneqy) Heqg'.
-destruct (Hdec x xSn) as [Heqx|_].
-contradiction Hneqx.
-destruct (Hdec y xSn) as [Heqy|_].
-contradiction Hneqy.
-assert (Heq : x=y).
-  apply Hginj; assumption.
-apply neq_dep_intro; assumption.
-(* g is surjective *)
-intros p Hlep.
-destruct (Hgsurj p (le_S _ _ Hlep)) as (x,Hx).
-assert (Hneq : x<>xSn).
-  intro Heq.
-  rewrite Heq in Hx.
-  rewrite Hx in HSnx.
-  rewrite HSnx in Hlep.
-  contradiction (le_Sn_n _ Hlep).
-exists (exist (fun a => a<>xSn) x Hneq).
-simpl.
-destruct (Hdec x xSn) as [Heqx|_].
-contradiction Hneq.
-assumption.
-Qed.
-
-(** Conclusion *)
-
-Theorem interval_discr :
-  forall n m, {p:nat|p<=n} = {p:nat|p<=m} -> n=m.
-Proof.
-intros n m Heq.
-apply card_inj with (A := {p:nat|p<=n}).
-apply interval_dec.
-apply card_interval.
-rewrite Heq.
-apply card_interval.
-Qed.

samples/D/aa.d

@@ -0,0 +1,440 @@
+/**
+ * Implementation of associative arrays.
+ *
+ * Copyright: Martin Nowak 2015 -.
+ * License:   $(LINK2 http://www.boost.org/LICENSE_1_0.txt, Boost License 1.0)
+ * Authors:   Martin Nowak
+ */
+module core.aa;
+
+import core.memory : GC;
+
+private
+{
+    // grow threshold
+    enum GROW_NUM = 4;
+    enum GROW_DEN = 5;
+    // shrink threshold
+    enum SHRINK_NUM = 1;
+    enum SHRINK_DEN = 8;
+    // grow factor
+    enum GROW_FAC = 4;
+    // growing the AA doubles it's size, so the shrink threshold must be
+    // smaller than half the grow threshold to have a hysteresis
+    static assert(GROW_FAC * SHRINK_NUM * GROW_DEN < GROW_NUM * SHRINK_DEN);
+    // initial load factor (for literals), mean of both thresholds
+    enum INIT_NUM = (GROW_DEN * SHRINK_NUM + GROW_NUM * SHRINK_DEN) / 2;
+    enum INIT_DEN = SHRINK_DEN * GROW_DEN;
+
+    // magic hash constants to distinguish empty, deleted, and filled buckets
+    enum HASH_EMPTY = 0;
+    enum HASH_DELETED = 0x1;
+    enum HASH_FILLED_MARK = size_t(1) << 8 * size_t.sizeof - 1;
+}
+
+enum INIT_NUM_BUCKETS = 8;
+
+struct AA(Key, Val)
+{
+    this(size_t sz)
+    {
+        impl = new Impl(nextpow2(sz));
+    }
+
+    @property bool empty() const pure nothrow @safe @nogc
+    {
+        return !length;
+    }
+
+    @property size_t length() const pure nothrow @safe @nogc
+    {
+        return impl is null ? 0 : impl.length;
+    }
+
+    void opIndexAssign(Val val, in Key key)
+    {
+        // lazily alloc implementation
+        if (impl is null)
+            impl = new Impl(INIT_NUM_BUCKETS);
+
+        // get hash and bucket for key
+        immutable hash = calcHash(key);
+
+        // found a value => assignment
+        if (auto p = impl.findSlotLookup(hash, key))
+        {
+            p.entry.val = val;
+            return;
+        }
+
+        auto p = findSlotInsert(hash);
+        if (p.deleted)
+            --deleted;
+        // check load factor and possibly grow
+        else if (++used * GROW_DEN > dim * GROW_NUM)
+        {
+            grow();
+            p = findSlotInsert(hash);
+            assert(p.empty);
+        }
+
+        // update search cache and allocate entry
+        firstUsed = min(firstUsed, cast(uint)(p - buckets.ptr));
+        p.hash = hash;
+        p.entry = new Impl.Entry(key, val); // TODO: move
+        return;
+    }
+
+    ref inout(Val) opIndex(in Key key) inout @trusted
+    {
+        auto p = opIn_r(key);
+        assert(p !is null);
+        return *p;
+    }
+
+    inout(Val)* opIn_r(in Key key) inout @trusted
+    {
+        if (empty)
+            return null;
+
+        immutable hash = calcHash(key);
+        if (auto p = findSlotLookup(hash, key))
+            return &p.entry.val;
+        return null;
+    }
+
+    bool remove(in Key key)
+    {
+        if (empty)
+            return false;
+
+        immutable hash = calcHash(key);
+        if (auto p = findSlotLookup(hash, key))
+        {
+            // clear entry
+            p.hash = HASH_DELETED;
+            p.entry = null;
+
+            ++deleted;
+            if (length * SHRINK_DEN < dim * SHRINK_NUM)
+                shrink();
+
+            return true;
+        }
+        return false;
+    }
+
+    Val get(in Key key, lazy Val val)
+    {
+        auto p = opIn_r(key);
+        return p is null ? val : *p;
+    }
+
+    ref Val getOrSet(in Key key, lazy Val val)
+    {
+        // lazily alloc implementation
+        if (impl is null)
+            impl = new Impl(INIT_NUM_BUCKETS);
+
+        // get hash and bucket for key
+        immutable hash = calcHash(key);
+
+        // found a value => assignment
+        if (auto p = impl.findSlotLookup(hash, key))
+            return p.entry.val;
+
+        auto p = findSlotInsert(hash);
+        if (p.deleted)
+            --deleted;
+        // check load factor and possibly grow
+        else if (++used * GROW_DEN > dim * GROW_NUM)
+        {
+            grow();
+            p = findSlotInsert(hash);
+            assert(p.empty);
+        }
+
+        // update search cache and allocate entry
+        firstUsed = min(firstUsed, cast(uint)(p - buckets.ptr));
+        p.hash = hash;
+        p.entry = new Impl.Entry(key, val);
+        return p.entry.val;
+    }
+
+    /**
+       Convert the AA to the type of the builtin language AA.
+     */
+    Val[Key] toBuiltinAA() pure nothrow
+    {
+        return cast(Val[Key]) _aaFromCoreAA(impl, rtInterface);
+    }
+
+private:
+
+    private this(inout(Impl)* impl) inout
+    {
+        this.impl = impl;
+    }
+
+    ref Val getLValue(in Key key)
+    {
+        // lazily alloc implementation
+        if (impl is null)
+            impl = new Impl(INIT_NUM_BUCKETS);
+
+        // get hash and bucket for key
+        immutable hash = calcHash(key);
+
+        // found a value => assignment
+        if (auto p = impl.findSlotLookup(hash, key))
+            return p.entry.val;
+
+        auto p = findSlotInsert(hash);
+        if (p.deleted)
+            --deleted;
+        // check load factor and possibly grow
+        else if (++used * GROW_DEN > dim * GROW_NUM)
+        {
+            grow();
+            p = findSlotInsert(hash);
+            assert(p.empty);
+        }
+
+        // update search cache and allocate entry
+        firstUsed = min(firstUsed, cast(uint)(p - buckets.ptr));
+        p.hash = hash;
+        p.entry = new Impl.Entry(key); // TODO: move
+        return p.entry.val;
+    }
+
+    static struct Impl
+    {
+        this(size_t sz)
+        {
+            buckets = allocBuckets(sz);
+        }
+
+        @property size_t length() const pure nothrow @nogc
+        {
+            assert(used >= deleted);
+            return used - deleted;
+        }
+
+        @property size_t dim() const pure nothrow @nogc
+        {
+            return buckets.length;
+        }
+
+        @property size_t mask() const pure nothrow @nogc
+        {
+            return dim - 1;
+        }
+
+        // find the first slot to insert a value with hash
+        inout(Bucket)* findSlotInsert(size_t hash) inout pure nothrow @nogc
+        {
+            for (size_t i = hash & mask, j = 1;; ++j)
+            {
+                if (!buckets[i].filled)
+                    return &buckets[i];
+                i = (i + j) & mask;
+            }
+        }
+
+        // lookup a key
+        inout(Bucket)* findSlotLookup(size_t hash, in Key key) inout
+        {
+            for (size_t i = hash & mask, j = 1;; ++j)
+            {
+                if (buckets[i].hash == hash && key == buckets[i].entry.key)
+                    return &buckets[i];
+                else if (buckets[i].empty)
+                    return null;
+                i = (i + j) & mask;
+            }
+        }
+
+        void grow()
+        {
+            // If there are so many deleted entries, that growing would push us
+            // below the shrink threshold, we just purge deleted entries instead.
+            if (length * SHRINK_DEN < GROW_FAC * dim * SHRINK_NUM)
+                resize(dim);
+            else
+                resize(GROW_FAC * dim);
+        }
+
+        void shrink()
+        {
+            if (dim > INIT_NUM_BUCKETS)
+                resize(dim / GROW_FAC);
+        }
+
+        void resize(size_t ndim) pure nothrow
+        {
+            auto obuckets = buckets;
+            buckets = allocBuckets(ndim);
+
+            foreach (ref b; obuckets)
+                if (b.filled)
+                    *findSlotInsert(b.hash) = b;
+
+            firstUsed = 0;
+            used -= deleted;
+            deleted = 0;
+            GC.free(obuckets.ptr); // safe to free b/c impossible to reference
+        }
+
+        static struct Entry
+        {
+            Key key;
+            Val val;
+        }
+
+        static struct Bucket
+        {
+            size_t hash;
+            Entry* entry;
+
+            @property bool empty() const
+            {
+                return hash == HASH_EMPTY;
+            }
+
+            @property bool deleted() const
+            {
+                return hash == HASH_DELETED;
+            }
+
+            @property bool filled() const
+            {
+                return cast(ptrdiff_t) hash < 0;
+            }
+        }
+
+        Bucket[] allocBuckets(size_t dim) @trusted pure nothrow
+        {
+            enum attr = GC.BlkAttr.NO_INTERIOR;
+            immutable sz = dim * Bucket.sizeof;
+            return (cast(Bucket*) GC.calloc(sz, attr))[0 .. dim];
+        }
+
+        Bucket[] buckets;
+        uint used;
+        uint deleted;
+        uint firstUsed;
+    }
+
+    RTInterface* rtInterface()() pure nothrow @nogc
+    {
+        static size_t aaLen(in void* pimpl) pure nothrow @nogc
+        {
+            auto aa = const(AA)(cast(const(Impl)*) pimpl);
+            return aa.length;
+        }
+
+        static void* aaGetY(void** pimpl, in void* pkey)
+        {
+            auto aa = AA(cast(Impl*)*pimpl);
+            auto res = &aa.getLValue(*cast(const(Key*)) pkey);
+            *pimpl = aa.impl; // might have changed
+            return res;
+        }
+
+        static inout(void)* aaInX(inout void* pimpl, in void* pkey)
+        {
+            auto aa = inout(AA)(cast(inout(Impl)*) pimpl);
+            return aa.opIn_r(*cast(const(Key*)) pkey);
+        }
+
+        static bool aaDelX(void* pimpl, in void* pkey)
+        {
+            auto aa = AA(cast(Impl*) pimpl);
+            return aa.remove(*cast(const(Key*)) pkey);
+        }
+
+        static immutable vtbl = RTInterface(&aaLen, &aaGetY, &aaInX, &aaDelX);
+        return cast(RTInterface*)&vtbl;
+    }
+
+    static size_t calcHash(in ref Key key)
+    {
+        return hashOf(key) | HASH_FILLED_MARK;
+    }
+
+    Impl* impl;
+    alias impl this;
+}
+
+package extern (C) void* _aaFromCoreAA(void* impl, RTInterface* rtIntf) pure nothrow;
+
+private:
+
+struct RTInterface
+{
+    alias AA = void*;
+
+    size_t function(in AA aa) pure nothrow @nogc len;
+    void* function(AA* aa, in void* pkey) getY;
+    inout(void)* function(inout AA aa, in void* pkey) inX;
+    bool function(AA aa, in void* pkey) delX;
+}
+
+unittest
+{
+    AA!(int, int) aa;
+    assert(aa.length == 0);
+    aa[0] = 1;
+    assert(aa.length == 1 && aa[0] == 1);
+    aa[1] = 2;
+    assert(aa.length == 2 && aa[1] == 2);
+    import core.stdc.stdio;
+
+    int[int] rtaa = aa.toBuiltinAA();
+    assert(rtaa.length == 2);
+    puts("length");
+    assert(rtaa[0] == 1);
+    assert(rtaa[1] == 2);
+    rtaa[2] = 3;
+
+    assert(aa[2] == 3);
+}
+
+unittest
+{
+    auto aa = AA!(int, int)(3);
+    aa[0] = 0;
+    aa[1] = 1;
+    aa[2] = 2;
+    assert(aa.length == 3);
+}
+
+//==============================================================================
+// Helper functions
+//------------------------------------------------------------------------------
+
+size_t nextpow2(in size_t n) pure nothrow @nogc
+{
+    import core.bitop : bsr;
+
+    if (n < 2)
+        return 1;
+    return size_t(1) << bsr(n - 1) + 1;
+}
+
+pure nothrow @nogc unittest
+{
+    //                            0, 1, 2, 3, 4, 5, 6, 7, 8,  9
+    foreach (const n, const pow2; [1, 1, 2, 4, 4, 8, 8, 8, 8, 16])
+        assert(nextpow2(n) == pow2);
+}
+
+T min(T)(T a, T b) pure nothrow @nogc
+{
+    return a < b ? a : b;
+}
+
+T max(T)(T a, T b) pure nothrow @nogc
+{
+    return b < a ? a : b;
+}

samples/D/arrayops.d

@@ -0,0 +1,187 @@
+/**
+ * Benchmark for array ops.
+ *
+ * Copyright: Copyright Martin Nowak 2016 -.
+ * License:   $(LINK2 http://www.boost.org/LICENSE_1_0.txt, Boost License 1.0)
+ * Authors:   Martin Nowak
+ */
+import core.cpuid, std.algorithm, std.datetime, std.meta, std.stdio, std.string,
+    std.range;
+
+float[6] getLatencies(T, string op)()
+{
+    enum N = (64 * (1 << 6) + 64) * T.sizeof;
+    auto a = Array!T(N), b = Array!T(N), c = Array!T(N);
+    float[6] latencies = float.max;
+    foreach (i, ref latency; latencies)
+    {
+        auto len = 1 << i;
+        foreach (_; 1 .. 32)
+        {
+            a[] = 24;
+            b[] = 4;
+            c[] = 2;
+            auto sw = StopWatch(AutoStart.yes);
+            foreach (off; size_t(0) .. size_t(64))
+            {
+                off = off * len + off;
+                enum op = op.replace("const", "2").replace("a",
+                        "a[off .. off + len]").replace("b",
+                        "b[off .. off + len]").replace("c", "c[off .. off + len]");
+                mixin(op ~ ";");
+            }
+            latency = min(latency, sw.peek.nsecs);
+        }
+    }
+    float[6] res = latencies[] / 1024;
+    return res;
+}
+
+float[4] getThroughput(T, string op)()
+{
+    enum N = (40 * 1024 * 1024 + 64 * T.sizeof) / T.sizeof;
+    auto a = Array!T(N), b = Array!T(N), c = Array!T(N);
+    float[4] latencies = float.max;
+    size_t[4] lengths = [
+        8 * 1024 / T.sizeof, 32 * 1024 / T.sizeof, 512 * 1024 / T.sizeof, 32 * 1024 * 1024 / T
+        .sizeof
+    ];
+    foreach (i, ref latency; latencies)
+    {
+        auto len = lengths[i] / 64;
+        foreach (_; 1 .. 4)
+        {
+            a[] = 24;
+            b[] = 4;
+            c[] = 2;
+            auto sw = StopWatch(AutoStart.yes);
+            foreach (off; size_t(0) .. size_t(64))
+            {
+                off = off * len + off;
+                enum op = op.replace("const", "2").replace("a",
+                        "a[off .. off + len]").replace("b",
+                        "b[off .. off + len]").replace("c", "c[off .. off + len]");
+                mixin(op ~ ";");
+            }
+            immutable nsecs = sw.peek.nsecs;
+            runMasked({latency = min(latency, nsecs);});
+        }
+    }
+    float[4] throughputs = void;
+    runMasked({throughputs = T.sizeof * lengths[] / latencies[];});
+    return throughputs;
+}
+
+string[] genOps()
+{
+    string[] ops;
+    foreach (op1; ["+", "-", "*", "/"])
+    {
+        ops ~= "a " ~ op1 ~ "= b";
+        ops ~= "a " ~ op1 ~ "= const";
+        foreach (op2; ["+", "-", "*", "/"])
+        {
+            ops ~= "a " ~ op1 ~ "= b " ~ op2 ~ " c";
+            ops ~= "a " ~ op1 ~ "= b " ~ op2 ~ " const";
+        }
+    }
+    return ops;
+}
+
+void runOp(string op)()
+{
+    foreach (T; AliasSeq!(ubyte, ushort, uint, ulong, byte, short, int, long, float,
+            double))
+        writefln("%s, %s, %(%.2f, %), %(%s, %)", T.stringof, op,
+            getLatencies!(T, op), getThroughput!(T, op));
+}
+
+struct Array(T)
+{
+    import core.stdc.stdlib : free, malloc;
+
+    this(size_t n)
+    {
+        ary = (cast(T*) malloc(T.sizeof * n))[0 .. n];
+    }
+
+    ~this()
+    {
+        free(ary.ptr);
+    }
+
+    T[] ary;
+    alias ary this;
+}
+
+version (X86)
+    version = SSE;
+else version (X86_64)
+    version = SSE;
+else
+    static assert(0, "unimplemented");
+
+version (SSE)
+{
+    uint mxcsr()
+    {
+        uint ret = void;
+        asm
+        {
+            stmxcsr ret;
+        }
+        return ret;
+    }
+
+    void mxcsr(uint val)
+    {
+        asm
+        {
+            ldmxcsr val;
+        }
+    }
+
+    // http://softpixel.com/~cwright/programming/simd/sse.php
+    enum FPU_EXCEPTION_MASKS = 1 << 12 | 1 << 11 | 1 << 10 | 1 << 9 | 1 << 8 | 1 << 7;
+    enum FPU_EXCEPTION_FLAGS = 1 << 5 | 1 << 4 | 1 << 3 | 1 << 2 | 1 << 1 | 1 << 0;
+
+    void maskFPUExceptions()
+    {
+        mxcsr = mxcsr | FPU_EXCEPTION_MASKS;
+    }
+
+    void unmaskFPUExceptions()
+    {
+        mxcsr = mxcsr & ~FPU_EXCEPTION_MASKS;
+    }
+
+    uint FPUExceptionFlags()
+    {
+        return mxcsr & FPU_EXCEPTION_FLAGS;
+    }
+
+    void clearFPUExceptionFlags()
+    {
+        mxcsr = mxcsr & ~FPU_EXCEPTION_FLAGS;
+    }
+}
+
+void runMasked(scope void delegate() dg)
+{
+    assert(FPUExceptionFlags == 0);
+    maskFPUExceptions;
+    dg();
+    clearFPUExceptionFlags;
+    unmaskFPUExceptions;
+}
+
+void main()
+{
+    unmaskFPUExceptions;
+
+    writefln("type, op, %(latency%s, %), %-(throughput%s, %)", iota(6)
+        .map!(i => 1 << i), ["8KB", "32KB", "512KB", "32MB"]);
+    foreach (op; mixin("AliasSeq!(%(%s, %))".format(genOps)))
+        runOp!op;
+    maskFPUExceptions;
+}

samples/D/function.d

@@ -0,0 +1,3 @@
+void foo()
+{
+}

samples/D/hello_world.d

@@ -0,0 +1,6 @@
+import std.stdio;
+
+void main()
+{
+    writeln("Hello World");
+}

samples/D/template.d

@@ -0,0 +1,7 @@
+template Fib(size_t N)
+{
+    static if (N < 2)
+        enum Fib = size_t(1);
+    else
+        enum Fib = Fib!(N - 2) + Fib!(N - 1);
+}

samples/D/template_function.d

@@ -0,0 +1,3 @@
+void bar(T)(T t)
+{
+}

samples/D/unittest1.d

@@ -0,0 +1,3 @@
+unittest
+{
+}

samples/D/unittest2.d

@@ -0,0 +1,3 @@
+unittest("optional name")
+{
+}

samples/DataWeave/customInterpolator.dwl

@@ -0,0 +1,12 @@
+fun SQL(literals, parts) = ''
+---
+[
+  SQL `SELECT * FROM table WHERE id = $(1) AND name = $('a')`,
+  SQL `$('p')`,
+  SQL `$('a')$('b')`,
+  SQL `$('a')---$('b')`,
+  SQL `---$('a')---$('b')---`,
+  SQL `$('p')bbb`,
+  SQL `aaa$('p')`,
+  SQL `aaa$('p')bbb`
+]

samples/DataWeave/directives.dwl

@@ -0,0 +1,9 @@
+%dw 2.0
+var number = 1234
+fun foo(func,name="Mariano") = func(name)
+input payload application/test arg="value"
+output application/json
+---
+{
+  foo: "bar"
+}

samples/DataWeave/functions.dwl

@@ -0,0 +1,27 @@
+%dw 2.0
+var x=(param1, param2) -> { "$param1": param2 }
+var y=(param1, param2 = "c") -> { "$param1": param2 }
+var toUser = (user) -> { name: user.name,	lastName: user.lastName }
+fun z(param1, param2) = { "$param1": param2 }
+var a = { name: "Mariano" , toUser: ((param1, param2) -> { "$param1": param2 }) }
+var applyFirst =  (array, func) -> (func(array[0]) ++  array[1 to -1])
+
+var nested =  (array, func) -> (a) -> (b) -> (c) -> array map func(a ++ b ++ c)
+
+
+fun f2(a1, a2) = ""
+fun f3(a1:String, a2:Number):String = a1
+fun f4(a1:String, a2:(a:Number) -> Number):String = a1
+---
+result: {
+  a: x("a", "b"),
+  b: y("a"),
+  c: y("a", "b"),
+  users: { (in1 map ((user) -> { user: (toUser(user) ++ user) })) },
+  d: z("a", "b"),
+  e: a.toUser("name","Mariano"),
+  f: a.toUser("name","Mariano").name,
+  f: applyFirst("mariano", (s) -> upper(s) ),
+  g: [] map (s) -> upper(s),
+  h: 1 f2 2
+}

samples/DataWeave/literals.dwl

@@ -0,0 +1,36 @@
+%dw 2.0
+---
+{
+  "boolean":{
+    "true" : true,
+    "false": false
+  },
+  "Number": {
+    "int": 123,
+    "decimal": 123.23
+  },
+  "string": {
+    "singleQuote" : 'A String',
+    "doubleQuote" : "A String"
+  },
+  "regex": /foo/,
+  "date": {
+    a: |2003-10-01|,
+    b: |2005-045|,
+    c: |2003-W14-3|,
+    d: |23:57:59|,
+    e: |23:57:30.700|,
+    f: |23:50:30Z|,
+    g: |+13:00|,
+    h: |Z|,
+    i: |-02:00|,
+    j: |2005-06-02T15:10:16|,
+    k: |2005-06-02T15:10:16Z|,
+    l: |2005-06-02T15:10:16+03:00|,
+    m: |P12Y7M11D|,
+    n: |P12Y5M|,
+    o: |P45DT9H20M8S|,
+    p: |PT9H20M8S|
+  }
+}
+

samples/DataWeave/match.dwl

@@ -0,0 +1,33 @@
+{
+  // Regex Pattern Matching (Can be named or unnamed)
+  a: in0.phones map $ match {
+    case matches /\+(\d+)\s\((\d+)\)\s(\d+\-\d+)/ -> { country: $[0], area: $[1], number: $[2] }
+    case matches /\((\d+)\)\s(\d+\-\d+)/ -> { area: $[1], number: $[2] }
+    case phone matches /\((\d+)\)\s(\d+\-\d+)/ -> { area: phone[1], number: phone[2] }
+  },
+  // Type Pattern Matching (Can be named or unnamed)
+  b: in0.object match {
+    case is Object -> { object: $ }
+    case is Number -> { number: $ }
+    // This is how you name variables if needed
+    case y is Boolean -> { boolean: y }
+  },
+  // Literal Pattern Matching (Can be named or unnamed)
+  c: in0.value match {
+    case "Emiliano" -> { string: $ }
+    case 123 -> { number: $ }
+    // This is how you name variables if needed
+    case value: "Mariano" -> { name: value }
+  },
+  // Boolean Expression Pattern Matching (Always named)
+  d: in0.value match {
+    case x if x > 30 -> { biggerThan30: x }
+    case x if x == 9 -> { nine: x }
+  },
+  // Default matches
+  e: in0.value match {
+    case "Emiliano" -> "string"
+    case 3.14 -> number
+    else -> "1234"
+  }
+}