Merge pull request #1803 from github/cut-release-v4.2.0

Bumping to v4.2.0

.gitignore

@@ -1 +1,6 @@
-Gemfile.lock
+/Gemfile.lock
+.bundle/
+benchmark/
+lib/linguist/samples.json
+/grammars
+/node_modules

.travis.yml

@@ -1,8 +1,13 @@
-before_install: sudo apt-get install libicu-dev -y
+before_install:
+  - git fetch origin master:master
+  - git fetch origin v2.0.0:v2.0.0
+  - git fetch origin test/attributes:test/attributes
+  - git fetch origin test/master:test/master
+  - sudo apt-get install libicu-dev -y
 rvm:
-  - 1.8.7
-  - 1.9.2
   - 1.9.3
-  - ree
+  - 2.0.0
+  - 2.1
+  - 2.2
 notifications:
   disabled: true

CONTRIBUTING.md

@@ -0,0 +1,31 @@
+## Contributing
+
+The majority of contributions won't need to touch any Ruby code at all. The [master language list][languages] is just a YAML configuration file.
+
+Almost all bug fixes or new language additions should come with some additional code samples. Just drop them under [`samples/`][samples] in the correct subdirectory and our test suite will automatically test them. In most cases you shouldn't need to add any new assertions.
+
+### My code is detected as the wrong language
+
+This can usually be solved either by adding a new filename or file name extension to the language's entry in [`languages.yml`][languages] or adding more [samples][samples] for your language to the repository to make Linguist's classifier smarter.
+
+### Syntax highlighting looks wrong
+
+Assuming your code is being detected as the right language (see above), in most cases this is due to a bug in the language grammar rather than a bug in Linguist. [`grammars.yml`][grammars] lists all the grammars we use for syntax highlighting on github.com. Find the one corresponding to your code's programming language and submit a bug report upstream.
+
+You can also try to fix the bug yourself and submit a Pull Request. [This piece from TextMate's documentation](http://manual.macromates.com/en/language_grammars) offers a good introduction on how to work with TextMate-compatible grammars. You can test grammars using [Lightshow](https://lightshow.githubapp.com).
+
+Once the bug has been fixed upstream, please let us know and we'll pick it up for GitHub.
+
+### I want to add support for the `X` programming language
+
+Great! You'll need to:
+
+0. Add an entry for your language to [`languages.yml`][languages].
+0. Add a grammar for your language to [`grammars.yml`][grammars] by running `script/download-grammars --add URL`. Please only add grammars that have a license that permits redistribution.
+0. Add samples for your language to the [samples directory][samples].
+
+We try only to add languages once they have some usage on GitHub, so please note in-the-wild usage examples in your pull request. In most cases we prefer that languages already be in use in hundreds of repositories before supporting them in Linguist.
+
+[grammars]: /grammars.yml
+[languages]: /lib/linguist/languages.yml
+[samples]: /samples

Gemfile

@@ -1,2 +1,5 @@
-source :rubygems
-gemspec
+source 'https://rubygems.org'
+gemspec :name => "github-linguist"
+gemspec :name => "github-linguist-grammars"
+gem 'test-unit', require: false if RUBY_VERSION >= '2.2'
+gem 'byebug' if RUBY_VERSION >= '2.0'

LICENSE

@@ -1,4 +1,4 @@
-Copyright (c) 2011 GitHub, Inc.
+Copyright (c) 2011-2014 GitHub, Inc.
 
 Permission is hereby granted, free of charge, to any person
 obtaining a copy of this software and associated documentation

README.md

@@ -1,76 +1,143 @@
 # Linguist
 
-We use this library at GitHub to detect blob languages, highlight code, ignore binary files, suppress generated files in diffs and generate language breakdown graphs.
+We use this library at GitHub to detect blob languages, ignore binary files, suppress generated files in diffs, and generate language breakdown graphs.
+
+Tips for filing issues and creating pull requests can be found in [`CONTRIBUTING.md`](/CONTRIBUTING.md).
 
 ## Features
 
 ### Language detection
 
-Linguist defines the list of all languages known to GitHub in a [yaml file](https://github.com/github/linguist/blob/master/lib/linguist/languages.yml). In order for a file to be highlighted, a language and lexer must be defined there.
+Linguist defines a list of all languages known to GitHub in a [yaml file](https://github.com/github/linguist/blob/master/lib/linguist/languages.yml).
 
-Most languages are detected by their file extension. This is the fastest and most common situation. For script files, which are usually extensionless, we do "deep content inspection"™ and check the shebang of the file. Checking the file's contents may also be used for disambiguating languages. C, C++ and Obj-C all use `.h` files. Looking for common keywords, we are usually able to guess the correct language.
+Most languages are detected by their file extension. For disambiguating between files with common extensions, we first apply some common-sense heuristics to pick out obvious languages. After that, we use a
+[statistical
+classifier](https://github.com/github/linguist/blob/master/lib/linguist/classifier.rb).
+This process can help us tell the difference between, for example, `.h` files which could be either C, C++, or Obj-C.
 
-In the actual GitHub app we deal with `Grit::Blob` objects. For testing, there is a simple `FileBlob` API.
+```ruby
 
-    Linguist::FileBlob.new("lib/linguist.rb").language.name #=> "Ruby"
+Linguist::FileBlob.new("lib/linguist.rb").language.name #=> "Ruby"
 
-    Linguist::FileBlob.new("bin/linguist").language.name #=> "Ruby"
+Linguist::FileBlob.new("bin/linguist").language.name #=> "Ruby"
+```
 
 See [lib/linguist/language.rb](https://github.com/github/linguist/blob/master/lib/linguist/language.rb) and [lib/linguist/languages.yml](https://github.com/github/linguist/blob/master/lib/linguist/languages.yml).
 
 ### Syntax Highlighting
 
-The actual syntax highlighting is handled by our Pygments wrapper, [pygments.rb](https://github.com/tmm1/pygments.rb). It also provides a [Lexer abstraction](https://github.com/tmm1/pygments.rb/blob/master/lib/pygments/lexer.rb) that determines which highlighter should be used on a file.
+Syntax highlighting in GitHub is performed using TextMate-compatible grammars. These are the same grammars that TextMate, Sublime Text and Atom use.
 
-We typically run on a prerelease version of Pygments to get early access to new lexers. The [lexers.yml](https://github.com/github/linguist/blob/master/lib/linguist/lexers.yml) file is a dump of the lexers we have available on our server. If there is a new lexer in pygments-main not on the list, [open an issue](https://github.com/github/linguist/issues) and we'll try to upgrade it soon.
-
-### MIME type detection
-
-Most of the MIME types handling is done by the Ruby [mime-types gem](https://github.com/halostatue/mime-types). But we have our own list of additions and overrides. To add or modify this list, see [lib/linguist/mimes.yml](https://github.com/github/linguist/blob/master/lib/linguist/mimes.yml).
-
-MIME types are used to set the Content-Type of raw binary blobs which are served from a special `raw.github.com` domain. However, all text blobs are served as `text/plain` regardless of their type to ensure they open in the browser rather than downloading.
-
-The MIME type also determines whether a blob is binary or plain text. So if you're seeing a blob that says "View Raw" and it is actually plain text, the mime type and encoding probably needs to be explicitly stated.
-
-    Linguist::FileBlob.new("linguist.zip").binary? #=> true
-
-See [lib/linguist/mimes.yml](https://github.com/github/linguist/blob/master/lib/linguist/mimes.yml).
+Every language in `languages.yml` is mapped to its corresponding TM `scope`. This scope will be used when picking up a grammar for highlighting. **When adding a new language to Linguist, please add its corresponding scope too (assuming there's an existing TextMate bundle, Sublime Text package, or Atom package) so syntax highlighting works for it**.
 
 ### Stats
 
-The [Language Graph](https://github.com/github/linguist/graphs/languages) is built by aggregating the languages of all repo's blobs. The top language in the graph determines the project's primary language. Collectively, these stats make up the [Top Languages](https://github.com/languages) page.
+The Language stats bar that you see on every repository is built by aggregating the languages of each file in that repository. The top language in the graph determines the project's primary language.
 
-The repository stats API can be used on a directory:
+The repository stats API, accessed through `#languages`, can be used on a directory:
 
-    project = Linguist::Repository.from_directory(".")
-    project.language.name  #=> "Ruby"
-    project.languages      #=> { "Ruby" => 0.98,
-                                 "Shell" => 0.02 }
+***API UPDATE***
 
-These stats are also printed out by the binary. Try running `linguist` on itself:
+Since [Version 3.0.0](https://github.com/github/linguist/releases/tag/v3.0.0) Linguist expects a git repository (in the form of a [Rugged::Repository](https://github.com/libgit2/rugged#repositories)) to be passed when initializing `Linguist::Repository`.
 
-    $ bundle exec linguist lib/
-    100%  Ruby
+
+```ruby
+require 'rugged'
+require 'linguist'
+
+repo = Rugged::Repository.new('.')
+project = Linguist::Repository.new(repo, repo.head.target_id)
+project.language       #=> "Ruby"
+project.languages      #=> { "Ruby" => 119387 }
+```
+
+These stats are also printed out by the `linguist` binary. You can use the
+`--breakdown` flag, and the binary will also output the breakdown of files by language.
+
+You can try running `linguist` on the root directory in this repository itself:
+
+    $ bundle exec linguist --breakdown
+
+    100.00% Ruby
+
+    Ruby:
+    Gemfile
+    Rakefile
+    bin/linguist
+    github-linguist.gemspec
+    lib/linguist.rb
+    lib/linguist/blob_helper.rb
+    lib/linguist/classifier.rb
+    lib/linguist/file_blob.rb
+    lib/linguist/generated.rb
+    lib/linguist/heuristics.rb
+    lib/linguist/language.rb
+    lib/linguist/lazy_blob.rb
+    lib/linguist/md5.rb
+    lib/linguist/repository.rb
+    lib/linguist/samples.rb
+    lib/linguist/tokenizer.rb
+    lib/linguist/version.rb
+    test/test_blob.rb
+    test/test_classifier.rb
+    test/test_heuristics.rb
+    test/test_language.rb
+    test/test_md5.rb
+    test/test_pedantic.rb
+    test/test_repository.rb
+    test/test_samples.rb
+    test/test_tokenizer.rb
 
 #### Ignore vendored files
 
 Checking other code into your git repo is a common practice. But this often inflates your project's language stats and may even cause your project to be labeled as another language. We are able to identify some of these files and directories and exclude them.
 
-    Linguist::FileBlob.new("vendor/plugins/foo.rb").vendored? # => true
+```ruby
+Linguist::FileBlob.new("vendor/plugins/foo.rb").vendored? # => true
+```
 
 See [Linguist::BlobHelper#vendored?](https://github.com/github/linguist/blob/master/lib/linguist/blob_helper.rb) and [lib/linguist/vendor.yml](https://github.com/github/linguist/blob/master/lib/linguist/vendor.yml).
 
 #### Generated file detection
 
-Not all plain text files are true source files. Generated files like minified js and compiled CoffeeScript can be detected and excluded from language stats. As an extra bonus, these files are suppressed in Diffs.
+Not all plain text files are true source files. Generated files like minified js and compiled CoffeeScript can be detected and excluded from language stats. As an extra bonus, these files are suppressed in diffs.
 
-    Linguist::FileBlob.new("underscore.min.js").generated? # => true
+```ruby
+Linguist::FileBlob.new("underscore.min.js").generated? # => true
+```
 
-See [Linguist::BlobHelper#generated?](https://github.com/github/linguist/blob/master/lib/linguist/blob_helper.rb).
+See [Linguist::Generated#generated?](https://github.com/github/linguist/blob/master/lib/linguist/generated.rb).
+
+## Overrides
+
+Linguist supports custom overrides for language definitions and vendored paths. Add a `.gitattributes` file to your project using the keys `linguist-language` and `linguist-vendored` with the standard git-style path matchers for the files you want to override.
+
+Please note that the overrides currently only affect the language statistics for a repository and not the syntax-highlighting of files.
+
+```
+$ cat .gitattributes
+*.rb linguist-language=Java
+
+$ linguist --breakdown
+100.00% Java
+
+Java:
+ruby_file.rb
+```
+
+By default, Linguist treats all of the paths defined in [lib/linguist/vendor.yml](https://github.com/github/linguist/blob/master/lib/linguist/vendor.yml) as vendored and therefore doesn't include them in the language statistics for a repository. Use the `linguist-vendored` attribute to vendor or un-vendor paths.
+
+```
+$ cat .gitattributes
+special-vendored-path/* linguist-vendored
+jquery.js linguist-vendored=false
+```
 
 ## Installation
 
-To get it, clone the repo and run [Bundler](http://gembundler.com/) to install its dependencies.
+Github.com is usually running the latest version of the `github-linguist` gem that is released on [RubyGems.org](http://rubygems.org/gems/github-linguist).
+
+But for development you are going to want to checkout out the source. To get it, clone the repo and run [Bundler](http://gembundler.com/) to install its dependencies.
 
     git clone https://github.com/github/linguist.git
     cd linguist/
@@ -80,17 +147,50 @@ To run the tests:
 
     bundle exec rake test
 
-*Since this code is specific to GitHub, is not published as a official rubygem.*
+### A note on language extensions
 
-If you are seeing errors like `StandardError: could not find any magic files!`, it means the CharlockHolmes gem didn’t install correctly. See the [installing section](https://github.com/brianmario/charlock_holmes/blob/master/README.md) of the CharlockHolmes README for more information.
+Linguist has a number of methods available to it for identifying the language of a particular file. The initial lookup is based upon the extension of the file, possible file extensions are defined in an array called `extensions`. Take a look at this example for example for `Perl`:
 
-## Contributing
+```
+Perl:
+  type: programming
+  ace_mode: perl
+  color: "#0298c3"
+  extensions:
+  - .pl
+  - .PL
+  - .perl
+  - .ph
+  - .plx
+  - .pm
+  - .pod
+  - .psgi
+  interpreters:
+  - perl
+```
+Any of the extensions defined are valid but the first in this array should be the most popular.
 
-1. Fork it.
-2. Create a branch (`git checkout -b detect-foo-language`)
-3. Make your changes
-4. Run the tests (`bundle install` then `bundle exec rake`)
-5. Commit your changes (`git commit -am "Added detection for the new Foo language"`)
-6. Push to the branch (`git push origin detect-foo-language`)
-7. Create a [Pull Request](http://help.github.com/pull-requests/) from your branch.
-8. Promote it. Get others to drop in and +1 it.
+### Testing
+
+Sometimes getting the tests running can be too much work, especially if you don't have much Ruby experience. It's okay: be lazy and let our build bot [Travis](http://travis-ci.org/#!/github/linguist) run the tests for you. Just open a pull request and the bot will start cranking away.
+
+Here's our current build status, which is hopefully green: [![Build Status](https://secure.travis-ci.org/github/linguist.png?branch=master)](http://travis-ci.org/github/linguist)
+
+### Releasing
+
+If you are the current maintainer of this gem:
+
+ 0. Create a branch for the release: `git checkout -b cut-release-vxx.xx.xx`
+ 0. Make sure your local dependencies are up to date: `bundle install`
+ 0. Ensure that samples are updated: `bundle exec rake samples`
+ 0. Ensure that tests are green: `bundle exec rake test`
+ 0. Bump gem version in `lib/linguist/version.rb`.  For example, [like this](https://github.com/github/linguist/commit/8d2ea90a5ba3b2fe6e1508b7155aa4632eea2985).
+ 0. Make a PR to github/linguist.  For example, [#1238](https://github.com/github/linguist/pull/1238).
+ 0. Build a local gem: `bundle exec rake build_gem`
+ 0. Testing:
+   0. Bump the Gemfile and Gemfile.lock versions for an app which relies on this gem
+   0. Install the new gem locally
+   0. Test behavior locally, branch deploy, whatever needs to happen
+ 0. Merge github/linguist PR
+ 0. Tag and push: `git tag vx.xx.xx; git push --tags`
+ 0. Push to rubygems.org -- `gem push github-linguist-3.0.0.gem`

Rakefile

@@ -1,7 +1,154 @@
+require 'bundler/setup'
+require 'rake/clean'
 require 'rake/testtask'
+require 'yaml'
+require 'yajl'
 
 task :default => :test
 
-Rake::TestTask.new do |t|
-  t.warning = true
+Rake::TestTask.new
+
+# Extend test task to check for samples
+task :test => :check_samples
+
+desc "Check that we have samples.json generated"
+task :check_samples do
+  unless File.exist?('lib/linguist/samples.json')
+    Rake::Task[:samples].invoke
+  end
+end
+
+task :samples do
+  require 'linguist/samples'
+  json = Yajl.dump(Linguist::Samples.data, :pretty => true)
+  File.write 'lib/linguist/samples.json', json
+end
+
+task :build_gem => :samples do
+  languages = YAML.load_file("lib/linguist/languages.yml")
+  File.write("lib/linguist/languages.json", Yajl.dump(languages))
+  `gem build github-linguist.gemspec`
+  File.delete("lib/linguist/languages.json")
+end
+
+task :build_grammars_gem do
+  rm_rf "grammars"
+  sh "script/download-grammars"
+  sh "gem", "build", "github-linguist-grammars.gemspec"
+end
+
+namespace :benchmark do
+  benchmark_path = "benchmark/results"
+
+  # $ bundle exec rake benchmark:generate CORPUS=path/to/samples
+  desc "Generate results for"
+  task :generate do
+    ref = `git rev-parse HEAD`.strip[0,8]
+
+    corpus = File.expand_path(ENV["CORPUS"] || "samples")
+
+    require 'linguist/language'
+
+    results = Hash.new
+    Dir.glob("#{corpus}/**/*").each do |file|
+      next unless File.file?(file)
+      filename = file.gsub("#{corpus}/", "")
+      results[filename] = Linguist::FileBlob.new(file).language
+    end
+
+    # Ensure results directory exists
+    FileUtils.mkdir_p("benchmark/results")
+
+    # Write results
+    if `git status`.include?('working directory clean')
+      result_filename = "benchmark/results/#{File.basename(corpus)}-#{ref}.json"
+    else
+      result_filename = "benchmark/results/#{File.basename(corpus)}-#{ref}-unstaged.json"
+    end
+
+    File.write(result_filename, results.to_json)
+    puts "wrote #{result_filename}"
+  end
+
+  # $ bundle exec rake benchmark:compare REFERENCE=path/to/reference.json CANDIDATE=path/to/candidate.json
+  desc "Compare results"
+  task :compare do
+    reference_file = ENV["REFERENCE"]
+    candidate_file = ENV["CANDIDATE"]
+
+    reference = Yajl.load(File.read(reference_file))
+    reference_counts = Hash.new(0)
+    reference.each { |filename, language| reference_counts[language] += 1 }
+
+    candidate = Yajl.load(File.read(candidate_file))
+    candidate_counts = Hash.new(0)
+    candidate.each { |filename, language| candidate_counts[language] += 1 }
+
+    changes = diff(reference_counts, candidate_counts)
+
+    if changes.any?
+      changes.each do |language, (before, after)|
+        before_percent = 100 * before / reference.size.to_f
+        after_percent = 100 * after / candidate.size.to_f
+        puts "%s changed from %.1f%% to %.1f%%" % [language || 'unknown', before_percent, after_percent]
+      end
+    else
+      puts "No changes"
+    end
+  end
+end
+
+namespace :classifier do
+  LIMIT = 1_000
+
+  desc "Run classifier against #{LIMIT} public gists"
+  task :test do
+    require 'linguist/classifier'
+    require 'linguist/samples'
+
+    total, correct, incorrect = 0, 0, 0
+    $stdout.sync = true
+
+    each_public_gist do |gist_url, file_url, file_language|
+      next if file_language.nil? || file_language == 'Text'
+      begin
+        data = open(file_url).read
+        guessed_language, score = Linguist::Classifier.classify(Linguist::Samples.cache, data).first
+
+        total += 1
+        guessed_language == file_language ? correct += 1 : incorrect += 1
+
+        print "\r\e[0K%d:%d  %g%%" % [correct, incorrect, (correct.to_f/total.to_f)*100]
+        $stdout.flush
+      rescue URI::InvalidURIError
+      else
+        break if total >= LIMIT
+      end
+    end
+    puts ""
+  end
+
+  def each_public_gist
+    require 'open-uri'
+    url = "https://api.github.com/gists/public"
+
+    loop do
+      resp = open(url)
+      url = resp.meta['link'][/<([^>]+)>; rel="next"/, 1]
+      gists = Yajl.load(resp.read)
+
+      for gist in gists
+        for filename, attrs in gist['files']
+          yield gist['url'], attrs['raw_url'], attrs['language']
+        end
+      end
+    end
+  end
+end
+
+
+def diff(a, b)
+  (a.keys | b.keys).each_with_object({}) do |key, diff|
+    diff[key] = [a[key], b[key]] unless a[key] == b[key]
+  end
 end

bin/linguist

@@ -1,15 +1,42 @@
 #!/usr/bin/env ruby
 
+# linguist — detect language type for a file, or, given a directory, determine language breakdown
+#     usage: linguist <path> [<--breakdown>]
+
 require 'linguist/file_blob'
+require 'linguist/language'
 require 'linguist/repository'
+require 'rugged'
 
 path = ARGV[0] || Dir.pwd
 
+# special case if not given a directory but still given the --breakdown option
+if path == "--breakdown"
+  path = Dir.pwd
+  breakdown = true
+end
+
+ARGV.shift
+breakdown = true if ARGV[0] == "--breakdown"
+
 if File.directory?(path)
-  repo = Linguist::Repository.from_directory(path)
+  rugged = Rugged::Repository.new(path)
+  repo = Linguist::Repository.new(rugged, rugged.head.target_id)
   repo.languages.sort_by { |_, size| size }.reverse.each do |language, size|
-    percentage = ((size / repo.size.to_f) * 100).round
-    puts "%-4s %s" % ["#{percentage}%", language]
+    percentage = ((size / repo.size.to_f) * 100)
+    percentage = sprintf '%.2f' % percentage
+    puts "%-7s %s" % ["#{percentage}%", language]
+  end
+  if breakdown
+    puts
+    file_breakdown = repo.breakdown_by_file
+    file_breakdown.each do |lang, files|
+      puts "#{lang}:"
+      files.each do |file|
+        puts file
+      end
+      puts
+    end
   end
 elsif File.file?(path)
   blob = Linguist::FileBlob.new(path, Dir.pwd)
@@ -23,12 +50,11 @@ elsif File.file?(path)
 
   puts "#{blob.name}: #{blob.loc} lines (#{blob.sloc} sloc)"
   puts "  type:      #{type}"
-  puts "  extension: #{blob.pathname.extname}"
   puts "  mime type: #{blob.mime_type}"
   puts "  language:  #{blob.language}"
 
   if blob.large?
-    puts "  blob is to large to be shown"
+    puts "  blob is too large to be shown"
   end
 
   if blob.generated?

github-linguist-grammars.gemspec

@@ -0,0 +1,14 @@
+require File.expand_path('../lib/linguist/version', __FILE__)
+
+Gem::Specification.new do |s|
+  s.name    = 'github-linguist-grammars'
+  s.version = Linguist::VERSION
+  s.summary = "Language grammars for use with github-linguist"
+
+  s.authors  = "GitHub"
+  s.homepage = "https://github.com/github/linguist"
+
+  s.files = ['lib/linguist/grammars.rb'] + Dir['grammars/*']
+
+  s.add_development_dependency 'plist', '~>3.1'
+end

github-linguist.gemspec

@@ -1,16 +1,25 @@
+require File.expand_path('../lib/linguist/version', __FILE__)
+
 Gem::Specification.new do |s|
   s.name    = 'github-linguist'
-  s.version = '1.0.0'
+  s.version = Linguist::VERSION
   s.summary = "GitHub Language detection"
+  s.description = 'We use this library at GitHub to detect blob languages, highlight code, ignore binary files, suppress generated files in diffs, and generate language breakdown graphs.'
 
-  s.authors = "GitHub"
+  s.authors  = "GitHub"
+  s.homepage = "https://github.com/github/linguist"
+  s.license  = "MIT"
 
-  s.files = Dir['lib/**/*']
+  s.files = Dir['lib/**/*'] - ['lib/linguist/grammars.rb']
   s.executables << 'linguist'
 
-  s.add_dependency 'charlock_holmes', '~> 0.6.6'
-  s.add_dependency 'escape_utils',    '~> 0.2.3'
-  s.add_dependency 'mime-types',      '~> 1.18'
-  s.add_dependency 'pygments.rb',     '~> 0.2.13'
+  s.add_dependency 'charlock_holmes', '~> 0.7.3'
+  s.add_dependency 'escape_utils',    '~> 1.0.1'
+  s.add_dependency 'mime-types',      '>= 1.19'
+  s.add_dependency 'rugged',          '~> 0.22.0b4'
+
+  s.add_development_dependency 'mocha'
+  s.add_development_dependency 'pry'
   s.add_development_dependency 'rake'
+  s.add_development_dependency 'yajl-ruby'
 end

grammars.yml

@@ -0,0 +1,422 @@
+---
+http://svn.edgewall.org/repos/genshi/contrib/textmate/Genshi.tmbundle/Syntaxes/Markup%20Template%20%28XML%29.tmLanguage:
+- text.xml.genshi
+http://svn.textmate.org/trunk/Review/Bundles/BlitzMax.tmbundle:
+- source.blitzmax
+http://svn.textmate.org/trunk/Review/Bundles/Cython.tmbundle:
+- source.cython
+http://svn.textmate.org/trunk/Review/Bundles/Forth.tmbundle:
+- source.forth
+http://svn.textmate.org/trunk/Review/Bundles/Parrot.tmbundle:
+- source.parrot.pir
+http://svn.textmate.org/trunk/Review/Bundles/Ruby%20Sass.tmbundle:
+- source.sass
+http://svn.textmate.org/trunk/Review/Bundles/SecondLife%20LSL.tmbundle:
+- source.lsl
+http://svn.textmate.org/trunk/Review/Bundles/VHDL.tmbundle:
+- source.vhdl
+http://svn.textmate.org/trunk/Review/Bundles/XQuery.tmbundle:
+- source.xquery
+https://bitbucket.org/Clams/sublimesystemverilog/get/default.tar.gz:
+- source.systemverilog
+- source.ucfconstraints
+https://bitbucket.org/bitlang/sublime_cobol/raw/b0e9c44ac5f7a2fb553421aa986b35854cbfda4a/COBOL.tmLanguage:
+- source.cobol
+https://fan.googlecode.com/hg-history/Build%201.0.55/adm/tools/textmate/Fan.tmbundle/Syntaxes/Fan.tmLanguage:
+- source.fan
+https://github.com/AlanQuatermain/go-tmbundle:
+- source.go
+https://github.com/Anomareh/PHP-Twig.tmbundle:
+- text.html.twig
+https://github.com/Cirru/sublime-cirru/raw/master/Cirru.tmLanguage:
+- source.cirru
+https://github.com/Cykey/Sublime-Logos:
+- source.logos
+https://github.com/Drako/SublimeBrainfuck/raw/master/Brainfuck.tmLanguage:
+- source.bf
+https://github.com/JohnNilsson/awk-sublime/raw/master/AWK.tmLanguage:
+- source.awk
+https://github.com/JonBons/Sublime-SQF-Language:
+- source.sqf
+https://github.com/MarioRicalde/SCSS.tmbundle:
+- source.scss
+https://github.com/Oldes/Sublime-REBOL:
+- source.rebol
+https://github.com/PogiNate/Sublime-Inform:
+- source.Inform7
+https://github.com/Red-Nova-Technologies/autoitv3-tmbundle:
+- source.autoit.3
+https://github.com/SalGnt/Sublime-VimL:
+- source.viml
+https://github.com/Shammah/boo-sublime/raw/master/Boo.tmLanguage:
+- source.boo
+https://github.com/SublimeText/ColdFusion:
+- source.cfscript
+- source.cfscript.cfc
+- text.cfml.basic
+- text.html.cfm
+https://github.com/SublimeText/NSIS:
+- source.nsis
+https://github.com/Varriount/NimLime:
+- source.nimrod
+- source.nimrod_filter
+- source.nimrodcfg
+https://github.com/alkemist/gradle.tmbundle:
+- source.groovy.gradle
+https://github.com/ambethia/Sublime-Loom:
+- source.loomscript
+https://github.com/angryant0007/VBDotNetSyntax:
+- source.vbnet
+https://github.com/anunayk/cool-tmbundle:
+- source.cool
+https://github.com/aroben/ada.tmbundle/raw/c45eed4d5f98fe3bcbbffbb9e436601ab5bbde4b/Syntaxes/Ada.plist:
+- source.ada
+https://github.com/aroben/ruby.tmbundle@4636a3023153c3034eb6ffc613899ba9cf33b41f:
+- source.ruby
+- text.html.erb
+https://github.com/asbjornenge/Docker.tmbundle:
+- source.dockerfile
+https://github.com/atom/language-clojure:
+- source.clojure
+https://github.com/atom/language-coffee-script:
+- source.coffee
+- source.litcoffee
+https://github.com/atom/language-csharp:
+- source.cs
+- source.csx
+- source.nant-build
+https://github.com/atom/language-javascript:
+- source.js
+- source.js.regexp
+https://github.com/atom/language-python:
+- source.python
+- source.regexp.python
+- text.python.traceback
+https://github.com/atom/language-shellscript:
+- source.shell
+- text.shell-session
+https://github.com/austinwagner/sublime-sourcepawn:
+- source.sp
+https://github.com/bfad/Sublime-Lasso:
+- file.lasso
+https://github.com/bholt/chapel-tmbundle:
+- source.chapel
+https://github.com/brandonwamboldt/sublime-nginx:
+- source.nginx
+https://github.com/bro/bro-sublime:
+- source.bro
+https://github.com/carsonoid/sublime_man_page_support/raw/master/man-groff.tmLanguage:
+- text.groff
+https://github.com/ccreutzig/sublime-MuPAD:
+- source.mupad
+https://github.com/cdwilson/nesC.tmbundle:
+- source.nesc
+https://github.com/christophevg/racket-tmbundle:
+- source.racket
+https://github.com/clemos/haxe-sublime-bundle:
+- source.erazor
+- source.haxe.2
+- source.hss.1
+- source.hxml
+- source.nmml
+https://github.com/cucumber/cucumber-tmbundle:
+- source.ruby.rspec.cucumber.steps
+- text.gherkin.feature
+https://github.com/daaain/Handlebars/raw/master/Handlebars.tmLanguage:
+- text.html.handlebars
+https://github.com/davidpeckham/powershell.tmbundle:
+- source.powershell
+https://github.com/davidrios/jade-tmbundle:
+- source.jade
+- source.pyjade
+https://github.com/elixir-lang/elixir-tmbundle:
+- source.elixir
+- text.elixir
+- text.html.elixir
+https://github.com/ericzou/ebundles/raw/master/Bundles/MSDOS%20batch%20file.tmbundle/Syntaxes/MSDOS%20batch%20file.tmLanguage:
+- source.dosbatch
+https://github.com/euler0/sublime-glsl/raw/master/GLSL.tmLanguage:
+- source.glsl
+https://github.com/fancy-lang/fancy-tmbundle:
+- source.fancy
+https://github.com/fsharp/fsharpbinding:
+- source.fsharp
+https://github.com/gingerbeardman/monkey.tmbundle:
+- source.monkey
+https://github.com/guillermooo/dart-sublime-bundle/raw/master/Dart.tmLanguage:
+- source.dart
+https://github.com/harrism/sublimetext-cuda-cpp/raw/master/cuda-c%2B%2B.tmLanguage:
+- source.cuda-c++
+https://github.com/hww3/pike-textmate:
+- source.pike
+https://github.com/jeancharles-roger/ceylon-sublimetext/raw/master/Ceylon.tmLanguage:
+- source.ceylon
+https://github.com/jfairbank/Sublime-Text-2-OpenEdge-ABL:
+- source.abl
+https://github.com/jhasse/sublime-rust:
+- source.rust
+https://github.com/johanasplund/sublime-befunge/raw/master/Befunge-93.tmLanguage:
+- source.befunge
+https://github.com/joshaven/RDoc.tmbundle:
+- text.rdoc
+https://github.com/jpcamara/Textmate-Gosu-Bundle/raw/master/Gosu.tmbundle/Syntaxes/Gosu.tmLanguage:
+- source.gosu.2
+https://github.com/kswedberg/jquery-tmbundle:
+- source.js.jquery
+https://github.com/laughedelic/sublime-idris/raw/master/Idris.tmLanguage:
+- source.idris
+https://github.com/lavrton/sublime-better-typescript:
+- source.ts
+https://github.com/leafo/moonscript-tmbundle:
+- source.moonscript
+https://github.com/lsf37/Isabelle.tmbundle:
+- source.isabelle.theory
+https://github.com/lunixbochs/x86-assembly-textmate-bundle:
+- source.asm.x86
+https://github.com/macekond/Alloy.tmbundle:
+- source.alloy
+https://github.com/mads379/opa.tmbundle:
+- source.opa
+https://github.com/mads379/scala.tmbundle:
+- source.sbt
+- source.scala
+https://github.com/marconi/mako-tmbundle:
+- text.html.mako
+https://github.com/mattfoster/gnuplot-tmbundle:
+- source.gnuplot
+https://github.com/mgalloy/idl.tmbundle:
+- source.idl
+- source.idl-dlm
+- text.idl-idldoc
+https://github.com/michaeledgar/protobuf-tmbundle:
+- source.protobuf
+https://github.com/mkolosick/Sublime-Coq/raw/master/Coq.tmLanguage:
+- source.coq
+https://github.com/mokus0/Agda.tmbundle:
+- source.agda
+https://github.com/nanoant/Julia.tmbundle:
+- source.julia
+https://github.com/nanoant/assembly.tmbundle/raw/master/Syntaxes/objdump%20C%2B%2B.tmLanguage:
+- objdump.x86asm
+https://github.com/nilium/ooc.tmbundle:
+- source.ooc
+https://github.com/paulmillr/LiveScript.tmbundle:
+- source.livescript
+https://github.com/pferruggiaro/sublime-tea:
+- source.tea
+https://github.com/puppet-textmate-bundle/puppet-textmate-bundle:
+- source.puppet
+https://github.com/pvl/abap.tmbundle:
+- source.abap
+https://github.com/scalate/Scalate.tmbundle:
+- source.scaml
+- text.html.ssp
+https://github.com/shadanan/mathematica-tmbundle:
+- source.mathematica
+https://github.com/shellderp/sublime-robot-plugin:
+- text.robot
+https://github.com/simongregory/actionscript3-tmbundle:
+- source.actionscript.3
+- text.html.asdoc
+- text.xml.flex-config
+https://github.com/skozlovf/Sublime-QML:
+- source.qml
+https://github.com/slash-lang/Slash.tmbundle:
+- text.html.slash
+https://github.com/slavapestov/factor/raw/master/misc/Factor.tmbundle/Syntaxes/Factor.tmLanguage:
+- source.factor
+https://github.com/slim-template/ruby-slim.tmbundle:
+- text.slim
+https://github.com/staltz/SublimeXtend:
+- source.xtend
+https://github.com/statatmbundle/Stata.tmbundle:
+- source.mata
+- source.stata
+https://github.com/technosophos/Vala-TMBundle:
+- source.vala
+https://github.com/textmate/ant.tmbundle:
+- text.xml.ant
+https://github.com/textmate/antlr.tmbundle:
+- source.antlr
+https://github.com/textmate/apache.tmbundle:
+- source.apache-config
+- source.apache-config.mod_perl
+https://github.com/textmate/applescript.tmbundle:
+- source.applescript
+https://github.com/textmate/asp.tmbundle:
+- source.asp
+- text.html.asp
+https://github.com/textmate/bison.tmbundle:
+- source.bison
+https://github.com/textmate/c.tmbundle:
+- source.c
+- source.c++
+- source.c.platform
+https://github.com/textmate/capnproto.tmbundle:
+- source.capnp
+https://github.com/textmate/cmake.tmbundle:
+- source.cache.cmake
+- source.cmake
+https://github.com/textmate/cpp-qt.tmbundle:
+- source.c++.qt
+- source.qmake
+https://github.com/textmate/css.tmbundle:
+- source.css
+https://github.com/textmate/d.tmbundle:
+- source.d
+https://github.com/textmate/diff.tmbundle:
+- source.diff
+https://github.com/textmate/dylan.tmbundle:
+- source.dylan
+- source.lid
+- source.makegen
+https://github.com/textmate/eiffel.tmbundle:
+- source.eiffel
+https://github.com/textmate/erlang.tmbundle:
+- source.erlang
+- text.html.erlang.yaws
+https://github.com/textmate/fortran.tmbundle:
+- source.fortran
+- source.fortran.modern
+https://github.com/textmate/gettext.tmbundle:
+- source.po
+https://github.com/textmate/graphviz.tmbundle:
+- source.dot
+https://github.com/textmate/groovy.tmbundle:
+- source.groovy
+https://github.com/textmate/haskell.tmbundle:
+- source.haskell
+- text.tex.latex.haskell
+https://github.com/textmate/html.tmbundle:
+- text.html.basic
+https://github.com/textmate/ini.tmbundle:
+- source.ini
+https://github.com/textmate/io.tmbundle:
+- source.io
+https://github.com/textmate/java.tmbundle:
+- source.java
+- source.java-properties
+- text.html.jsp
+- text.junit-test-report
+https://github.com/textmate/javadoc.tmbundle:
+- text.html.javadoc
+https://github.com/textmate/javascript-objective-j.tmbundle:
+- source.js.objj
+https://github.com/textmate/json.tmbundle:
+- source.json
+https://github.com/textmate/latex.tmbundle:
+- text.bibtex
+- text.log.latex
+- text.tex
+- text.tex.latex
+- text.tex.latex.beamer
+- text.tex.latex.memoir
+https://github.com/textmate/less.tmbundle:
+- source.css.less
+https://github.com/textmate/lilypond.tmbundle:
+- source.lilypond
+https://github.com/textmate/lisp.tmbundle:
+- source.lisp
+https://github.com/textmate/logtalk.tmbundle:
+- source.logtalk
+https://github.com/textmate/lua.tmbundle:
+- source.lua
+https://github.com/textmate/make.tmbundle:
+- source.makefile
+https://github.com/textmate/markdown.tmbundle:
+- text.html.markdown
+https://github.com/textmate/matlab.tmbundle:
+- source.matlab
+- source.octave
+https://github.com/textmate/maven.tmbundle:
+- text.xml.pom
+https://github.com/textmate/nemerle.tmbundle:
+- source.nemerle
+https://github.com/textmate/ninja.tmbundle:
+- source.ninja
+https://github.com/textmate/objective-c.tmbundle:
+- source.objc
+- source.objc++
+- source.objc.platform
+- source.strings
+https://github.com/textmate/ocaml.tmbundle:
+- source.camlp4.ocaml
+- source.ocaml
+- source.ocamllex
+- source.ocamlyacc
+https://github.com/textmate/pascal.tmbundle:
+- source.pascal
+https://github.com/textmate/perl.tmbundle:
+- source.perl
+https://github.com/textmate/php-smarty.tmbundle:
+- source.smarty
+https://github.com/textmate/php.tmbundle:
+- text.html.php
+https://github.com/textmate/postscript.tmbundle:
+- source.postscript
+https://github.com/textmate/processing.tmbundle:
+- source.processing
+https://github.com/textmate/prolog.tmbundle:
+- source.prolog
+https://github.com/textmate/python-django.tmbundle:
+- source.python.django
+- text.html.django
+https://github.com/textmate/r.tmbundle:
+- source.r
+- text.tex.latex.rd
+https://github.com/textmate/restructuredtext.tmbundle:
+- text.restructuredtext
+https://github.com/textmate/ruby-haml.tmbundle:
+- text.haml
+https://github.com/textmate/ruby-on-rails-tmbundle:
+- source.js.erb.rails
+- source.ruby.rails
+- source.ruby.rails.rjs
+- source.sql.ruby
+- text.html.erb.rails
+https://github.com/textmate/scheme.tmbundle:
+- source.scheme
+https://github.com/textmate/scilab.tmbundle:
+- source.scilab
+https://github.com/textmate/sql.tmbundle:
+- source.sql
+https://github.com/textmate/standard-ml.tmbundle:
+- source.cm
+- source.ml
+https://github.com/textmate/swift.tmbundle:
+- source.swift
+https://github.com/textmate/tcl.tmbundle:
+- source.tcl
+- text.html.tcl
+https://github.com/textmate/text.tmbundle:
+- text.plain
+https://github.com/textmate/textile.tmbundle:
+- text.html.textile
+https://github.com/textmate/textmate.tmbundle:
+- source.regexp.oniguruma
+- source.tm-properties
+https://github.com/textmate/thrift.tmbundle:
+- source.thrift
+https://github.com/textmate/toml.tmbundle:
+- source.toml
+https://github.com/textmate/verilog.tmbundle:
+- source.verilog
+https://github.com/textmate/xml.tmbundle:
+- text.xml
+- text.xml.xsl
+https://github.com/textmate/yaml.tmbundle:
+- source.yaml
+https://github.com/tomas-stefano/smalltalk-tmbundle:
+- source.smalltalk
+https://github.com/vic/ioke-outdated/raw/master/share/TextMate/Ioke.tmbundle/Syntaxes/Ioke.tmLanguage:
+- source.ioke
+https://github.com/vkostyukov/kotlin-sublime-package:
+- source.Kotlin
+https://github.com/vmg/zephir-sublime:
+- source.php.zephir
+https://github.com/whitequark/llvm.tmbundle:
+- source.llvm
+https://github.com/wmertens/sublime-nix:
+- source.nix
+https://raw.githubusercontent.com/eregon/oz-tmbundle/master/Syntaxes/Oz.tmLanguage:
+- source.oz

lib/linguist.rb

@@ -1,5 +1,8 @@
 require 'linguist/blob_helper'
+require 'linguist/generated'
+require 'linguist/heuristics'
 require 'linguist/language'
-require 'linguist/mime'
-require 'linguist/pathname'
 require 'linguist/repository'
+require 'linguist/samples'
+require 'linguist/shebang'
+require 'linguist/version'

lib/linguist/blob_helper.rb

@@ -1,23 +1,19 @@
-require 'linguist/language'
-require 'linguist/mime'
-require 'linguist/pathname'
-
+require 'linguist/generated'
 require 'charlock_holmes'
 require 'escape_utils'
-require 'pygments'
+require 'mime/types'
 require 'yaml'
 
 module Linguist
+  # DEPRECATED Avoid mixing into Blob classes. Prefer functional interfaces
+  # like `Language.detect` over `Blob#language`. Functions are much easier to
+  # cache and compose.
+  #
+  # Avoid adding additional bloat to this module.
+  #
   # BlobHelper is a mixin for Blobish classes that respond to "name",
   # "data" and "size" such as Grit::Blob.
   module BlobHelper
-    # Internal: Get a Pathname wrapper for Blob#name
-    #
-    # Returns a Pathname.
-    def pathname
-      Pathname.new(name || "")
-    end
-
     # Public: Get the extname of the path
     #
     # Examples
@@ -27,7 +23,23 @@ module Linguist
     #
     # Returns a String
     def extname
-      pathname.extname
+      File.extname(name.to_s)
+    end
+
+    # Internal: Lookup mime type for extension.
+    #
+    # Returns a MIME::Type
+    def _mime_type
+      if defined? @_mime_type
+        @_mime_type
+      else
+        guesses = ::MIME::Types.type_for(extname.to_s)
+
+        # Prefer text mime types over binary
+        @_mime_type = guesses.detect { |type| type.ascii? } ||
+          # Otherwise use the first guess
+          guesses.first
+      end
     end
 
     # Public: Get the actual blob mime type
@@ -39,7 +51,23 @@ module Linguist
     #
     # Returns a mime type String.
     def mime_type
-      @mime_type ||= pathname.mime_type
+      _mime_type ? _mime_type.to_s : 'text/plain'
+    end
+
+    # Internal: Is the blob binary according to its mime type
+    #
+    # Return true or false
+    def binary_mime_type?
+      _mime_type ? _mime_type.binary? : false
+    end
+
+    # Internal: Is the blob binary according to its mime type,
+    # overriding it if we have better data from the languages.yml
+    # database.
+    #
+    # Return true or false
+    def likely_binary?
+      binary_mime_type? && !Language.find_by_filename(name)
     end
 
     # Public: Get the Content-Type header value
@@ -71,7 +99,7 @@ module Linguist
       elsif name.nil?
         "attachment"
       else
-        "attachment; filename=#{EscapeUtils.escape_url(pathname.basename)}"
+        "attachment; filename=#{EscapeUtils.escape_url(File.basename(name))}"
       end
     end
 
@@ -81,6 +109,12 @@ module Linguist
       end
     end
 
+    def ruby_encoding
+      if hash = detect_encoding
+        hash[:ruby_encoding]
+      end
+    end
+
     # Try to guess the encoding
     #
     # Returns: a Hash, with :encoding, :confidence, :type
@@ -90,15 +124,6 @@ module Linguist
       @detect_encoding ||= CharlockHolmes::EncodingDetector.new.detect(data) if data
     end
 
-    # Public: Is the blob binary according to its mime type
-    #
-    # Return true or false
-    def binary_mime_type?
-      if mime_type = Mime.lookup_mime_type_for(pathname.extname)
-        mime_type.binary?
-      end
-    end
-
     # Public: Is the blob binary?
     #
     # Return true or false
@@ -121,6 +146,13 @@ module Linguist
       end
     end
 
+    # Public: Is the blob empty?
+    #
+    # Return true or false
+    def empty?
+      data.nil? || data == ""
+    end
+
     # Public: Is the blob text?
     #
     # Return true or false
@@ -132,23 +164,28 @@ module Linguist
     #
     # Return true or false
     def image?
-      ['.png', '.jpg', '.jpeg', '.gif'].include?(extname)
+      ['.png', '.jpg', '.jpeg', '.gif'].include?(extname.downcase)
     end
 
-    # Public: Is the blob a possible drupal php file?
+    # Public: Is the blob a supported 3D model format?
     #
     # Return true or false
-    def drupal_extname?
-      ['.module', '.install', '.test', '.inc'].include?(extname)
+    def solid?
+      extname.downcase == '.stl'
     end
 
-    # Public: Is the blob likely to have a shebang?
+    # Public: Is this blob a CSV file?
     #
     # Return true or false
-    def shebang_extname?
-      extname.empty? &&
-        mode &&
-        (mode.to_i(8) & 05) == 05
+    def csv?
+      text? && extname.downcase == '.csv'
+    end
+
+    # Public: Is the blob a PDF?
+    #
+    # Return true or false
+    def pdf?
+      extname.downcase == '.pdf'
     end
 
     MEGABYTE = 1024 * 1024
@@ -162,21 +199,13 @@ module Linguist
 
     # Public: Is the blob safe to colorize?
     #
-    # We use Pygments.rb for syntax highlighting blobs, which
-    # has some quirks and also is essentially 'un-killable' via
-    # normal timeout.  To workaround this we try to
-    # carefully handling Pygments.rb anything it can't handle.
-    #
     # Return true or false
     def safe_to_colorize?
-      text? && !large? && !high_ratio_of_long_lines?
+      !large? && text? && !high_ratio_of_long_lines?
     end
 
     # Internal: Does the blob have a ratio of long lines?
     #
-    # These types of files are usually going to make Pygments.rb
-    # angry if we try to colorize them.
-    #
     # Return true or false
     def high_ratio_of_long_lines?
       return false if loc == 0
@@ -213,7 +242,36 @@ module Linguist
     #
     # Returns an Array of lines
     def lines
-      @lines ||= (viewable? && data) ? data.split("\n", -1) : []
+      @lines ||=
+        if viewable? && data
+          # `data` is usually encoded as ASCII-8BIT even when the content has
+          # been detected as a different encoding. However, we are not allowed
+          # to change the encoding of `data` because we've made the implicit
+          # guarantee that each entry in `lines` is encoded the same way as
+          # `data`.
+          #
+          # Instead, we re-encode each possible newline sequence as the
+          # detected encoding, then force them back to the encoding of `data`
+          # (usually a binary encoding like ASCII-8BIT). This means that the
+          # byte sequence will match how newlines are likely encoded in the
+          # file, but we don't have to change the encoding of `data` as far as
+          # Ruby is concerned. This allows us to correctly parse out each line
+          # without changing the encoding of `data`, and
+          # also--importantly--without having to duplicate many (potentially
+          # large) strings.
+          begin
+            encoded_newlines = ["\r\n", "\r", "\n"].
+              map { |nl| nl.encode(ruby_encoding, "ASCII-8BIT").force_encoding(data.encoding) }
+
+            data.split(Regexp.union(encoded_newlines), -1)
+          rescue Encoding::ConverterNotFoundError
+            # The data is not splittable in the detected encoding.  Assume it's
+            # one big line.
+            [data]
+          end
+        else
+          []
+        end
     end
 
     # Public: Get number of lines of code
@@ -234,171 +292,16 @@ module Linguist
       lines.grep(/\S/).size
     end
 
-    # Internal: Compute average line length.
-    #
-    # Returns Integer.
-    def average_line_length
-      if lines.any?
-        lines.inject(0) { |n, l| n += l.length } / lines.length
-      else
-        0
-      end
-    end
-
     # Public: Is the blob a generated file?
     #
-    # Generated source code is supressed in diffs and is ignored by
+    # Generated source code is suppressed in diffs and is ignored by
     # language statistics.
     #
-    # Requires Blob#data
-    #
-    # Includes:
-    # - XCode project XML files
-    # - Minified JavaScript
-    # - Compiled CoffeeScript
-    # - PEG.js-generated parsers
-    #
-    # Please add additional test coverage to
-    # `test/test_blob.rb#test_generated` if you make any changes.
+    # May load Blob#data
     #
     # Return true or false
     def generated?
-      if name == 'Gemfile.lock' || minified_javascript? || compiled_coffeescript? ||
-      xcode_project_file? || generated_net_docfile? || generated_parser?
-        true
-      else
-        false
-      end
-    end
-
-    # Internal: Is the blob an XCode project file?
-    #
-    # Generated if the file extension is an XCode project
-    # file extension.
-    #
-    # Returns true of false.
-    def xcode_project_file?
-      ['.xib', '.nib', '.storyboard', '.pbxproj', '.xcworkspacedata', '.xcuserstate'].include?(extname)
-    end
-
-    # Internal: Is the blob minified JS?
-    #
-    # Consider JS minified if the average line length is
-    # greater then 100c.
-    #
-    # Returns true or false.
-    def minified_javascript?
-      return unless extname == '.js'
-      average_line_length > 100
-    end
-
-    # Internal: Is the blob of JS a parser generated by PEG.js?
-    #
-    # Requires Blob#data
-    #
-    # PEG.js-generated parsers are not meant to be consumed by humans.
-    #
-    # Return true or false
-    def generated_parser?
-      return false unless extname == '.js'
-
-      # PEG.js-generated parsers include a comment near the top  of the file
-      # that marks them as such.
-      if lines[0..4].join('') =~ /^(?:[^\/]|\/[^\*])*\/\*(?:[^\*]|\*[^\/])*Generated by PEG.js/
-        return true
-      end
-
-      false
-    end
-
-    # Internal: Is the blob of JS generated by CoffeeScript?
-    #
-    # Requires Blob#data
-    #
-    # CoffeScript is meant to output JS that would be difficult to
-    # tell if it was generated or not. Look for a number of patterns
-    # output by the CS compiler.
-    #
-    # Return true or false
-    def compiled_coffeescript?
-      return false unless extname == '.js'
-
-      # CoffeeScript generated by > 1.2 include a comment on the first line
-      if lines[0] =~ /^\/\/ Generated by /
-        return true
-      end
-
-      if lines[0] == '(function() {' &&     # First line is module closure opening
-          lines[-2] == '}).call(this);' &&  # Second to last line closes module closure
-          lines[-1] == ''                   # Last line is blank
-
-        score = 0
-
-        lines.each do |line|
-          if line =~ /var /
-            # Underscored temp vars are likely to be Coffee
-            score += 1 * line.gsub(/(_fn|_i|_len|_ref|_results)/).count
-
-            # bind and extend functions are very Coffee specific
-            score += 3 * line.gsub(/(__bind|__extends|__hasProp|__indexOf|__slice)/).count
-          end
-        end
-
-        # Require a score of 3. This is fairly arbitrary. Consider
-        # tweaking later.
-        score >= 3
-      else
-        false
-      end
-    end
-
-    # Internal: Is this a generated documentation file for a .NET assembly?
-    #
-    # Requires Blob#data
-    #
-    # .NET developers often check in the XML Intellisense file along with an
-    # assembly - however, these don't have a special extension, so we have to
-    # dig into the contents to determine if it's a docfile. Luckily, these files
-    # are extremely structured, so recognizing them is easy.
-    #
-    # Returns true or false
-    def generated_net_docfile?
-      return false unless extname.downcase == ".xml"
-      return false unless lines.count > 3
-
-      # .NET Docfiles always open with <doc> and their first tag is an
-      # <assembly> tag
-      return lines[1].include?("<doc>") &&
-        lines[2].include?("<assembly>") &&
-        lines[-2].include?("</doc>")
-    end
-
-    # Public: Should the blob be indexed for searching?
-    #
-    # Excluded:
-    # - Files over 0.1MB
-    # - Non-text files
-    # - Langauges marked as not searchable
-    # - Generated source files
-    #
-    # Please add additional test coverage to
-    # `test/test_blob.rb#test_indexable` if you make any changes.
-    #
-    # Return true or false
-    def indexable?
-      if binary?
-        false
-      elsif language.nil?
-        false
-      elsif !language.searchable?
-        false
-      elsif generated?
-        false
-      elsif size > 100 * 1024
-        false
-      else
-        true
-      end
+      @_generated ||= Generated.generated?(name, lambda { data })
     end
 
     # Public: Detects the Language of the blob.
@@ -407,307 +310,12 @@ module Linguist
     #
     # Returns a Language or nil if none is detected
     def language
-      if defined? @language
-        @language
-      else
-        @language = guess_language
-      end
+      @language ||= Language.detect(self)
     end
 
-    # Internal: Guess language
-    #
-    # Please add additional test coverage to
-    # `test/test_blob.rb#test_language` if you make any changes.
-    #
-    # Returns a Language or nil
-    def guess_language
-      return if binary_mime_type?
-
-      # Disambiguate between multiple language extensions
-      disambiguate_extension_language ||
-
-        # See if there is a Language for the extension
-        pathname.language ||
-
-        # Look for idioms in first line
-        first_line_language ||
-
-        # Try to detect Language from shebang line
-        shebang_language
-    end
-
-    # Internal: Get the lexer of the blob.
-    #
-    # Returns a Lexer.
-    def lexer
-      language ? language.lexer : Pygments::Lexer.find_by_name('Text only')
-    end
-
-    # Internal: Disambiguates between multiple language extensions.
-    #
-    # Delegates to "guess_EXTENSION_language".
-    #
-    # Please add additional test coverage to
-    # `test/test_blob.rb#test_language` if you add another method.
-    #
-    # Returns a Language or nil.
-    def disambiguate_extension_language
-      if Language.ambiguous?(extname)
-        name = "guess_#{extname.sub(/^\./, '')}_language"
-        send(name) if respond_to?(name)
-      end
-    end
-
-    # Internal: Guess language of .cls files
-    #
-    # Returns a Language.
-    def guess_cls_language
-      if lines.grep(/^(%|\\)/).any?
-        Language['TeX']
-      elsif lines.grep(/^\s*(CLASS|METHOD|INTERFACE).*:\s*/i).any? || lines.grep(/^\s*(USING|DEFINE)/i).any?
-        Language['OpenEdge ABL']
-      elsif lines.grep(/\{$/).any? || lines.grep(/\}$/).any?
-        Language['Apex']
-      elsif lines.grep(/^(\'\*|Attribute|Option|Sub|Private|Protected|Public|Friend)/i).any?
-        Language['Visual Basic']
-      else
-        # The most common language should be the fallback
-        Language['TeX']
-      end
-    end
-
-    # Internal: Guess language of header files (.h).
-    #
-    # Returns a Language.
-    def guess_h_language
-      if lines.grep(/^@(interface|property|private|public|end)/).any?
-        Language['Objective-C']
-      elsif lines.grep(/^class |^\s+(public|protected|private):/).any?
-        Language['C++']
-      else
-        Language['C']
-      end
-    end
-
-    # Internal: Guess language of .m files.
-    #
-    # Objective-C heuristics:
-    # * Keywords  ("#import", "#include", "#ifdef", #define, "@end") or "//" and opening "\*" comments
-    #
-    # Matlab heuristics:
-    # * Leading "function " of "classdef " keyword
-    # * "%" comments
-    #
-    # Note: All "#" keywords, e.g., "#import", are guaranteed to be Objective-C. Because the ampersand
-    # is used to created function handles and anonymous functions in Matlab, most "@" keywords are not
-    # safe heuristics. However, "end" is a reserved term in Matlab and can't be used to create a valid
-    # function handle. Because @end is required to close any @implementation, @property, @interface,
-    # @synthesize, etc. directive in Objective-C, only @end needs to be checked for.
-    #
-    # Returns a Language.
-    def guess_m_language
-      # Objective-C keywords or comments
-      if lines.grep(/^#(import|include|ifdef|define)|@end/).any? || lines.grep(/^\s*\/\//).any? || lines.grep(/^\s*\/\*/).any?
-        Language['Objective-C']
-
-      # Matlab file function or class or comments
-      elsif lines.any? && lines.first.match(/^\s*(function |classdef )/) || lines.grep(/^\s*%/).any?
-        Language['Matlab']
-
-      # Fallback to Objective-C, don't want any Matlab false positives
-      else
-        Language['Objective-C']
-      end
-    end
-
-    # Internal: Guess language of .pl files
-    #
-    # The rules for disambiguation are:
-    #
-    # 1. Many perl files begin with a shebang
-    # 2. Most Prolog source files have a rule somewhere (marked by the :- operator)
-    # 3. Default to Perl, because it is more popular
-    #
-    # Returns a Language.
-    def guess_pl_language
-      if shebang_script == 'perl'
-        Language['Perl']
-      elsif lines.grep(/:-/).any?
-        Language['Prolog']
-      else
-        Language['Perl']
-      end
-    end
-
-    # Internal: Guess language of .r files.
-    #
-    # Returns a Language.
-    def guess_r_language
-      if lines.grep(/(rebol|(:\s+func|make\s+object!|^\s*context)\s*\[)/i).any?
-        Language['Rebol']
-      else
-        Language['R']
-      end
-    end
-
-    # Internal: Guess language of .t files.
-    #
-    # Returns a Language.
-    def guess_t_language
-      score = 0
-      score += 1 if lines.grep(/^% /).any?
-      score += data.gsub(/ := /).count
-      score += data.gsub(/proc |procedure |fcn |function /).count
-      score += data.gsub(/var \w+: \w+/).count
-
-      # Tell-tale signs its gotta be Perl
-      if lines.grep(/^(my )?(sub |\$|@|%)\w+/).any?
-        score = 0
-      end
-
-      if score >= 3
-        Language['Turing']
-      else
-        Language['Perl']
-      end
-    end
-
-    # Internal: Guess language of .v files.
-    #
-    # Returns a Language
-    def guess_v_language
-      if lines.grep(/^(\/\*|\/\/|module|parameter|input|output|wire|reg|always|initial|begin|\`)/).any?
-        Language['Verilog']
-      else
-        Language['Coq']
-      end
-    end
-
-    # Internal: Guess language of .gsp files.
-    #
-    # Returns a Language.
-    def guess_gsp_language
-      if lines.grep(/<%|<%@|\$\{|<%|<g:|<meta name="layout"|<r:/).any?
-        Language['Groovy Server Pages']
-      else
-        Language['Gosu']
-      end
-    end
-
-    # Internal: Guess language from the first line.
-    #
-    # Look for leading "<?php" in Drupal files
-    #
-    # Returns a Language.
-    def first_line_language
-      # Only check files with drupal php extensions
-      return unless drupal_extname?
-
-      # Fail fast if blob isn't viewable?
-      return unless viewable?
-
-      if lines.first.to_s =~ /^<\?php/
-        Language['PHP']
-      end
-    end
-
-    # Internal: Extract the script name from the shebang line
-    #
-    # Requires Blob#data
-    #
-    # Examples
-    #
-    #   '#!/usr/bin/ruby'
-    #   # => 'ruby'
-    #
-    #   '#!/usr/bin/env ruby'
-    #   # => 'ruby'
-    #
-    #   '#!/usr/bash/python2.4'
-    #   # => 'python'
-    #
-    # Please add additional test coverage to
-    # `test/test_blob.rb#test_shebang_script` if you make any changes.
-    #
-    # Returns a script name String or nil
-    def shebang_script
-      # Fail fast if blob isn't viewable?
-      return unless viewable?
-
-      if lines.any? && (match = lines[0].match(/(.+)\n?/)) && (bang = match[0]) =~ /^#!/
-        bang.sub!(/^#! /, '#!')
-        tokens = bang.split(' ')
-        pieces = tokens.first.split('/')
-        if pieces.size > 1
-          script = pieces.last
-        else
-          script = pieces.first.sub('#!', '')
-        end
-
-        script = script == 'env' ? tokens[1] : script
-
-        # python2.4 => python
-        if script =~ /((?:\d+\.?)+)/
-          script.sub! $1, ''
-        end
-
-        # Check for multiline shebang hacks that exec themselves
-        #
-        #   #!/bin/sh
-        #   exec foo "$0" "$@"
-        #
-        if script == 'sh' &&
-            lines[0...5].any? { |l| l.match(/exec (\w+).+\$0.+\$@/) }
-          script = $1
-        end
-
-        script
-      end
-    end
-
-    # Internal: Get Language for shebang script
-    #
-    # Returns the Language or nil
-    def shebang_language
-      # Skip file extensions unlikely to have shebangs
-      return unless shebang_extname?
-
-      if script = shebang_script
-        Language[script]
-      end
-    end
-
-    # Public: Highlight syntax of blob
-    #
-    # options - A Hash of options (defaults to {})
-    #
-    # Returns html String
-    def colorize(options = {})
-      return unless safe_to_colorize?
-      options[:options] ||= {}
-      options[:options][:encoding] ||= encoding
-      lexer.highlight(data, options)
-    end
-
-    # Public: Highlight syntax of blob without the outer highlight div
-    # wrapper.
-    #
-    # options - A Hash of options (defaults to {})
-    #
-    # Returns html String
-    def colorize_without_wrapper(options = {})
-      if text = colorize(options)
-        text[%r{<div class="highlight"><pre>(.*?)</pre>\s*</div>}m, 1]
-      else
-        ''
-      end
-    end
-
-    Language.overridden_extensions.each do |extension|
-      name = "guess_#{extension.sub(/^\./, '')}_language".to_sym
-      unless instance_methods.map(&:to_sym).include?(name)
-        raise NotImplementedError, "Language##{name} was not defined"
-      end
+    # Internal: Get the TextMate compatible scope for the blob
+    def tm_scope
+      language && language.tm_scope
     end
   end
 end

lib/linguist/classifier.rb

@@ -0,0 +1,190 @@
+require 'linguist/tokenizer'
+
+module Linguist
+  # Language bayesian classifier.
+  class Classifier
+    # Public: Use the classifier to detect language of the blob.
+    #
+    # blob               - An object that quacks like a blob.
+    # possible_languages - Array of Language objects
+    #
+    # Examples
+    #
+    #   Classifier.call(FileBlob.new("path/to/file"), [
+    #     Language["Ruby"], Language["Python"]
+    #   ])
+    #
+    # Returns an Array of Language objects, most probable first.
+    def self.call(blob, possible_languages)
+      language_names = possible_languages.map(&:name)
+      classify(Samples.cache, blob.data, language_names).map do |name, _|
+        Language[name] # Return the actual Language objects
+      end
+    end
+
+    # Public: Train classifier that data is a certain language.
+    #
+    # db       - Hash classifier database object
+    # language - String language of data
+    # data     - String contents of file
+    #
+    # Examples
+    #
+    #   Classifier.train(db, 'Ruby', "def hello; end")
+    #
+    # Returns nothing.
+    #
+    # Set LINGUIST_DEBUG=1 or =2 to see probabilities per-token or
+    # per-language.  See also #dump_all_tokens, below.
+    def self.train!(db, language, data)
+      tokens = Tokenizer.tokenize(data)
+
+      db['tokens_total'] ||= 0
+      db['languages_total'] ||= 0
+      db['tokens'] ||= {}
+      db['language_tokens'] ||= {}
+      db['languages'] ||= {}
+
+      tokens.each do |token|
+        db['tokens'][language] ||= {}
+        db['tokens'][language][token] ||= 0
+        db['tokens'][language][token] += 1
+        db['language_tokens'][language] ||= 0
+        db['language_tokens'][language] += 1
+        db['tokens_total'] += 1
+      end
+      db['languages'][language] ||= 0
+      db['languages'][language] += 1
+      db['languages_total'] += 1
+
+      nil
+    end
+
+    # Public: Guess language of data.
+    #
+    # db        - Hash of classifier tokens database.
+    # data      - Array of tokens or String data to analyze.
+    # languages - Array of language name Strings to restrict to.
+    #
+    # Examples
+    #
+    #   Classifier.classify(db, "def hello; end")
+    #   # => [ 'Ruby', 0.90], ['Python', 0.2], ... ]
+    #
+    # Returns sorted Array of result pairs. Each pair contains the
+    # String language name and a Float score.
+    def self.classify(db, tokens, languages = nil)
+      languages ||= db['languages'].keys
+      new(db).classify(tokens, languages)
+    end
+
+    # Internal: Initialize a Classifier.
+    def initialize(db = {})
+      @tokens_total    = db['tokens_total']
+      @languages_total = db['languages_total']
+      @tokens          = db['tokens']
+      @language_tokens = db['language_tokens']
+      @languages       = db['languages']
+    end
+
+    # Internal: Guess language of data
+    #
+    # data      - Array of tokens or String data to analyze.
+    # languages - Array of language name Strings to restrict to.
+    #
+    # Returns sorted Array of result pairs. Each pair contains the
+    # String language name and a Float score.
+    def classify(tokens, languages)
+      return [] if tokens.nil?
+      tokens = Tokenizer.tokenize(tokens) if tokens.is_a?(String)
+      scores = {}
+
+      debug_dump_all_tokens(tokens, languages) if verbosity >= 2
+
+      languages.each do |language|
+        scores[language] = tokens_probability(tokens, language) + language_probability(language)
+        debug_dump_probabilities(tokens, language, scores[language]) if verbosity >= 1
+      end
+
+      scores.sort { |a, b| b[1] <=> a[1] }.map { |score| [score[0], score[1]] }
+    end
+
+    # Internal: Probably of set of tokens in a language occurring - P(D | C)
+    #
+    # tokens   - Array of String tokens.
+    # language - Language to check.
+    #
+    # Returns Float between 0.0 and 1.0.
+    def tokens_probability(tokens, language)
+      tokens.inject(0.0) do |sum, token|
+        sum += Math.log(token_probability(token, language))
+      end
+    end
+
+    # Internal: Probably of token in language occurring - P(F | C)
+    #
+    # token    - String token.
+    # language - Language to check.
+    #
+    # Returns Float between 0.0 and 1.0.
+    def token_probability(token, language)
+      if @tokens[language][token].to_f == 0.0
+        1 / @tokens_total.to_f
+      else
+        @tokens[language][token].to_f / @language_tokens[language].to_f
+      end
+    end
+
+    # Internal: Probably of a language occurring - P(C)
+    #
+    # language - Language to check.
+    #
+    # Returns Float between 0.0 and 1.0.
+    def language_probability(language)
+      Math.log(@languages[language].to_f / @languages_total.to_f)
+    end
+
+    private
+      def verbosity
+        @verbosity ||= (ENV['LINGUIST_DEBUG'] || 0).to_i
+      end
+
+      def debug_dump_probabilities(tokens, language, score)
+        printf("%10s = %10.3f + %7.3f = %10.3f\n",
+            language, tokens_probability(tokens, language), language_probability(language), score)
+      end
+
+      # Internal: show a table of probabilities for each <token,language> pair.
+      #
+      # The number in each table entry is the number of "points" that each
+      # token contributes toward the belief that the file under test is a
+      # particular language.  Points are additive.
+      #
+      # Points are the number of times a token appears in the file, times
+      # how much more likely (log of probability ratio) that token is to
+      # appear in one language vs. the least-likely language.  Dashes
+      # indicate the least-likely language (and zero points) for each token.
+      def debug_dump_all_tokens(tokens, languages)
+        maxlen = tokens.map { |tok| tok.size }.max
+
+        printf "%#{maxlen}s", ""
+        puts "    #" + languages.map { |lang| sprintf("%10s", lang) }.join
+
+        token_map = Hash.new(0)
+        tokens.each { |tok| token_map[tok] += 1 }
+
+        token_map.sort.each { |tok, count|
+          arr = languages.map { |lang| [lang, token_probability(tok, lang)] }
+          min = arr.map { |a,b| b }.min
+          minlog = Math.log(min)
+          if !arr.inject(true) { |result, n| result && n[1] == arr[0][1] }
+            printf "%#{maxlen}s%5d", tok, count
+
+            puts arr.map { |ent|
+              ent[1] == min ? "         -" : sprintf("%10.3f", count * (Math.log(ent[1]) - minlog))
+            }.join
+          end
+        }
+      end
+  end
+end

lib/linguist/file_blob.rb

@@ -52,5 +52,26 @@ module Linguist
     def size
       File.size(@path)
     end
+
+    # Public: Get file extension.
+    #
+    # Returns a String.
+    def extension
+      extensions.last || ""
+    end
+
+    # Public: Return an array of the file extensions
+    #
+    #     >> Linguist::FileBlob.new("app/views/things/index.html.erb").extensions
+    #     => [".html.erb", ".erb"]
+    #
+    # Returns an Array
+    def extensions
+      basename, *segments = File.basename(name).split(".")
+
+      segments.map.with_index do |segment, index|
+        "." + segments[index..-1].join(".")
+      end
+    end
   end
 end

lib/linguist/generated.rb

@@ -0,0 +1,266 @@
+module Linguist
+  class Generated
+    # Public: Is the blob a generated file?
+    #
+    # name - String filename
+    # data - String blob data. A block also maybe passed in for lazy
+    #        loading. This behavior is deprecated and you should always
+    #        pass in a String.
+    #
+    # Return true or false
+    def self.generated?(name, data)
+      new(name, data).generated?
+    end
+
+    # Internal: Initialize Generated instance
+    #
+    # name - String filename
+    # data - String blob data
+    def initialize(name, data)
+      @name = name
+      @extname = File.extname(name)
+      @_data = data
+    end
+
+    attr_reader :name, :extname
+
+    # Lazy load blob data if block was passed in.
+    #
+    # Awful, awful stuff happening here.
+    #
+    # Returns String data.
+    def data
+      @data ||= @_data.respond_to?(:call) ? @_data.call() : @_data
+    end
+
+    # Public: Get each line of data
+    #
+    # Returns an Array of lines
+    def lines
+      # TODO: data should be required to be a String, no nils
+      @lines ||= data ? data.split("\n", -1) : []
+    end
+
+    # Internal: Is the blob a generated file?
+    #
+    # Generated source code is suppressed in diffs and is ignored by
+    # language statistics.
+    #
+    # Please add additional test coverage to
+    # `test/test_blob.rb#test_generated` if you make any changes.
+    #
+    # Return true or false
+    def generated?
+      minified_files? ||
+      compiled_coffeescript? ||
+      xcode_file? ||
+      generated_parser? ||
+      generated_net_docfile? ||
+      generated_net_designer_file? ||
+      generated_postscript? ||
+      generated_protocol_buffer? ||
+      generated_jni_header? ||
+      composer_lock? ||
+      node_modules? ||
+      godeps? ||
+      vcr_cassette? ||
+      generated_by_zephir?
+    end
+
+    # Internal: Is the blob an Xcode file?
+    #
+    # Generated if the file extension is an Xcode
+    # file extension.
+    #
+    # Returns true of false.
+    def xcode_file?
+      ['.nib', '.xcworkspacedata', '.xcuserstate'].include?(extname)
+    end
+
+    # Internal: Is the blob minified files?
+    #
+    # Consider a file minified if the average line length is
+    # greater then 110c.
+    #
+    # Currently, only JS and CSS files are detected by this method.
+    #
+    # Returns true or false.
+    def minified_files?
+      return unless ['.js', '.css'].include? extname
+      if lines.any?
+        (lines.inject(0) { |n, l| n += l.length } / lines.length) > 110
+      else
+        false
+      end
+    end
+
+    # Internal: Is the blob of JS generated by CoffeeScript?
+    #
+    # CoffeeScript is meant to output JS that would be difficult to
+    # tell if it was generated or not. Look for a number of patterns
+    # output by the CS compiler.
+    #
+    # Return true or false
+    def compiled_coffeescript?
+      return false unless extname == '.js'
+
+      # CoffeeScript generated by > 1.2 include a comment on the first line
+      if lines[0] =~ /^\/\/ Generated by /
+        return true
+      end
+
+      if lines[0] == '(function() {' &&     # First line is module closure opening
+          lines[-2] == '}).call(this);' &&  # Second to last line closes module closure
+          lines[-1] == ''                   # Last line is blank
+
+        score = 0
+
+        lines.each do |line|
+          if line =~ /var /
+            # Underscored temp vars are likely to be Coffee
+            score += 1 * line.gsub(/(_fn|_i|_len|_ref|_results)/).count
+
+            # bind and extend functions are very Coffee specific
+            score += 3 * line.gsub(/(__bind|__extends|__hasProp|__indexOf|__slice)/).count
+          end
+        end
+
+        # Require a score of 3. This is fairly arbitrary. Consider
+        # tweaking later.
+        score >= 3
+      else
+        false
+      end
+    end
+
+    # Internal: Is this a generated documentation file for a .NET assembly?
+    #
+    # .NET developers often check in the XML Intellisense file along with an
+    # assembly - however, these don't have a special extension, so we have to
+    # dig into the contents to determine if it's a docfile. Luckily, these files
+    # are extremely structured, so recognizing them is easy.
+    #
+    # Returns true or false
+    def generated_net_docfile?
+      return false unless extname.downcase == ".xml"
+      return false unless lines.count > 3
+
+      # .NET Docfiles always open with <doc> and their first tag is an
+      # <assembly> tag
+      return lines[1].include?("<doc>") &&
+        lines[2].include?("<assembly>") &&
+        lines[-2].include?("</doc>")
+    end
+
+    # Internal: Is this a codegen file for a .NET project?
+    #
+    # Visual Studio often uses code generation to generate partial classes, and
+    # these files can be quite unwieldy. Let's hide them.
+    #
+    # Returns true or false
+    def generated_net_designer_file?
+      name.downcase =~ /\.designer\.cs$/
+    end
+
+    # Internal: Is the blob of JS a parser generated by PEG.js?
+    #
+    # PEG.js-generated parsers are not meant to be consumed by humans.
+    #
+    # Return true or false
+    def generated_parser?
+      return false unless extname == '.js'
+
+      # PEG.js-generated parsers include a comment near the top  of the file
+      # that marks them as such.
+      if lines[0..4].join('') =~ /^(?:[^\/]|\/[^\*])*\/\*(?:[^\*]|\*[^\/])*Generated by PEG.js/
+        return true
+      end
+
+      false
+    end
+
+    # Internal: Is the blob of PostScript generated?
+    #
+    # PostScript files are often generated by other programs. If they tell us so,
+    # we can detect them.
+    #
+    # Returns true or false.
+    def generated_postscript?
+      return false unless ['.ps', '.eps'].include? extname
+
+      # We analyze the "%%Creator:" comment, which contains the author/generator
+      # of the file. If there is one, it should be in one of the first few lines.
+      creator = lines[0..9].find {|line| line =~ /^%%Creator: /}
+      return false if creator.nil?
+
+      # Most generators write their version number, while human authors' or companies'
+      # names don't contain numbers. So look if the line contains digits. Also
+      # look for some special cases without version numbers.
+      return creator =~ /[0-9]/ ||
+        creator.include?("mpage") ||
+        creator.include?("draw") ||
+        creator.include?("ImageMagick")
+    end
+
+    # Internal: Is the blob a C++, Java or Python source file generated by the
+    # Protocol Buffer compiler?
+    #
+    # Returns true of false.
+    def generated_protocol_buffer?
+      return false unless ['.py', '.java', '.h', '.cc', '.cpp'].include?(extname)
+      return false unless lines.count > 1
+
+      return lines[0].include?("Generated by the protocol buffer compiler.  DO NOT EDIT!")
+    end
+
+    # Internal: Is the blob a C/C++ header generated by the Java JNI tool javah?
+    #
+    # Returns true of false.
+    def generated_jni_header?
+      return false unless extname == '.h'
+      return false unless lines.count > 2
+
+      return lines[0].include?("/* DO NOT EDIT THIS FILE - it is machine generated */") &&
+               lines[1].include?("#include <jni.h>")
+    end
+
+    # Internal: Is the blob part of node_modules/, which are not meant for humans in pull requests.
+    #
+    # Returns true or false.
+    def node_modules?
+      !!name.match(/node_modules\//)
+    end
+
+    # Internal: Is the blob part of Godeps/,
+    # which are not meant for humans in pull requests.
+    #
+    # Returns true or false.
+    def godeps?
+      !!name.match(/Godeps\//)
+    end
+
+    # Internal: Is the blob a generated php composer lock file?
+    #
+    # Returns true or false.
+    def composer_lock?
+      !!name.match(/composer\.lock/)
+    end
+
+    # Internal: Is the blob a generated by Zephir
+    #
+    # Returns true or false.
+    def generated_by_zephir?
+      !!name.match(/.\.zep\.(?:c|h|php)$/)
+    end
+
+    # Is the blob a VCR Cassette file?
+    #
+    # Returns true or false
+    def vcr_cassette?
+      return false unless extname == '.yml'
+      return false unless lines.count > 2
+      # VCR Cassettes have "recorded_with: VCR" in the second last line.
+      return lines[-2].include?("recorded_with: VCR")
+    end
+  end
+end

lib/linguist/grammars.rb

@@ -0,0 +1,13 @@
+# Note: This file is included in the github-linguist-grammars gem, not the
+# github-linguist gem.
+
+module Linguist
+  module Grammars
+    # Get the path to the directory containing the language grammar JSON files.
+    #
+    # Returns a String.
+    def self.path
+      File.expand_path("../../../grammars", __FILE__)
+    end
+  end
+end

lib/linguist/heuristics.rb

@@ -0,0 +1,160 @@
+module Linguist
+  # A collection of simple heuristics that can be used to better analyze languages.
+  class Heuristics
+    # Public: Use heuristics to detect language of the blob.
+    #
+    # blob               - An object that quacks like a blob.
+    # possible_languages - Array of Language objects
+    #
+    # Examples
+    #
+    #   Heuristics.call(FileBlob.new("path/to/file"), [
+    #     Language["Ruby"], Language["Python"]
+    #   ])
+    #
+    # Returns an Array of languages, or empty if none matched or were inconclusive.
+    def self.call(blob, languages)
+      data = blob.data
+
+      @heuristics.each do |heuristic|
+        return Array(heuristic.call(data)) if heuristic.matches?(languages)
+      end
+
+      [] # No heuristics matched
+    end
+
+    # Internal: Define a new heuristic.
+    #
+    # languages - String names of languages to disambiguate.
+    # heuristic - Block which takes data as an argument and returns a Language or nil.
+    #
+    # Examples
+    #
+    #     disambiguate "Perl", "Prolog" do |data|
+    #       if data.include?("use strict")
+    #         Language["Perl"]
+    #       elsif data.include?(":-")
+    #         Language["Prolog"]
+    #       end
+    #     end
+    #
+    def self.disambiguate(*languages, &heuristic)
+      @heuristics << new(languages, &heuristic)
+    end
+
+    # Internal: Array of defined heuristics
+    @heuristics = []
+
+    # Internal
+    def initialize(languages, &heuristic)
+      @languages = languages
+      @heuristic = heuristic
+    end
+
+    # Internal: Check if this heuristic matches the candidate languages.
+    def matches?(candidates)
+      candidates.all? { |l| @languages.include?(l.name) }
+    end
+
+    # Internal: Perform the heuristic
+    def call(data)
+      @heuristic.call(data)
+    end
+
+    disambiguate "Objective-C", "C++", "C" do |data|
+      if (/@(interface|class|protocol|property|end|synchronised|selector|implementation)\b/.match(data))
+        Language["Objective-C"]
+      elsif (/^\s*#\s*include <(cstdint|string|vector|map|list|array|bitset|queue|stack|forward_list|unordered_map|unordered_set|(i|o|io)stream)>/.match(data) ||
+        /^\s*template\s*</.match(data) || /^[^@]class\s+\w+/.match(data) || /^[^@](private|public|protected):$/.match(data) || /std::.+$/.match(data))
+        Language["C++"]
+      end
+    end
+
+    disambiguate "Perl", "Perl6", "Prolog" do |data|
+      if data.include?("use v6")
+        Language["Perl6"]
+      elsif data.include?("use strict")
+        Language["Perl"]
+      elsif data.include?(":-")
+        Language["Prolog"]
+      end
+    end
+
+    disambiguate "ECL", "Prolog" do |data|
+      if data.include?(":-")
+        Language["Prolog"]
+      elsif data.include?(":=")
+        Language["ECL"]
+      end
+    end
+
+    disambiguate "IDL", "Prolog" do |data|
+      if data.include?(":-")
+        Language["Prolog"]
+      else
+        Language["IDL"]
+      end
+    end
+
+    disambiguate "Common Lisp", "OpenCL", "Cool" do |data|
+      if data.include?("(defun ")
+        Language["Common Lisp"]
+      elsif /^class/x.match(data)
+        Language["Cool"]
+      elsif /\/\* |\/\/ |^\}/.match(data)
+        Language["OpenCL"]
+      end
+    end
+
+    disambiguate "Hack", "PHP" do |data|
+      if data.include?("<?hh")
+        Language["Hack"]
+      elsif /<?[^h]/.match(data)
+        Language["PHP"]
+      end
+    end
+
+    disambiguate "Scala", "SuperCollider" do |data|
+      if /\^(this|super)\./.match(data) || /^\s*(\+|\*)\s*\w+\s*{/.match(data) || /^\s*~\w+\s*=\./.match(data)
+        Language["SuperCollider"]
+      elsif /^\s*import (scala|java)\./.match(data) || /^\s*val\s+\w+\s*=/.match(data) || /^\s*class\b/.match(data)
+        Language["Scala"]
+      end
+    end
+
+    disambiguate "AsciiDoc", "AGS Script" do |data|
+      Language["AsciiDoc"] if /^=+(\s|\n)/.match(data)
+    end
+
+    disambiguate "FORTRAN", "Forth" do |data|
+      if /^: /.match(data)
+        Language["Forth"]
+      elsif /^([c*][^a-z]|      subroutine\s)/i.match(data)
+        Language["FORTRAN"]
+      end
+    end
+
+    disambiguate "F#", "Forth", "GLSL" do |data|
+      if /^(: |new-device)/.match(data)
+        Language["Forth"]
+      elsif /^(#light|import|let|module|namespace|open|type)/.match(data)
+        Language["F#"]
+      elsif /^(#include|#pragma|precision|uniform|varying|void)/.match(data)
+        Language["GLSL"]
+      end
+    end
+
+    disambiguate "Gosu", "JavaScript" do |data|
+      Language["Gosu"] if /^uses java\./.match(data)
+    end
+
+    disambiguate "LoomScript", "LiveScript" do |data|
+      if /^\s*package\s*[\w\.\/\*\s]*\s*{/.match(data)
+        Language["LoomScript"]
+      else
+        Language["LiveScript"]
+      end
+    end
+
+  end
+end

lib/linguist/language.rb

@@ -1,6 +1,17 @@
 require 'escape_utils'
-require 'pygments'
 require 'yaml'
+begin
+  require 'yajl'
+rescue LoadError
+end
+
+require 'linguist/classifier'
+require 'linguist/heuristics'
+require 'linguist/samples'
+require 'linguist/file_blob'
+require 'linguist/blob_helper'
+require 'linguist/strategy/filename'
+require 'linguist/shebang'
 
 module Linguist
   # Language names that are recognizable by GitHub. Defined languages
@@ -9,28 +20,31 @@ module Linguist
   # Languages are defined in `lib/linguist/languages.yml`.
   class Language
     @languages       = []
-    @overrides       = {}
     @index           = {}
     @name_index      = {}
     @alias_index     = {}
-    @extension_index = {}
-    @filename_index  = {}
+
+    @extension_index          = Hash.new { |h,k| h[k] = [] }
+    @interpreter_index        = Hash.new { |h,k| h[k] = [] }
+    @filename_index           = Hash.new { |h,k| h[k] = [] }
 
     # Valid Languages types
-    TYPES = [:data, :markup, :programming]
+    TYPES = [:data, :markup, :programming, :prose]
 
-    # Internal: Test if extension maps to multiple Languages.
+    # Names of non-programming languages that we will still detect
     #
-    # Returns true or false.
-    def self.ambiguous?(extension)
-      @overrides.include?(extension)
+    # Returns an array
+    def self.detectable_markup
+      ["CSS", "Less", "Sass", "SCSS", "Stylus", "TeX"]
     end
 
-    # Include?: Return overridden extensions.
+    # Detect languages by a specific type
     #
-    # Returns extensions Array.
-    def self.overridden_extensions
-      @overrides.keys
+    # type - A symbol that exists within TYPES
+    #
+    # Returns an array
+    def self.by_type(type)
+      all.select { |h| h.type == type }
     end
 
     # Internal: Create a new Language object
@@ -49,7 +63,7 @@ module Linguist
       end
 
       # Language name index
-      @index[language.name] = @name_index[language.name] = language
+      @index[language.name.downcase] = @name_index[language.name.downcase] = language
 
       language.aliases.each do |name|
         # All Language aliases should be unique. Raise if there is a duplicate.
@@ -57,7 +71,7 @@ module Linguist
           raise ArgumentError, "Duplicate alias: #{name}"
         end
 
-        @index[name] = @alias_index[name] = language
+        @index[name.downcase] = @alias_index[name.downcase] = language
       end
 
       language.extensions.each do |extension|
@@ -65,34 +79,52 @@ module Linguist
           raise ArgumentError, "Extension is missing a '.': #{extension.inspect}"
         end
 
-        unless ambiguous?(extension)
-          # Index the extension with a leading ".": ".rb"
-          @extension_index[extension] = language
-
-          # Index the extension without a leading ".": "rb"
-          @extension_index[extension.sub(/^\./, '')] = language
-        end
+        @extension_index[extension] << language
       end
 
-      language.overrides.each do |extension|
-        if extension !~ /^\./
-          raise ArgumentError, "Extension is missing a '.': #{extension.inspect}"
-        end
-
-        if l = @overrides[extension]
-          raise ArgumentError, "#{extension} is already overridden by #{l.name}"
-        end
-
-        @overrides[extension] = language
+      language.interpreters.each do |interpreter|
+        @interpreter_index[interpreter] << language
       end
 
       language.filenames.each do |filename|
-        @filename_index[filename] = language
+        @filename_index[filename] << language
       end
 
       language
     end
 
+    STRATEGIES = [
+      Linguist::Strategy::Filename,
+      Linguist::Shebang,
+      Linguist::Heuristics,
+      Linguist::Classifier
+    ]
+
+    # Public: Detects the Language of the blob.
+    #
+    # blob - an object that includes the Linguist `BlobHelper` interface;
+    #       see Linguist::LazyBlob and Linguist::FileBlob for examples
+    #
+    # Returns Language or nil.
+    def self.detect(blob)
+      # Bail early if the blob is binary or empty.
+      return nil if blob.likely_binary? || blob.binary? || blob.empty?
+
+      # Call each strategy until one candidate is returned.
+      STRATEGIES.reduce([]) do |languages, strategy|
+        candidates = strategy.call(blob, languages)
+        if candidates.size == 1
+          return candidates.first
+        elsif candidates.size > 1
+          # More than one candidate was found, pass them to the next strategy.
+          candidates
+        else
+          # No candiates were found, pass on languages from the previous strategy.
+          languages
+        end
+      end.first
+    end
+
     # Public: Get all Languages
     #
     # Returns an Array of Languages
@@ -111,7 +143,7 @@ module Linguist
     #
     # Returns the Language or nil if none was found.
     def self.find_by_name(name)
-      @name_index[name]
+      name && @name_index[name.downcase]
     end
 
     # Public: Look up Language by one of its aliases.
@@ -125,38 +157,68 @@ module Linguist
     #
     # Returns the Lexer or nil if none was found.
     def self.find_by_alias(name)
-      @alias_index[name]
+      name && @alias_index[name.downcase]
     end
 
-    # Public: Look up Language by extension.
-    #
-    # extension - The extension String. May include leading "."
-    #
-    # Examples
-    #
-    #   Language.find_by_extension('.rb')
-    #   # => #<Language name="Ruby">
-    #
-    # Returns the Language or nil if none was found.
-    def self.find_by_extension(extension)
-      @extension_index[extension]
-    end
-
-    # Public: Look up Language by filename.
+    # Public: Look up Languages by filename.
     #
     # filename - The path String.
     #
     # Examples
     #
     #   Language.find_by_filename('foo.rb')
-    #   # => #<Language name="Ruby">
+    #   # => [#<Language name="Ruby">]
     #
-    # Returns the Language or nil if none was found.
+    # Returns all matching Languages or [] if none were found.
     def self.find_by_filename(filename)
-      basename, extname = File.basename(filename), File.extname(filename)
-      @filename_index[basename] || @extension_index[extname]
+      basename = File.basename(filename)
+
+      # find the first extension with language definitions
+      extname = FileBlob.new(filename).extensions.detect do |e|
+        !@extension_index[e].empty?
+      end
+
+      (@filename_index[basename] + @extension_index[extname]).compact.uniq
     end
 
+    # Public: Look up Languages by file extension.
+    #
+    # extname - The extension String.
+    #
+    # Examples
+    #
+    #   Language.find_by_extension('.rb')
+    #   # => [#<Language name="Ruby">]
+    #
+    #   Language.find_by_extension('rb')
+    #   # => [#<Language name="Ruby">]
+    #
+    # Returns all matching Languages or [] if none were found.
+    def self.find_by_extension(extname)
+      extname = ".#{extname}" unless extname.start_with?(".")
+      @extension_index[extname]
+    end
+
+    # DEPRECATED
+    def self.find_by_shebang(data)
+      @interpreter_index[Shebang.interpreter(data)]
+    end
+
+    # Public: Look up Languages by interpreter.
+    #
+    # interpreter - String of interpreter name
+    #
+    # Examples
+    #
+    #   Language.find_by_interpreter("bash")
+    #   # => [#<Language name="Bash">]
+    #
+    # Returns the matching Language
+    def self.find_by_interpreter(interpreter)
+      @interpreter_index[interpreter]
+    end
+
+
     # Public: Look up Language by its name or lexer.
     #
     # name - The String name of the Language
@@ -171,7 +233,7 @@ module Linguist
     #
     # Returns the Language or nil if none was found.
     def self.[](name)
-      @index[name]
+      name && @index[name.downcase]
     end
 
     # Public: A List of popular languages
@@ -230,27 +292,28 @@ module Linguist
       # Set aliases
       @aliases = [default_alias_name] + (attributes[:aliases] || [])
 
-      # Lookup Lexer object
-      @lexer = Pygments::Lexer.find_by_name(attributes[:lexer] || name) ||
-        raise(ArgumentError, "#{@name} is missing lexer")
+      # Load the TextMate scope name or try to guess one
+      @tm_scope = attributes[:tm_scope] || begin
+        context = case @type
+                  when :data, :markup, :prose
+                    'text'
+                  when :programming, nil
+                    'source'
+                  end
+        "#{context}.#{@name.downcase}"
+      end
 
       @ace_mode = attributes[:ace_mode]
+      @wrap = attributes[:wrap] || false
 
       # Set legacy search term
       @search_term = attributes[:search_term] || default_alias_name
 
       # Set extensions or default to [].
       @extensions = attributes[:extensions] || []
-      @overrides  = attributes[:overrides]  || []
+      @interpreters = attributes[:interpreters]   || []
       @filenames  = attributes[:filenames]  || []
 
-      @primary_extension = attributes[:primary_extension] || default_primary_extension || extensions.first
-
-      # Prepend primary extension unless its already included
-      if primary_extension && !extensions.include?(primary_extension)
-        @extensions = [primary_extension] + extensions
-      end
-
       # Set popular, and searchable flags
       @popular    = attributes.key?(:popular)    ? attributes[:popular]    : false
       @searchable = attributes.key?(:searchable) ? attributes[:searchable] : true
@@ -313,6 +376,11 @@ module Linguist
     # Returns the Lexer
     attr_reader :lexer
 
+    # Public: Get the name of a TextMate-compatible scope
+    #
+    # Returns the scope
+    attr_reader :tm_scope
+
     # Public: Get Ace mode
     #
     # Examples
@@ -324,6 +392,11 @@ module Linguist
     # Returns a String name or nil
     attr_reader :ace_mode
 
+    # Public: Should language lines be wrapped
+    #
+    # Returns true or false
+    attr_reader :wrap
+
     # Public: Get extensions
     #
     # Examples
@@ -333,24 +406,14 @@ module Linguist
     # Returns the extensions Array
     attr_reader :extensions
 
-    # Deprecated: Get primary extension
+    # Public: Get interpreters
     #
-    # Defaults to the first extension but can be overriden
-    # in the languages.yml.
+    # Examples
     #
-    # The primary extension can not be nil. Tests should verify this.
+    #   # => ['awk', 'gawk', 'mawk' ...]
     #
-    # This attribute is only used by app/helpers/gists_helper.rb for
-    # creating the language dropdown. It really should be using `name`
-    # instead. Would like to drop primary extension.
-    #
-    # Returns the extension String.
-    attr_reader :primary_extension
-
-    # Internal: Get overridden extensions.
-    #
-    # Returns the extensions Array.
-    attr_reader :overrides
+    # Returns the interpreters Array
+    attr_reader :interpreters
 
     # Public: Get filenames
     #
@@ -361,6 +424,22 @@ module Linguist
     # Returns the extensions Array
     attr_reader :filenames
 
+    # Deprecated: Get primary extension
+    #
+    # Defaults to the first extension but can be overridden
+    # in the languages.yml.
+    #
+    # The primary extension can not be nil. Tests should verify this.
+    #
+    # This method is only used by app/helpers/gists_helper.rb for creating
+    # the language dropdown. It really should be using `name` instead.
+    # Would like to drop primary extension.
+    #
+    # Returns the extension String.
+    def primary_extension
+      extensions.first
+    end
+
     # Public: Get URL escaped name.
     #
     # Examples
@@ -381,13 +460,6 @@ module Linguist
       name.downcase.gsub(/\s/, '-')
     end
 
-    # Internal: Get default primary extension.
-    #
-    # Returns the extension String.
-    def default_primary_extension
-      extensions.first
-    end
-
     # Public: Get Language group
     #
     # Returns a Language
@@ -426,7 +498,7 @@ module Linguist
     #
     # Returns html String
     def colorize(text, options = {})
-      lexer.highlight(text, options = {})
+      lexer.highlight(text, options)
     end
 
     # Public: Return name as String representation
@@ -451,22 +523,68 @@ module Linguist
     end
   end
 
+  extensions = Samples.cache['extnames']
+  interpreters = Samples.cache['interpreters']
+  filenames = Samples.cache['filenames']
   popular = YAML.load_file(File.expand_path("../popular.yml", __FILE__))
 
-  YAML.load_file(File.expand_path("../languages.yml", __FILE__)).each do |name, options|
+  languages_yml = File.expand_path("../languages.yml", __FILE__)
+  languages_json = File.expand_path("../languages.json", __FILE__)
+
+  if File.exist?(languages_json) && defined?(Yajl)
+    languages = Yajl.load(File.read(languages_json))
+  else
+    languages = YAML.load_file(languages_yml)
+  end
+
+  languages.each do |name, options|
+    options['extensions'] ||= []
+    options['interpreters'] ||= []
+    options['filenames'] ||= []
+
+    if extnames = extensions[name]
+      extnames.each do |extname|
+        if !options['extensions'].include?(extname)
+          warn "#{name} has a sample with extension (#{extname}) that isn't explicitly defined in languages.yml" unless extname == '.script!'
+          options['extensions'] << extname
+        end
+      end
+    end
+
+    if interpreters == nil
+      interpreters = {}
+    end
+
+    if interpreter_names = interpreters[name]
+      interpreter_names.each do |interpreter|
+        if !options['interpreters'].include?(interpreter)
+          options['interpreters'] << interpreter
+        end
+      end
+    end
+
+    if fns = filenames[name]
+      fns.each do |filename|
+        if !options['filenames'].include?(filename)
+          options['filenames'] << filename
+        end
+      end
+    end
+
     Language.create(
       :name              => name,
       :color             => options['color'],
       :type              => options['type'],
       :aliases           => options['aliases'],
       :lexer             => options['lexer'],
+      :tm_scope          => options['tm_scope'],
       :ace_mode          => options['ace_mode'],
+      :wrap              => options['wrap'],
       :group_name        => options['group'],
-      :searchable        => options.key?('searchable') ? options['searchable'] : true,
+      :searchable        => options.fetch('searchable', true),
       :search_term       => options['search_term'],
-      :extensions        => options['extensions'],
-      :primary_extension => options['primary_extension'],
-      :overrides         => options['overrides'],
+      :extensions        => Array(options['extensions']),
+      :interpreters      => options['interpreters'].sort,
       :filenames         => options['filenames'],
       :popular           => popular.include?(name)
     )

lib/linguist/lazy_blob.rb

@@ -0,0 +1,71 @@
+require 'linguist/blob_helper'
+require 'linguist/language'
+require 'rugged'
+
+module Linguist
+  class LazyBlob
+    GIT_ATTR = ['linguist-language', 'linguist-vendored']
+    GIT_ATTR_OPTS = { :priority => [:index], :skip_system => true }
+    GIT_ATTR_FLAGS = Rugged::Repository::Attributes.parse_opts(GIT_ATTR_OPTS)
+
+    include BlobHelper
+
+    MAX_SIZE = 128 * 1024
+
+    attr_reader :repository
+    attr_reader :oid
+    attr_reader :name
+    attr_reader :mode
+
+    def initialize(repo, oid, name, mode = nil)
+      @repository = repo
+      @oid = oid
+      @name = name
+      @mode = mode
+    end
+
+    def git_attributes
+      @git_attributes ||= repository.fetch_attributes(
+        name, GIT_ATTR, GIT_ATTR_FLAGS)
+    end
+
+    def vendored?
+      if attr = git_attributes['linguist-vendored']
+        return boolean_attribute(attr)
+      else
+        return super
+      end
+    end
+
+    def language
+      return @language if defined?(@language)
+
+      @language = if lang = git_attributes['linguist-language']
+        Language.find_by_name(lang)
+      else
+        super
+      end
+    end
+
+    def data
+      load_blob!
+      @data
+    end
+
+    def size
+      load_blob!
+      @size
+    end
+
+    protected
+
+    # Returns true if the attribute is present and not the string "false".
+    def boolean_attribute(attr)
+      attr != "false"
+    end
+
+    def load_blob!
+      @data, @size = Rugged::Blob.to_buffer(repository, oid, MAX_SIZE) if @data.nil?
+    end
+  end
+end

lib/linguist/md5.rb

@@ -0,0 +1,38 @@
+require 'digest/md5'
+
+module Linguist
+  module MD5
+    # Public: Create deep nested digest of value object.
+    #
+    # Useful for object comparison.
+    #
+    # obj - Object to digest.
+    #
+    # Returns String hex digest
+    def self.hexdigest(obj)
+      digest = Digest::MD5.new
+
+      case obj
+      when String, Symbol, Integer
+        digest.update "#{obj.class}"
+        digest.update "#{obj}"
+      when TrueClass, FalseClass, NilClass
+        digest.update "#{obj.class}"
+      when Array
+        digest.update "#{obj.class}"
+        for e in obj
+          digest.update(hexdigest(e))
+        end
+      when Hash
+        digest.update "#{obj.class}"
+        for e in obj.map { |(k, v)| hexdigest([k, v]) }.sort
+          digest.update(e)
+        end
+      else
+        raise TypeError, "can't convert #{obj.inspect} into String"
+      end
+
+      digest.hexdigest
+    end
+  end
+end

lib/linguist/mime.rb

@@ -1,91 +0,0 @@
-require 'mime/types'
-require 'yaml'
-
-class MIME::Type
-  attr_accessor :override
-end
-
-# Register additional mime type extensions
-#
-# Follows same format as mime-types data file
-#   https://github.com/halostatue/mime-types/blob/master/lib/mime/types.rb.data
-File.read(File.expand_path("../mimes.yml", __FILE__)).lines.each do |line|
-  # Regexp was cargo culted from mime-types lib
-  next unless line =~ %r{^
-    #{MIME::Type::MEDIA_TYPE_RE}
-    (?:\s@([^\s]+))?
-    (?:\s:(#{MIME::Type::ENCODING_RE}))?
-  }x
-
-  mediatype  = $1
-  subtype    = $2
-  extensions = $3
-  encoding   = $4
-
-  # Lookup existing mime type
-  mime_type = MIME::Types["#{mediatype}/#{subtype}"].first ||
-    # Or create a new instance
-    MIME::Type.new("#{mediatype}/#{subtype}")
-
-  if extensions
-    extensions.split(/,/).each do |extension|
-      mime_type.extensions << extension
-    end
-  end
-
-  if encoding
-    mime_type.encoding = encoding
-  end
-
-  mime_type.override = true
-
-  # Kind of hacky, but we need to reindex the mime type after making changes
-  MIME::Types.add_type_variant(mime_type)
-  MIME::Types.index_extensions(mime_type)
-end
-
-module Linguist
-  module Mime
-    # Internal: Look up mime type for extension.
-    #
-    # ext - The extension String. May include leading "."
-    #
-    # Examples
-    #
-    #   Mime.mime_for('.html')
-    #   # => 'text/html'
-    #
-    #   Mime.mime_for('txt')
-    #   # => 'text/plain'
-    #
-    # Return mime type String otherwise falls back to 'text/plain'.
-    def self.mime_for(ext)
-      mime_type = lookup_mime_type_for(ext)
-      mime_type ? mime_type.to_s : 'text/plain'
-    end
-
-    # Internal: Lookup mime type for extension or mime type
-    #
-    # ext_or_mime_type - A file extension ".txt" or mime type "text/plain".
-    #
-    # Returns a MIME::Type
-    def self.lookup_mime_type_for(ext_or_mime_type)
-      ext_or_mime_type ||= ''
-
-      if ext_or_mime_type =~ /\w+\/\w+/
-        guesses = ::MIME::Types[ext_or_mime_type]
-      else
-        guesses = ::MIME::Types.type_for(ext_or_mime_type)
-      end
-
-      # Use custom override first
-      guesses.detect { |type| type.override } ||
-
-        # Prefer text mime types over binary
-        guesses.detect { |type| type.ascii? } ||
-
-        # Otherwise use the first guess
-        guesses.first
-    end
-  end
-end

lib/linguist/mimes.yml

@@ -1,62 +0,0 @@
-# Additional types to add to MIME::Types
-#
-# MIME types are used to set the Content-Type of raw binary blobs. All text
-# blobs are served as text/plain regardless of their type to ensure they
-# open in the browser rather than downloading.
-#
-# The encoding helps determine whether a file should be treated as plain
-# text or binary. By default, a mime type's encoding is base64 (binary).
-# These types will show a "View Raw" link. To force a type to render as
-# plain text, set it to 8bit for UTF-8. text/* types will be treated as
-# text by default.
-#
-#   <type> @<extensions> :<encoding>
-#
-# type       - mediatype/subtype
-# extensions - comma seperated extension list
-# encoding   - base64 (binary), 7bit (ASCII), 8bit (UTF-8), or
-#              quoted-printable (Printable ASCII).
-#
-# Follows same format as mime-types data file
-#   https://github.com/halostatue/mime-types/blob/master/lib/mime/types.rb.data
-#
-# Any additions or modifications (even trivial) should have corresponding
-# test change in `test/test_mime.rb`.
-
-# TODO: Lookup actual types
-application/octet-stream @a,blend,gem,graffle,ipa,lib,mcz,nib,o,ogv,otf,pfx,pigx,plgx,psd,sib,spl,sqlite3,swc,ucode,xpi
-
-# Please keep this list alphabetized
-application/java-archive @ear,war
-application/netcdf :8bit
-application/ogg @ogg
-application/postscript :base64
-application/vnd.adobe.air-application-installer-package+zip @air
-application/vnd.mozilla.xul+xml :8bit
-application/vnd.oasis.opendocument.presentation @odp
-application/vnd.oasis.opendocument.spreadsheet @ods
-application/vnd.oasis.opendocument.text @odt
-application/vnd.openofficeorg.extension @oxt
-application/vnd.openxmlformats-officedocument.presentationml.presentation @pptx
-application/x-chrome-extension @crx
-application/x-iwork-keynote-sffkey @key
-application/x-iwork-numbers-sffnumbers @numbers
-application/x-iwork-pages-sffpages @pages
-application/x-ms-xbap @xbap :8bit
-application/x-parrot-bytecode @pbc
-application/x-shockwave-flash @swf
-application/x-silverlight-app @xap
-application/x-supercollider @sc :8bit
-application/x-troff-ms :8bit
-application/x-wais-source :8bit
-application/xaml+xml @xaml :8bit
-application/xslt+xml @xslt :8bit
-image/x-icns @icns
-text/cache-manifest @manifest
-text/plain @cu,cxx
-text/x-logtalk @lgt
-text/x-nemerle @n
-text/x-nimrod @nim
-text/x-ocaml @ml,mli,mll,mly,sig,sml
-text/x-rust @rs,rc
-text/x-scheme @rkt,scm,sls,sps,ss

lib/linguist/pathname.rb

@@ -1,92 +0,0 @@
-require 'linguist/language'
-require 'linguist/mime'
-require 'pygments'
-
-module Linguist
-  # Similar to ::Pathname, Linguist::Pathname wraps a path string and
-  # provides helpful query methods. Its useful when you only have a
-  # filename but not a blob and need to figure out the language of the file.
-  class Pathname
-    # Public: Initialize a Pathname
-    #
-    # path - A filename String. The file may or maybe actually exist.
-    #
-    # Returns a Pathname.
-    def initialize(path)
-      @path = path
-    end
-
-    # Public: Get the basename of the path
-    #
-    # Examples
-    #
-    #   Pathname.new('sub/dir/file.rb').basename
-    #   # => 'file.rb'
-    #
-    # Returns a String.
-    def basename
-      File.basename(@path)
-    end
-
-    # Public: Get the extname of the path
-    #
-    # Examples
-    #
-    #   Pathname.new('.rb').extname
-    #   # => '.rb'
-    #
-    #   Pathname.new('file.rb').extname
-    #   # => '.rb'
-    #
-    # Returns a String.
-    def extname
-      File.extname(@path)
-    end
-
-    # Public: Get the language of the path
-    #
-    # The path extension name is the only heuristic used to detect the
-    # language name.
-    #
-    # Examples
-    #
-    #   Pathname.new('file.rb').language
-    #   # => Language['Ruby']
-    #
-    # Returns a Language or nil if none was found.
-    def language
-      @language ||= Language.find_by_filename(@path)
-    end
-
-    # Internal: Get the lexer of the path
-    #
-    # Returns a Lexer.
-    def lexer
-      language ? language.lexer : Pygments::Lexer.find_by_name('Text only')
-    end
-
-    # Public: Get the mime type
-    #
-    # Examples
-    #
-    #   Pathname.new('index.html').mime_type
-    #   # => 'text/html'
-    #
-    # Returns a mime type String.
-    def mime_type
-      @mime_type ||= Mime.mime_for(extname)
-    end
-
-    # Public: Return self as String
-    #
-    # Returns a String
-    def to_s
-      @path.dup
-    end
-
-    def eql?(other)
-      other.is_a?(self.class) && @path == other.to_s
-    end
-    alias_method :==, :eql?
-  end
-end

lib/linguist/popular.yml

@@ -8,6 +8,8 @@
 - C#
 - C++
 - CSS
+- Clojure
+- CoffeeScript
 - Common Lisp
 - Diff
 - Emacs Lisp
@@ -25,5 +27,3 @@
 - SQL
 - Scala
 - Scheme
-- TeX
-- XML

lib/linguist/repository.rb

@@ -1,4 +1,5 @@
-require 'linguist/file_blob'
+require 'linguist/lazy_blob'
+require 'rugged'
 
 module Linguist
   # A Repository is an abstraction of a Grit::Repo or a basic file
@@ -7,89 +8,165 @@ module Linguist
   # Its primary purpose is for gathering language statistics across
   # the entire project.
   class Repository
-    # Public: Initialize a new Repository from a File directory
-    #
-    # base_path - A path String
-    #
-    # Returns a Repository
-    def self.from_directory(base_path)
-      new Dir["#{base_path}/**/*"].
-        select { |f| File.file?(f) }.
-        map { |path| FileBlob.new(path, base_path) }
+    attr_reader :repository
+
+    # Public: Create a new Repository based on the stats of
+    # an existing one
+    def self.incremental(repo, commit_oid, old_commit_oid, old_stats)
+      repo = self.new(repo, commit_oid)
+      repo.load_existing_stats(old_commit_oid, old_stats)
+      repo
     end
 
-    # Public: Initialize a new Repository
+    # Public: Initialize a new Repository to be analyzed for language
+    # data
     #
-    # enum - Enumerator that responds to `each` and
-    #        yields Blob objects
+    # repo - a Rugged::Repository object
+    # commit_oid - the sha1 of the commit that will be analyzed;
+    #              this is usually the master branch
     #
     # Returns a Repository
-    def initialize(enum)
-      @enum = enum
-      @computed_stats = false
-      @language = @size = nil
-      @sizes = Hash.new { 0 }
+    def initialize(repo, commit_oid)
+      @repository = repo
+      @commit_oid = commit_oid
+
+      raise TypeError, 'commit_oid must be a commit SHA1' unless commit_oid.is_a?(String)
+    end
+
+    # Public: Load the results of a previous analysis on this repository
+    # to speed up the new scan.
+    #
+    # The new analysis will be performed incrementally as to only take
+    # into account the file changes since the last time the repository
+    # was scanned
+    #
+    # old_commit_oid - the sha1 of the commit that was previously analyzed
+    # old_stats - the result of the previous analysis, obtained by calling
+    #             Repository#cache on the old repository
+    #
+    # Returns nothing
+    def load_existing_stats(old_commit_oid, old_stats)
+      @old_commit_oid = old_commit_oid
+      @old_stats = old_stats
+      nil
     end
 
     # Public: Returns a breakdown of language stats.
     #
     # Examples
     #
-    #   # => { Language['Ruby'] => 46319,
-    #          Language['JavaScript'] => 258 }
+    #   # => { 'Ruby' => 46319,
+    #          'JavaScript' => 258 }
     #
-    # Returns a Hash of Language keys and Integer size values.
+    # Returns a Hash of language names and Integer size values.
     def languages
-      compute_stats
-      @sizes
+      @sizes ||= begin
+        sizes = Hash.new { 0 }
+        cache.each do |_, (language, size)|
+          sizes[language] += size
+        end
+        sizes
+      end
     end
 
     # Public: Get primary Language of repository.
     #
-    # Returns a Language
+    # Returns a language name
     def language
-      compute_stats
-      @language
+      @language ||= begin
+        primary = languages.max_by { |(_, size)| size }
+        primary && primary[0]
+      end
     end
 
     # Public: Get the total size of the repository.
     #
     # Returns a byte size Integer
     def size
-      compute_stats
-      @size
+      @size ||= languages.inject(0) { |s,(_,v)| s + v }
     end
 
-    # Internal: Compute language breakdown for each blob in the Repository.
+    # Public: Return the language breakdown of this repository by file
     #
-    # Returns nothing
-    def compute_stats
-      return if @computed_stats
+    # Returns a map of language names => [filenames...]
+    def breakdown_by_file
+      @file_breakdown ||= begin
+        breakdown = Hash.new { |h,k| h[k] = Array.new }
+        cache.each do |filename, (language, _)|
+          breakdown[language] << filename
+        end
+        breakdown
+      end
+    end
 
-      @enum.each do |blob|
-        # Skip binary file extensions
-        next if blob.binary_mime_type?
+    # Public: Return the cached results of the analysis
+    #
+    # This is a per-file breakdown that can be passed to other instances
+    # of Linguist::Repository to perform incremental scans
+    #
+    # Returns a map of filename => [language, size]
+    def cache
+      @cache ||= begin
+        if @old_commit_oid == @commit_oid
+          @old_stats
+        else
+          compute_stats(@old_commit_oid, @old_stats)
+        end
+      end
+    end
 
-        # Skip vendored or generated blobs
-        next if blob.vendored? || blob.generated? || blob.language.nil?
+    def read_index
+      attr_index = Rugged::Index.new
+      attr_index.read_tree(current_tree)
+      repository.index = attr_index
+    end
 
-        # Only include programming languages
-        if blob.language.type == :programming
-          @sizes[blob.language.group] += blob.size
+    def current_tree
+      @tree ||= Rugged::Commit.lookup(repository, @commit_oid).tree
+    end
+
+    protected
+
+    def compute_stats(old_commit_oid, cache = nil)
+      old_tree = old_commit_oid && Rugged::Commit.lookup(repository, old_commit_oid).tree
+
+      read_index
+
+      diff = Rugged::Tree.diff(repository, old_tree, current_tree)
+
+      # Clear file map and fetch full diff if any .gitattributes files are changed
+      if cache && diff.each_delta.any? { |delta| File.basename(delta.new_file[:path]) == ".gitattributes" }
+        diff = Rugged::Tree.diff(repository, old_tree = nil, current_tree)
+        file_map = {}
+      else
+        file_map = cache ? cache.dup : {}
+      end
+
+      diff.each_delta do |delta|
+        old = delta.old_file[:path]
+        new = delta.new_file[:path]
+
+        file_map.delete(old)
+        next if delta.binary
+
+        if [:added, :modified].include? delta.status
+          # Skip submodules
+          mode = delta.new_file[:mode]
+          next if (mode & 040000) != 0
+
+          blob = Linguist::LazyBlob.new(repository, delta.new_file[:oid], new, mode.to_s(8))
+
+          # Skip vendored or generated blobs
+          next if blob.vendored? || blob.generated? || blob.language.nil?
+
+          # Only include programming languages and acceptable markup languages
+          if blob.language.type == :programming || Language.detectable_markup.include?(blob.language.name)
+            file_map[new] = [blob.language.group.name, blob.size]
+          end
         end
       end
 
-      # Compute total size
-      @size = @sizes.inject(0) { |s,(k,v)| s + v }
-
-      # Get primary language
-      if primary = @sizes.max_by { |(_, size)| size }
-        @language = primary[0]
-      end
-
-      @computed_stats = true
-
-      nil
+      file_map
     end
   end
 end

lib/linguist/samples.rb

@@ -0,0 +1,118 @@
+begin
+  require 'yajl'
+rescue LoadError
+  require 'yaml'
+end
+
+require 'linguist/md5'
+require 'linguist/classifier'
+require 'linguist/shebang'
+
+module Linguist
+  # Model for accessing classifier training data.
+  module Samples
+    # Path to samples root directory
+    ROOT = File.expand_path("../../../samples", __FILE__)
+
+    # Path for serialized samples db
+    PATH = File.expand_path('../samples.json', __FILE__)
+
+    # Hash of serialized samples object
+    def self.cache
+      @cache ||= begin
+        serializer = defined?(Yajl) ? Yajl : YAML
+        serializer.load(File.read(PATH))
+      end
+    end
+
+    # Public: Iterate over each sample.
+    #
+    # &block - Yields Sample to block
+    #
+    # Returns nothing.
+    def self.each(&block)
+      Dir.entries(ROOT).sort!.each do |category|
+        next if category == '.' || category == '..'
+
+        # Skip text and binary for now
+        # Possibly reconsider this later
+        next if category == 'Text' || category == 'Binary'
+
+        dirname = File.join(ROOT, category)
+        Dir.entries(dirname).each do |filename|
+          next if filename == '.' || filename == '..'
+
+          if filename == 'filenames'
+            Dir.entries(File.join(dirname, filename)).each do |subfilename|
+              next if subfilename == '.' || subfilename == '..'
+
+              yield({
+                :path    => File.join(dirname, filename, subfilename),
+                :language => category,
+                :filename => subfilename
+              })
+            end
+          else
+            path = File.join(dirname, filename)
+
+            if File.extname(filename) == ""
+              raise "#{path} is missing an extension, maybe it belongs in filenames/ subdir"
+            end
+
+            yield({
+              :path     => path,
+              :language => category,
+              :interpreter => Shebang.interpreter(File.read(path)),
+              :extname  => File.extname(filename)
+            })
+          end
+        end
+      end
+
+      nil
+    end
+
+    # Public: Build Classifier from all samples.
+    #
+    # Returns trained Classifier.
+    def self.data
+      db = {}
+      db['extnames'] = {}
+      db['interpreters'] = {}
+      db['filenames'] = {}
+
+      each do |sample|
+        language_name = sample[:language]
+
+        if sample[:extname]
+          db['extnames'][language_name] ||= []
+          if !db['extnames'][language_name].include?(sample[:extname])
+            db['extnames'][language_name] << sample[:extname]
+            db['extnames'][language_name].sort!
+          end
+        end
+
+        if sample[:interpreter]
+          db['interpreters'][language_name] ||= []
+          if !db['interpreters'][language_name].include?(sample[:interpreter])
+            db['interpreters'][language_name] << sample[:interpreter]
+            db['interpreters'][language_name].sort!
+          end
+        end
+
+        if sample[:filename]
+          db['filenames'][language_name] ||= []
+          db['filenames'][language_name] << sample[:filename]
+          db['filenames'][language_name].sort!
+        end
+
+        data = File.read(sample[:path])
+        Classifier.train!(db, language_name, data)
+      end
+
+      db['md5'] = Linguist::MD5.hexdigest(db)
+
+      db
+    end
+  end
+end

lib/linguist/shebang.rb

@@ -0,0 +1,44 @@
+module Linguist
+  class Shebang
+    # Public: Use shebang to detect language of the blob.
+    #
+    # blob               - An object that quacks like a blob.
+    #
+    # Examples
+    #
+    #   Shebang.call(FileBlob.new("path/to/file"))
+    #
+    # Returns an Array with one Language if the blob has a shebang with a valid
+    # interpreter, or empty if there is no shebang.
+    def self.call(blob, _ = nil)
+      Language.find_by_interpreter interpreter(blob.data)
+    end
+
+    # Public: Get the interpreter from the shebang
+    #
+    # Returns a String or nil
+    def self.interpreter(data)
+      lines = data.lines
+      return unless match = /^#! ?(.*)$/.match(lines.first)
+
+      tokens = match[1].split(' ')
+      script = tokens.first.split('/').last
+
+      script = tokens[1] if script == 'env'
+
+      # If script has an invalid shebang, we might get here
+      return unless script
+
+      # "python2.6" -> "python2"
+      script.sub! $1, '' if script =~ /(\.\d+)$/
+
+      # Check for multiline shebang hacks that call `exec`
+      if script == 'sh' &&
+        lines.first(5).any? { |l| l.match(/exec (\w+).+\$0.+\$@/) }
+        script = $1
+      end
+
+      File.basename(script)
+    end
+  end
+end

lib/linguist/strategy/filename.rb

@@ -0,0 +1,20 @@
+module Linguist
+  module Strategy
+    # Detects language based on filename and/or extension
+    class Filename
+      def self.call(blob, _)
+        name = blob.name.to_s
+
+        # A bit of an elegant hack. If the file is executable but extensionless,
+        # append a "magic" extension so it can be classified with other
+        # languages that have shebang scripts.
+        extensions = FileBlob.new(name).extensions
+        if extensions.empty? && blob.mode && (blob.mode.to_i(8) & 05) == 05
+          name += ".script!"
+        end
+
+        Language.find_by_filename(name)
+      end
+    end
+  end
+end

lib/linguist/tokenizer.rb

@@ -0,0 +1,198 @@
+require 'strscan'
+
+module Linguist
+  # Generic programming language tokenizer.
+  #
+  # Tokens are designed for use in the language bayes classifier.
+  # It strips any data strings or comments and preserves significant
+  # language symbols.
+  class Tokenizer
+    # Public: Extract tokens from data
+    #
+    # data - String to tokenize
+    #
+    # Returns Array of token Strings.
+    def self.tokenize(data)
+      new.extract_tokens(data)
+    end
+
+    # Read up to 100KB
+    BYTE_LIMIT = 100_000
+
+    # Start state on token, ignore anything till the next newline
+    SINGLE_LINE_COMMENTS = [
+      '//', # C
+      '#',  # Ruby
+      '%',  # Tex
+    ]
+
+    # Start state on opening token, ignore anything until the closing
+    # token is reached.
+    MULTI_LINE_COMMENTS = [
+      ['/*', '*/'],    # C
+      ['<!--', '-->'], # XML
+      ['{-', '-}'],    # Haskell
+      ['(*', '*)'],    # Coq
+      ['"""', '"""']   # Python
+    ]
+
+    START_SINGLE_LINE_COMMENT =  Regexp.compile(SINGLE_LINE_COMMENTS.map { |c|
+      "\s*#{Regexp.escape(c)} "
+    }.join("|"))
+
+    START_MULTI_LINE_COMMENT =  Regexp.compile(MULTI_LINE_COMMENTS.map { |c|
+      Regexp.escape(c[0])
+    }.join("|"))
+
+    # Internal: Extract generic tokens from data.
+    #
+    # data - String to scan.
+    #
+    # Examples
+    #
+    #   extract_tokens("printf('Hello')")
+    #   # => ['printf', '(', ')']
+    #
+    # Returns Array of token Strings.
+    def extract_tokens(data)
+      s = StringScanner.new(data)
+
+      tokens = []
+      until s.eos?
+        break if s.pos >= BYTE_LIMIT
+
+        if token = s.scan(/^#!.+$/)
+          if name = extract_shebang(token)
+            tokens << "SHEBANG#!#{name}"
+          end
+
+        # Single line comment
+        elsif s.beginning_of_line? && token = s.scan(START_SINGLE_LINE_COMMENT)
+          # tokens << token.strip
+          s.skip_until(/\n|\Z/)
+
+        # Multiline comments
+        elsif token = s.scan(START_MULTI_LINE_COMMENT)
+          # tokens << token
+          close_token = MULTI_LINE_COMMENTS.assoc(token)[1]
+          s.skip_until(Regexp.compile(Regexp.escape(close_token)))
+          # tokens << close_token
+
+        # Skip single or double quoted strings
+        elsif s.scan(/"/)
+          if s.peek(1) == "\""
+            s.getch
+          else
+            s.skip_until(/[^\\]"/)
+          end
+        elsif s.scan(/'/)
+          if s.peek(1) == "'"
+            s.getch
+          else
+            s.skip_until(/[^\\]'/)
+          end
+
+        # Skip number literals
+        elsif s.scan(/(0x)?\d(\d|\.)*/)
+
+        # SGML style brackets
+        elsif token = s.scan(/<[^\s<>][^<>]*>/)
+          extract_sgml_tokens(token).each { |t| tokens << t }
+
+        # Common programming punctuation
+        elsif token = s.scan(/;|\{|\}|\(|\)|\[|\]/)
+          tokens << token
+
+        # Regular token
+        elsif token = s.scan(/[\w\.@#\/\*]+/)
+          tokens << token
+
+        # Common operators
+        elsif token = s.scan(/<<?|\+|\-|\*|\/|%|&&?|\|\|?/)
+          tokens << token
+
+        else
+          s.getch
+        end
+      end
+
+      tokens
+    end
+
+    # Internal: Extract normalized shebang command token.
+    #
+    # Examples
+    #
+    #   extract_shebang("#!/usr/bin/ruby")
+    #   # => "ruby"
+    #
+    #   extract_shebang("#!/usr/bin/env node")
+    #   # => "node"
+    #
+    # Returns String token or nil it couldn't be parsed.
+    def extract_shebang(data)
+      s = StringScanner.new(data)
+
+      if path = s.scan(/^#!\s*\S+/)
+        script = path.split('/').last
+        if script == 'env'
+          s.scan(/\s+/)
+          script = s.scan(/\S+/)
+        end
+        script = script[/[^\d]+/, 0] if script
+        return script
+      end
+
+      nil
+    end
+
+    # Internal: Extract tokens from inside SGML tag.
+    #
+    # data - SGML tag String.
+    #
+    # Examples
+    #
+    #   extract_sgml_tokens("<a href='' class=foo>")
+    #   # => ["<a>", "href="]
+    #
+    # Returns Array of token Strings.
+    def extract_sgml_tokens(data)
+      s = StringScanner.new(data)
+
+      tokens = []
+
+      until s.eos?
+        # Emit start token
+        if token = s.scan(/<\/?[^\s>]+/)
+          tokens << "#{token}>"
+
+        # Emit attributes with trailing =
+        elsif token = s.scan(/\w+=/)
+          tokens << token
+
+          # Then skip over attribute value
+          if s.scan(/"/)
+            s.skip_until(/[^\\]"/)
+          elsif s.scan(/'/)
+            s.skip_until(/[^\\]'/)
+          else
+            s.skip_until(/\w+/)
+          end
+
+        # Emit lone attributes
+        elsif token = s.scan(/\w+/)
+          tokens << token
+
+        # Stop at the end of the tag
+        elsif s.scan(/>/)
+          s.terminate
+
+        else
+          s.getch
+        end
+      end
+
+      tokens
+    end
+  end
+end

lib/linguist/vendor.yml

@@ -10,25 +10,75 @@
 ## Vendor Conventions ##
 
 # Caches
-- cache/
+- (^|/)cache/
+
+# Dependencies
+- ^[Dd]ependencies/
 
 # C deps
 #  https://github.com/joyent/node
 - ^deps/
 - ^tools/
+- (^|/)configure$
+- (^|/)configure.ac$
+- (^|/)config.guess$
+- (^|/)config.sub$
 
-# Node depedencies
+# Node dependencies
 - node_modules/
 
-# Vendored depedencies
-- vendor/
+# Bower Components
+- bower_components/
 
+# Erlang bundles
+- ^rebar$
+
+# Go dependencies
+- Godeps/_workspace/
+
+# Minified JavaScript and CSS
+- (\.|-)min\.(js|css)$
+
+# Bootstrap css and js
+- (^|/)bootstrap([^.]*)\.(js|css)$
+
+# Font Awesome
+- font-awesome.css
+
+# Foundation css
+- foundation.css
+
+# Normalize.css
+- normalize.css
+
+# Bourbon SCSS
+- (^|/)[Bb]ourbon/.*\.css$
+- (^|/)[Bb]ourbon/.*\.scss$
+
+# Animate.css
+- animate.css
+
+# Vendored dependencies
+- third[-_]?party/
+- 3rd[-_]?party/
+- vendors?/
+- extern(al)?/
+
+# Debian packaging
+- ^debian/
+
+# Haxelib projects often contain a neko bytecode file named run.n
+- run.n$
 
 ## Commonly Bundled JavaScript frameworks ##
 
 # jQuery
-- (^|/)jquery([^.]*)(\.min)?\.js$
-- (^|/)jquery\-\d\.\d(\.\d)?(\.min)?\.js$
+- (^|/)jquery([^.]*)\.js$
+- (^|/)jquery\-\d\.\d+(\.\d+)?\.js$
+
+# jQuery UI
+- (^|/)jquery\-ui(\-\d\.\d+(\.\d+)?)?(\.\w+)?\.(js|css)$
+- (^|/)jquery\.(ui|effects)\.([^.]*)\.(js|css)$
 
 # Prototype
 - (^|/)prototype(.*)\.js$
@@ -36,6 +86,9 @@
 - (^|/)controls\.js$
 - (^|/)dragdrop\.js$
 
+# Typescript definition files
+- (.*?)\.d\.ts$
+
 # MooTools
 - (^|/)mootools([^.]*)\d+\.\d+.\d+([^.]*)\.js$
 
@@ -49,10 +102,6 @@
 - (^|/)yahoo-([^.]*)\.js$
 - (^|/)yui([^.]*)\.js$
 
-# LESS css
-- (^|/)less([^.]*)(\.min)?\.js$
-- (^|/)less\-\d+\.\d+\.\d+(\.min)?\.js$
-
 # WYS editors
 - (^|/)ckeditor\.js$
 - (^|/)tiny_mce([^.]*)\.js$
@@ -61,36 +110,140 @@
 # MathJax
 - (^|/)MathJax/
 
+# Chart.js
+- (^|/)Chart\.js$
+
+# Codemirror
+- (^|/)[Cc]ode[Mm]irror/(lib|mode|theme|addon|keymap)
+
+# SyntaxHighlighter - http://alexgorbatchev.com/
+- (^|/)shBrush([^.]*)\.js$
+- (^|/)shCore\.js$
+- (^|/)shLegacy\.js$
+
+# AngularJS
+- (^|/)angular([^.]*)\.js$
+
+# D3.js
+- (^|\/)d3(\.v\d+)?([^.]*)\.js$
+
+# React
+- (^|/)react(-[^.]*)?\.js$
+
+# Modernizr
+- (^|/)modernizr\-\d\.\d+(\.\d+)?\.js$
+- (^|/)modernizr\.custom\.\d+\.js$
+
+# Knockout
+- (^|/)knockout-(\d+\.){3}(debug\.)?js$
+
 ## Python ##
 
+# django
+- (^|/)admin_media/
+
 # Fabric
 - ^fabfile\.py$
 
 # WAF
 - ^waf$
 
+# .osx
+- ^.osx$
 
 ## Obj-C ##
 
+# Cocoapods
+- ^Pods/
+
 # Sparkle
 - (^|/)Sparkle/
 
+## Groovy ##
+
+# Gradle
+- (^|/)gradlew$
+- (^|/)gradlew\.bat$
+- (^|/)gradle/wrapper/
+
 ## .NET ##
 
 # Visual Studio IntelliSense
 - -vsdoc\.js$
+- \.intellisense\.js$
 
 # jQuery validation plugin (MS bundles this with asp.net mvc)
-- (^|/)jquery([^.]*)\.validate(\.min)?\.js$
+- (^|/)jquery([^.]*)\.validate(\.unobtrusive)?\.js$
+- (^|/)jquery([^.]*)\.unobtrusive\-ajax\.js$
 
 # Microsoft Ajax
 - (^|/)[Mm]icrosoft([Mm]vc)?([Aa]jax|[Vv]alidation)(\.debug)?\.js$
 
 # NuGet
-- ^[Pp]ackages/
+- ^[Pp]ackages\/.+\.\d+\/
 
 # ExtJS
-- (^|/)extjs/
+- (^|/)extjs/.*?\.js$
+- (^|/)extjs/.*?\.xml$
+- (^|/)extjs/.*?\.txt$
+- (^|/)extjs/.*?\.html$
+- (^|/)extjs/.*?\.properties$
+- (^|/)extjs/.sencha/
+- (^|/)extjs/docs/
+- (^|/)extjs/builds/
+- (^|/)extjs/cmd/
+- (^|/)extjs/examples/
+- (^|/)extjs/locale/
+- (^|/)extjs/packages/
+- (^|/)extjs/plugins/
+- (^|/)extjs/resources/
+- (^|/)extjs/src/
+- (^|/)extjs/welcome/
+
+# Html5shiv
+- (^|/)html5shiv\.js$
 
 # Samples folders
 - ^[Ss]amples/
+
+# LICENSE, README, git config files
+- ^COPYING$
+- LICENSE$
+- License$
+- gitattributes$
+- gitignore$
+- gitmodules$
+- ^README$
+- ^readme$
+
+# Test fixtures
+- ^[Tt]est/fixtures/
+
+# PhoneGap/Cordova
+- (^|/)cordova([^.]*)\.js$
+- (^|/)cordova\-\d\.\d(\.\d)?\.js$
+
+# Foundation js
+- foundation(\..*)?\.js$
+
+# Vagrant
+- ^Vagrantfile$
+
+# .DS_Store's
+- .[Dd][Ss]_[Ss]tore$
+
+# R packages
+- ^vignettes/
+- ^inst/extdata/
+
+# Octicons
+- octicons.css
+- sprockets-octicons.scss
+
+# Typesafe Activator
+- (^|/)activator$
+- (^|/)activator\.bat$
+
+# ProGuard
+- proguard.pro
+- proguard-rules.pro

lib/linguist/version.rb

@@ -0,0 +1,3 @@
+module Linguist
+  VERSION = "4.2.0"
+end

package.json

@@ -0,0 +1,6 @@
+{
+  "repository": "https://github.com/github/linguist",
+  "dependencies": {
+    "season": "~>3.0"
+  }
+}

samples/ABAP/cl_csv_parser.abap

@@ -0,0 +1,219 @@
+*/**
+* The MIT License (MIT)
+* Copyright (c) 2012 René van Mil
+* 
+* Permission is hereby granted, free of charge, to any person obtaining
+* a copy of this software and associated documentation files (the
+* "Software"), to deal in the Software without restriction, including
+* without limitation the rights to use, copy, modify, merge, publish,
+* distribute, sublicense, and/or sell copies of the Software, and to
+* permit persons to whom the Software is furnished to do so, subject to
+* the following conditions:
+* 
+* The above copyright notice and this permission notice shall be
+* included in all copies or substantial portions of the Software.
+* 
+* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+* CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+* TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+*/
+
+*----------------------------------------------------------------------*
+*       CLASS CL_CSV_PARSER DEFINITION
+*----------------------------------------------------------------------*
+*
+*----------------------------------------------------------------------*
+class cl_csv_parser definition
+  public
+  inheriting from cl_object
+  final
+  create public .
+
+  public section.
+*"* public components of class CL_CSV_PARSER
+*"* do not include other source files here!!!
+
+    type-pools abap .
+    methods constructor
+      importing
+        !delegate type ref to if_csv_parser_delegate
+        !csvstring type string
+        !separator type c
+        !skip_first_line type abap_bool .
+    methods parse
+      raising
+        cx_csv_parse_error .
+  protected section.
+*"* protected components of class CL_CSV_PARSER
+*"* do not include other source files here!!!
+  private section.
+*"* private components of class CL_CSV_PARSER
+*"* do not include other source files here!!!
+
+    constants _textindicator type c value '"'.              "#EC NOTEXT
+    data _delegate type ref to if_csv_parser_delegate .
+    data _csvstring type string .
+    data _separator type c .
+    type-pools abap .
+    data _skip_first_line type abap_bool .
+
+    methods _lines
+      returning
+        value(returning) type stringtab .
+    methods _parse_line
+      importing
+        !line type string
+      returning
+        value(returning) type stringtab
+      raising
+        cx_csv_parse_error .
+endclass.                    "CL_CSV_PARSER DEFINITION
+
+
+
+*----------------------------------------------------------------------*
+*       CLASS CL_CSV_PARSER IMPLEMENTATION
+*----------------------------------------------------------------------*
+*
+*----------------------------------------------------------------------*
+class cl_csv_parser implementation.
+
+
+* <SIGNATURE>---------------------------------------------------------------------------------------+
+* | Instance Public Method CL_CSV_PARSER->CONSTRUCTOR
+* +-------------------------------------------------------------------------------------------------+
+* | [--->] DELEGATE                       TYPE REF TO IF_CSV_PARSER_DELEGATE
+* | [--->] CSVSTRING                      TYPE        STRING
+* | [--->] SEPARATOR                      TYPE        C
+* | [--->] SKIP_FIRST_LINE                TYPE        ABAP_BOOL
+* +--------------------------------------------------------------------------------------</SIGNATURE>
+  method constructor.
+    super->constructor( ).
+    _delegate = delegate.
+    _csvstring = csvstring.
+    _separator = separator.
+    _skip_first_line = skip_first_line.
+  endmethod.                    "constructor
+
+
+* <SIGNATURE>---------------------------------------------------------------------------------------+
+* | Instance Public Method CL_CSV_PARSER->PARSE
+* +-------------------------------------------------------------------------------------------------+
+* | [!CX!] CX_CSV_PARSE_ERROR
+* +--------------------------------------------------------------------------------------</SIGNATURE>
+  method parse.
+    data msg type string.
+    if _csvstring is initial.
+      message e002(csv) into msg.
+      raise exception type cx_csv_parse_error
+        exporting
+          message = msg.
+    endif.
+
+    " Get the lines
+    data is_first_line type abap_bool value abap_true.
+    data lines type standard table of string.
+    lines = _lines( ).
+    field-symbols <line> type string.
+    loop at lines assigning <line>.
+      " Should we skip the first line?
+      if _skip_first_line = abap_true and is_first_line = abap_true.
+        is_first_line = abap_false.
+        continue.
+      endif.
+      " Parse the line
+      data values type standard table of string.
+      values = _parse_line( <line> ).
+      " Send values to delegate
+      _delegate->values_found( values ).
+    endloop.
+  endmethod.                    "parse
+
+
+* <SIGNATURE>---------------------------------------------------------------------------------------+
+* | Instance Private Method CL_CSV_PARSER->_LINES
+* +-------------------------------------------------------------------------------------------------+
+* | [<-()] RETURNING                      TYPE        STRINGTAB
+* +--------------------------------------------------------------------------------------</SIGNATURE>
+  method _lines.
+    split _csvstring at cl_abap_char_utilities=>cr_lf into table returning.
+  endmethod.                    "_lines
+
+
+* <SIGNATURE>---------------------------------------------------------------------------------------+
+* | Instance Private Method CL_CSV_PARSER->_PARSE_LINE
+* +-------------------------------------------------------------------------------------------------+
+* | [--->] LINE                           TYPE        STRING
+* | [<-()] RETURNING                      TYPE        STRINGTAB
+* | [!CX!] CX_CSV_PARSE_ERROR
+* +--------------------------------------------------------------------------------------</SIGNATURE>
+  method _parse_line.
+    data msg type string.
+
+    data csvvalue type string.
+    data csvvalues type standard table of string.
+
+    data char type c.
+    data pos type i value 0.
+    data len type i.
+    len = strlen( line ).
+    while pos < len.
+      char = line+pos(1).
+      if char <> _separator.
+        if char = _textindicator.
+          data text_ended type abap_bool.
+          text_ended = abap_false.
+          while text_ended = abap_false.
+            pos = pos + 1.
+            if pos < len.
+              char = line+pos(1).
+              if char = _textindicator.
+                text_ended = abap_true.
+              else.
+                if char is initial. " Space
+                  concatenate csvvalue ` ` into csvvalue.
+                else.
+                  concatenate csvvalue char into csvvalue.
+                endif.
+              endif.
+            else.
+              " Reached the end of the line while inside a text value
+              " This indicates an error in the CSV formatting
+              text_ended = abap_true.
+              message e003(csv) into msg.
+              raise exception type cx_csv_parse_error
+                exporting
+                  message = msg.
+            endif.
+          endwhile.
+          " Check if next character is a separator, otherwise the CSV formatting is incorrect
+          data nextpos type i.
+          nextpos = pos + 1.
+          if nextpos < len and line+nextpos(1) <> _separator.
+            message e003(csv) into msg.
+            raise exception type cx_csv_parse_error
+              exporting
+                message = msg.
+          endif.
+        else.
+          if char is initial. " Space
+            concatenate csvvalue ` ` into csvvalue.
+          else.
+            concatenate csvvalue char into csvvalue.
+          endif.
+        endif.
+      else.
+        append csvvalue to csvvalues.
+        clear csvvalue.
+      endif.
+      pos = pos + 1.
+    endwhile.
+    append csvvalue to csvvalues. " Don't forget the last value
+
+    returning = csvvalues.
+  endmethod.                    "_parse_line
+endclass.                    "CL_CSV_PARSER IMPLEMENTATION

samples/AGS Script/GlobalScript.asc

@@ -0,0 +1,521 @@
+// main global script file
+
+// A function that initializes a bunch of stuff.
+function initialize_control_panel() { 
+  // Centre the control panel
+  gPanel.Centre(); 
+  // Centre the Restart dialog as well
+  gRestartYN.Centre(); 
+  if (!IsSpeechVoxAvailable()) { 
+    // If there is no speech-vox file, and therefore no speech,
+    // disable all the controls related with speech.
+    lblVoice.Visible = false;  
+    btnVoice.Visible = false;  
+    sldVoice.Visible = false;
+  }
+  else {
+    // If there *is*, then set it to voice and text. It's best to use
+    // both whenever possible, for the player's sake.
+    SetVoiceMode(eSpeechVoiceAndText); 
+    // And reflect this in the control panel.
+    btnVoice.Text = "Voice and Text"; 
+  }
+  if (!System.SupportsGammaControl) {
+    // If we can't change the gamma settings, disable the relevant options.
+    sldGamma.Visible = false; 
+    lblGamma.Visible = false;
+  } 
+  
+  //And now, set all the defaults
+  System.Volume = 100;
+  sldAudio.Value = System.Volume;
+  SetGameSpeed(40);
+  sldSpeed.Value = 40;
+  if (IsSpeechVoxAvailable()) {
+     SetVoiceMode(eSpeechVoiceAndText);
+     btnVoice.Text = "Voice and Text";
+     sldVoice.Value = 255;
+     SetSpeechVolume(255);
+  }
+  if (System.SupportsGammaControl) {
+    System.Gamma = 100;
+    sldGamma.Value = 100;
+  }
+}
+
+// Called when the game starts, before the first room is loaded
+function game_start() {   
+  // Put the code all in a function and then just call the function. 
+  // It saves cluttering up places like game_start.
+  initialize_control_panel(); 
+  // Use the KeyboardMovement module to, per default, replicate the standard
+  // keyboard movement of most Sierra games. See KeyboardMovement.txt for more info
+  KeyboardMovement.SetMode(eKeyboardMovement_Tapping); 
+}
+
+function repeatedly_execute() {
+  
+  // Put here anything you want to happen every game cycle, even when
+  // the game is paused. This will not run when the game is blocked
+  // inside a command like a blocking Walk()
+  
+  if (IsGamePaused() == 1) return;
+
+  // Put here anything you want to happen every game cycle, but not
+  // when the game is paused.
+}
+
+function repeatedly_execute_always() {
+  
+  // Put anything you want to happen every game cycle, even
+  // when the game is blocked inside a command like a
+  // blocking Walk().
+  // You cannot run blocking commands from this function.
+  
+}
+
+function show_inventory_window () 
+{
+  gInventory.Visible = true;
+  // switch to the Use cursor (to select items with)
+  mouse.Mode = eModeInteract;
+  // But, override the appearance to look like the arrow
+  mouse.UseModeGraphic(eModePointer);
+}
+
+function show_save_game_dialog()
+{
+  gSaveGame.Visible = true;
+  // Get the list of save games
+  lstSaveGamesList.FillSaveGameList();
+  if (lstSaveGamesList.ItemCount > 0)
+  {
+    // If there is at least one, set the default text
+    // to be the first game's name
+    txtNewSaveName.Text = lstSaveGamesList.Items[0];
+  }
+  else
+  {
+    // No save games yet, default empty text.
+    txtNewSaveName.Text = "";
+  }
+  mouse.UseModeGraphic(eModePointer);
+  gIconbar.Visible = false;
+}
+
+function show_restore_game_dialog()
+{
+  gRestoreGame.Visible = true;
+  lstRestoreGamesList.FillSaveGameList();
+  mouse.UseModeGraphic(eModePointer);
+  gIconbar.Visible = false;
+}
+
+function close_save_game_dialog()
+{
+  gSaveGame.Visible = false;
+  mouse.UseDefaultGraphic();
+  gIconbar.Visible = true;
+}
+
+function close_restore_game_dialog()
+{
+  gRestoreGame.Visible = false;
+  mouse.UseDefaultGraphic();
+  gIconbar.Visible = true;
+}
+
+// Called when a key is pressed. keycode holds the key's ASCII code
+function on_key_press(eKeyCode keycode) {
+  // The following is called before "if game is paused keycode=0", so
+  // it'll happen even when the game is paused.
+  
+  if ((keycode == eKeyEscape) && gRestartYN.Visible) {
+    //Use ESC to cancel restart.
+    gRestartYN.Visible = false; 
+    gIconbar.Visible = true;
+    // If the panel's not ON, then the player must have gotten here by tapping F9,
+    // therefore his cursor needs restoring. If the panel IS on, then it doesn't,
+    // because it's already a pointer. Get used to thinking like this!!
+    if (!gPanel.Visible) mouse.UseDefaultGraphic(); 
+    return;
+  }
+  if ((keycode == eKeyEscape) && gPanel.Visible) {
+    // Use ESC to turn the panel off.
+    gPanel.Visible = false; 
+    mouse.UseDefaultGraphic();
+    gIconbar.Visible = true;
+    return;
+  }
+  if ((keycode == eKeyEscape) && (gSaveGame.Visible))
+  {
+    // Use ESC to close the save game dialog
+    close_save_game_dialog();
+    return;
+  }
+  if ((keycode == eKeyEscape) && (gRestoreGame.Visible))
+  {
+    // Use ESC to close the restore game dialog
+    close_restore_game_dialog();
+    return;
+  }
+  
+  if (keycode == eKeyReturn) { 
+    // ENTER, in this case merely confirms restart
+    if (gRestartYN.Visible) RestartGame();
+  }
+
+  if (IsGamePaused() || (IsInterfaceEnabled() == 0))
+  {
+    // If the game is paused with a modal GUI on the
+    // screen, or the player interface is disabled in
+    // a cut scene, ignore any keypresses.
+    return;
+  }
+
+  // FUNCTION KEYS AND SYSTEM SHORTCUTS
+  if (keycode == eKeyEscape) {
+    // ESC
+    gPanel.Visible = true; 
+    gIconbar.Visible = false;
+    mouse.UseModeGraphic(eModePointer);
+  }
+  if (keycode == eKeyCtrlQ)  QuitGame(1);   // Ctrl-Q
+  if (keycode == eKeyF5) show_save_game_dialog();   // F5
+  if (keycode == eKeyF7) show_restore_game_dialog();  // F7
+  if (keycode == eKeyF9) {
+    // F9, asks the player to confirm restarting (so much better to always confirm first)
+    gRestartYN.Visible = true;  
+    gIconbar.Visible = false;
+    mouse.UseModeGraphic(eModePointer);
+  }
+  if (keycode == eKeyF12) SaveScreenShot("scrnshot.bmp");  // F12
+  if (keycode == eKeyTab)   show_inventory_window();  // Tab, show inventory
+
+  // GAME COMMAND SHORTCUTS
+  if (keycode == 'W') mouse.Mode=eModeWalkto; //Notice this alternate way to indicate keycodes.
+  if (keycode == 'L') mouse.Mode=eModeLookat; //Note that all we do here is set modes.
+  if (keycode == 'U') mouse.Mode=eModeInteract; //If you want something else to happen, such as GUI buttons highlighting,
+  if (keycode == 'T') mouse.Mode=eModeTalkto; //you'll need some more scripting done.
+  if (keycode == 'I') mouse.Mode=eModeUseinv; //But this will, as-is, give you some standard keyboard shortcuts your players will very much appreciate.
+
+  // For extra cursor modes, such as pick up, feel free to add as you will.
+  // Uncomment the line below if you use the "Pick Up" mode.
+  //if (keycode == 'P' || keycode == 'G') mouse.Mode=eModePickup; 
+
+  // DEBUG FUNCTIONS
+  if (keycode == eKeyCtrlS)  Debug(0,0);  // Ctrl-S, give all inventory
+  if (keycode == eKeyCtrlV)  Debug(1,0);  // Ctrl-V, version
+  if (keycode == eKeyCtrlA)  Debug(2,0);  // Ctrl-A, show walkable areas
+  if (keycode == eKeyCtrlX)  Debug(3,0);  // Ctrl-X, teleport to room
+  if (keycode == eKeyCtrlW && game.debug_mode) 
+    player.PlaceOnWalkableArea(); //Ctrl-W, move to walkable area 
+}
+
+
+function on_mouse_click(MouseButton button) {
+  // called when a mouse button is clicked. button is either LEFT or RIGHT
+  if (IsGamePaused() == 1) {
+    // Game is paused, so do nothing (ie. don't allow mouse click)
+  }
+  else if (button == eMouseLeft) {
+    ProcessClick(mouse.x, mouse.y, mouse.Mode );
+  }
+  else if (button == eMouseRight || button == eMouseWheelSouth){
+    // right-click our mouse-wheel down, so cycle cursor
+    mouse.SelectNextMode();
+  }
+  else if (button == eMouseMiddle) { 
+    // Middle-button-click, default make character walk to clicked area (a little shortcut)
+    // Could have been just "player.Walk(mouse.x,mouse.y)", but it's best to
+    // leave our options open - what if you have a special script triggered
+    // on "walking" mode?
+    ProcessClick(mouse.x, mouse.y, eModeWalkto); 
+  }
+  else if (button == eMouseWheelNorth) { 
+    // Mouse-wheel up, cycle cursors 
+    // If mode isn't WALK, set the previous mode (notice usage of numbers instead
+    // of eNums, when it suits us)...
+    if (mouse.Mode>0) mouse.Mode=mouse.Mode-1; 
+    else 
+    { 
+      // ...but if it is WALK mode...
+      if (player.ActiveInventory!=null) 
+      {
+        //...and the player has a selected inventory item, set mouse mode to UseInv. 
+        mouse.Mode=eModeUseinv; 
+      }
+      else 
+      {
+        // If they don't, however, just set it to mode TALK (change this line if you add more cursor modes)
+        mouse.Mode=eModeTalkto; 
+      }
+    }
+  }
+}
+
+function interface_click(int interface, int button) {
+  // This function is obsolete, from 2.62 and earlier versions.
+}
+
+function btnInvUp_Click(GUIControl *control, MouseButton button) {
+  invCustomInv.ScrollUp();
+}
+
+function btnInvDown_Click(GUIControl *control, MouseButton button) {
+  invCustomInv.ScrollDown();
+}
+
+function btnInvOK_Click(GUIControl *control, MouseButton button) {
+	// They pressed the OK button, close the GUI
+	gInventory.Visible = false;
+	mouse.UseDefaultGraphic();
+}
+
+function btnInvSelect_Click(GUIControl *control, MouseButton button) {
+  
+	// They pressed SELECT, so switch to the Get cursor
+	mouse.Mode = eModeInteract;
+	// But, override the appearance to look like the arrow
+	mouse.UseModeGraphic(eModePointer);
+}
+
+function btnIconInv_Click(GUIControl *control, MouseButton button) {
+  
+  show_inventory_window();
+}
+
+function btnIconCurInv_Click(GUIControl *control, MouseButton button) {
+  
+  if (player.ActiveInventory != null)
+    mouse.Mode = eModeUseinv;
+}
+
+function btnIconSave_Click(GUIControl *control, MouseButton button) 
+{
+  show_save_game_dialog();
+}
+
+function btnIconLoad_Click(GUIControl *control, MouseButton button) 
+{
+  show_restore_game_dialog();
+}
+
+function btnIconExit_Click(GUIControl *control, MouseButton button) {
+  
+  QuitGame(1);
+}
+
+function btnIconAbout_Click(GUIControl *control, MouseButton button) {
+  
+  gPanel.Visible=true;
+  gIconbar.Visible=false;
+  mouse.UseModeGraphic(eModePointer);
+}
+
+function cEgo_Look()
+{
+  Display("Damn, I'm looking good!");
+}
+
+function cEgo_Interact()
+{
+  Display("You rub your hands up and down your clothes.");
+}
+
+function cEgo_Talk()
+{
+  Display("Talking to yourself is a sign of madness!");
+}
+
+//START OF CONTROL PANEL FUNCTIONS
+function btnSave_OnClick(GUIControl *control, MouseButton button)
+{
+  gPanel.Visible = false;
+  mouse.UseDefaultGraphic();
+  gIconbar.Visible = true;
+  Wait(1);
+  btnIconSave_Click(btnIconSave, eMouseLeft);
+}
+
+function gControl_OnClick(GUI *theGui, MouseButton button)
+{
+
+}
+
+function btnAbout_OnClick(GUIControl *control, MouseButton button)
+{
+Display("Adventure Game Studio run-time engine default game.");
+}
+
+function btnQuit_OnClick(GUIControl *control, MouseButton button)
+{
+  gPanel.Visible = false;
+  Wait(1);
+  QuitGame(1);
+  gPanel.Visible = true;
+  gIconbar.Visible = false;
+  mouse.UseModeGraphic(eModePointer);
+}
+
+function btnLoad_OnClick(GUIControl *control, MouseButton button)
+{
+  gPanel.Visible = false;
+  mouse.UseDefaultGraphic();
+  gIconbar.Visible = true;
+  Wait(1);
+  btnIconLoad_Click(btnIconLoad, eMouseLeft);
+}
+
+function btnResume_OnClick(GUIControl *control, MouseButton button)
+{
+  gPanel.Visible = false;
+  mouse.UseDefaultGraphic();
+  gIconbar.Visible = true;
+}
+
+function sldAudio_OnChange(GUIControl *control)
+{
+  System.Volume = sldAudio.Value;
+}
+
+function sldVoice_OnChange(GUIControl *control)
+{
+  // Sets voice volume. Note that we don't check for the existence of speech.vox - 
+  // we did that in game_start, so if it's not there the slider won't even be available.
+  SetSpeechVolume(sldVoice.Value); 
+}
+
+function btnVoice_OnClick(GUIControl *control, MouseButton button)
+{
+  // Note that we don't check for the existence of speech.vox - we did that in game_start,
+  // so if it's not there the button won't even be available.
+  if (btnVoice.Text == "Voice and Text") { 
+    SetVoiceMode(eSpeechVoiceOnly); 
+    btnVoice.Text = "Voice only";
+  }
+  else if (btnVoice.Text == "Voice only") {
+    SetVoiceMode(eSpeechTextOnly);
+    btnVoice.Text = "Text only";
+  }
+  else if (btnVoice.Text == "Text only") {
+    SetVoiceMode(eSpeechVoiceAndText);
+    btnVoice.Text = "Voice and Text";
+  }
+}
+
+function sldGamma_OnChange(GUIControl *control)
+{
+  // Set the gamma. Note there's no need to check for anything else, as we ensured,
+  // in game_start, that the slider won't even appear if it's not possible to do this.
+  System.Gamma = sldGamma.Value; 
+}
+
+function btnDefault_OnClick(GUIControl *control, MouseButton button)
+{
+  // Reset everything to default. You'll have to edit these as well as the sliders
+  // if you'd rather have different default parameters.
+  System.Volume = 100;
+  sldAudio.Value = System.Volume;
+  sldSpeed.Value = 40;
+  SetGameSpeed(40);
+  if (IsSpeechVoxAvailable()) {
+     SetVoiceMode(eSpeechVoiceAndText);
+     btnVoice.Text = "Voice and Text";
+     sldVoice.Value = 255;
+     SetSpeechVolume(255);
+  }
+  if (System.SupportsGammaControl) {
+    System.Gamma = 100;
+    sldGamma.Value = 100;
+  }
+}
+//END OF CONTROL PANEL FUNCTIONS
+
+function dialog_request(int param) 
+{
+  // This is used by the dialog text parser if you need to process
+  // text that the player types in to the parser.
+  // It is not used by default.
+}
+
+function sldSpeed_OnChange(GUIControl *control)
+{
+  SetGameSpeed(sldSpeed.Value);
+}
+
+function btnRestart_OnClick(GUIControl *control, MouseButton button)
+{
+  gRestartYN.Visible=true;
+  gIconbar.Visible=false;
+}
+
+function btnRestartYes_OnClick(GUIControl *control, MouseButton button)
+{
+  RestartGame();
+}
+
+function btnRestartNo_OnClick(GUIControl *control, MouseButton button)
+{
+  gRestartYN.Visible = false;
+  gIconbar.Visible = true;
+  // If the panel's not ON, then the player must have gotten here by tapping F9,
+  // therefore his cursor needs restoring. If the panel IS on, then it doesn't,
+  // because it's already a pointer. Get used to thinking like this!!
+  if (!gPanel.Visible) mouse.UseDefaultGraphic(); 
+}
+
+function btnCancelSave_OnClick(GUIControl *control, MouseButton button)
+{
+  close_save_game_dialog();
+}
+
+function btnSaveGame_OnClick(GUIControl *control, MouseButton button)
+{
+  int gameSlotToSaveInto = lstSaveGamesList.ItemCount + 1;
+  int i = 0;
+  while (i < lstSaveGamesList.ItemCount)
+  {
+    if (lstSaveGamesList.Items[i] == txtNewSaveName.Text)
+    {
+      gameSlotToSaveInto = lstSaveGamesList.SaveGameSlots[i];
+    }
+    i++;
+  }
+  SaveGameSlot(gameSlotToSaveInto, txtNewSaveName.Text);
+  close_save_game_dialog();
+}
+
+function btnCancelRestore_OnClick(GUIControl *control, MouseButton button)
+{
+  close_restore_game_dialog();
+}
+
+function btnRestoreGame_OnClick(GUIControl *control, MouseButton button)
+{
+  if (lstRestoreGamesList.SelectedIndex >= 0)
+  {
+    RestoreGameSlot(lstRestoreGamesList.SaveGameSlots[lstRestoreGamesList.SelectedIndex]);
+  }
+  close_restore_game_dialog();
+}
+
+function lstSaveGamesList_OnSelectionCh(GUIControl *control)
+{
+  txtNewSaveName.Text = lstSaveGamesList.Items[lstSaveGamesList.SelectedIndex];
+}
+
+function txtNewSaveName_OnActivate(GUIControl *control)
+{
+  // Pressing return in the text box simulates clicking the Save button
+  btnSaveGame_OnClick(control, eMouseLeft);
+}
+
+function btnDeleteSave_OnClick(GUIControl *control, MouseButton button)
+{
+  if (lstSaveGamesList.SelectedIndex >= 0)
+  {
+    DeleteSaveSlot(lstSaveGamesList.SaveGameSlots[lstSaveGamesList.SelectedIndex]);
+    lstSaveGamesList.FillSaveGameList();
+  }
+}

samples/AGS Script/GlobalScript.ash

@@ -0,0 +1,4 @@
+// Main header script - this will be included into every script in
+// the game (local and global). Do not place functions here; rather,
+// place import definitions and #define names here to be used by all
+// scripts.

samples/AGS Script/KeyboardMovement_102.asc

@@ -0,0 +1,216 @@
+// Main script for module 'KeyboardMovement'
+
+//****************************************************************************************************
+// DEFINITIONS
+//****************************************************************************************************
+
+#define DISTANCE 10000// distance player walks in Tapping mode before he stops
+
+enum KeyboardMovement_Directions {
+	eKeyboardMovement_Stop, 
+	eKeyboardMovement_DownLeft, 
+	eKeyboardMovement_Down, 
+	eKeyboardMovement_DownRight, 
+	eKeyboardMovement_Left, 
+	eKeyboardMovement_Right, 
+	eKeyboardMovement_UpLeft, 
+	eKeyboardMovement_Up, 
+	eKeyboardMovement_UpRight
+};
+
+//****************************************************************************************************
+// VARIABLES
+//****************************************************************************************************
+
+// keycodes as variables for future key customization functions (static variables?):
+int KeyboardMovement_KeyDown = 380; // down arrow
+int KeyboardMovement_KeyLeft = 375; // left arrow
+int KeyboardMovement_KeyRight = 377; // right arrow
+int KeyboardMovement_KeyUp = 372; // up arrow
+int KeyboardMovement_KeyDownRight = 381; // PgDn (numpad)
+int KeyboardMovement_KeyUpRight = 373; // PgUp (numpad)
+int KeyboardMovement_KeyDownLeft = 379; // End (numpad)
+int KeyboardMovement_KeyUpLeft = 371; // Home (numpad)
+int KeyboardMovement_KeyStop = 376; // 5 (numpad)
+
+KeyboardMovement_Modes KeyboardMovement_Mode = eKeyboardMovement_None; // stores current keyboard control mode (disabled by default)
+
+KeyboardMovement_Directions KeyboardMovement_CurrentDirection = eKeyboardMovement_Stop; // stores current walking direction of player character
+
+//****************************************************************************************************
+// USER FUNCTIONS
+//****************************************************************************************************
+
+//====================================================================================================
+
+static function KeyboardMovement::SetMode(KeyboardMovement_Modes mode) {
+	KeyboardMovement_Mode = mode;
+}
+
+//====================================================================================================
+
+// key customization functions here
+
+//====================================================================================================
+
+//****************************************************************************************************
+// EVENT HANDLER FUNCTIONS
+//****************************************************************************************************
+
+//====================================================================================================
+
+function repeatedly_execute() {
+
+	//--------------------------------------------------
+	// Pressing mode
+	//--------------------------------------------------
+
+	if ((IsGamePaused() == true) || (KeyboardMovement_Mode != eKeyboardMovement_Pressing) || (IsInterfaceEnabled() == false) || (player.on == false)) return 0;
+	  // if game is paused, module or mode disabled, interface disabled or player character hidden, quit function
+
+	KeyboardMovement_Directions newdirection; // declare variable storing new direction
+
+	// get new direction:
+	if ( ((IsKeyPressed(KeyboardMovement_KeyDown)) && (IsKeyPressed(KeyboardMovement_KeyRight))) || (IsKeyPressed(KeyboardMovement_KeyDownRight)) ) newdirection = eKeyboardMovement_DownRight; // if down&right arrows or PgDn (numeric pad) held down, set new direction to Down-Right
+	else if ( ((IsKeyPressed(KeyboardMovement_KeyUp)) && (IsKeyPressed(KeyboardMovement_KeyRight))) || (IsKeyPressed(KeyboardMovement_KeyUpRight)) ) newdirection = eKeyboardMovement_UpRight; // up&right arrows or PgUp (numpad)
+	else if ( ((IsKeyPressed(KeyboardMovement_KeyDown)) && (IsKeyPressed(KeyboardMovement_KeyLeft))) || (IsKeyPressed(KeyboardMovement_KeyDownLeft)) ) newdirection = eKeyboardMovement_DownLeft; // down&left arrows or End (numpad)
+	else if ( ((IsKeyPressed(KeyboardMovement_KeyUp)) && (IsKeyPressed(KeyboardMovement_KeyLeft))) || (IsKeyPressed(KeyboardMovement_KeyUpLeft)) ) newdirection = eKeyboardMovement_UpLeft; // up&left arrows or Home (numpad)
+	else if (IsKeyPressed(KeyboardMovement_KeyDown)) newdirection = eKeyboardMovement_Down; // down arrow
+	else if (IsKeyPressed(KeyboardMovement_KeyLeft)) newdirection = eKeyboardMovement_Left; // left arrow
+	else if (IsKeyPressed(KeyboardMovement_KeyRight)) newdirection = eKeyboardMovement_Right; // right arrow
+	else if (IsKeyPressed(KeyboardMovement_KeyUp)) newdirection = eKeyboardMovement_Up; // up arrow
+	else newdirection = eKeyboardMovement_Stop; // if none of the above held down, set it to stop player character
+
+	if (IsKeyPressed(KeyboardMovement_KeyStop)) newdirection = eKeyboardMovement_Stop; // if 5 (numeric pad) held down, stop player character, regardless of whether some of the above are held down
+
+	if (newdirection != KeyboardMovement_CurrentDirection) { // if new direction is different from current direction
+
+		if (newdirection == eKeyboardMovement_Stop) player.StopMoving(); // if new direction is the Stop command, stop movement of player character
+		else { // if new direction is NOT the Stop command
+
+			int dx, dy; // declare variables storing new walk coordinates
+			if (newdirection == eKeyboardMovement_DownRight) {
+				dx = DISTANCE;
+				dy = DISTANCE;
+			}
+			else if (newdirection == eKeyboardMovement_UpRight) {
+				dx = DISTANCE;
+				dy = -DISTANCE;
+			}
+			else if (newdirection == eKeyboardMovement_DownLeft) {
+				dx = -DISTANCE;
+				dy = DISTANCE;
+			}
+			else if (newdirection == eKeyboardMovement_UpLeft) {
+				dx = -DISTANCE;
+				dy = -DISTANCE;
+			}
+			else if (newdirection == eKeyboardMovement_Down) {
+				dx = 0;
+				dy = DISTANCE;
+			}
+			else if (newdirection == eKeyboardMovement_Left) {
+				dx = -DISTANCE;
+				dy = 0;
+			}
+			else if (newdirection == eKeyboardMovement_Right) {
+				dx = DISTANCE;
+				dy = 0;
+			}
+			else if (newdirection == eKeyboardMovement_Up) {
+				dx = 0;
+				dy = -DISTANCE;
+			}
+
+			player.WalkStraight(player.x + dx, player.y + dy, eNoBlock); // walk player character to the new coordinates
+		}
+		KeyboardMovement_CurrentDirection = newdirection; // update current direction to new direction
+
+	}
+
+}
+
+//====================================================================================================
+
+function on_key_press(int keycode) {
+
+	//--------------------------------------------------
+	// Tapping mode
+	//--------------------------------------------------
+
+	if ((IsGamePaused() == true) || (KeyboardMovement_Mode != eKeyboardMovement_Tapping) || (IsInterfaceEnabled() == false) || (player.on == false)) return 0;
+	  // if game is paused, module or mode disabled, interface disabled or player character hidden, quit function
+
+	KeyboardMovement_Directions newdirection; // declare variable storing new direction
+
+	// get new direction:
+	if (keycode == KeyboardMovement_KeyDownRight) newdirection = eKeyboardMovement_DownRight; // if down-right key pressed, set new direction to Down-Right
+	else if (keycode == KeyboardMovement_KeyUpRight) newdirection = eKeyboardMovement_UpRight;
+	else if (keycode == KeyboardMovement_KeyDownLeft) newdirection = eKeyboardMovement_DownLeft;
+	else if (keycode == KeyboardMovement_KeyUpLeft) newdirection = eKeyboardMovement_UpLeft;
+	else if (keycode == KeyboardMovement_KeyDown) newdirection = eKeyboardMovement_Down;
+	else if (keycode == KeyboardMovement_KeyLeft) newdirection = eKeyboardMovement_Left;
+	else if (keycode == KeyboardMovement_KeyRight) newdirection = eKeyboardMovement_Right;
+	else if (keycode == KeyboardMovement_KeyUp) newdirection = eKeyboardMovement_Up;
+	else if (keycode == KeyboardMovement_KeyStop) newdirection = eKeyboardMovement_Stop; // if stop key pressed, set to stop player character
+
+	if (newdirection != KeyboardMovement_CurrentDirection) { // if new direction is different from current direction
+
+		if (newdirection == eKeyboardMovement_Stop) player.StopMoving(); // if new direction is the Stop command, stop movement of player character
+		else { // if new direction is NOT the Stop command
+
+			int dx, dy; // declare variables storing new walk coordinates
+			if (newdirection == eKeyboardMovement_DownRight) {
+				dx = DISTANCE;
+				dy = DISTANCE;
+			}
+			else if (newdirection == eKeyboardMovement_UpRight) {
+				dx = DISTANCE;
+				dy = -DISTANCE;
+			}
+			else if (newdirection == eKeyboardMovement_DownLeft) {
+				dx = -DISTANCE;
+				dy = DISTANCE;
+			}
+			else if (newdirection == eKeyboardMovement_UpLeft) {
+				dx = -DISTANCE;
+				dy = -DISTANCE;
+			}
+			else if (newdirection == eKeyboardMovement_Down) {
+				dx = 0;
+				dy = DISTANCE;
+			}
+			else if (newdirection == eKeyboardMovement_Left) {
+				dx = -DISTANCE;
+				dy = 0;
+			}
+			else if (newdirection == eKeyboardMovement_Right) {
+				dx = DISTANCE;
+				dy = 0;
+			}
+			else if (newdirection == eKeyboardMovement_Up) {
+				dx = 0;
+				dy = -DISTANCE;
+			}
+
+			player.WalkStraight(player.x + dx, player.y + dy, eNoBlock); // walk player character to the new coordinates
+		}
+		KeyboardMovement_CurrentDirection = newdirection; // update current direction to new direction
+
+	}
+	else { // if new direction is same as current direction
+		player.StopMoving(); // stop player character
+		KeyboardMovement_CurrentDirection = eKeyboardMovement_Stop; // update current direction
+	}
+
+}
+
+//====================================================================================================
+
+function on_event(EventType event, int data) {
+
+	if (event == eEventLeaveRoom) KeyboardMovement_CurrentDirection = eKeyboardMovement_Stop;
+
+}
+
+//====================================================================================================

samples/AGS Script/KeyboardMovement_102.ash

@@ -0,0 +1,13 @@
+// Script header for module 'KeyboardMovement'
+
+#define KeyboardMovement_VERSION 101
+
+enum KeyboardMovement_Modes {
+	eKeyboardMovement_None, 
+	eKeyboardMovement_Tapping, 
+	eKeyboardMovement_Pressing
+};
+
+struct KeyboardMovement {
+	import static function SetMode(KeyboardMovement_Modes mode);
+};

samples/APL/DeepakChopra.apl

@@ -0,0 +1,18 @@
+⍝   You can try this at http://tryapl.org/
+⍝   I can not explain how much I suddenly love this crypto-language
+
+
+
+Starts     ← 'Experiential truth ' 'The physical world ' 'Non-judgment '       'Quantum physics '
+Middles    ← 'nurtures an '        'projects onto '      'imparts reality to ' 'constructs with '
+Qualifiers ← 'abundance of '       'the barrier of '     'self-righteous '     'potential '
+Finishes   ← 'marvel.'             'choices.'            'creativity.'         'actions.'
+
+rf     ← {(?⍴⍵)⊃⍵}
+erf    ← {rf ¨ ⍵}
+
+deepak ← {erf Starts Middles Qualifiers Finishes}
+
+
+
+deepak ⍬

samples/APL/UT.dyalog

@@ -0,0 +1,367 @@
+:NameSpace UT
+
+    sac ← 0
+    expect_orig ← expect ← ⎕NS⍬
+    exception ← ⍬
+    nexpect_orig ← nexpect ← ⎕NS⍬
+
+    ∇ {Z}←{Conf}run Argument;PRE_test;POST_test;TEST_step;COVER_step;FromSpace
+     
+      load_display_if_not_already_loaded
+      load_salt_scripts_into_current_namespace_if_configured
+     
+      FromSpace←1⊃⎕RSI
+     
+      PRE_test←{}
+      POST_test←{}
+      COVER_step←{}
+      :If 0≠⎕NC'Conf'
+          :If Conf has'cover_target'
+              PRE_test←{{}⎕PROFILE'start'}
+              POST_test←{{}⎕PROFILE'stop'}
+          :EndIf
+      :EndIf
+     
+      :If is_function Argument
+          TEST_step←single_function_test_function
+          COVER_file←Argument,'_coverage.html'
+     
+      :ElseIf is_list_of_functions Argument
+          TEST_step←list_of_functions_test_function
+          COVER_file←'list_coverage.html'
+     
+      :ElseIf is_file Argument
+          TEST_step←file_test_function
+          COVER_file←(get_file_name Argument),'_coverage.html'
+     
+      :ElseIf is_dir Argument
+          test_files←test_files_in_dir Argument
+          TEST_step←test_dir_function
+          Argument←test_files
+      :EndIf
+     
+      :If 0≠⎕NC'Conf'
+          :If Conf has'cover_target'
+              COVER_step←{Conf,←⊂('cover_file'COVER_file)
+                  generate_coverage_page Conf}
+          :EndIf
+      :EndIf
+     
+      PRE_test ⍬
+      Z←FromSpace TEST_step Argument
+      POST_test ⍬
+      COVER_step ⍬
+    ∇
+
+    ∇ load_display_if_not_already_loaded
+      :If 0=⎕NC'#.DISPLAY'
+          'DISPLAY'#.⎕CY'display'
+      :EndIf
+    ∇
+
+    ∇ load_salt_scripts_into_current_namespace_if_configured
+      :If 0≠⎕NC'#.UT.appdir'
+          :If ⍬≢#.UT.appdir
+              ⎕SE.SALT.Load #.UT.appdir,'src/*.dyalog -target=#'
+              ⎕SE.SALT.Load #.UT.appdir,'test/*.dyalog -target=#'
+          :EndIf
+      :EndIf
+    ∇
+
+    ∇ Z←FromSpace single_function_test_function TestName
+      Z←run_ut FromSpace TestName
+    ∇
+
+    ∇ Z←FromSpace list_of_functions_test_function ListOfNames;t
+      t←⎕TS
+      Z←run_ut¨{FromSpace ⍵}¨ListOfNames
+      t←⎕TS-t
+      ('Test execution report')print_passed_crashed_failed Z t
+    ∇
+
+    ∇ Z←FromSpace file_test_function FilePath;FileNS;Functions;TestFunctions;t
+      FileNS←⎕SE.SALT.Load FilePath,' -target=#'
+      Functions←↓FileNS.⎕NL 3
+      TestFunctions←(is_test¨Functions)/Functions
+      :If (0/⍬,⊂0/'')≡TestFunctions
+          ⎕←'No test functions found'
+          Z←⍬
+      :Else
+          t←⎕TS
+          Z←run_ut¨{FileNS ⍵}¨TestFunctions
+          t←⎕TS-t
+          (FilePath,' tests')print_passed_crashed_failed Z t
+      :EndIf
+    ∇
+
+    ∇ Z←FromSpace test_dir_function Test_files
+      :If Test_files≡⍬/⍬,⊂''
+          ⎕←'No test files found'
+          Z←⍬
+      :Else
+          Z←#.UT.run¨Test_files
+      :EndIf
+    ∇
+
+    ∇ Z←get_file_name Argument;separator
+      separator←⊃⌽(Argument∊'/\')/⍳⍴Argument
+      Z←¯7↓separator↓Argument
+    ∇
+
+    ∇ generate_coverage_page Conf;ProfileData;CoverResults;HTML
+      ProfileData←⎕PROFILE'data'
+      ToCover←retrieve_coverables¨(⊃'cover_target'in Conf)
+      :If (⍴ToCover)≡(⍴⊂1)
+          ToCover←⊃ToCover
+      :EndIf
+      Representations←get_representation¨ToCover
+      CoverResults←ProfileData∘generate_cover_result¨↓ToCover,[1.5]Representations
+      HTML←generate_html CoverResults
+      Conf write_html_to_page HTML
+      ⎕PROFILE'clear'
+    ∇
+
+    ∇ Z←retrieve_coverables Something;nc;functions
+      nc←⎕NC Something
+      :If nc=3
+          Z←Something
+      :ElseIf nc=9
+          functions←strip¨↓⍎Something,'.⎕NL 3'
+          Z←{(Something,'.',⍵)}¨functions
+      :EndIf
+    ∇
+
+    ∇ Z←strip input
+      Z←(input≠' ')/input
+    ∇
+
+    ∇ Z←get_representation Function;nc;rep
+      nc←⎕NC⊂Function
+      :If nc=3.1
+          rep←↓⎕CR Function
+          rep[1]←⊂'∇',⊃rep[1]
+          rep,←⊂'∇'
+          rep←↑rep
+      :Else
+          rep←⎕CR Function
+      :EndIf
+      Z←rep
+    ∇
+
+    ∇ Z←ProfileData generate_cover_result(name representation);Indices;lines;functionlines;covered_lines
+      Indices←({name≡⍵}¨ProfileData[;1])/⍳⍴ProfileData[;1]
+      lines←ProfileData[Indices;2]
+      nc←⎕NC⊂name
+      :If 3.1=nc
+          functionlines←¯2+⍴↓representation
+      :Else
+          functionlines←⊃⍴↓representation
+      :EndIf
+      covered_lines←(⍬∘≢¨lines)/lines
+      Z←(nc lines functionlines covered_lines representation)
+    ∇
+
+    ∇ Z←generate_html CoverResults;Covered;Total;Percentage;CoverageText;ColorizedCode;Timestamp;Page
+      Covered←⊃⊃+/{⍴4⊃⍵}¨CoverResults
+      Total←⊃⊃+/{3⊃⍵}¨CoverResults
+      Percentage←100×Covered÷Total
+      CoverageText←'Coverage: ',Percentage,'% (',Covered,'/',Total,')'
+      ColorizedCode←⊃,/{colorize_code_by_coverage ⍵}¨CoverResults
+      Timestamp←generate_timestamp_text
+      Page←⍬
+      Page,←⊂⍬,'<html>'
+      Page,←⊂⍬,'<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>'
+      Page,←⊂⍬,'<style>pre cov {line-height:80%;}'
+      Page,←⊂⍬,'pre cov {color: green;}'
+      Page,←⊂⍬,'pre uncov {line-height:80%;}'
+      Page,←⊂⍬,'pre uncov {color:red;}</style>'
+      Page,←⊂⍬,CoverageText
+      Page,←⊂⍬,'<pre>'
+      Page,←ColorizedCode
+      Page,←⊂⍬,'</pre>'
+      Page,←Timestamp
+      Page,←⊂⍬,'</html>'
+      Z←Page
+    ∇
+
+    ∇ Z←colorize_code_by_coverage CoverResult;Colors;Ends;Code
+      :If 3.1=⊃CoverResult
+          Colors←(2+3⊃CoverResult)⍴⊂'<uncov>'
+          Colors[1]←⊂''
+          Colors[⍴Colors]←⊂''
+          Ends←(2+3⊃CoverResult)⍴⊂'</uncov>'
+          Ends[1]←⊂''
+          Ends[⍴Ends]←⊂''
+      :Else
+          Colors←(3⊃CoverResult)⍴⊂'<uncov>'
+          Ends←(3⊃CoverResult)⍴⊂'</uncov>'
+      :EndIf
+      Colors[1+4⊃CoverResult]←⊂'<cov>'
+      Ends[1+4⊃CoverResult]←⊂'</cov>'
+      Code←↓5⊃CoverResult
+      Z←Colors,[1.5]Code
+      Z←{⍺,(⎕UCS 13),⍵}/Z,Ends
+    ∇
+
+    ∇ Z←generate_timestamp_text;TS;YYMMDD;HHMMSS
+      TS←⎕TS
+      YYMMDD←⊃{⍺,'-',⍵}/3↑TS
+      HHMMSS←⊃{⍺,':',⍵}/3↑3↓TS
+      Z←'Page generated: ',YYMMDD,'|',HHMMSS
+    ∇
+
+    ∇ Conf write_html_to_page Page;tie;filename
+      filename←(⊃'cover_out'in Conf),(⊃'cover_file'in Conf)
+      :Trap 22
+          tie←filename ⎕NTIE 0
+          filename ⎕NERASE tie
+          filename ⎕NCREATE tie
+      :Else
+          tie←filename ⎕NCREATE 0
+      :EndTrap
+      Simple_array←⍕⊃,/Page
+      (⎕UCS'UTF-8'⎕UCS Simple_array)⎕NAPPEND tie
+    ∇
+
+    ∇ Z←is_function Argument
+      Z←'_TEST'≡¯5↑Argument
+    ∇
+
+    ∇ Z←is_list_of_functions Argument
+      Z←2=≡Argument
+    ∇
+
+    ∇ Z←is_file Argument
+      Z←'.dyalog'≡¯7↑Argument
+    ∇
+
+    ∇ Z←is_dir Argument;attr
+      :If 'Linux'≡5↑⊃'.'⎕WG'APLVersion'
+          Z←'yes'≡⊃⎕CMD'test -d ',Argument,' && echo yes || echo no'
+      :Else
+          'gfa'⎕NA'I kernel32|GetFileAttributes* <0t'
+          :If Z←¯1≠attr←gfa⊂Argument ⍝ If file exists
+              Z←⊃2 16⊤attr           ⍝ Return bit 4
+          :EndIf
+      :EndIf
+    ∇
+
+
+    ∇ Z←test_files_in_dir Argument
+      :If 'Linux'≡5↑⊃'.'⎕WG'APLVersion'
+          Z←⎕SH'find ',Argument,' -name \*_tests.dyalog'
+      :Else
+          #.⎕CY'files'
+          Z←#.Files.Dir Argument,'\*_tests.dyalog'
+          Z←(Argument,'\')∘,¨Z
+      :EndIf
+    ∇
+
+    ∇ Z←run_ut ut_data;returned;crashed;pass;crash;fail;message
+      (returned crashed time)←execute_function ut_data
+      (pass crash fail)←determine_pass_crash_or_fail returned crashed
+      message←determine_message pass fail crashed(2⊃ut_data)returned time
+      print_message_to_screen message
+      Z←(pass crash fail)
+    ∇
+
+    ∇ Z←execute_function ut_data;function;t
+      reset_UT_globals
+      function←(⍕(⊃ut_data[1])),'.',⊃ut_data[2]
+      :Trap sac
+          :If 3.2≡⎕NC⊂function
+              t←⎕TS
+              Z←(⍎function,' ⍬')0
+              t←⎕TS-t
+          :Else
+              t←⎕TS
+              Z←(⍎function)0
+              t←⎕TS-t
+          :EndIf
+     
+      :Else
+          Z←(↑⎕DM)1
+          :If exception≢⍬
+              expect←exception
+              Z[2]←0
+              t←⎕TS-t
+          :EndIf
+      :EndTrap
+      Z,←⊂t
+    ∇
+
+    ∇ reset_UT_globals
+      expect_orig ← expect← ⎕NS⍬
+      exception←⍬
+      nexpect_orig ← nexpect← ⎕NS⍬
+    ∇
+
+    ∇ Z←is_test FunctionName;wsIndex
+      wsIndex←FunctionName⍳' '
+      FunctionName←(wsIndex-1)↑FunctionName
+      Z←'_TEST'≡¯5↑FunctionName
+    ∇
+
+    ∇ Heading print_passed_crashed_failed(ArrayRes time)
+      ⎕←'-----------------------------------------'
+      ⎕←Heading
+      ⎕←'    ⍋  Passed: ',+/{1⊃⍵}¨ArrayRes
+      ⎕←'    ⍟ Crashed: ',+/{2⊃⍵}¨ArrayRes
+      ⎕←'    ⍒  Failed: ',+/{3⊃⍵}¨ArrayRes
+      ⎕←'    ○ Runtime: ',time[5],'m',time[6],'s',time[7],'ms'
+    ∇
+    
+    determine_pass_crash_or_fail←{
+      r c←⍵ ⋄ 0≠c:0 1 0 ⋄ z←(0 0 1)(1 0 0)
+      expect_orig≢expect:(⎕IO+expect≡r)⊃z ⋄ (⎕IO+nexpect≢r)⊃z
+    }
+
+    ∇ Z←determine_message(pass fail crashed name returned time)
+      :If crashed
+          Z←'CRASHED: 'failure_message name returned
+      :ElseIf pass
+          Z←'Passed ',time[5],'m',time[6],'s',time[7],'ms'
+      :Else
+          Z←'FAILED: 'failure_message name returned
+      :EndIf
+    ∇
+
+    ∇ print_message_to_screen message
+      ⎕←message
+    ∇
+
+    ∇ Z←term_to_text Term;Text;Rows
+      Text←#.DISPLAY Term
+      Rows←1⊃⍴Text
+      Z←(Rows 4⍴''),Text
+    ∇
+
+    ∇ Z←Cause failure_message(name returned);hdr;exp;expterm;got;gotterm
+      hdr←Cause,name
+      exp←'Expected'
+      expterm←term_to_text #.UT.expect
+      got←'Got'
+      gotterm←term_to_text returned
+      Z←align_and_join_message_parts hdr exp expterm got gotterm
+    ∇
+
+    ∇ Z←align_and_join_message_parts Parts;hdr;exp;expterm;got;gotterm;R1;C1;R2;C2;W
+      (hdr exp expterm got gotterm)←Parts
+      (R1 C1)←⍴expterm
+      (R2 C2)←⍴gotterm
+      W←⊃⊃⌈/C1 C2(⍴hdr)(⍴exp)(⍴got)
+      Z←(W↑hdr),[0.5](W↑exp)
+      Z←Z⍪(R1 W↑expterm)
+      Z←Z⍪(W↑got)
+      Z←Z⍪(R2 W↑gotterm)
+    ∇
+
+    ∇ Z←confparam in config
+      Z←1↓⊃({confparam≡⊃⍵}¨config)/config
+    ∇
+
+    ∇ Z←config has confparam
+      Z←∨/{confparam≡⊃⍵}¨config
+    ∇
+
+:EndNameSpace

samples/ATS/CoYonedaLemma.dats

@@ -0,0 +1,110 @@
+(* ****** ****** *)
+//
+// HX-2014-01
+// CoYoneda Lemma:
+//
+(* ****** ****** *)
+//
+#include
+"share/atspre_staload.hats"
+//
+(* ****** ****** *)
+
+staload
+"libats/ML/SATS/basis.sats"
+staload
+"libats/ML/SATS/list0.sats"
+
+(* ****** ****** *)
+
+staload _ = "libats/ML/DATS/list0.dats"
+
+(* ****** ****** *)
+
+sortdef ftype = type -> type
+
+(* ****** ****** *)
+
+infixr (->) ->>
+typedef ->> (a:type, b:type) = a -<cloref1> b
+
+(* ****** ****** *)
+
+typedef
+functor(F:ftype) =
+  {a,b:type} (a ->> b) ->> F(a) ->> F(b)
+
+(* ****** ****** *)
+
+typedef
+list0 (a:type) = list0 (a)
+extern
+val functor_list0 : functor (list0)
+
+(* ****** ****** *)
+
+implement
+functor_list0{a,b}
+  (f) = lam xs => list0_map<a><b> (xs, f)
+
+(* ****** ****** *)
+
+datatype
+CoYoneda
+ (F:ftype, r:type) = {a:type} CoYoneda of (a ->> r, F(a))
+// end of [CoYoneda]
+
+(* ****** ****** *)
+//
+extern
+fun CoYoneda_phi
+  : {F:ftype}functor(F) -> {r:type} (F (r) ->> CoYoneda (F, r))
+extern
+fun CoYoneda_psi
+  : {F:ftype}functor(F) -> {r:type} (CoYoneda (F, r) ->> F (r))
+//
+(* ****** ****** *)
+
+implement
+CoYoneda_phi(ftor) = lam (fx) => CoYoneda (lam x => x, fx)
+implement
+CoYoneda_psi(ftor) = lam (CoYoneda(f, fx)) => ftor (f) (fx)
+
+(* ****** ****** *)
+
+datatype int0 = I of (int)
+datatype bool = True | False // boxed boolean
+
+(* ****** ****** *)
+//
+fun bool2string
+  (x:bool): string =
+(
+  case+ x of True() => "True" | False() => "False"
+)
+//
+implement
+fprint_val<bool> (out, x) = fprint (out, bool2string(x))
+//
+(* ****** ****** *)
+
+fun int2bool (i: int0): bool =
+  let val+I(i) = i in if i > 0 then True else False end
+
+(* ****** ****** *)
+
+val myintlist0 = g0ofg1($list{int0}((I)1, (I)0, (I)1, (I)0, (I)0))
+val myboolist0 = CoYoneda{list0,bool}{int0}(lam (i) => int2bool(i), myintlist0)
+val myboolist0 = CoYoneda_psi{list0}(functor_list0){bool}(myboolist0)
+
+(* ****** ****** *)
+
+val ((*void*)) = fprintln! (stdout_ref, "myboolist0 = ", myboolist0)
+
+(* ****** ****** *)
+
+implement main0 () = ()
+
+(* ****** ****** *)
+
+(* end of [CoYonedaLemma.dats] *)

samples/ATS/DiningPhil2.dats

@@ -0,0 +1,178 @@
+(* ****** ****** *)
+//
+// HX-2013-11
+//
+// Implementing a variant of
+// the problem of Dining Philosophers
+//
+(* ****** ****** *)
+//
+#include
+"share/atspre_define.hats"
+#include
+"share/atspre_staload.hats"
+//
+(* ****** ****** *)
+
+staload
+UN = "prelude/SATS/unsafe.sats"
+
+(* ****** ****** *)
+
+staload "libc/SATS/stdlib.sats"
+staload "libc/SATS/unistd.sats"
+
+(* ****** ****** *)
+
+staload "{$LIBATSHWXI}/teaching/mythread/SATS/channel.sats"
+
+(* ****** ****** *)
+
+staload _ = "libats/DATS/deqarray.dats"
+staload _ = "{$LIBATSHWXI}/teaching/mythread/DATS/channel.dats"
+
+(* ****** ****** *)
+
+staload "./DiningPhil2.sats"
+
+(* ****** ****** *)
+
+implement phil_left (n) = n
+implement phil_right (n) = (n+1) \nmod NPHIL
+
+(* ****** ****** *)
+//
+extern
+fun randsleep (n: intGte(1)): void
+//
+implement
+randsleep (n) =
+  ignoret (sleep($UN.cast{uInt}(rand() mod n + 1)))
+// end of [randsleep]
+//
+(* ****** ****** *)
+
+implement
+phil_think (n) =
+{
+val () = println! ("phil_think(", n, ") starts")
+val () = randsleep (6)
+val () = println! ("phil_think(", n, ") finishes")
+}
+
+(* ****** ****** *)
+
+implement
+phil_dine (n, lf, rf) =
+{
+val () = println! ("phil_dine(", n, ") starts")
+val () = randsleep (3)
+val () = println! ("phil_dine(", n, ") finishes")
+}
+
+(* ****** ****** *)
+
+implement
+phil_loop (n) = let
+//
+val () = phil_think (n)
+//
+val nl = phil_left (n)
+val nr = phil_right (n)
+//
+val ch_lfork = fork_changet (nl)
+val ch_rfork = fork_changet (nr)
+//
+val lf = channel_takeout (ch_lfork)
+val () = println! ("phil_loop(", n, ") picks left fork")
+//
+val () = randsleep (2) // HX: try to actively induce deadlock
+//
+val rf = channel_takeout (ch_rfork)
+val () = println! ("phil_loop(", n, ") picks right fork")
+//
+val () = phil_dine (n, lf, rf)
+//
+val ch_forktray = forktray_changet ()
+val () = channel_insert (ch_forktray, lf)
+val () = channel_insert (ch_forktray, rf)
+//
+in
+  phil_loop (n)
+end // end of [phil_loop]
+
+(* ****** ****** *)
+
+implement
+cleaner_wash (f) =
+{
+val f = fork_get_num (f)
+val () = println! ("cleaner_wash(", f, ") starts")
+val () = randsleep (1)
+val () = println! ("cleaner_wash(", f, ") finishes")
+}
+
+(* ****** ****** *)
+
+implement
+cleaner_return (f) =
+{
+  val n = fork_get_num (f)
+  val ch = fork_changet (n)
+  val () = channel_insert (ch, f)
+}
+
+(* ****** ****** *)
+
+implement
+cleaner_loop () = let
+//
+val ch = forktray_changet ()
+val f0 = channel_takeout (ch)
+//
+val () = cleaner_wash (f0)
+val () = cleaner_return (f0)
+//
+in
+  cleaner_loop ()
+end // end of [cleaner_loop]
+
+(* ****** ****** *)
+
+dynload "DiningPhil2.sats"
+dynload "DiningPhil2_fork.dats"
+dynload "DiningPhil2_thread.dats"
+
+(* ****** ****** *)
+
+local
+//
+staload
+"{$LIBATSHWXI}/teaching/mythread/SATS/mythread.sats"
+//
+in (* in of [local] *)
+//
+val () = mythread_create_cloptr (llam () => phil_loop (0))
+val () = mythread_create_cloptr (llam () => phil_loop (1))
+val () = mythread_create_cloptr (llam () => phil_loop (2))
+val () = mythread_create_cloptr (llam () => phil_loop (3))
+val () = mythread_create_cloptr (llam () => phil_loop (4))
+//
+val () = mythread_create_cloptr (llam () => cleaner_loop ())
+//
+end // end of [local]
+
+(* ****** ****** *)
+
+implement
+main0 () =
+{
+//
+val () = println! ("DiningPhil2: starting")
+val ((*void*)) = while (true) ignoret (sleep(1))
+//
+} (* end of [main0] *)
+
+(* ****** ****** *)
+
+(* end of [DiningPhil2.dats] *)

samples/ATS/DiningPhil2.sats

@@ -0,0 +1,71 @@
+(* ****** ****** *)
+//
+// HX-2013-11
+//
+// Implementing a variant of
+// the problem of Dining Philosophers
+//
+(* ****** ****** *)
+
+#include
+"share/atspre_define.hats"
+
+(* ****** ****** *)
+
+staload "{$LIBATSHWXI}/teaching/mythread/SATS/channel.sats"
+
+(* ****** ****** *)
+
+%{#
+#define NPHIL 5
+%} // end of [%{#]
+#define NPHIL 5
+
+(* ****** ****** *)
+
+typedef nphil = natLt(NPHIL)
+
+(* ****** ****** *)
+
+fun phil_left (n: nphil): nphil
+fun phil_right (n: nphil): nphil
+
+(* ****** ****** *)
+//
+fun phil_loop (n: nphil): void
+//
+(* ****** ****** *)
+
+fun cleaner_loop ((*void*)): void
+
+(* ****** ****** *)
+
+absvtype fork_vtype = ptr
+vtypedef fork = fork_vtype
+
+(* ****** ****** *)
+
+fun fork_get_num (!fork): nphil
+
+(* ****** ****** *)
+
+fun phil_dine
+  (n: nphil, lf: !fork, rf: !fork): void
+// end of [phil_dine]
+
+fun phil_think (n: nphil): void
+
+(* ****** ****** *)
+
+fun cleaner_wash (f: !fork): void
+fun cleaner_return (f: fork): void
+
+(* ****** ****** *)
+//
+fun fork_changet (n: nphil): channel(fork)
+//
+fun forktray_changet ((*void*)): channel(fork)
+//
+(* ****** ****** *)
+
+(* end of [DiningPhil2.sats] *)

samples/ATS/DiningPhil2_fork.dats

@@ -0,0 +1,89 @@
+(* ****** ****** *)
+//
+// HX-2013-11
+//
+// Implementing a variant of
+// the problem of Dining Philosophers
+//
+(* ****** ****** *)
+//
+#include
+"share/atspre_define.hats"
+#include
+"share/atspre_staload.hats"
+//
+(* ****** ****** *)
+
+staload
+UN = "prelude/SATS/unsafe.sats"
+
+(* ****** ****** *)
+
+staload "{$LIBATSHWXI}/teaching/mythread/SATS/channel.sats"
+
+(* ****** ****** *)
+
+staload _ = "libats/DATS/deqarray.dats"
+staload _ = "{$LIBATSHWXI}/teaching/mythread/DATS/channel.dats"
+
+(* ****** ****** *)
+
+staload "./DiningPhil2.sats"
+
+(* ****** ****** *)
+
+datavtype fork = FORK of (nphil)
+
+(* ****** ****** *)
+
+assume fork_vtype = fork
+
+(* ****** ****** *)
+
+implement
+fork_get_num (f) = let val FORK(n) = f in n end
+
+(* ****** ****** *)
+
+local
+
+val
+the_forkarray = let
+//
+typedef t = channel(fork)
+//
+implement
+array_tabulate$fopr<t>
+  (n) = ch where
+{
+  val n = $UN.cast{nphil}(n)
+  val ch = channel_create_exn<fork> (i2sz(2))
+  val () = channel_insert (ch, FORK (n))
+}
+//
+in
+  arrayref_tabulate<t> (i2sz(NPHIL))
+end // end of [val]
+
+in (* in of [local] *)
+
+implement fork_changet (n) = the_forkarray[n]
+
+end // end of [local]
+
+(* ****** ****** *)
+
+local
+
+val the_forktray =
+  channel_create_exn<fork> (i2sz(NPHIL+1))
+
+in (* in of [local] *)
+
+implement forktray_changet () = the_forktray
+
+end // end of [local]
+
+(* ****** ****** *)
+
+(* end of [DiningPhil2_fork.dats] *)

samples/ATS/DiningPhil2_thread.dats

@@ -0,0 +1,43 @@
+(* ****** ****** *)
+//
+// HX-2013-11
+//
+// Implementing a variant of
+// the problem of Dining Philosophers
+//
+(* ****** ****** *)
+//
+#include "share/atspre_define.hats"
+#include "share/atspre_staload.hats"
+//
+(* ****** ****** *)
+
+staload "{$LIBATSHWXI}/teaching/mythread/SATS/mythread.sats"
+
+(* ****** ****** *)
+
+local
+//
+#include "{$LIBATSHWXI}/teaching/mythread/DATS/mythread.dats"
+//
+in (* in of [local] *)
+//
+// HX: it is intentionally left to be empty
+//
+end // end of [local]
+
+(* ****** ****** *)
+
+local
+//
+#include "{$LIBATSHWXI}/teaching/mythread/DATS/mythread_posix.dats"
+//
+in (* in of [local] *)
+//
+// HX: it is intentionally left to be empty
+//
+end // end of [local]
+
+(* ****** ****** *)
+
+(* end of [DiningPhil2_thread.dats] *)

samples/ATS/YonedaLemma.dats

@@ -0,0 +1,178 @@
+(* ****** ****** *)
+//
+// HX-2014-01
+// Yoneda Lemma:
+// The hardest "trivial" theorem :)
+//
+(* ****** ****** *)
+//
+#include
+"share/atspre_staload.hats"
+//
+(* ****** ****** *)
+
+staload
+"libats/ML/SATS/basis.sats"
+staload
+"libats/ML/SATS/list0.sats"
+staload
+"libats/ML/SATS/option0.sats"
+
+(* ****** ****** *)
+
+staload _ = "libats/ML/DATS/list0.dats"
+staload _ = "libats/ML/DATS/option0.dats"
+
+(* ****** ****** *)
+
+sortdef ftype = type -> type
+
+(* ****** ****** *)
+
+infixr (->) ->>
+typedef ->> (a:type, b:type) = a -<cloref1> b
+
+(* ****** ****** *)
+
+typedef
+functor(F:ftype) =
+  {a,b:type} (a ->> b) ->> F(a) ->> F(b)
+
+(* ****** ****** *)
+
+typedef
+list0 (a:type) = list0 (a)
+extern
+val functor_list0 : functor (list0)
+
+(* ****** ****** *)
+
+implement
+functor_list0{a,b}
+  (f) = lam xs => list0_map<a><b> (xs, f)
+
+(* ****** ****** *)
+
+typedef
+option0 (a:type) = option0 (a)  
+extern
+val functor_option0 : functor (option0)
+  
+(* ****** ****** *)
+
+implement
+functor_option0{a,b}
+  (f) = lam opt => option0_map<a><b> (opt, f)
+
+(* ****** ****** *)
+
+extern
+val functor_homres
+  : {c:type} functor (lam(r:type) => c ->> r)
+
+(* ****** ****** *)
+
+implement
+functor_homres{c}{a,b} (f) = lam (r) => lam (x) => f (r(x))
+
+(* ****** ****** *)
+//
+extern
+fun Yoneda_phi : {F:ftype}functor(F) ->
+  {a:type}F(a) ->> ({r:type}(a ->> r) ->> F(r))
+extern
+fun Yoneda_psi : {F:ftype}functor(F) ->
+  {a:type}({r:type}(a ->> r) ->> F(r)) ->> F(a)
+//
+(* ****** ****** *)
+//
+implement
+Yoneda_phi
+  (ftor) = lam(fx) => lam (m) => ftor(m)(fx)
+//
+implement
+Yoneda_psi (ftor) = lam(mf) => mf(lam x => x)
+//
+(* ****** ****** *)
+
+(*
+
+(* ****** ****** *)
+//
+// HX-2014-01-05:
+// Another version based on Natural Transformation
+//
+(* ****** ****** *)
+
+typedef
+natrans(F:ftype, G:ftype) = {x:type} (F(x) ->> G(x))
+
+(* ****** ****** *)
+//
+extern
+fun Yoneda_phi_nat : {F:ftype}functor(F) ->
+  {a:type} F(a) ->> natrans(lam (r:type) => (a ->> r), F)
+extern
+fun Yoneda_psi_nat : {F:ftype}functor(F) ->
+  {a:type} natrans(lam (r:type) => (a ->> r), F) ->> F(a)
+//
+(* ****** ****** *)
+//
+implement
+Yoneda_phi_nat
+  (ftor) = lam(fx) => lam (m) => ftor(m)(fx)
+//
+implement
+Yoneda_psi_nat (ftor) = lam(mf) => mf(lam x => x)
+//
+(* ****** ****** *)
+
+*)
+
+(* ****** ****** *)
+
+datatype bool = True | False // boxed boolean
+
+(* ****** ****** *)
+//
+fun bool2string
+  (x:bool): string =
+(
+  case+ x of True() => "True" | False() => "False"
+)
+//
+implement
+fprint_val<bool> (out, x) = fprint (out, bool2string(x))
+//
+(* ****** ****** *)
+//
+val myboolist0 =
+  $list_t{bool}(True, False, True, False, False)
+val myboolist0 = g0ofg1_list (myboolist0)
+//
+(* ****** ****** *)
+//
+extern
+val Yoneda_bool_list0 : {r:type} (bool ->> r) ->> list0(r)
+//
+implement
+Yoneda_bool_list0 =
+  Yoneda_phi(functor_list0){bool}(myboolist0)
+//
+(* ****** ****** *)
+//
+val myboolist1 =
+  Yoneda_psi(functor_list0){bool}(Yoneda_bool_list0)
+//
+(* ****** ****** *)
+
+val () = fprintln! (stdout_ref, "myboolist0 = ", myboolist0)
+val () = fprintln! (stdout_ref, "myboolist1 = ", myboolist1)
+
+(* ****** ****** *)
+
+implement main0 () = ()
+
+(* ****** ****** *)
+
+(* end of [YonedaLemma.dats] *)

samples/ATS/linset.hats

@@ -0,0 +1,187 @@
+(***********************************************************************)
+(*                                                                     *)
+(*                         Applied Type System                         *)
+(*                                                                     *)
+(***********************************************************************)
+
+(*
+** ATS/Postiats - Unleashing the Potential of Types!
+** Copyright (C) 2011-2013 Hongwei Xi, ATS Trustful Software, Inc.
+** All rights reserved
+**
+** ATS is free software;  you can  redistribute it and/or modify it under
+** the terms of  the GNU GENERAL PUBLIC LICENSE (GPL) as published by the
+** Free Software Foundation; either version 3, or (at  your  option)  any
+** later version.
+** 
+** ATS is distributed in the hope that it will be useful, but WITHOUT ANY
+** WARRANTY; without  even  the  implied  warranty  of MERCHANTABILITY or
+** FITNESS FOR A PARTICULAR PURPOSE.  See the  GNU General Public License
+** for more details.
+** 
+** You  should  have  received  a  copy of the GNU General Public License
+** along  with  ATS;  see the  file COPYING.  If not, please write to the
+** Free Software Foundation,  51 Franklin Street, Fifth Floor, Boston, MA
+** 02110-1301, USA.
+*)
+
+(* ****** ****** *)
+
+(* Author: Hongwei Xi *)
+(* Authoremail: hwxi AT cs DOT bu DOT edu *)
+(* Start time: December, 2012 *)
+
+(* ****** ****** *)
+//
+// HX: shared by linset_listord (* ordered list *)
+// HX: shared by linset_avltree (* AVL-tree-based *)
+//
+(* ****** ****** *)
+//
+// HX-2013-02:
+// for sets of nonlinear elements
+//
+absvtype set_vtype (a:t@ype+) = ptr
+//
+(* ****** ****** *)
+
+vtypedef set (a:t0p) = set_vtype (a)
+
+(* ****** ****** *)
+
+fun{a:t0p}
+compare_elt_elt (x1: a, x2: a):<> int
+
+(* ****** ****** *)
+
+fun{} linset_nil{a:t0p} ():<> set(a)
+fun{} linset_make_nil{a:t0p} ():<> set(a)
+
+(* ****** ****** *)
+
+fun{a:t0p} linset_sing (x: a):<!wrt> set(a)
+fun{a:t0p} linset_make_sing (x: a):<!wrt> set(a)
+
+(* ****** ****** *)
+
+fun{a:t0p}
+linset_make_list (xs: List(INV(a))):<!wrt> set(a)
+
+(* ****** ****** *)
+
+fun{}
+linset_is_nil {a:t0p} (xs: !set(INV(a))):<> bool
+fun{}
+linset_isnot_nil {a:t0p} (xs: !set(INV(a))):<> bool
+
+(* ****** ****** *)
+
+fun{a:t0p} linset_size (!set(INV(a))): size_t
+
+(* ****** ****** *)
+
+fun{a:t0p}
+linset_is_member (xs: !set(INV(a)), x0: a):<> bool
+fun{a:t0p}
+linset_isnot_member (xs: !set(INV(a)), x0: a):<> bool
+
+(* ****** ****** *)
+
+fun{a:t0p}
+linset_copy (!set(INV(a))):<!wrt> set(a)
+fun{a:t0p}
+linset_free (xs: set(INV(a))):<!wrt> void
+
+(* ****** ****** *)
+//
+fun{a:t0p}
+linset_insert
+  (xs: &set(INV(a)) >> _, x0: a):<!wrt> bool
+//
+(* ****** ****** *)
+//
+fun{a:t0p}
+linset_takeout
+(
+  &set(INV(a)) >> _, a, res: &(a?) >> opt(a, b)
+) :<!wrt> #[b:bool] bool(b) // endfun
+fun{a:t0p}
+linset_takeout_opt (&set(INV(a)) >> _, a):<!wrt> Option_vt(a)
+//
+(* ****** ****** *)
+//
+fun{a:t0p}
+linset_remove
+  (xs: &set(INV(a)) >> _, x0: a):<!wrt> bool
+//
+(* ****** ****** *)
+//
+// HX: choosing an element in an unspecified manner
+//
+fun{a:t0p}
+linset_choose
+(
+  xs: !set(INV(a)), x: &a? >> opt (a, b)
+) :<!wrt> #[b:bool] bool(b)
+//
+fun{a:t0p}
+linset_choose_opt (xs: !set(INV(a))):<!wrt> Option_vt(a)
+//
+(* ****** ****** *)
+
+fun{a:t0p}
+linset_takeoutmax
+(
+  xs: &set(INV(a)) >> _, res: &a? >> opt(a, b)
+) :<!wrt> #[b:bool] bool (b)
+fun{a:t0p}
+linset_takeoutmax_opt (xs: &set(INV(a)) >> _):<!wrt> Option_vt(a)
+
+(* ****** ****** *)
+
+fun{a:t0p}
+linset_takeoutmin
+(
+  xs: &set(INV(a)) >> _, res: &a? >> opt(a, b)
+) :<!wrt> #[b:bool] bool (b)
+fun{a:t0p}
+linset_takeoutmin_opt (xs: &set(INV(a)) >> _):<!wrt> Option_vt(a)
+
+(* ****** ****** *)
+//
+fun{}
+fprint_linset$sep (FILEref): void // ", "
+//
+fun{a:t0p}
+fprint_linset (out: FILEref, xs: !set(INV(a))): void
+//
+overload fprint with fprint_linset
+//
+(* ****** ****** *)
+//
+fun{
+a:t0p}{env:vt0p
+} linset_foreach$fwork
+  (x: a, env: &(env) >> _): void
+//
+fun{a:t0p}
+linset_foreach (set: !set(INV(a))): void
+fun{
+a:t0p}{env:vt0p
+} linset_foreach_env
+  (set: !set(INV(a)), env: &(env) >> _): void
+// end of [linset_foreach_env]
+//
+(* ****** ****** *)
+
+fun{a:t0p}
+linset_listize (xs: set(INV(a))): List0_vt (a)
+
+(* ****** ****** *)
+
+fun{a:t0p}
+linset_listize1 (xs: !set(INV(a))): List0_vt (a)
+
+(* ****** ****** *)
+
+(* end of [linset.hats] *)

samples/ATS/linset_listord.dats

@@ -0,0 +1,504 @@
+(***********************************************************************)
+(*                                                                     *)
+(*                         Applied Type System                         *)
+(*                                                                     *)
+(***********************************************************************)
+
+(*
+** ATS/Postiats - Unleashing the Potential of Types!
+** Copyright (C) 2011-2013 Hongwei Xi, ATS Trustful Software, Inc.
+** All rights reserved
+**
+** ATS is free software;  you can  redistribute it and/or modify it under
+** the terms of  the GNU GENERAL PUBLIC LICENSE (GPL) as published by the
+** Free Software Foundation; either version 3, or (at  your  option)  any
+** later version.
+** 
+** ATS is distributed in the hope that it will be useful, but WITHOUT ANY
+** WARRANTY; without  even  the  implied  warranty  of MERCHANTABILITY or
+** FITNESS FOR A PARTICULAR PURPOSE.  See the  GNU General Public License
+** for more details.
+** 
+** You  should  have  received  a  copy of the GNU General Public License
+** along  with  ATS;  see the  file COPYING.  If not, please write to the
+** Free Software Foundation,  51 Franklin Street, Fifth Floor, Boston, MA
+** 02110-1301, USA.
+*)
+
+(* ****** ****** *)
+
+(* Author: Hongwei Xi *)
+(* Authoremail: hwxi AT cs DOT bu DOT edu *)
+(* Start time: February, 2013 *)
+
+(* ****** ****** *)
+//
+// HX-2013-08:
+// a set is represented as a sorted list in descending order;
+// note that descending order is chosen to faciliate set comparison
+//
+(* ****** ****** *)
+
+staload
+UN = "prelude/SATS/unsafe.sats"
+
+(* ****** ****** *)
+
+staload "libats/SATS/linset_listord.sats"
+
+(* ****** ****** *)
+
+#include "./SHARE/linset.hats" // code reuse
+#include "./SHARE/linset_node.hats" // code reuse
+
+(* ****** ****** *)
+
+assume
+set_vtype (elt:t@ype) = List0_vt (elt)
+
+(* ****** ****** *)
+
+implement{}
+linset_nil () = list_vt_nil ()
+implement{}
+linset_make_nil () = list_vt_nil ()
+
+(* ****** ****** *)
+
+implement
+{a}(*tmp*)
+linset_sing
+  (x) = list_vt_cons{a}(x, list_vt_nil)
+// end of [linset_sing]
+implement{a}
+linset_make_sing
+  (x) = list_vt_cons{a}(x, list_vt_nil)
+// end of [linset_make_sing]
+
+(* ****** ****** *)
+
+implement{}
+linset_is_nil (xs) = list_vt_is_nil (xs)
+implement{}
+linset_isnot_nil (xs) = list_vt_is_cons (xs)
+
+(* ****** ****** *)
+
+implement{a}
+linset_size (xs) =
+  let val n = list_vt_length(xs) in i2sz(n) end
+// end of [linset_size]
+
+(* ****** ****** *)
+
+implement{a}
+linset_is_member
+  (xs, x0) = let
+//
+fun aux
+  {n:nat} .<n>.
+(
+  xs: !list_vt (a, n)
+) :<> bool = let
+in
+//
+case+ xs of
+| list_vt_cons (x, xs) => let
+    val sgn = compare_elt_elt<a> (x0, x) in
+    if sgn > 0 then false else (if sgn < 0 then aux (xs) else true)
+  end // end of [list_vt_cons]
+| list_vt_nil ((*void*)) => false
+//
+end // end of [aux]
+//
+in
+  aux (xs)
+end // end of [linset_is_member]
+
+(* ****** ****** *)
+
+implement{a}
+linset_copy (xs) = list_vt_copy<a> (xs)
+implement{a}
+linset_free (xs) = list_vt_free<a> (xs)
+
+(* ****** ****** *)
+
+implement{a}
+linset_insert
+  (xs, x0) = let
+//
+fun
+mynode_cons
+  {n:nat} .<>.
+(
+  nx: mynode1 (a), xs: list_vt (a, n)
+) : list_vt (a, n+1) = let
+//
+val xs1 =
+$UN.castvwtp0{List1_vt(a)}(nx)
+val+@list_vt_cons (_, xs2) = xs1
+prval () = $UN.cast2void (xs2); val () = (xs2 := xs)
+//
+in
+  fold@ (xs1); xs1
+end // end of [mynode_cons]
+//
+fun ins
+  {n:nat} .<n>. // tail-recursive
+(
+  xs: &list_vt (a, n) >> list_vt (a, n1)
+) : #[n1:nat | n <= n1; n1 <= n+1] bool =
+(
+case+ xs of
+| @list_vt_cons
+    (x, xs1) => let
+    val sgn =
+      compare_elt_elt<a> (x0, x)
+    // end of [val]
+  in
+    if sgn > 0 then let
+      prval () = fold@ (xs)
+      val nx = mynode_make_elt<a> (x0)
+      val ((*void*)) = xs := mynode_cons (nx, xs)
+    in
+      false
+    end else if sgn < 0 then let
+      val ans = ins (xs1)
+      prval () = fold@ (xs)
+    in
+      ans
+    end else let // [x0] is found
+      prval () = fold@ (xs)
+    in
+      true (* [x0] in [xs] *)
+    end (* end of [if] *)
+  end // end of [list_vt_cons]
+| list_vt_nil () => let
+    val nx = mynode_make_elt<a> (x0)
+    val ((*void*)) = xs := mynode_cons (nx, xs)
+  in
+    false
+  end // end of [list_vt_nil]
+) (* end of [ins] *)
+//
+in
+  $effmask_all (ins (xs))
+end // end of [linset_insert]
+
+(* ****** ****** *)
+
+(*
+//
+HX-2013-08:
+[linset_remove] moved up
+//
+implement{a}
+linset_remove
+  (xs, x0) = let
+//
+fun rem
+  {n:nat} .<n>. // tail-recursive
+(
+  xs: &list_vt (a, n) >> list_vt (a, n1)
+) : #[n1:nat | n1 <= n; n <= n1+1] bool =
+(
+case+ xs of
+| @list_vt_cons
+    (x, xs1) => let
+    val sgn =
+      compare_elt_elt<a> (x0, x)
+    // end of [val]
+  in
+    if sgn > 0 then let
+      prval () = fold@ (xs)
+    in
+      false
+    end else if sgn < 0 then let
+      val ans = rem (xs1)
+      prval () = fold@ (xs)
+    in
+      ans
+    end else let // x0 = x
+      val xs1_ = xs1
+      val ((*void*)) = free@{a}{0}(xs)
+      val () = xs := xs1_
+    in
+      true // [x0] in [xs]
+    end (* end of [if] *)
+  end // end of [list_vt_cons]
+| list_vt_nil () => false
+) (* end of [rem] *)
+//
+in
+  $effmask_all (rem (xs))
+end // end of [linset_remove]
+*)
+
+(* ****** ****** *)
+(*
+** By Brandon Barker
+*)
+implement
+{a}(*tmp*)
+linset_choose
+  (xs, x0) = let
+in
+//
+case+ xs of
+| list_vt_cons
+    (x, xs1) => let
+    val () = x0 := x
+    prval () = opt_some{a}(x0)
+  in
+    true
+  end // end of [list_vt_cons]
+| list_vt_nil () => let
+    prval () = opt_none{a}(x0)
+  in
+    false
+  end // end of [list_vt_nil]
+//
+end // end of [linset_choose]
+
+(* ****** ****** *)
+
+implement
+{a}{env}
+linset_foreach_env (xs, env) = let
+//
+implement
+list_vt_foreach$fwork<a><env>
+  (x, env) = linset_foreach$fwork<a><env> (x, env)
+//
+in
+  list_vt_foreach_env<a><env> (xs, env)
+end // end of [linset_foreach_env]
+
+(* ****** ****** *)
+
+implement{a}
+linset_listize (xs) = xs
+
+(* ****** ****** *)
+
+implement{a}
+linset_listize1 (xs) = list_vt_copy (xs)
+
+(* ****** ****** *)
+//
+// HX: functions for processing mynodes
+//
+(* ****** ****** *)
+
+implement{
+} mynode_null{a} () =
+  $UN.castvwtp0{mynode(a,null)}(the_null_ptr)
+// end of [mynode_null]
+
+(* ****** ****** *)
+
+implement
+{a}(*tmp*)
+mynode_make_elt
+  (x) = let
+//
+val nx = list_vt_cons{a}{0}(x, _ )
+//
+in
+  $UN.castvwtp0{mynode1(a)}(nx)
+end // end of [mynode_make_elt]
+
+(* ****** ****** *)
+
+implement{
+} mynode_free
+  {a}(nx) = () where {
+val nx =
+  $UN.castvwtp0{List1_vt(a)}(nx)
+//
+val+~list_vt_cons (_, nx2) = nx
+//
+prval ((*void*)) = $UN.cast2void (nx2)
+//
+} (* end of [mynode_free] *)
+
+(* ****** ****** *)
+
+implement
+{a}(*tmp*)
+mynode_get_elt
+  (nx) = (x) where {
+//
+val nx1 =
+  $UN.castvwtp1{List1_vt(a)}(nx)
+//
+val+list_vt_cons (x, _) = nx1
+//
+prval ((*void*)) = $UN.cast2void (nx1)
+//
+} (* end of [mynode_get_elt] *)
+
+(* ****** ****** *)
+
+implement
+{a}(*tmp*)
+mynode_set_elt
+  {l} (nx, x0) =
+{
+//
+val nx1 =
+  $UN.castvwtp1{List1_vt(a)}(nx)
+//
+val+@list_vt_cons (x, _) = nx1
+//
+val () = x := x0
+//
+prval () = fold@ (nx1)
+prval () = $UN.cast2void (nx1)
+//
+prval () = __assert (nx) where
+{
+  extern praxi __assert (nx: !mynode(a?, l) >> mynode (a, l)): void
+} (* end of [prval] *)
+//
+} (* end of [mynode_set_elt] *)
+
+(* ****** ****** *)
+
+implement
+{a}(*tmp*)
+mynode_getfree_elt
+  (nx) = (x) where {
+//
+val nx =
+  $UN.castvwtp0{List1_vt(a)}(nx)
+//
+val+~list_vt_cons (x, nx2) = nx
+//
+prval ((*void*)) = $UN.cast2void (nx2)
+//
+} (* end of [mynode_getfree_elt] *)
+
+(* ****** ****** *)
+
+(*
+fun{a:t0p}
+linset_takeout_ngc
+  (set: &set(INV(a)) >> _, x0: a):<!wrt> mynode0 (a)
+// end of [linset_takeout_ngc]
+*)
+implement
+{a}(*tmp*)
+linset_takeout_ngc
+  (set, x0) = let
+//
+fun takeout
+(
+  xs: &List0_vt (a) >> _
+) : mynode0(a) = let
+in
+//
+case+ xs of
+| @list_vt_cons
+    (x, xs1) => let
+    prval pf_x = view@x
+    prval pf_xs1 = view@xs1
+    val sgn =
+      compare_elt_elt<a> (x0, x)
+    // end of [val]
+  in
+    if sgn > 0 then let
+      prval () = fold@ (xs)
+    in
+      mynode_null{a}((*void*))
+    end else if sgn < 0 then let
+      val res = takeout (xs1)
+      prval ((*void*)) = fold@ (xs)
+    in
+      res
+    end else let // x0 = x
+      val xs1_ = xs1
+      val res = $UN.castvwtp0{mynode1(a)}((pf_x, pf_xs1 | xs))
+      val () = xs := xs1_
+    in
+      res // [x0] in [xs]
+    end (* end of [if] *)
+  end // end of [list_vt_cons]
+| list_vt_nil () => mynode_null{a}((*void*))
+//
+end (* end of [takeout] *)
+//
+in
+  $effmask_all (takeout (set))
+end // end of [linset_takeout_ngc]
+
+(* ****** ****** *)
+
+implement
+{a}(*tmp*)
+linset_takeoutmax_ngc
+  (xs) = let
+in
+//
+case+ xs of
+| @list_vt_cons
+    (x, xs1) => let
+    prval pf_x = view@x
+    prval pf_xs1 = view@xs1
+    val xs_ = xs
+    val () = xs := xs1
+  in
+    $UN.castvwtp0{mynode1(a)}((pf_x, pf_xs1 | xs_))
+  end // end of [list_vt_cons]
+| @list_vt_nil () => let
+    prval () = fold@ (xs)
+  in
+    mynode_null{a}((*void*))
+  end // end of [list_vt_nil]
+//
+end // end of [linset_takeoutmax_ngc]
+
+(* ****** ****** *)
+
+implement
+{a}(*tmp*)
+linset_takeoutmin_ngc
+  (xs) = let
+//
+fun unsnoc
+  {n:pos} .<n>.
+(
+  xs: &list_vt (a, n) >> list_vt (a, n-1)
+) :<!wrt> mynode1 (a) = let
+//
+val+@list_vt_cons (x, xs1) = xs
+//
+prval pf_x = view@x and pf_xs1 = view@xs1
+//
+in
+//
+case+ xs1 of
+| list_vt_cons _ =>
+    let val res = unsnoc(xs1) in fold@xs; res end
+  // end of [list_vt_cons]
+| list_vt_nil () => let
+    val xs_ = xs
+    val () = xs := list_vt_nil{a}()
+  in
+    $UN.castvwtp0{mynode1(a)}((pf_x, pf_xs1 | xs_))
+  end // end of [list_vt_nil]
+//
+end // end of [unsnoc]
+//
+in
+//
+case+ xs of
+| list_vt_cons _ => unsnoc (xs)
+| list_vt_nil () => mynode_null{a}((*void*))
+//
+end // end of [linset_takeoutmin_ngc]
+
+(* ****** ****** *)
+
+(* end of [linset_listord.dats] *)

samples/ATS/linset_listord.sats

@@ -0,0 +1,51 @@
+(***********************************************************************)
+(*                                                                     *)
+(*                         Applied Type System                         *)
+(*                                                                     *)
+(***********************************************************************)
+
+(*
+** ATS/Postiats - Unleashing the Potential of Types!
+** Copyright (C) 2011-2013 Hongwei Xi, ATS Trustful Software, Inc.
+** All rights reserved
+**
+** ATS is free software;  you can  redistribute it and/or modify it under
+** the terms of  the GNU GENERAL PUBLIC LICENSE (GPL) as published by the
+** Free Software Foundation; either version 3, or (at  your  option)  any
+** later version.
+** 
+** ATS is distributed in the hope that it will be useful, but WITHOUT ANY
+** WARRANTY; without  even  the  implied  warranty  of MERCHANTABILITY or
+** FITNESS FOR A PARTICULAR PURPOSE.  See the  GNU General Public License
+** for more details.
+** 
+** You  should  have  received  a  copy of the GNU General Public License
+** along  with  ATS;  see the  file COPYING.  If not, please write to the
+** Free Software Foundation,  51 Franklin Street, Fifth Floor, Boston, MA
+** 02110-1301, USA.
+*)
+
+(* ****** ****** *)
+//
+// Author: Hongwei Xi
+// Authoremail: hwxiATcsDOTbuDOTedu
+// Time: October, 2010
+//
+(* ****** ****** *)
+
+#define ATS_PACKNAME "ATSLIB.libats.linset_listord"
+#define ATS_STALOADFLAG 0 // no static loading at run-time
+
+(* ****** ****** *)
+
+#include "./SHARE/linset.hats"
+#include "./SHARE/linset_node.hats"
+
+(* ****** ****** *)
+
+castfn
+linset2list {a:t0p} (xs: set (INV(a))):<> List0_vt (a)
+
+(* ****** ****** *)
+
+(* end of [linset_listord.sats] *)

samples/ATS/main.atxt

@@ -0,0 +1,215 @@
+%{
+#include "./../ATEXT/atextfun.hats"
+%}
+
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
+   "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml">
+
+<head>
+<meta http-equiv="content-type" content="text/html; charset=UTF-8" />
+<title>EFFECTIVATS-DiningPhil2</title>
+#patscode_style()
+</head>
+
+<body>
+
+<h1>
+Effective ATS: Dining Philosophers
+</h1>
+
+In this article, I present an implementation of a slight variant of the
+famous problem of 5-Dining-Philosophers by Dijkstra that makes simple but
+convincing use of linear types.
+
+<h2>
+The Original Problem 
+</h2>
+
+There are five philosophers sitting around a table and there are also 5
+forks placed on the table such that each fork is located between the left
+hand of a philosopher and the right hand of another philosopher. Each
+philosopher does the following routine repeatedly: thinking and dining.  In
+order to dine, a philosopher needs to first acquire two forks: one located
+on his left-hand side and the other on his right-hand side. After
+finishing dining, a philosopher puts the two acquired forks onto the table:
+one on his left-hand side and the other on his right-hand side.
+
+<h2>
+A Variant of the Original Problem 
+</h2>
+
+The following twist is added to the original version:
+
+<p>
+
+After a fork is used, it becomes a "dirty" fork and needs to be put in a
+tray for dirty forks. There is a cleaner who cleans dirty forks and then
+puts them back on the table.
+
+<h2>
+Channels for Communication
+</h2>
+
+A channel is just a shared queue of fixed capacity. The following two
+functions are for inserting an element into and taking an element out of a
+given channel:
+
+<pre
+class="patsyntax">
+#pats2xhtml_sats("\
+fun{a:vt0p} channel_insert (channel (a), a): void
+fun{a:vt0p} channel_takeout (chan: channel (a)): (a) 
+")</pre>
+
+If [channel_insert] is called on a channel that is full, then the caller is
+blocked until an element is taken out of the channel.  If [channel_takeout]
+is called on a channel that is empty, then the caller is blocked until an
+element is inserted into the channel.
+
+<h2>
+A Channel for Each Fork
+</h2>
+
+Forks are resources given a linear type. Each fork is initially stored in a
+channel, which can be obtained by calling the following function:
+
+<pre
+class="patsyntax">
+#pats2xhtml_sats("\
+fun fork_changet (n: nphil): channel(fork)
+")</pre>
+
+where the type [nphil] is defined to be [natLt(5)] (for natural numbers
+less than 5). The channels for storing forks are chosen to be of capacity
+2. The reason that channels of capacity 2 are chosen to store at most one
+element (in each of them) is to guarantee that these channels can never be
+full (so that there is no attempt made to send signals to awake callers
+supposedly being blocked due to channels being full).
+
+
+<h2>
+A Channel for the Fork Tray
+</h2>
+
+A tray for storing "dirty" forks is also a channel, which can be obtained
+by calling the following function:
+
+<pre
+class="patsyntax">
+#pats2xhtml_sats("\
+fun forktray_changet ((*void*)): channel(fork)
+")</pre>
+
+The capacity chosen for the channel is 6 (instead of 5) so that it can
+never become full (as there are only 5 forks in total).
+
+<h2>
+Philosopher Loop
+</h2>
+
+Each philosopher is implemented as a loop:
+
+<pre
+class="patsyntax">
+#pats2xhtml_dats('\
+implement
+phil_loop (n) = let
+//
+val () = phil_think (n)
+//
+val nl = phil_left (n) // = n
+val nr = phil_right (n) // = (n+1) % 5
+//
+val ch_lfork = fork_changet (nl)
+val ch_rfork = fork_changet (nr)
+//
+val lf = channel_takeout (ch_lfork)
+val () = println! ("phil_loop(", n, ") picks left fork")
+//
+val () = randsleep (2) // sleep up to 2 seconds
+//
+val rf = channel_takeout (ch_rfork)
+val () = println! ("phil_loop(", n, ") picks right fork")
+//
+val () = phil_dine (n, lf, rf)
+//
+val ch_forktray = forktray_changet ()
+val () = channel_insert (ch_forktray, lf) // left fork to dirty tray
+val () = channel_insert (ch_forktray, rf) // right fork to dirty tray
+//
+in
+  phil_loop (n)
+end // end of [phil_loop]
+')</pre>
+
+It should be straighforward to follow the code for [phil_loop].
+
+<h2>
+Fork Cleaner Loop
+</h2>
+
+A cleaner is implemented as a loop:
+
+<pre
+class="patsyntax">
+#pats2xhtml_dats('\
+implement
+cleaner_loop () = let
+//
+val ch = forktray_changet ()
+val f0 = channel_takeout (ch) // [f0] is dirty
+//
+val () = cleaner_wash (f0) // washes dirty [f0]
+val () = cleaner_return (f0) // puts back cleaned [f0]
+//
+in
+  cleaner_loop ()
+end // end of [cleaner_loop]
+')</pre>
+
+The function [cleaner_return] first finds out the number of a given fork
+and then uses the number to locate the channel for storing the fork. Its
+actual implementation is given as follows:
+
+<pre
+class="patsyntax">
+#pats2xhtml_dats('\
+implement
+cleaner_return (f) =
+{
+  val n = fork_get_num (f)
+  val ch = fork_changet (n)
+  val () = channel_insert (ch, f)
+}
+')</pre>
+
+It should now be straighforward to follow the code for [cleaner_loop].
+
+<h2>
+Testing
+</h2>
+
+The entire code of this implementation is stored in the following files:
+
+<pre>
+DiningPhil2.sats
+DiningPhil2.dats
+DiningPhil2_fork.dats
+DiningPhil2_thread.dats
+</pre>
+
+There is also a Makefile available for compiling the ATS source code into
+an excutable for testing. One should be able to encounter a deadlock after
+running the simulation for a while.
+
+<hr size="2">
+
+This article is written by <a href="http://www.cs.bu.edu/~hwxi/">Hongwei Xi</a>.
+
+</body>
+</html>
+
+%{
+implement main () = fprint_filsub (stdout_ref, "main_atxt.txt")
+%}

samples/Agda/NatCat.agda

@@ -0,0 +1,39 @@
+module NatCat where
+
+open import Relation.Binary.PropositionalEquality
+
+-- If you can show that a relation only ever has one inhabitant
+-- you get the category laws for free
+module
+  EasyCategory
+  (obj : Set)
+  (_⟶_ : obj → obj → Set)
+  (_∘_ : ∀ {x y z} → x ⟶ y → y ⟶ z → x ⟶ z)
+  (id : ∀ x → x ⟶ x)
+  (single-inhabitant : (x y : obj) (r s : x ⟶ y) → r ≡ s)
+  where
+
+  idʳ : ∀ x y (r : x ⟶ y) → r ∘ id y ≡ r
+  idʳ x y r = single-inhabitant x y (r ∘ id y) r 
+
+  idˡ : ∀ x y (r : x ⟶ y) → id x ∘ r ≡ r
+  idˡ x y r = single-inhabitant x y (id x ∘ r) r
+
+  ∘-assoc : ∀ w x y z (r : w ⟶ x) (s : x ⟶ y) (t : y ⟶ z) → (r ∘ s) ∘ t ≡ r ∘ (s ∘ t)
+  ∘-assoc w x y z r s t = single-inhabitant w z ((r ∘ s) ∘ t) (r ∘ (s ∘ t))
+
+open import Data.Nat
+
+same : (x y : ℕ) (r s : x ≤ y) → r ≡ s
+same .0 y z≤n z≤n = refl
+same .(suc m) .(suc n) (s≤s {m} {n} r) (s≤s s) = cong s≤s (same m n r s)
+
+≤-trans : ∀ x y z → x ≤ y → y ≤ z → x ≤ z
+≤-trans .0 y z z≤n s = z≤n
+≤-trans .(suc m) .(suc n) .(suc n₁) (s≤s {m} {n} r) (s≤s {.n} {n₁} s) = s≤s (≤-trans m n n₁ r s)
+
+≤-refl : ∀ x → x ≤ x
+≤-refl zero = z≤n
+≤-refl (suc x) = s≤s (≤-refl x)
+
+module Nat-EasyCategory = EasyCategory ℕ _≤_ (λ {x}{y}{z} → ≤-trans x y z) ≤-refl same

samples/Alloy/file_system.als

@@ -0,0 +1,59 @@
+module examples/systems/file_system
+
+/*
+ * Model of a generic file system.
+ */
+
+abstract sig Object {}
+
+sig Name {}
+
+sig File extends Object {} { some d: Dir | this in d.entries.contents }
+
+sig Dir extends Object {
+  entries: set DirEntry,
+  parent: lone Dir
+} {
+  parent = this.~@contents.~@entries
+  all e1, e2 : entries | e1.name = e2.name => e1 = e2
+  this !in this.^@parent
+  this != Root => Root in this.^@parent
+}
+
+one sig Root extends Dir {} { no parent }
+
+lone sig Cur extends Dir {}
+
+sig DirEntry {
+  name: Name,
+  contents: Object
+} {
+  one this.~entries
+}
+
+
+/**
+ * all directories besides root have one parent
+ */
+pred OneParent_buggyVersion {
+    all d: Dir - Root | one d.parent
+}
+
+/**
+ * all directories besides root have one parent
+ */
+pred OneParent_correctVersion {
+    all d: Dir - Root | (one d.parent && one contents.d)
+}
+
+/**
+ * Only files may be linked (that is, have more than one entry)
+ * That is, all directories are the contents of at most one directory entry
+ */
+pred NoDirAliases {
+    all o: Dir | lone o.~contents
+}
+
+check { OneParent_buggyVersion => NoDirAliases } for 5 expect 1
+
+check { OneParent_correctVersion => NoDirAliases } for 5 expect 0

samples/Alloy/marksweepgc.als

@@ -0,0 +1,83 @@
+module examples/systems/marksweepgc
+
+/*
+ * Model of mark and sweep garbage collection.
+ */
+
+// a node in the heap
+sig Node {}
+
+sig HeapState {
+  left, right : Node -> lone Node,
+  marked : set Node,
+  freeList : lone Node
+}
+
+pred clearMarks[hs, hs' : HeapState] {
+  // clear marked set
+  no hs'.marked
+  // left and right fields are unchanged
+  hs'.left = hs.left
+  hs'.right = hs.right
+}
+
+/**
+ * simulate the recursion of the mark() function using transitive closure
+ */
+fun reachable[hs: HeapState, n: Node] : set Node {
+  n + n.^(hs.left + hs.right)
+}
+
+pred mark[hs: HeapState, from : Node, hs': HeapState] {
+  hs'.marked = hs.reachable[from]
+  hs'.left = hs.left
+  hs'.right = hs.right
+}
+
+/**
+ * complete hack to simulate behavior of code to set freeList
+ */
+pred setFreeList[hs, hs': HeapState] {
+  // especially hackish
+  hs'.freeList.*(hs'.left) in (Node - hs.marked)
+  all n: Node |
+    (n !in hs.marked) => {
+      no hs'.right[n]
+      hs'.left[n] in (hs'.freeList.*(hs'.left))
+      n in hs'.freeList.*(hs'.left)
+    } else {
+      hs'.left[n] = hs.left[n]
+      hs'.right[n] = hs.right[n]
+    }
+  hs'.marked = hs.marked
+}
+
+pred GC[hs: HeapState, root : Node, hs': HeapState] {
+  some hs1, hs2: HeapState |
+    hs.clearMarks[hs1] && hs1.mark[root, hs2] && hs2.setFreeList[hs']
+}
+
+assert Soundness1 {
+  all h, h' : HeapState, root : Node |
+    h.GC[root, h'] =>
+      (all live : h.reachable[root] | {
+        h'.left[live] = h.left[live]
+        h'.right[live] = h.right[live]
+      })
+}
+
+assert Soundness2 {
+  all h, h' : HeapState, root : Node |
+    h.GC[root, h'] =>
+      no h'.reachable[root] & h'.reachable[h'.freeList]
+}
+
+assert Completeness {
+  all h, h' : HeapState, root : Node |
+    h.GC[root, h'] =>
+      (Node - h'.reachable[root]) in h'.reachable[h'.freeList]
+}
+
+check Soundness1 for 3 expect 0
+check Soundness2 for 3 expect 0
+check Completeness for 3 expect 0

samples/Alloy/views.als

@@ -0,0 +1,217 @@
+module examples/systems/views
+
+/*
+ * Model of views in object-oriented programming.
+ *
+ * Two object references, called the view and the backing,
+ * are related by a view mechanism when changes to the
+ * backing are automatically propagated to the view. Note
+ * that the state of a view need not be a projection of the
+ * state of the backing; the keySet method of Map, for
+ * example, produces two view relationships, and for the
+ * one in which the map is modified by changes to the key
+ * set, the value of the new map cannot be determined from
+ * the key set. Note that in the iterator view mechanism,
+ * the iterator is by this definition the backing object,
+ * since changes are propagated from iterator to collection
+ * and not vice versa. Oddly, a reference may be a view of
+ * more than one backing: there can be two iterators on the
+ * same collection, eg. A reference cannot be a view under
+ * more than one view type.
+ *
+ * A reference is made dirty when it is a backing for a view
+ * with which it is no longer related by the view invariant.
+ * This usually happens when a view is modified, either
+ * directly or via another backing. For example, changing a
+ * collection directly when it has an iterator invalidates
+ * it, as does changing the collection through one iterator
+ * when there are others.
+ *
+ * More work is needed if we want to model more closely the
+ * failure of an iterator when its collection is invalidated.
+ *
+ * As a terminological convention, when there are two
+ * complementary view relationships, we will give them types
+ * t and t'. For example, KeySetView propagates from map to
+ * set, and KeySetView' propagates from set to map.
+ *
+ * author: Daniel Jackson
+ */
+
+open util/ordering[State] as so
+open util/relation as rel
+
+sig Ref {}
+sig Object {}
+
+-- t->b->v in views when v is view of type t of backing b
+-- dirty contains refs that have been invalidated
+sig State {
+  refs: set Ref,
+  obj: refs -> one Object,
+  views: ViewType -> refs -> refs,
+  dirty: set refs
+--  , anyviews: Ref -> Ref -- for visualization
+  }
+-- {anyviews = ViewType.views}
+
+sig Map extends Object {
+  keys: set Ref,
+  map: keys -> one Ref
+  }{all s: State |  keys + Ref.map in s.refs}
+sig MapRef extends Ref {}
+fact {State.obj[MapRef] in Map}
+
+sig Iterator extends Object {
+  left, done: set Ref,
+  lastRef: lone done
+  }{all s: State | done + left + lastRef in s.refs}
+sig IteratorRef extends Ref {}
+fact {State.obj[IteratorRef] in Iterator}
+
+sig Set extends Object {
+  elts: set Ref
+  }{all s: State | elts in s.refs}
+sig SetRef extends Ref {}
+fact {State.obj[SetRef] in Set}
+
+abstract sig ViewType {}
+one sig KeySetView, KeySetView', IteratorView extends ViewType {}
+fact ViewTypes {
+  State.views[KeySetView] in MapRef -> SetRef
+  State.views[KeySetView'] in SetRef -> MapRef
+  State.views[IteratorView] in IteratorRef -> SetRef
+  all s: State | s.views[KeySetView] = ~(s.views[KeySetView'])
+  }
+
+/**
+ * mods is refs modified directly or by view mechanism
+ * doesn't handle possibility of modifying an object and its view at once?
+ * should we limit frame conds to non-dirty refs?
+ */
+pred modifies [pre, post: State, rs: set Ref] {
+  let vr = pre.views[ViewType], mods = rs.*vr {
+    all r: pre.refs - mods | pre.obj[r] = post.obj[r]
+    all b: mods, v: pre.refs, t: ViewType |
+      b->v in pre.views[t] => viewFrame [t, pre.obj[v], post.obj[v], post.obj[b]]
+    post.dirty = pre.dirty +
+      {b: pre.refs | some v: Ref, t: ViewType |
+          b->v in pre.views[t] && !viewFrame [t, pre.obj[v], post.obj[v], post.obj[b]]
+      }
+    }
+  }
+
+pred allocates [pre, post: State, rs: set Ref] {
+  no rs & pre.refs
+  post.refs = pre.refs + rs
+  }
+
+/** 
+ * models frame condition that limits change to view object from v to v' when backing object changes to b'
+ */
+pred viewFrame [t: ViewType, v, v', b': Object] {
+  t in KeySetView => v'.elts = dom [b'.map]
+  t in KeySetView' => b'.elts = dom [v'.map]
+  t in KeySetView' => (b'.elts) <: (v.map) = (b'.elts) <: (v'.map)
+  t in IteratorView => v'.elts = b'.left + b'.done
+  }
+
+pred MapRef.keySet [pre, post: State, setRefs: SetRef] {
+  post.obj[setRefs].elts = dom [pre.obj[this].map]
+  modifies [pre, post, none]
+  allocates [pre, post, setRefs]
+  post.views = pre.views + KeySetView->this->setRefs + KeySetView'->setRefs->this
+  }
+
+pred MapRef.put [pre, post: State, k, v: Ref] {
+  post.obj[this].map = pre.obj[this].map ++ k->v
+  modifies [pre, post, this]
+  allocates [pre, post, none]
+  post.views = pre.views
+  }
+
+pred SetRef.iterator [pre, post: State, iterRef: IteratorRef] {
+  let i = post.obj[iterRef] {
+    i.left = pre.obj[this].elts
+    no i.done + i.lastRef
+    }
+  modifies [pre,post,none]
+  allocates [pre, post, iterRef]
+  post.views = pre.views + IteratorView->iterRef->this
+  }
+
+pred IteratorRef.remove [pre, post: State] {
+  let i = pre.obj[this], i' = post.obj[this] {
+    i'.left = i.left
+    i'.done = i.done - i.lastRef
+    no i'.lastRef
+    }
+  modifies [pre,post,this]
+  allocates [pre, post, none]
+  pre.views = post.views
+  }
+
+pred IteratorRef.next [pre, post: State, ref: Ref] {
+  let i = pre.obj[this], i' = post.obj[this] {
+    ref in i.left
+    i'.left = i.left - ref
+    i'.done = i.done + ref
+    i'.lastRef = ref
+    }
+  modifies [pre, post, this]
+  allocates [pre, post, none]
+  pre.views = post.views
+  }
+
+pred IteratorRef.hasNext [s: State] {
+  some s.obj[this].left
+  }
+
+assert zippishOK {
+  all
+    ks, vs: SetRef,
+    m: MapRef,
+    ki, vi: IteratorRef,
+    k, v: Ref |
+    let s0=so/first,
+    s1=so/next[s0],
+    s2=so/next[s1],
+    s3=so/next[s2],
+    s4=so/next[s3],
+    s5=so/next[s4],
+    s6=so/next[s5],
+    s7=so/next[s6] |
+  ({
+    precondition [s0, ks, vs, m]
+    no s0.dirty
+    ks.iterator [s0, s1, ki]
+    vs.iterator [s1, s2, vi]
+    ki.hasNext [s2]
+    vi.hasNext [s2]
+    ki.this/next [s2, s3, k]
+    vi.this/next [s3, s4, v]
+    m.put [s4, s5, k, v]
+    ki.remove [s5, s6]
+    vi.remove [s6, s7]
+  } => no State.dirty)
+  }
+
+pred precondition [pre: State, ks, vs, m: Ref] {
+  // all these conditions and other errors discovered in scope of 6 but 8,3
+  // in initial state, must have view invariants hold
+  (all t: ViewType, b, v: pre.refs |
+    b->v in pre.views[t] => viewFrame [t, pre.obj[v], pre.obj[v], pre.obj[b]])
+  // sets are not aliases
+--  ks != vs
+  // sets are not views of map
+--  no (ks+vs)->m & ViewType.pre.views
+  // no iterator currently on either set
+--  no Ref->(ks+vs) & ViewType.pre.views
+  }
+
+check zippishOK for 6 but 8 State, 3 ViewType expect 1
+
+/** 
+ * experiment with controlling heap size
+ */
+fact {all s: State | #s.obj < 5}

samples/Ant Build System/filenames/ant.xml

@@ -0,0 +1,110 @@
+<?xml version="1.0" encoding="iso-8859-1"?>
+<project name="WebBuild">
+
+    <!-- generate timestamps -->
+    <tstamp />
+
+    <!-- Debugging Macro -->
+    <import file="echopath.xml" />
+
+    <!-- JS build files macro -->
+    <import file="rhinoscript.xml" />
+
+    <!-- Component Build Files -->
+    <import file="setup.xml" />
+    <import file="clean.xml" />
+    <import file="copy.xml" />
+    <import file="file.transform.xml" />
+    <import file="external.tools.xml" />
+    <import file="rename.xml" />
+    <import file="js.xml" />
+    <import file="css.xml" />
+    <import file="img.xml" />
+    <import file="png8.xml" />
+    <import file="yui.xml" />
+    <import file="cdn.xml" />
+    <import file="datauri.xml" />
+    <import file="devlive.xml" />
+
+    <!-- This dirname is the only complete path we know for sure, everything builds off of it -->
+    <dirname property="dir.build" file="${ant.file.WebBuild}" />
+
+    <!-- get name for newly built folder -->
+    <basename property="app.name"       file="${basedir}" />
+
+    <!-- read global properties file -->
+    <property file="${dir.build}\build.properties" />
+
+    <!-- Build Directories -->
+    <property name="dir.build.js"   location="${dir.build}/js" />
+
+    <!-- App Directories -->
+    <property name="dir.app"        location="${dir.result}/${app.name}" />
+    <property name="dir.app.temp"   location="${dir.temp}/${app.name}" />
+    <property name="dir.app.files"  location="${dir.app.temp}/${dir.files}" />
+
+    <!-- Files -->
+    <property name="mapping.js"     location="${dir.app.temp}/${mapping.file.js}" />
+    <property name="mapping.css"    location="${dir.app.temp}/${mapping.file.css}" />
+    <property name="mapping.img"    location="${dir.app.temp}/${mapping.file.img}" />
+    <property name="mapping.swf"    location="${dir.app.temp}/${mapping.file.swf}" />
+    <property name="mapping.fonts"  location="${dir.app.temp}/${mapping.file.fonts}" />
+
+    <!-- Tool Directories -->
+    <property name="dir.bin"    location="${dir.build}/Bin" />
+    <property name="dir.jar"    location="${dir.bin}/jar" />
+
+    <!-- Tool Files -->
+	<property name="tools.compressor"     location="${dir.jar}/${tools.file.compressor}" />
+	<property name="tools.cssembed"       location="${dir.jar}/${tools.file.cssembed}" />
+    <property name="tools.filetransform"  location="${dir.jar}/${tools.file.filetransform}" />
+    <property name="tools.optipng"        location="${dir.bin}/${tools.file.optipng}" />
+    <property name="tools.jpegtran"       location="${dir.bin}/${tools.file.jpegtran}" />
+
+
+    <!-- BUILD TARGETS -->
+
+    <!-- low level utility build targets -->
+
+    <!-- Build the tools -->
+    <target name="-setup.build.tools"
+            depends="-define.filetransform, -define.cssembed, -define.yuicompressor, -define.jsclasspath"
+    />
+
+    <!-- set up filesystem properties -->
+    <target
+        name="-setup"
+        depends="-setup.mode, -setup.conditions, -setup.js, -setup.css, -setup.swf, -setup.img, -setup.fonts, -setup.yui"
+    />
+
+    <!-- utility-ish targets -->
+    <target name="copy"         depends="clean, tools, -copy" />
+    <target name="tools"        depends="-setup.build.tools" />
+    <target name="finalize"     depends="copy, -finalize" />
+    <target name="-prepare"     depends="copy, -setup" />
+
+    <!-- individual component build targets (empty descriptions are to make sure they show in "ant -p") -->
+    <target name="devlive"      depends="-prepare, -devlive"            description="" />
+    <target name="js"           depends="-prepare, -js"                 description="" />
+    <target name="css"          depends="-prepare, -css"                description="" />
+    <target name="rename"       depends="-prepare, -rename"             description="" />
+    <target name="yui"          depends="-prepare, rename, -yui"        description="" />
+    <target name="cdn"          depends="-prepare, -cdn"                description="" />
+
+    <!-- high level build targets (Excluding of images is on purpose here, it's slow) -->
+    <target name="core"
+            depends="devlive, js, css, cdn, rename, yui, -js.inline"
+            description="Core build work"
+    />
+
+    <target name="prod"
+            depends="core, finalize"
+            description="Full Production Build"
+    />
+
+    <!-- debug target -->
+    <target name="debug" depends="-setup">
+        <echoproperties/>
+    </target>
+
+</project>

samples/Ant Build System/filenames/build.xml

@@ -0,0 +1 @@
+ant.xml

samples/ApacheConf/filenames/.htaccess

@@ -0,0 +1,26 @@
+ServerSignature Off
+RewriteCond %{REQUEST_METHOD} ^(HEAD|TRACE|DELETE|TRACK) [NC,OR]
+RewriteCond %{THE_REQUEST} (\\r|\\n|%0A|%0D) [NC,OR]
+
+RewriteCond %{HTTP_REFERER} (<|>|’|%0A|%0D|%27|%3C|%3E|%00) [NC,OR]
+RewriteCond %{HTTP_COOKIE} (<|>|’|%0A|%0D|%27|%3C|%3E|%00) [NC,OR]
+RewriteCond %{REQUEST_URI} ^/(,|;|:|<|>|”>|”<|/|\\\.\.\\).{0,9999} [NC,OR]
+
+RewriteCond %{HTTP_USER_AGENT} ^$ [OR]
+RewriteCond %{HTTP_USER_AGENT} ^(java|curl|wget) [NC,OR]
+RewriteCond %{HTTP_USER_AGENT} (winhttp|HTTrack|clshttp|archiver|loader|email|harvest|extract|grab|miner) [NC,OR]
+RewriteCond %{HTTP_USER_AGENT} (libwww-perl|curl|wget|python|nikto|scan) [NC,OR]
+RewriteCond %{HTTP_USER_AGENT} (<|>|’|%0A|%0D|%27|%3C|%3E|%00) [NC,OR]
+
+#Block mySQL injects
+RewriteCond %{QUERY_STRING} (;|<|>|’|”|\)|%0A|%0D|%22|%27|%3C|%3E|%00).*(/\*|union|select|insert|cast|set|declare|drop|update|md5|benchmark) [NC,OR]
+
+RewriteCond %{QUERY_STRING} \.\./\.\. [OR]
+
+RewriteCond %{QUERY_STRING} (localhost|loopback|127\.0\.0\.1) [NC,OR]
+RewriteCond %{QUERY_STRING} \.[a-z0-9] [NC,OR]
+RewriteCond %{QUERY_STRING} (<|>|’|%0A|%0D|%27|%3C|%3E|%00) [NC]
+# Note: The final RewriteCond must NOT use the [OR] flag.
+
+# Return 403 Forbidden error.
+RewriteRule .* index.php [F]

samples/ApacheConf/filenames/apache2.conf

@@ -0,0 +1,470 @@
+# This is the main Apache HTTP server configuration file.  It contains the
+# configuration directives that give the server its instructions.
+# See <URL:http://httpd.apache.org/docs/2.2> for detailed information.
+# In particular, see
+# <URL:http://httpd.apache.org/docs/2.2/mod/directives.html>
+# for a discussion of each configuration directive.
+#
+# Do NOT simply read the instructions in here without understanding
+# what they do.  They're here only as hints or reminders.  If you are unsure
+# consult the online docs. You have been warned.
+#
+# Configuration and logfile names: If the filenames you specify for many
+# of the server's control files begin with "/" (or "drive:/" for Win32), the
+# server will use that explicit path.  If the filenames do *not* begin
+# with "/", the value of ServerRoot is prepended -- so "/var/log/apache2/foo.log"
+# with ServerRoot set to "" will be interpreted by the
+# server as "//var/log/apache2/foo.log".
+
+#
+# ServerRoot: The top of the directory tree under which the server's
+# configuration, error, and log files are kept.
+#
+# Do not add a slash at the end of the directory path.  If you point
+# ServerRoot at a non-local disk, be sure to point the LockFile directive
+# at a local disk.  If you wish to share the same ServerRoot for multiple
+# httpd daemons, you will need to change at least LockFile and PidFile.
+#
+ServerRoot ""
+
+#
+# Listen: Allows you to bind Apache to specific IP addresses and/or
+# ports, instead of the default. See also the <VirtualHost>
+# directive.
+#
+# Change this to Listen on specific IP addresses as shown below to
+# prevent Apache from glomming onto all bound IP addresses.
+#
+#Listen 12.34.56.78:80
+Listen 80
+
+#
+# Dynamic Shared Object (DSO) Support
+#
+# To be able to use the functionality of a module which was built as a DSO you
+# have to place corresponding `LoadModule' lines at this location so the
+# directives contained in it are actually available _before_ they are used.
+# Statically compiled modules (those listed by `httpd -l') do not need
+# to be loaded here.
+#
+# Example:
+# LoadModule foo_module modules/mod_foo.so
+#
+LoadModule authn_file_module /usr/lib/apache2/modules/mod_authn_file.so
+LoadModule authn_dbm_module /usr/lib/apache2/modules/mod_authn_dbm.so
+LoadModule authn_anon_module /usr/lib/apache2/modules/mod_authn_anon.so
+LoadModule authn_dbd_module /usr/lib/apache2/modules/mod_authn_dbd.so
+LoadModule authn_default_module /usr/lib/apache2/modules/mod_authn_default.so
+LoadModule authn_alias_module /usr/lib/apache2/modules/mod_authn_alias.so
+LoadModule authz_host_module /usr/lib/apache2/modules/mod_authz_host.so
+LoadModule authz_groupfile_module /usr/lib/apache2/modules/mod_authz_groupfile.so
+LoadModule authz_user_module /usr/lib/apache2/modules/mod_authz_user.so
+LoadModule authz_dbm_module /usr/lib/apache2/modules/mod_authz_dbm.so
+LoadModule authz_owner_module /usr/lib/apache2/modules/mod_authz_owner.so
+LoadModule authnz_ldap_module /usr/lib/apache2/modules/mod_authnz_ldap.so
+LoadModule authz_default_module /usr/lib/apache2/modules/mod_authz_default.so
+LoadModule auth_basic_module /usr/lib/apache2/modules/mod_auth_basic.so
+LoadModule auth_digest_module /usr/lib/apache2/modules/mod_auth_digest.so
+LoadModule file_cache_module /usr/lib/apache2/modules/mod_file_cache.so
+LoadModule cache_module /usr/lib/apache2/modules/mod_cache.so
+LoadModule disk_cache_module /usr/lib/apache2/modules/mod_disk_cache.so
+LoadModule mem_cache_module /usr/lib/apache2/modules/mod_mem_cache.so
+LoadModule dbd_module /usr/lib/apache2/modules/mod_dbd.so
+LoadModule dumpio_module /usr/lib/apache2/modules/mod_dumpio.so
+LoadModule ext_filter_module /usr/lib/apache2/modules/mod_ext_filter.so
+LoadModule include_module /usr/lib/apache2/modules/mod_include.so
+LoadModule filter_module /usr/lib/apache2/modules/mod_filter.so
+LoadModule charset_lite_module /usr/lib/apache2/modules/mod_charset_lite.so
+LoadModule deflate_module /usr/lib/apache2/modules/mod_deflate.so
+LoadModule ldap_module /usr/lib/apache2/modules/mod_ldap.so
+LoadModule log_forensic_module /usr/lib/apache2/modules/mod_log_forensic.so
+LoadModule env_module /usr/lib/apache2/modules/mod_env.so
+LoadModule mime_magic_module /usr/lib/apache2/modules/mod_mime_magic.so
+LoadModule cern_meta_module /usr/lib/apache2/modules/mod_cern_meta.so
+LoadModule expires_module /usr/lib/apache2/modules/mod_expires.so
+LoadModule headers_module /usr/lib/apache2/modules/mod_headers.so
+LoadModule ident_module /usr/lib/apache2/modules/mod_ident.so
+LoadModule usertrack_module /usr/lib/apache2/modules/mod_usertrack.so
+LoadModule unique_id_module /usr/lib/apache2/modules/mod_unique_id.so
+LoadModule setenvif_module /usr/lib/apache2/modules/mod_setenvif.so
+LoadModule version_module /usr/lib/apache2/modules/mod_version.so
+LoadModule proxy_module /usr/lib/apache2/modules/mod_proxy.so
+LoadModule proxy_connect_module /usr/lib/apache2/modules/mod_proxy_connect.so
+LoadModule proxy_ftp_module /usr/lib/apache2/modules/mod_proxy_ftp.so
+LoadModule proxy_http_module /usr/lib/apache2/modules/mod_proxy_http.so
+LoadModule proxy_ajp_module /usr/lib/apache2/modules/mod_proxy_ajp.so
+LoadModule proxy_balancer_module /usr/lib/apache2/modules/mod_proxy_balancer.so
+LoadModule ssl_module /usr/lib/apache2/modules/mod_ssl.so
+LoadModule mime_module /usr/lib/apache2/modules/mod_mime.so
+LoadModule dav_module /usr/lib/apache2/modules/mod_dav.so
+LoadModule status_module /usr/lib/apache2/modules/mod_status.so
+LoadModule autoindex_module /usr/lib/apache2/modules/mod_autoindex.so
+LoadModule asis_module /usr/lib/apache2/modules/mod_asis.so
+LoadModule info_module /usr/lib/apache2/modules/mod_info.so
+LoadModule suexec_module /usr/lib/apache2/modules/mod_suexec.so
+LoadModule cgid_module /usr/lib/apache2/modules/mod_cgid.so
+LoadModule cgi_module /usr/lib/apache2/modules/mod_cgi.so
+LoadModule dav_fs_module /usr/lib/apache2/modules/mod_dav_fs.so
+LoadModule dav_lock_module /usr/lib/apache2/modules/mod_dav_lock.so
+LoadModule vhost_alias_module /usr/lib/apache2/modules/mod_vhost_alias.so
+LoadModule negotiation_module /usr/lib/apache2/modules/mod_negotiation.so
+LoadModule dir_module /usr/lib/apache2/modules/mod_dir.so
+LoadModule imagemap_module /usr/lib/apache2/modules/mod_imagemap.so
+LoadModule actions_module /usr/lib/apache2/modules/mod_actions.so
+LoadModule speling_module /usr/lib/apache2/modules/mod_speling.so
+LoadModule userdir_module /usr/lib/apache2/modules/mod_userdir.so
+LoadModule alias_module /usr/lib/apache2/modules/mod_alias.so
+LoadModule rewrite_module /usr/lib/apache2/modules/mod_rewrite.so
+
+<IfModule !mpm_netware_module>
+#
+# If you wish httpd to run as a different user or group, you must run
+# httpd as root initially and it will switch.
+#
+# User/Group: The name (or #number) of the user/group to run httpd as.
+# It is usually good practice to create a dedicated user and group for
+# running httpd, as with most system services.
+#
+User daemon
+Group daemon
+</IfModule>
+
+# 'Main' server configuration
+#
+# The directives in this section set up the values used by the 'main'
+# server, which responds to any requests that aren't handled by a
+# <VirtualHost> definition.  These values also provide defaults for
+# any <VirtualHost> containers you may define later in the file.
+#
+# All of these directives may appear inside <VirtualHost> containers,
+# in which case these default settings will be overridden for the
+# virtual host being defined.
+#
+
+#
+# ServerAdmin: Your address, where problems with the server should be
+# e-mailed.  This address appears on some server-generated pages, such
+# as error documents.  e.g. admin@your-domain.com
+#
+ServerAdmin you@example.com
+
+#
+# ServerName gives the name and port that the server uses to identify itself.
+# This can often be determined automatically, but we recommend you specify
+# it explicitly to prevent problems during startup.
+#
+# If your host doesn't have a registered DNS name, enter its IP address here.
+#
+#ServerName www.example.com:80
+
+#
+# DocumentRoot: The directory out of which you will serve your
+# documents. By default, all requests are taken from this directory, but
+# symbolic links and aliases may be used to point to other locations.
+#
+DocumentRoot "/usr/share/apache2/default-site/htdocs"
+
+#
+# Each directory to which Apache has access can be configured with respect
+# to which services and features are allowed and/or disabled in that
+# directory (and its subdirectories).
+#
+# First, we configure the "default" to be a very restrictive set of
+# features.
+#
+<Directory />
+    Options FollowSymLinks
+    AllowOverride None
+    Order deny,allow
+    Deny from all
+</Directory>
+
+#
+# Note that from this point forward you must specifically allow
+# particular features to be enabled - so if something's not working as
+# you might expect, make sure that you have specifically enabled it
+# below.
+#
+
+#
+# This should be changed to whatever you set DocumentRoot to.
+#
+<Directory "/usr/share/apache2/default-site/htdocs">
+    #
+    # Possible values for the Options directive are "None", "All",
+    # or any combination of:
+    #   Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews
+    #
+    # Note that "MultiViews" must be named *explicitly* --- "Options All"
+    # doesn't give it to you.
+    #
+    # The Options directive is both complicated and important.  Please see
+    # http://httpd.apache.org/docs/2.2/mod/core.html#options
+    # for more information.
+    #
+    Options Indexes FollowSymLinks
+
+    #
+    # AllowOverride controls what directives may be placed in .htaccess files.
+    # It can be "All", "None", or any combination of the keywords:
+    #   Options FileInfo AuthConfig Limit
+    #
+    AllowOverride None
+
+    #
+    # Controls who can get stuff from this server.
+    #
+    Order allow,deny
+    Allow from all
+
+</Directory>
+
+#
+# DirectoryIndex: sets the file that Apache will serve if a directory
+# is requested.
+#
+<IfModule dir_module>
+    DirectoryIndex index.html
+</IfModule>
+
+#
+# The following lines prevent .htaccess and .htpasswd files from being
+# viewed by Web clients.
+#
+<FilesMatch "^\.ht">
+    Order allow,deny
+    Deny from all
+    Satisfy All
+</FilesMatch>
+
+#
+# ErrorLog: The location of the error log file.
+# If you do not specify an ErrorLog directive within a <VirtualHost>
+# container, error messages relating to that virtual host will be
+# logged here.  If you *do* define an error logfile for a <VirtualHost>
+# container, that host's errors will be logged there and not here.
+#
+ErrorLog /var/log/apache2/error_log
+
+#
+# LogLevel: Control the number of messages logged to the error_log.
+# Possible values include: debug, info, notice, warn, error, crit,
+# alert, emerg.
+#
+LogLevel warn
+
+<IfModule log_config_module>
+    #
+    # The following directives define some format nicknames for use with
+    # a CustomLog directive (see below).
+    #
+    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
+    LogFormat "%h %l %u %t \"%r\" %>s %b" common
+
+    <IfModule logio_module>
+      # You need to enable mod_logio.c to use %I and %O
+      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
+    </IfModule>
+
+    #
+    # The location and format of the access logfile (Common Logfile Format).
+    # If you do not define any access logfiles within a <VirtualHost>
+    # container, they will be logged here.  Contrariwise, if you *do*
+    # define per-<VirtualHost> access logfiles, transactions will be
+    # logged therein and *not* in this file.
+    #
+    CustomLog /var/log/apache2/access_log common
+
+    #
+    # If you prefer a logfile with access, agent, and referer information
+    # (Combined Logfile Format) you can use the following directive.
+    #
+    #CustomLog /var/log/apache2/access_log combined
+</IfModule>
+
+<IfModule alias_module>
+    #
+    # Redirect: Allows you to tell clients about documents that used to
+    # exist in your server's namespace, but do not anymore. The client
+    # will make a new request for the document at its new location.
+    # Example:
+    # Redirect permanent /foo http://www.example.com/bar
+
+    #
+    # Alias: Maps web paths into filesystem paths and is used to
+    # access content that does not live under the DocumentRoot.
+    # Example:
+    # Alias /webpath /full/filesystem/path
+    #
+    # If you include a trailing / on /webpath then the server will
+    # require it to be present in the URL.  You will also likely
+    # need to provide a <Directory> section to allow access to
+    # the filesystem path.
+
+    #
+    # ScriptAlias: This controls which directories contain server scripts.
+    # ScriptAliases are essentially the same as Aliases, except that
+    # documents in the target directory are treated as applications and
+    # run by the server when requested rather than as documents sent to the
+    # client.  The same rules about trailing "/" apply to ScriptAlias
+    # directives as to Alias.
+    #
+    ScriptAlias /cgi-bin/ "/usr/lib/cgi-bin/"
+
+</IfModule>
+
+<IfModule cgid_module>
+    #
+    # ScriptSock: On threaded servers, designate the path to the UNIX
+    # socket used to communicate with the CGI daemon of mod_cgid.
+    #
+    #Scriptsock /var/run/apache2/cgisock
+</IfModule>
+
+#
+# "/usr/lib/cgi-bin" should be changed to whatever your ScriptAliased
+# CGI directory exists, if you have that configured.
+#
+<Directory "/usr/lib/cgi-bin">
+    AllowOverride None
+    Options None
+    Order allow,deny
+    Allow from all
+</Directory>
+
+#
+# DefaultType: the default MIME type the server will use for a document
+# if it cannot otherwise determine one, such as from filename extensions.
+# If your server contains mostly text or HTML documents, "text/plain" is
+# a good value.  If most of your content is binary, such as applications
+# or images, you may want to use "application/octet-stream" instead to
+# keep browsers from trying to display binary files as though they are
+# text.
+#
+DefaultType text/plain
+
+<IfModule mime_module>
+    #
+    # TypesConfig points to the file containing the list of mappings from
+    # filename extension to MIME-type.
+    #
+    TypesConfig /etc/apache2/mime.types
+
+    #
+    # AddType allows you to add to or override the MIME configuration
+    # file specified in TypesConfig for specific file types.
+    #
+    #AddType application/x-gzip .tgz
+    #
+    # AddEncoding allows you to have certain browsers uncompress
+    # information on the fly. Note: Not all browsers support this.
+    #
+    #AddEncoding x-compress .Z
+    #AddEncoding x-gzip .gz .tgz
+    #
+    # If the AddEncoding directives above are commented-out, then you
+    # probably should define those extensions to indicate media types:
+    #
+    AddType application/x-compress .Z
+    AddType application/x-gzip .gz .tgz
+
+    #
+    # AddHandler allows you to map certain file extensions to "handlers":
+    # actions unrelated to filetype. These can be either built into the server
+    # or added with the Action directive (see below)
+    #
+    # To use CGI scripts outside of ScriptAliased directories:
+    # (You will also need to add "ExecCGI" to the "Options" directive.)
+    #
+    #AddHandler cgi-script .cgi
+
+    # For type maps (negotiated resources):
+    #AddHandler type-map var
+
+    #
+    # Filters allow you to process content before it is sent to the client.
+    #
+    # To parse .shtml files for server-side includes (SSI):
+    # (You will also need to add "Includes" to the "Options" directive.)
+    #
+    #AddType text/html .shtml
+    #AddOutputFilter INCLUDES .shtml
+</IfModule>
+
+#
+# The mod_mime_magic module allows the server to use various hints from the
+# contents of the file itself to determine its type.  The MIMEMagicFile
+# directive tells the module where the hint definitions are located.
+#
+#MIMEMagicFile /etc/apache2/magic
+
+#
+# Customizable error responses come in three flavors:
+# 1) plain text 2) local redirects 3) external redirects
+#
+# Some examples:
+#ErrorDocument 500 "The server made a boo boo."
+#ErrorDocument 404 /missing.html
+#ErrorDocument 404 "/cgi-bin/missing_handler.pl"
+#ErrorDocument 402 http://www.example.com/subscription_info.html
+#
+
+#
+# EnableMMAP and EnableSendfile: On systems that support it,
+# memory-mapping or the sendfile syscall is used to deliver
+# files.  This usually improves server performance, but must
+# be turned off when serving from networked-mounted
+# filesystems or if support for these functions is otherwise
+# broken on your system.
+#
+#EnableMMAP off
+#EnableSendfile off
+
+# Supplemental configuration
+#
+# The configuration files in the /etc/apache2/extra/ directory can be
+# included to add extra features or to modify the default configuration of
+# the server, or you may simply copy their contents here and change as
+# necessary.
+
+# Server-pool management (MPM specific)
+#Include /etc/apache2/extra/httpd-mpm.conf
+
+# Multi-language error messages
+#Include /etc/apache2/extra/httpd-multilang-errordoc.conf
+
+# Fancy directory listings
+#Include /etc/apache2/extra/httpd-autoindex.conf
+
+# Language settings
+#Include /etc/apache2/extra/httpd-languages.conf
+
+# User home directories
+#Include /etc/apache2/extra/httpd-userdir.conf
+
+# Real-time info on requests and configuration
+#Include /etc/apache2/extra/httpd-info.conf
+
+# Virtual hosts
+#Include /etc/apache2/extra/httpd-vhosts.conf
+
+# Local access to the Apache HTTP Server Manual
+#Include /etc/apache2/extra/httpd-manual.conf
+
+# Distributed authoring and versioning (WebDAV)
+#Include /etc/apache2/extra/httpd-dav.conf
+
+# Various default settings
+#Include /etc/apache2/extra/httpd-default.conf
+
+# Secure (SSL/TLS) connections
+#Include /etc/apache2/extra/httpd-ssl.conf
+#
+# Note: The following must must be present to support
+#       starting without SSL on platforms with no /dev/random equivalent
+#       but a statically compiled-in mod_ssl.
+#
+<IfModule ssl_module>
+SSLRandomSeed startup builtin
+SSLRandomSeed connect builtin
+</IfModule>

samples/ApacheConf/filenames/httpd.conf

@@ -0,0 +1,500 @@
+#
+# This is the main Apache HTTP server configuration file.  It contains the
+# configuration directives that give the server its instructions.
+# See <URL:http://httpd.apache.org/docs/2.2> for detailed information.
+# In particular, see 
+# <URL:http://httpd.apache.org/docs/2.2/mod/directives.html>
+# for a discussion of each configuration directive.
+#
+# Do NOT simply read the instructions in here without understanding
+# what they do.  They're here only as hints or reminders.  If you are unsure
+# consult the online docs. You have been warned.  
+#
+# Configuration and logfile names: If the filenames you specify for many
+# of the server's control files begin with "/" (or "drive:/" for Win32), the
+# server will use that explicit path.  If the filenames do *not* begin
+# with "/", the value of ServerRoot is prepended -- so "log/foo_log"
+# with ServerRoot set to "/usr" will be interpreted by the
+# server as "/usr/log/foo_log".
+
+#
+# ServerRoot: The top of the directory tree under which the server's
+# configuration, error, and log files are kept.
+#
+# Do not add a slash at the end of the directory path.  If you point
+# ServerRoot at a non-local disk, be sure to point the LockFile directive
+# at a local disk.  If you wish to share the same ServerRoot for multiple
+# httpd daemons, you will need to change at least LockFile and PidFile.
+#
+ServerRoot "/usr"
+
+#
+# Listen: Allows you to bind Apache to specific IP addresses and/or
+# ports, instead of the default. See also the <VirtualHost>
+# directive.
+#
+# Change this to Listen on specific IP addresses as shown below to 
+# prevent Apache from glomming onto all bound IP addresses.
+#
+#Listen 12.34.56.78:80
+Listen 80
+
+#
+# Dynamic Shared Object (DSO) Support
+#
+# To be able to use the functionality of a module which was built as a DSO you
+# have to place corresponding `LoadModule' lines at this location so the
+# directives contained in it are actually available _before_ they are used.
+# Statically compiled modules (those listed by `httpd -l') do not need
+# to be loaded here.
+#
+# Example:
+# LoadModule foo_module modules/mod_foo.so
+#
+LoadModule authn_file_module libexec/apache2/mod_authn_file.so
+LoadModule authn_dbm_module libexec/apache2/mod_authn_dbm.so
+LoadModule authn_anon_module libexec/apache2/mod_authn_anon.so
+LoadModule authn_dbd_module libexec/apache2/mod_authn_dbd.so
+LoadModule authn_default_module libexec/apache2/mod_authn_default.so
+LoadModule authz_host_module libexec/apache2/mod_authz_host.so
+LoadModule authz_groupfile_module libexec/apache2/mod_authz_groupfile.so
+LoadModule authz_user_module libexec/apache2/mod_authz_user.so
+LoadModule authz_dbm_module libexec/apache2/mod_authz_dbm.so
+LoadModule authz_owner_module libexec/apache2/mod_authz_owner.so
+LoadModule authz_default_module libexec/apache2/mod_authz_default.so
+LoadModule auth_basic_module libexec/apache2/mod_auth_basic.so
+LoadModule auth_digest_module libexec/apache2/mod_auth_digest.so
+LoadModule cache_module libexec/apache2/mod_cache.so
+LoadModule disk_cache_module libexec/apache2/mod_disk_cache.so
+LoadModule mem_cache_module libexec/apache2/mod_mem_cache.so
+LoadModule dbd_module libexec/apache2/mod_dbd.so
+LoadModule dumpio_module libexec/apache2/mod_dumpio.so
+LoadModule reqtimeout_module libexec/apache2/mod_reqtimeout.so
+LoadModule ext_filter_module libexec/apache2/mod_ext_filter.so
+LoadModule include_module libexec/apache2/mod_include.so
+LoadModule filter_module libexec/apache2/mod_filter.so
+LoadModule substitute_module libexec/apache2/mod_substitute.so
+LoadModule deflate_module libexec/apache2/mod_deflate.so
+LoadModule log_config_module libexec/apache2/mod_log_config.so
+LoadModule log_forensic_module libexec/apache2/mod_log_forensic.so
+LoadModule logio_module libexec/apache2/mod_logio.so
+LoadModule env_module libexec/apache2/mod_env.so
+LoadModule mime_magic_module libexec/apache2/mod_mime_magic.so
+LoadModule cern_meta_module libexec/apache2/mod_cern_meta.so
+LoadModule expires_module libexec/apache2/mod_expires.so
+LoadModule headers_module libexec/apache2/mod_headers.so
+LoadModule ident_module libexec/apache2/mod_ident.so
+LoadModule usertrack_module libexec/apache2/mod_usertrack.so
+#LoadModule unique_id_module libexec/apache2/mod_unique_id.so
+LoadModule setenvif_module libexec/apache2/mod_setenvif.so
+LoadModule version_module libexec/apache2/mod_version.so
+LoadModule proxy_module libexec/apache2/mod_proxy.so
+LoadModule proxy_connect_module libexec/apache2/mod_proxy_connect.so
+LoadModule proxy_ftp_module libexec/apache2/mod_proxy_ftp.so
+LoadModule proxy_http_module libexec/apache2/mod_proxy_http.so
+LoadModule proxy_scgi_module libexec/apache2/mod_proxy_scgi.so
+LoadModule proxy_ajp_module libexec/apache2/mod_proxy_ajp.so
+LoadModule proxy_balancer_module libexec/apache2/mod_proxy_balancer.so
+LoadModule ssl_module libexec/apache2/mod_ssl.so
+LoadModule mime_module libexec/apache2/mod_mime.so
+LoadModule dav_module libexec/apache2/mod_dav.so
+LoadModule status_module libexec/apache2/mod_status.so
+LoadModule autoindex_module libexec/apache2/mod_autoindex.so
+LoadModule asis_module libexec/apache2/mod_asis.so
+LoadModule info_module libexec/apache2/mod_info.so
+LoadModule cgi_module libexec/apache2/mod_cgi.so
+LoadModule dav_fs_module libexec/apache2/mod_dav_fs.so
+LoadModule vhost_alias_module libexec/apache2/mod_vhost_alias.so
+LoadModule negotiation_module libexec/apache2/mod_negotiation.so
+LoadModule dir_module libexec/apache2/mod_dir.so
+LoadModule imagemap_module libexec/apache2/mod_imagemap.so
+LoadModule actions_module libexec/apache2/mod_actions.so
+LoadModule speling_module libexec/apache2/mod_speling.so
+LoadModule userdir_module libexec/apache2/mod_userdir.so
+LoadModule alias_module libexec/apache2/mod_alias.so
+LoadModule rewrite_module libexec/apache2/mod_rewrite.so
+#LoadModule perl_module libexec/apache2/mod_perl.so
+#LoadModule php5_module libexec/apache2/libphp5.so
+#LoadModule hfs_apple_module libexec/apache2/mod_hfs_apple.so
+
+<IfModule !mpm_netware_module>
+<IfModule !mpm_winnt_module>
+#
+# If you wish httpd to run as a different user or group, you must run
+# httpd as root initially and it will switch.  
+#
+# User/Group: The name (or #number) of the user/group to run httpd as.
+# It is usually good practice to create a dedicated user and group for
+# running httpd, as with most system services.
+#
+User _www
+Group _www
+
+</IfModule>
+</IfModule>
+
+# 'Main' server configuration
+#
+# The directives in this section set up the values used by the 'main'
+# server, which responds to any requests that aren't handled by a
+# <VirtualHost> definition.  These values also provide defaults for
+# any <VirtualHost> containers you may define later in the file.
+#
+# All of these directives may appear inside <VirtualHost> containers,
+# in which case these default settings will be overridden for the
+# virtual host being defined.
+#
+
+#
+# ServerAdmin: Your address, where problems with the server should be
+# e-mailed.  This address appears on some server-generated pages, such
+# as error documents.  e.g. admin@your-domain.com
+#
+ServerAdmin you@example.com
+
+#
+# ServerName gives the name and port that the server uses to identify itself.
+# This can often be determined automatically, but we recommend you specify
+# it explicitly to prevent problems during startup.
+#
+# If your host doesn't have a registered DNS name, enter its IP address here.
+#
+#ServerName www.example.com:80
+
+#
+# DocumentRoot: The directory out of which you will serve your
+# documents. By default, all requests are taken from this directory, but
+# symbolic links and aliases may be used to point to other locations.
+#
+DocumentRoot "/Library/WebServer/Documents"
+
+#
+# Each directory to which Apache has access can be configured with respect
+# to which services and features are allowed and/or disabled in that
+# directory (and its subdirectories). 
+#
+# First, we configure the "default" to be a very restrictive set of 
+# features.  
+#
+<Directory />
+    Options FollowSymLinks
+    AllowOverride None
+    Order deny,allow
+    Deny from all
+</Directory>
+
+#
+# Note that from this point forward you must specifically allow
+# particular features to be enabled - so if something's not working as
+# you might expect, make sure that you have specifically enabled it
+# below.
+#
+
+#
+# This should be changed to whatever you set DocumentRoot to.
+#
+<Directory "/Library/WebServer/Documents">
+    #
+    # Possible values for the Options directive are "None", "All",
+    # or any combination of:
+    #   Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews
+    #
+    # Note that "MultiViews" must be named *explicitly* --- "Options All"
+    # doesn't give it to you.
+    #
+    # The Options directive is both complicated and important.  Please see
+    # http://httpd.apache.org/docs/2.2/mod/core.html#options
+    # for more information.
+    #
+    Options Indexes FollowSymLinks MultiViews
+
+    #
+    # AllowOverride controls what directives may be placed in .htaccess files.
+    # It can be "All", "None", or any combination of the keywords:
+    #   Options FileInfo AuthConfig Limit
+    #
+    AllowOverride None
+
+    #
+    # Controls who can get stuff from this server.
+    #
+    Order allow,deny
+    Allow from all
+
+</Directory>
+
+#
+# DirectoryIndex: sets the file that Apache will serve if a directory
+# is requested.
+#
+<IfModule dir_module>
+    DirectoryIndex index.html
+</IfModule>
+
+#
+# The following lines prevent .htaccess and .htpasswd files from being 
+# viewed by Web clients. 
+#
+<FilesMatch "^\.([Hh][Tt]|[Dd][Ss]_[Ss])">
+    Order allow,deny
+    Deny from all
+    Satisfy All
+</FilesMatch>
+
+#
+# Apple specific filesystem protection.
+#
+<Files "rsrc">
+    Order allow,deny
+    Deny from all
+    Satisfy All
+</Files>
+<DirectoryMatch ".*\.\.namedfork">
+    Order allow,deny
+    Deny from all
+    Satisfy All
+</DirectoryMatch>
+
+#
+# ErrorLog: The location of the error log file.
+# If you do not specify an ErrorLog directive within a <VirtualHost>
+# container, error messages relating to that virtual host will be
+# logged here.  If you *do* define an error logfile for a <VirtualHost>
+# container, that host's errors will be logged there and not here.
+#
+ErrorLog "/private/var/log/apache2/error_log"
+
+#
+# LogLevel: Control the number of messages logged to the error_log.
+# Possible values include: debug, info, notice, warn, error, crit,
+# alert, emerg.
+#
+LogLevel warn
+
+<IfModule log_config_module>
+    #
+    # The following directives define some format nicknames for use with
+    # a CustomLog directive (see below).
+    #
+    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
+    LogFormat "%h %l %u %t \"%r\" %>s %b" common
+
+    <IfModule logio_module>
+      # You need to enable mod_logio.c to use %I and %O
+      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
+    </IfModule>
+
+    #
+    # The location and format of the access logfile (Common Logfile Format).
+    # If you do not define any access logfiles within a <VirtualHost>
+    # container, they will be logged here.  Contrariwise, if you *do*
+    # define per-<VirtualHost> access logfiles, transactions will be
+    # logged therein and *not* in this file.
+    #
+    CustomLog "/private/var/log/apache2/access_log" common
+
+    #
+    # If you prefer a logfile with access, agent, and referer information
+    # (Combined Logfile Format) you can use the following directive.
+    #
+    #CustomLog "/private/var/log/apache2/access_log" combined
+</IfModule>
+
+<IfModule alias_module>
+    #
+    # Redirect: Allows you to tell clients about documents that used to 
+    # exist in your server's namespace, but do not anymore. The client 
+    # will make a new request for the document at its new location.
+    # Example:
+    # Redirect permanent /foo http://www.example.com/bar
+
+    #
+    # Alias: Maps web paths into filesystem paths and is used to
+    # access content that does not live under the DocumentRoot.
+    # Example:
+    # Alias /webpath /full/filesystem/path
+    #
+    # If you include a trailing / on /webpath then the server will
+    # require it to be present in the URL.  You will also likely
+    # need to provide a <Directory> section to allow access to
+    # the filesystem path.
+
+    #
+    # ScriptAlias: This controls which directories contain server scripts. 
+    # ScriptAliases are essentially the same as Aliases, except that
+    # documents in the target directory are treated as applications and
+    # run by the server when requested rather than as documents sent to the
+    # client.  The same rules about trailing "/" apply to ScriptAlias
+    # directives as to Alias.
+    #
+    ScriptAliasMatch ^/cgi-bin/((?!(?i:webobjects)).*$) "/Library/WebServer/CGI-Executables/$1"
+
+</IfModule>
+
+<IfModule cgid_module>
+    #
+    # ScriptSock: On threaded servers, designate the path to the UNIX
+    # socket used to communicate with the CGI daemon of mod_cgid.
+    #
+    #Scriptsock /private/var/run/cgisock
+</IfModule>
+
+#
+# "/Library/WebServer/CGI-Executables" should be changed to whatever your ScriptAliased
+# CGI directory exists, if you have that configured.
+#
+<Directory "/Library/WebServer/CGI-Executables">
+    AllowOverride None
+    Options None
+    Order allow,deny
+    Allow from all
+</Directory>
+
+#
+# DefaultType: the default MIME type the server will use for a document
+# if it cannot otherwise determine one, such as from filename extensions.
+# If your server contains mostly text or HTML documents, "text/plain" is
+# a good value.  If most of your content is binary, such as applications
+# or images, you may want to use "application/octet-stream" instead to
+# keep browsers from trying to display binary files as though they are
+# text.
+#
+DefaultType text/plain
+
+<IfModule mime_module>
+    #
+    # TypesConfig points to the file containing the list of mappings from
+    # filename extension to MIME-type.
+    #
+    TypesConfig /private/etc/apache2/mime.types
+
+    #
+    # AddType allows you to add to or override the MIME configuration
+    # file specified in TypesConfig for specific file types.
+    #
+    #AddType application/x-gzip .tgz
+    #
+    # AddEncoding allows you to have certain browsers uncompress
+    # information on the fly. Note: Not all browsers support this.
+    #
+    #AddEncoding x-compress .Z
+    #AddEncoding x-gzip .gz .tgz
+    #
+    # If the AddEncoding directives above are commented-out, then you
+    # probably should define those extensions to indicate media types:
+    #
+    AddType application/x-compress .Z
+    AddType application/x-gzip .gz .tgz
+
+    #
+    # AddHandler allows you to map certain file extensions to "handlers":
+    # actions unrelated to filetype. These can be either built into the server
+    # or added with the Action directive (see below)
+    #
+    # To use CGI scripts outside of ScriptAliased directories:
+    # (You will also need to add "ExecCGI" to the "Options" directive.)
+    #
+    #AddHandler cgi-script .cgi
+
+    # For type maps (negotiated resources):
+    #AddHandler type-map var
+
+    #
+    # Filters allow you to process content before it is sent to the client.
+    #
+    # To parse .shtml files for server-side includes (SSI):
+    # (You will also need to add "Includes" to the "Options" directive.)
+    #
+    #AddType text/html .shtml
+    #AddOutputFilter INCLUDES .shtml
+</IfModule>
+
+#
+# The mod_mime_magic module allows the server to use various hints from the
+# contents of the file itself to determine its type.  The MIMEMagicFile
+# directive tells the module where the hint definitions are located.
+#
+#MIMEMagicFile /private/etc/apache2/magic
+
+#
+# Customizable error responses come in three flavors:
+# 1) plain text 2) local redirects 3) external redirects
+#
+# Some examples:
+#ErrorDocument 500 "The server made a boo boo."
+#ErrorDocument 404 /missing.html
+#ErrorDocument 404 "/cgi-bin/missing_handler.pl"
+#ErrorDocument 402 http://www.example.com/subscription_info.html
+#
+
+#
+# MaxRanges: Maximum number of Ranges in a request before
+# returning the entire resource, or one of the special
+# values 'default', 'none' or 'unlimited'.
+# Default setting is to accept 200 Ranges.
+#MaxRanges unlimited
+
+#
+# EnableMMAP and EnableSendfile: On systems that support it, 
+# memory-mapping or the sendfile syscall is used to deliver
+# files.  This usually improves server performance, but must
+# be turned off when serving from networked-mounted 
+# filesystems or if support for these functions is otherwise
+# broken on your system.
+#
+#EnableMMAP off
+#EnableSendfile off
+
+# 6894961
+TraceEnable off
+
+# Supplemental configuration
+#
+# The configuration files in the /private/etc/apache2/extra/ directory can be 
+# included to add extra features or to modify the default configuration of 
+# the server, or you may simply copy their contents here and change as 
+# necessary.
+
+# Server-pool management (MPM specific)
+Include /private/etc/apache2/extra/httpd-mpm.conf
+
+# Multi-language error messages
+#Include /private/etc/apache2/extra/httpd-multilang-errordoc.conf
+
+# Fancy directory listings
+Include /private/etc/apache2/extra/httpd-autoindex.conf
+
+# Language settings
+Include /private/etc/apache2/extra/httpd-languages.conf
+
+# User home directories
+Include /private/etc/apache2/extra/httpd-userdir.conf
+
+# Real-time info on requests and configuration
+#Include /private/etc/apache2/extra/httpd-info.conf
+
+# Virtual hosts
+#Include /private/etc/apache2/extra/httpd-vhosts.conf
+
+# Local access to the Apache HTTP Server Manual
+Include /private/etc/apache2/extra/httpd-manual.conf
+
+# Distributed authoring and versioning (WebDAV)
+#Include /private/etc/apache2/extra/httpd-dav.conf
+
+# Various default settings
+#Include /private/etc/apache2/extra/httpd-default.conf
+
+# Secure (SSL/TLS) connections
+#Include /private/etc/apache2/extra/httpd-ssl.conf
+#
+# Note: The following must must be present to support
+#       starting without SSL on platforms with no /dev/random equivalent
+#       but a statically compiled-in mod_ssl.
+#
+<IfModule ssl_module>
+SSLRandomSeed startup builtin
+SSLRandomSeed connect builtin
+</IfModule>
+
+Include /private/etc/apache2/other/*.conf

samples/AppleScript/Convert To PDF.applescript

@@ -0,0 +1,87 @@
+(*
+Copyright 2003 Apple Computer, Inc.
+
+You may incorporate this Apple sample code into your program(s) without
+restriction.  This Apple sample code has been provided "AS IS" and the
+responsibility for its operation is yours.  You are not permitted to
+redistribute this Apple sample code as "Apple sample code" after having
+made changes.  If you're going to redistribute the code, we require
+that you make it clear that the code was descended from Apple sample
+code, but that you've made changes.
+*)
+
+property type_list : {"JPEG", "GIFf", "PICT", "TIFF", "PDF", "TEXT"}
+property extension_list : {"jpg", "gif", "pct", "tif", "pdf", "rtf"}
+--html is not currently handled
+
+on run {}
+	tell application "Finder" to set FinderSelection to the selection as alias list
+	
+	set FS to FinderSelection
+	--Ideally, this list could be passed to the open handler
+	
+	set SelectionCount to number of FS -- count	
+	if SelectionCount is 0 then
+		set FS to userPicksFolder()
+	else if the SelectionCount is 1 then
+		set MyPath to path to me
+		if MyPath is item 1 of FS then
+			--If I'm a droplet then I was double-clicked
+			set FS to userPicksFolder()
+		end if
+	else
+		--I'm not a double-clicked droplet
+	end if
+	open FS
+end run
+
+on userPicksFolder()
+	set these_items to {}
+	set these_items to (choose file with prompt "Select a file to convert to PDF:" of type {"JPEG", "GIFf", "PICT", "TIFF", "TEXT", "RTF"}) as list
+end userPicksFolder
+
+on open these_items
+	set thesefiles to {}
+	set the item_info to {}
+	repeat with i from 1 to the count of these_items
+		set this_item to (item i of these_items)
+		set the item_info to info for this_item
+		
+		if folder of the item_info is true then --if the item is a folder
+			processFolder(this_item)
+		else if ((folder of the item_info is false) and (alias of the item_info is false)) and (the file type of the item_info is in the type_list) or ((the name extension of the item_info) is in the extension_list) then
+			
+			set theFilePath to (item i of these_items as string)
+			set thePOSIXFilePath to POSIX path of theFilePath as string
+			processFile(thePOSIXFilePath)
+		end if
+	end repeat
+end open
+
+--process folders 
+on processFolder(theFolder)
+	set these_items to list folder theFolder without invisibles
+	repeat with i from 1 to the count of these_items
+		set this_item to alias ((theFolder as text) & (item i of these_items))
+		set the item_info to info for this_item
+		if folder of the item_info is true then
+			processFolder(this_item)
+		else if (alias of the item_info is false) and ((the file type of the item_info is in the type_list) or the name extension of the item_info is in the extension_list) then
+			set theFilePath to (this_item as string)
+			set thePOSIXFilePath to POSIX path of theFilePath as string
+			processFile(thePOSIXFilePath)
+		end if
+	end repeat
+end processFolder
+
+on processFile(thePOSIXFileName)
+	try
+		set terminalCommand to ""
+		set convertCommand to "/System/Library/Printers/Libraries/./convert "
+		set newFileName to thePOSIXFileName & ".pdf"
+		set terminalCommand to convertCommand & "-f " & "\"" & thePOSIXFileName & "\"" & " -o " & "\"" & newFileName & "\"" & " -j \"application/pdf\""
+		
+		do shell script terminalCommand
+	end try
+end processFile
+

samples/AppleScript/Convert To PostScript.applescript

@@ -0,0 +1,91 @@
+(*
+
+Copyright 2003 Apple Computer, Inc.
+
+You may incorporate this Apple sample code into your program(s) without
+restriction.  This Apple sample code has been provided "AS IS" and the
+responsibility for its operation is yours.  You are not permitted to
+redistribute this Apple sample code as "Apple sample code" after having
+made changes.  If you're going to redistribute the code, we require
+that you make it clear that the code was descended from Apple sample
+code, but that you've made changes.
+*)
+
+property type_list : {"JPEG", "GIFf", "PICT", "TIFF", "PDF", "TEXT"}
+property extension_list : {"jpg", "gif", "pct", "tif", "pdf", "rtf"}
+--html is not currently handled
+
+on run {}
+	tell application "Finder" to set FinderSelection to the selection as alias list
+	
+	set FS to FinderSelection
+	--Ideally, this list could be passed to the open handler
+	
+	set SelectionCount to number of FS -- count	
+	if SelectionCount is 0 then
+		set FS to userPicksFolder()
+	else if the SelectionCount is 1 then
+		set MyPath to path to me
+		if MyPath is item 1 of FS then
+			--If I'm a droplet then I was double-clicked
+			set FS to userPicksFolder()
+		end if
+	else
+		--I'm not a double-clicked droplet
+	end if
+	open FS
+end run
+
+on userPicksFolder()
+	set these_items to {}
+	set these_items to (choose file with prompt "Select a file to convert to PostScript:" of type {"JPEG", "GIFf", "PICT", "TIFF", "TEXT", "RTF"}) as list
+end userPicksFolder
+
+
+on open these_items
+	set thesefiles to {}
+	set the item_info to {}
+	repeat with i from 1 to the count of these_items
+		set this_item to (item i of these_items)
+		set the item_info to info for this_item
+		
+		if folder of the item_info is true then --if the item is a folder
+			processFolder(this_item)
+		else if ((folder of the item_info is false) and (alias of the item_info is false)) and (the file type of the item_info is in the type_list) or ((the name extension of the item_info) is in the extension_list) then
+			
+			set theFilePath to (item i of these_items as string)
+			set thePOSIXFilePath to POSIX path of theFilePath as string
+			processFile(thePOSIXFilePath)
+		end if
+	end repeat
+end open
+
+--process folders 
+on processFolder(theFolder)
+	set these_items to list folder theFolder without invisibles
+	repeat with i from 1 to the count of these_items
+		set this_item to alias ((theFolder as text) & (item i of these_items))
+		set the item_info to info for this_item
+		if folder of the item_info is true then
+			processFolder(this_item)
+		else if (alias of the item_info is false) and ((the file type of the item_info is in the type_list) or the name extension of the item_info is in the extension_list) then
+			set theFilePath to (this_item as string)
+			set thePOSIXFilePath to POSIX path of theFilePath as string
+			processFile(thePOSIXFilePath)
+		end if
+	end repeat
+end processFolder
+
+--need to pass the URL to Terminal
+on processFile(thePOSIXFileName)
+	try
+		set terminalCommand to ""
+		set convertCommand to "/System/Library/Printers/Libraries/./convert "
+		set newFileName to thePOSIXFileName & ".ps"
+		set terminalCommand to convertCommand & "-f " & "\"" & thePOSIXFileName & "\"" & " -o " & "\"" & newFileName & "\"" & " -j \"application/postscript\""
+		
+		do shell script terminalCommand
+	end try
+end processFile
+
+

samples/AppleScript/Count Messages in All Mailboxes.applescript

@@ -0,0 +1,80 @@
+(*
+Count Messages in All Mailboxes
+
+Copyright 2002-2012 Apple Inc. All rights reserved.
+
+You may incorporate this Apple sample code into your program(s) without
+restriction.  This Apple sample code has been provided "AS IS" and the
+responsibility for its operation is yours.  You are not permitted to
+redistribute this Apple sample code as "Apple sample code" after having
+made changes.  If you're going to redistribute the code, we require
+that you make it clear that the code was descended from Apple sample
+code, but that you've made changes.
+*)
+
+(*
+This script goes through each mailbox, gets the total message count and
+the unread count, then displays the final output in a new email message.
+*)
+
+tell application "Mail"
+	set localMailboxes to every mailbox
+	if (count of localMailboxes) is greater than 0 then
+		set messageCountDisplay to "Local mailboxes (On My Mac)" & return & my getMessageCountsForMailboxes(localMailboxes)
+	else
+		set messageCountDisplay to ""
+	end if
+	
+	set everyAccount to every account
+	repeat with eachAccount in everyAccount
+		set accountMailboxes to every mailbox of eachAccount
+		if (count of accountMailboxes) is greater than 0 then
+			set messageCountDisplay to messageCountDisplay & return & "Mailboxes for Account: " & name of eachAccount & return & my getMessageCountsForMailboxes(accountMailboxes)
+		end if
+	end repeat
+	
+	set outputMessage to make new outgoing message with properties {content:messageCountDisplay, subject:"Message counts for all my mailboxes", visible:true}
+	tell outputMessage
+		set font to "Courier"
+		set size to 12
+	end tell
+end tell
+
+on getMessageCountsForMailboxes(theMailboxes)
+	-- (list of mailboxes)
+	-- returns string
+	
+	set displayString to ""
+	
+	tell application "Mail"
+		repeat with eachMailbox in theMailboxes
+			set mailboxName to name of eachMailbox
+			set messageCount to (count of (messages of eachMailbox)) as string
+			set unreadCount to unread count of eachMailbox as string
+			
+			set displayString to displayString & "    " & my padString(mailboxName, 40) & "  " & messageCount & " (" & unreadCount & " unread)" & return
+		end repeat
+	end tell
+	
+	return displayString
+end getMessageCountsForMailboxes
+
+on padString(theString, fieldLength)
+	-- (string, integer)
+	-- returns string
+	
+	set stringLength to length of theString
+	
+	if stringLength is greater than fieldLength then
+		set paddedString to (text from character 1 to character (fieldLength - 3) of theString) & "..."
+	else -- stringLength is less than or equal to fieldLength
+		set paddedString to theString
+		
+		set paddingLength to fieldLength - stringLength
+		repeat paddingLength times
+			set paddedString to paddedString & space
+		end repeat
+	end if
+	
+	return paddedString
+end padString

samples/AppleScript/Crazy Message Text.applescript

@@ -0,0 +1,68 @@
+(*
+Crazy Message Text
+
+Copyright 2002-2012 Apple Inc. All rights reserved.
+
+You may incorporate this Apple sample code into your program(s) without
+restriction.  This Apple sample code has been provided "AS IS" and the
+responsibility for its operation is yours.  You are not permitted to
+redistribute this Apple sample code as "Apple sample code" after having
+made changes.  If you're going to redistribute the code, we require
+that you make it clear that the code was descended from Apple sample
+code, but that you've made changes.
+*)
+
+(*
+This script takes a string from the user and then makes a new message
+where each letter has a different font, size, and color.
+*)
+
+property lowFontSize : 36
+property highFontSize : 72
+property messageText : "Happy Birthday!"
+
+repeat
+	set userInput to display dialog "Enter some message text:" & return & return & "Minimum Character Size: " & (lowFontSize as string) & return & "Maximum Character Size: " & (highFontSize as string) default answer messageText buttons {"Cancel", "Set Prefs", "Continue"} default button 3
+	
+	if the button returned of userInput is "Set Prefs" then
+		set minimumFontSize to 9
+		
+		display dialog "Enter the minimum font size to use:" & return & return & "(Must be at least " & (minimumFontSize as string) & ")" default answer lowFontSize buttons {"OK"}
+		set newFontSize to text returned of the result as integer
+		if newFontSize is greater than or equal to minimumFontSize then
+			set lowFontSize to newFontSize
+		else
+			set lowFontSize to minimumFontSize
+		end if
+		
+		display dialog "Enter the maximum font size to use:" & return & return & "(Must be greater than " & (lowFontSize as string) & ")" default answer highFontSize buttons {"OK"}
+		set newFontSize to text returned of the result as integer
+		if newFontSize is greater than lowFontSize then
+			set highFontSize to newFontSize
+		else
+			set highFontSize to lowFontSize
+		end if
+		
+	else -- button returned of userInput is "Continue"
+		set theText to text returned of userInput
+		if theText is not "" then
+			set messageText to theText
+		end if
+		exit repeat
+	end if
+end repeat
+
+set fontList to {"American Typewriter", "American Typewriter Light", "American Typewriter Bold", "American Typewriter Condensed", "American Typewriter Condensed Light", "American Typewriter Condensed Bold", "Arial", "Arial Italic", "Arial Bold", "Arial Bold Italic", "Arial Black", "Baskerville", "Baskerville Italic", "Baskerville SemiBold", "Baskerville Bold", "Baskerville SemiBold Italic", "Baskerville Bold Italic", "Big Caslon Medium", "Comic Sans MS", "Comic Sans MS Bold", "Copperplate", "Copperplate Light", "Copperplate Bold", "Didot", "Didot Italic", "Didot Bold", "Futura Medium", "Futura Medium Italic", "Futura Condensed Medium", "Futura Condensed ExtraBold", "Geneva", "Gill Sans", "Gill Sans Italic", "Gill Sans Light", "Gill Sans Light Italic", "Gill Sans Bold", "Gill Sans Bold Italic", "Herculanum", "Lucida Grande", "Lucida Grande Bold", "Marker Felt Thin", "Marker Felt Wide", "Optima Regular", "Optima Italic", "Optima Bold", "Optima Bold Italic", "Optima ExtraBlack", "Papyrus", "Verdana", "Verdana Italic", "Verdana Bold", "Verdana Bold Italic", "Zapfino"}
+
+tell application "Mail"
+	activate
+	set crazyTextMessage to make new outgoing message with properties {content:messageText, visible:true}
+	
+	tell crazyTextMessage
+		repeat with eachCharacter in characters
+			set font of eachCharacter to (some item of fontList)
+			set size of eachCharacter to (random number from lowFontSize to highFontSize)
+			set color of eachCharacter to {random number from 0 to 65535, random number from 0 to 65535, random number from 0 to 65535}
+		end repeat
+	end tell
+end tell

samples/AppleScript/Get User Name.applescript

@@ -0,0 +1,41 @@
+(*
+Get User Name
+
+This script uses UI element scripting to get the name for the
+current user.
+
+If "Enable access for assistive devices" is not checked,
+this script will open the Universal Access System Preference and ask
+the user to check the checkbox.
+
+Copyright 2007 Apple Inc.
+
+You may incorporate this Apple sample code into your program(s) without
+restriction.  This Apple sample code has been provided "AS IS" and the
+responsibility for its operation is yours.  You are not permitted to
+redistribute this Apple sample code as "Apple sample code" after having
+made changes.  If you're going to redistribute the code, we require
+that you make it clear that the code was descended from Apple sample
+code, but that you've made changes.
+*)
+
+tell application "System Preferences"
+	activate
+	set current pane to pane "com.apple.preferences.users"
+end tell
+
+tell application "System Events"
+	if UI elements enabled then
+		tell tab group 1 of window "Accounts" of process "System Preferences"
+			click radio button 1
+			delay 2
+			get value of text field 1
+		end tell
+	else
+		tell application "System Preferences"
+			activate
+			set current pane to pane "com.apple.preference.universalaccess"
+			display dialog "UI element scripting is not enabled. Check \"Enable access for assistive devices\""
+		end tell
+	end if
+end tell

samples/AppleScript/Time Of Day.applescript

@@ -0,0 +1,75 @@
+(* 
+ Speaks the  date and time of day
+ 
+ Copyright 2008 Apple Inc. All rights reserved.
+ 
+ You may incorporate this Apple sample code into your program(s) without
+ restriction.  This Apple sample code has been provided "AS IS" and the
+ responsibility for its operation is yours.  You are not permitted to
+ redistribute this Apple sample code as "Apple sample code" after having
+ made changes.  If you're going to redistribute the code, we require
+ that you make it clear that the code was descended from Apple sample
+ code, but that you've made changes.
+ *)
+
+on isVoiceOverRunning()
+	set isRunning to false
+	tell application "System Events"
+		set isRunning to (name of processes) contains "VoiceOver"
+	end tell
+	return isRunning
+end isVoiceOverRunning
+
+on isVoiceOverRunningWithAppleScript()
+	if isVoiceOverRunning() then
+		set isRunningWithAppleScript to true
+		
+		-- is AppleScript enabled on VoiceOver --
+		tell application "VoiceOver"
+			try
+				set x to bounds of vo cursor
+			on error
+				set isRunningWithAppleScript to false
+			end try
+		end tell
+		return isRunningWithAppleScript
+	end if
+	return false
+end isVoiceOverRunningWithAppleScript
+
+set currentDate to current date
+set amPM to "AM"
+set currentHour to (currentDate's hours)
+set currentMinutes to currentDate's minutes
+
+if (currentHour > 12 and currentHour < 24) then
+	set amPM to "PM"
+else
+	set amPM to "AM"
+end if
+
+--  make minutes below 10 sound nice
+if currentMinutes < 10 then
+	set currentMinutes to ("0" & currentMinutes) as text
+end if
+
+--  ensure 0:nn gets set to 12:nn AM
+if currentHour is equal to 0 then
+	set currentHour to 12
+end if
+
+--  readjust for 12 hour time
+if (currentHour > 12) then
+	set currentHour to (currentHour - 12)
+end if
+
+set currentTime to ((currentDate's month) as text) & " " & ((currentDate's day) as text) & ", " & (currentHour as text) & ":" & ((currentMinutes) as text) & " " & amPM as text
+
+if isVoiceOverRunningWithAppleScript() then
+	tell application "VoiceOver"
+		output currentTime
+	end tell
+else
+	say currentTime
+	delay 2
+end if

samples/AsciiDoc/encoding.asciidoc

@@ -0,0 +1,13 @@
+Gregory Romé has written an AsciiDoc plugin for the Redmine project management application.
+
+https://github.com/foo-users/foo
+へと `vicmd` キーマップを足してみている試み、
+アニメーションgifです。
+
+tag::romé[]
+Gregory Romé has written an AsciiDoc plugin for the Redmine project management application.
+end::romé[]
+
+== Überschrift
+
+* Codierungen sind verrückt auf älteren Versionen von Ruby

samples/AsciiDoc/list.asc

@@ -0,0 +1,10 @@
+AsciiDoc Home Page
+==================
+
+Example Articles
+~~~~~~~~~~~~~~~~
+- Item 1
+
+- Item 2
+
+- Item 3

samples/AsciiDoc/sample.adoc

@@ -0,0 +1,25 @@
+Document Title
+==============
+Doc Writer <thedoc@asciidoctor.org>
+:idprefix: id_
+
+Preamble paragraph.
+
+NOTE: This is test, only a test.
+
+== Section A
+
+*Section A* paragraph.
+
+=== Section A Subsection
+
+*Section A* 'subsection' paragraph.
+
+== Section B
+
+*Section B* paragraph.
+
+.Section B list
+* Item 1
+* Item 2
+* Item 3

samples/AspectJ/CacheAspect.aj

@@ -0,0 +1,41 @@
+package com.blogspot.miguelinlas3.aspectj.cache;
+
+import java.util.Map;
+import java.util.WeakHashMap;
+
+import org.aspectj.lang.JoinPoint;
+
+import com.blogspot.miguelinlas3.aspectj.cache.marker.Cachable;
+
+/**
+ * This simple aspect simulates the behaviour of a very simple cache
+ *  
+ * @author migue
+ *
+ */
+public aspect CacheAspect {
+
+	public pointcut cache(Cachable cachable): execution(@Cachable * * (..)) && @annotation(cachable);
+	
+	Object around(Cachable cachable): cache(cachable){
+	
+		String evaluatedKey = this.evaluateKey(cachable.scriptKey(), thisJoinPoint);
+		
+		if(cache.containsKey(evaluatedKey)){
+			System.out.println("Cache hit for key " + evaluatedKey);
+			return this.cache.get(evaluatedKey);
+		}
+		
+		System.out.println("Cache miss for key " + evaluatedKey);
+		Object value = proceed(cachable);
+		cache.put(evaluatedKey, value);
+		return value;
+	}
+	
+	protected String evaluateKey(String key, JoinPoint joinPoint) {
+		// TODO add some smart staff to allow simple scripting in @Cachable annotation
+		return key;
+	}
+	
+	protected Map<String, Object> cache = new WeakHashMap<String, Object>();
+}

samples/AspectJ/OptimizeRecursionCache.aj

@@ -0,0 +1,50 @@
+package aspects.caching;
+
+import java.util.Map;
+
+/**
+ * Cache aspect for optimize recursive functions.
+ * 
+ * @author Migueli
+ * @date 05/11/2013
+ * @version 1.0
+ *
+ */
+public abstract aspect OptimizeRecursionCache {
+		
+	@SuppressWarnings("rawtypes")
+	private Map _cache;
+	
+	public OptimizeRecursionCache() {
+		_cache = getCache();
+	}
+	
+	@SuppressWarnings("rawtypes")
+	abstract public Map getCache();
+	
+	abstract public pointcut operation(Object o);
+
+	pointcut topLevelOperation(Object o): operation(o) && !cflowbelow(operation(Object));
+
+	before(Object o) : topLevelOperation(o) {
+		System.out.println("Seeking value for " + o);
+	}
+
+	Object around(Object o) : operation(o) {
+		Object cachedValue = _cache.get(o);
+		if (cachedValue != null) {
+			System.out.println("Found cached value for " + o + ": " + cachedValue);
+			return cachedValue;
+		}
+		return proceed(o);
+	}
+
+	@SuppressWarnings("unchecked")
+	after(Object o) returning(Object result) : topLevelOperation(o) {
+		_cache.put(o, result);
+	}
+	
+	after(Object o) returning(Object result) : topLevelOperation(o) {
+		System.out.println("cache size: " + _cache.size());
+	}
+}

samples/Assembly/External Interrupt.a51

@@ -0,0 +1,66 @@
+	ORG	0000h
+	SJMP	START
+	ORG	0003h
+	LCALL	INT0_ISR
+	RETI
+	ORG	000Bh
+	LCALL	T0_ISR
+	RETI
+	ORG	0013h
+	LCALL	INT1_ISR
+	RETI
+	ORG	001Bh
+	LCALL	T1_ISR
+	RETI
+	ORG	0023h
+	LCALL	UART_ISR
+	RETI
+	ORG	0030h
+START:
+	MOV	A,#11111110b
+	SETB	IT0	; Set External Interrupt 0 to be falling edge triggered
+	SETB	EX0	; Enable External Interrut 0
+	SETB	EA	; Enable Interrupt
+LEFT:			
+	CJNE	A,#01111111b,LOOP1
+	JMP	RIGHT
+LOOP1:
+	MOV	P1,A
+	RL	A	
+	LCALL	DELAY
+	SJMP	LEFT	
+RIGHT:
+	CJNE	A,#11111110b,LOOP2
+	JMP	LEFT
+LOOP2:
+	MOV	P1,A
+	RR	A	
+	LCALL	DELAY
+	SJMP	RIGHT
+	
+INT0_ISR:
+	MOV	R1,#3
+FLASH:
+	MOV	P1,#00h
+	LCALL	DELAY
+	MOV	P1,#0FFh
+	LCALL	DELAY
+	DJNZ	R1,FLASH
+	RET
+T0_ISR:
+	RET
+INT1_ISR:
+	RET
+T1_ISR:
+	RET
+UART_ISR:
+	RET
+
+DELAY:	MOV	R5,#20	;R5*20 mS
+D1:     MOV	R6,#40
+D2:     MOV	R7,#249
+	DJNZ	R7,$
+ 	DJNZ	R6,D2
+  	DJNZ	R5,D1
+   	RET
+	END

samples/Assembly/FASM.asm

@@ -0,0 +1,350 @@
+
+; flat assembler interface for Win32
+; Copyright (c) 1999-2014, Tomasz Grysztar.
+; All rights reserved.
+
+	format	PE console
+
+section '.text' code readable executable
+
+start:
+
+	mov	[con_handle],STD_OUTPUT_HANDLE
+	mov	esi,_logo
+	call	display_string
+
+	call	get_params
+	jc	information
+
+	call	init_memory
+
+	mov	esi,_memory_prefix
+	call	display_string
+	mov	eax,[memory_end]
+	sub	eax,[memory_start]
+	add	eax,[additional_memory_end]
+	sub	eax,[additional_memory]
+	shr	eax,10
+	call	display_number
+	mov	esi,_memory_suffix
+	call	display_string
+
+	call	[GetTickCount]
+	mov	[start_time],eax
+
+	call	preprocessor
+	call	parser
+	call	assembler
+	call	formatter
+
+	call	display_user_messages
+	movzx	eax,[current_pass]
+	inc	eax
+	call	display_number
+	mov	esi,_passes_suffix
+	call	display_string
+	call	[GetTickCount]
+	sub	eax,[start_time]
+	xor	edx,edx
+	mov	ebx,100
+	div	ebx
+	or	eax,eax
+	jz	display_bytes_count
+	xor	edx,edx
+	mov	ebx,10
+	div	ebx
+	push	edx
+	call	display_number
+	mov	dl,'.'
+	call	display_character
+	pop	eax
+	call	display_number
+	mov	esi,_seconds_suffix
+	call	display_string
+      display_bytes_count:
+	mov	eax,[written_size]
+	call	display_number
+	mov	esi,_bytes_suffix
+	call	display_string
+	xor	al,al
+	jmp	exit_program
+
+information:
+	mov	esi,_usage
+	call	display_string
+	mov	al,1
+	jmp	exit_program
+
+get_params:
+	mov	[input_file],0
+	mov	[output_file],0
+	mov	[symbols_file],0
+	mov	[memory_setting],0
+	mov	[passes_limit],100
+	call	[GetCommandLine]
+	mov	esi,eax
+	mov	edi,params
+    find_command_start:
+	lodsb
+	cmp	al,20h
+	je	find_command_start
+	cmp	al,22h
+	je	skip_quoted_name
+    skip_name:
+	lodsb
+	cmp	al,20h
+	je	find_param
+	or	al,al
+	jz	all_params
+	jmp	skip_name
+    skip_quoted_name:
+	lodsb
+	cmp	al,22h
+	je	find_param
+	or	al,al
+	jz	all_params
+	jmp	skip_quoted_name
+    find_param:
+	lodsb
+	cmp	al,20h
+	je	find_param
+	cmp	al,'-'
+	je	option_param
+	cmp	al,0Dh
+	je	all_params
+	or	al,al
+	jz	all_params
+	cmp	[input_file],0
+	jne	get_output_file
+	mov	[input_file],edi
+	jmp	process_param
+      get_output_file:
+	cmp	[output_file],0
+	jne	bad_params
+	mov	[output_file],edi
+    process_param:
+	cmp	al,22h
+	je	string_param
+    copy_param:
+	stosb
+	lodsb
+	cmp	al,20h
+	je	param_end
+	cmp	al,0Dh
+	je	param_end
+	or	al,al
+	jz	param_end
+	jmp	copy_param
+    string_param:
+	lodsb
+	cmp	al,22h
+	je	string_param_end
+	cmp	al,0Dh
+	je	param_end
+	or	al,al
+	jz	param_end
+	stosb
+	jmp	string_param
+    option_param:
+	lodsb
+	cmp	al,'m'
+	je	memory_option
+	cmp	al,'M'
+	je	memory_option
+	cmp	al,'p'
+	je	passes_option
+	cmp	al,'P'
+	je	passes_option
+	cmp	al,'s'
+	je	symbols_option
+	cmp	al,'S'
+	je	symbols_option
+    bad_params:
+	stc
+	ret
+    get_option_value:
+	xor	eax,eax
+	mov	edx,eax
+    get_option_digit:
+	lodsb
+	cmp	al,20h
+	je	option_value_ok
+	cmp	al,0Dh
+	je	option_value_ok
+	or	al,al
+	jz	option_value_ok
+	sub	al,30h
+	jc	invalid_option_value
+	cmp	al,9
+	ja	invalid_option_value
+	imul	edx,10
+	jo	invalid_option_value
+	add	edx,eax
+	jc	invalid_option_value
+	jmp	get_option_digit
+    option_value_ok:
+	dec	esi
+	clc
+	ret
+    invalid_option_value:
+	stc
+	ret
+    memory_option:
+	lodsb
+	cmp	al,20h
+	je	memory_option
+	cmp	al,0Dh
+	je	bad_params
+	or	al,al
+	jz	bad_params
+	dec	esi
+	call	get_option_value
+	or	edx,edx
+	jz	bad_params
+	cmp	edx,1 shl (32-10)
+	jae	bad_params
+	mov	[memory_setting],edx
+	jmp	find_param
+    passes_option:
+	lodsb
+	cmp	al,20h
+	je	passes_option
+	cmp	al,0Dh
+	je	bad_params
+	or	al,al
+	jz	bad_params
+	dec	esi
+	call	get_option_value
+	or	edx,edx
+	jz	bad_params
+	cmp	edx,10000h
+	ja	bad_params
+	mov	[passes_limit],dx
+	jmp	find_param
+    symbols_option:
+	mov	[symbols_file],edi
+      find_symbols_file_name:
+	lodsb
+	cmp	al,20h
+	jne	process_param
+	jmp	find_symbols_file_name
+    param_end:
+	dec	esi
+    string_param_end:
+	xor	al,al
+	stosb
+	jmp	find_param
+    all_params:
+	cmp	[input_file],0
+	je	bad_params
+	clc
+	ret
+
+include 'system.inc'
+
+include '..\errors.inc'
+include '..\symbdump.inc'
+include '..\preproce.inc'
+include '..\parser.inc'
+include '..\exprpars.inc'
+include '..\assemble.inc'
+include '..\exprcalc.inc'
+include '..\formats.inc'
+include '..\x86_64.inc'
+include '..\avx.inc'
+
+include '..\tables.inc'
+include '..\messages.inc'
+
+section '.data' data readable writeable
+
+include '..\version.inc'
+
+_copyright db 'Copyright (c) 1999-2014, Tomasz Grysztar',0Dh,0Ah,0
+
+_logo db 'flat assembler  version ',VERSION_STRING,0
+_usage db 0Dh,0Ah
+       db 'usage: fasm <source> [output]',0Dh,0Ah
+       db 'optional settings:',0Dh,0Ah
+       db ' -m <limit>    set the limit in kilobytes for the available memory',0Dh,0Ah
+       db ' -p <limit>    set the maximum allowed number of passes',0Dh,0Ah
+       db ' -s <file>     dump symbolic information for debugging',0Dh,0Ah
+       db 0
+_memory_prefix db '  (',0
+_memory_suffix db ' kilobytes memory)',0Dh,0Ah,0
+_passes_suffix db ' passes, ',0
+_seconds_suffix db ' seconds, ',0
+_bytes_suffix db ' bytes.',0Dh,0Ah,0
+
+align 4
+
+include '..\variable.inc'
+
+con_handle dd ?
+memory_setting dd ?
+start_time dd ?
+bytes_count dd ?
+displayed_count dd ?
+character db ?
+last_displayed rb 2
+
+params rb 1000h
+options rb 1000h
+buffer rb 4000h
+
+stack 10000h
+
+section '.idata' import data readable writeable
+
+  dd 0,0,0,rva kernel_name,rva kernel_table
+  dd 0,0,0,0,0
+
+  kernel_table:
+    ExitProcess dd rva _ExitProcess
+    CreateFile dd rva _CreateFileA
+    ReadFile dd rva _ReadFile
+    WriteFile dd rva _WriteFile
+    CloseHandle dd rva _CloseHandle
+    SetFilePointer dd rva _SetFilePointer
+    GetCommandLine dd rva _GetCommandLineA
+    GetEnvironmentVariable dd rva _GetEnvironmentVariable
+    GetStdHandle dd rva _GetStdHandle
+    VirtualAlloc dd rva _VirtualAlloc
+    VirtualFree dd rva _VirtualFree
+    GetTickCount dd rva _GetTickCount
+    GetSystemTime dd rva _GetSystemTime
+    GlobalMemoryStatus dd rva _GlobalMemoryStatus
+    dd 0
+
+  kernel_name db 'KERNEL32.DLL',0
+
+  _ExitProcess dw 0
+    db 'ExitProcess',0
+  _CreateFileA dw 0
+    db 'CreateFileA',0
+  _ReadFile dw 0
+    db 'ReadFile',0
+  _WriteFile dw 0
+    db 'WriteFile',0
+  _CloseHandle dw 0
+    db 'CloseHandle',0
+  _SetFilePointer dw 0
+    db 'SetFilePointer',0
+  _GetCommandLineA dw 0
+    db 'GetCommandLineA',0
+  _GetEnvironmentVariable dw 0
+    db 'GetEnvironmentVariableA',0
+  _GetStdHandle dw 0
+    db 'GetStdHandle',0
+  _VirtualAlloc dw 0
+    db 'VirtualAlloc',0
+  _VirtualFree dw 0
+    db 'VirtualFree',0
+  _GetTickCount dw 0
+    db 'GetTickCount',0
+  _GetSystemTime dw 0
+    db 'GetSystemTime',0
+  _GlobalMemoryStatus dw 0
+    db 'GlobalMemoryStatus',0
+
+section '.reloc' fixups data readable discardable

samples/Awk/test.awk

@@ -0,0 +1,121 @@
+#!/bin/awk -f
+
+BEGIN {
+  # It is not possible to define output file names here because
+  # FILENAME is not define in the BEGIN section
+  n = "";
+  printf "Generating data files ...";
+  network_max_bandwidth_in_byte = 10000000;
+  network_max_packet_per_second = 1000000;
+  last3 = 0;
+  last4 = 0;
+  last5 = 0;
+  last6 = 0;
+}
+{
+  if ($1 ~ /Average/)
+    { # Skip the Average values
+      n = "";
+      next;
+    }
+
+  if ($2 ~ /all/)
+    { # This is the cpu info
+      print $3 > FILENAME".cpu.user.dat";
+#	  print $4 > FILENAME".cpu.nice.dat";
+      print $5 > FILENAME".cpu.system.dat";
+#     print $6 > FILENAME".cpu.iowait.dat";
+      print $7 > FILENAME".cpu.idle.dat";
+      print 100-$7 > FILENAME".cpu.busy.dat";
+    }
+  if ($2 ~ /eth0/)
+    { # This is the eth0 network info
+      if ($3 > network_max_packet_per_second)
+	print last3 > FILENAME".net.rxpck.dat"; # Total number of packets received per second.
+      else
+	{
+	  last3 = $3;
+	  print $3 > FILENAME".net.rxpck.dat"; # Total number of packets received per second.
+	}
+      if ($4 > network_max_packet_per_second)
+	print last4 > FILENAME".net.txpck.dat"; # Total number of packets transmitted per second.
+      else
+	{
+	  last4 = $4;
+	  print $4 > FILENAME".net.txpck.dat"; # Total number of packets transmitted per second.
+	}
+      if ($5 > network_max_bandwidth_in_byte)
+	print last5 > FILENAME".net.rxbyt.dat"; # Total number of bytes received per second.
+      else
+	{
+	  last5 = $5;
+	  print $5 > FILENAME".net.rxbyt.dat"; # Total number of bytes received per second.
+	}
+      if ($6 > network_max_bandwidth_in_byte)
+	print last6 > FILENAME".net.txbyt.dat"; # Total number of bytes transmitted per second.
+      else
+	{
+	  last6 = $6;
+	  print $6 > FILENAME".net.txbyt.dat"; # Total number of bytes transmitted per second.
+	}
+#     print $7 > FILENAME".net.rxcmp.dat"; # Number of compressed packets received per second (for cslip etc.).
+#     print $8 > FILENAME".net.txcmp.dat"; # Number of compressed packets transmitted per second.
+#     print $9 > FILENAME".net.rxmcst.dat"; # Number of multicast packets received per second.
+    }
+
+  # Detect which is the next info to be parsed
+  if ($2 ~ /proc|cswch|tps|kbmemfree|totsck/)
+    {
+      n = $2;
+    }
+
+  # Only get lines with numbers (real data !)
+  if ($2 ~ /[0-9]/)
+    {
+      if (n == "proc/s")
+	{ # This is the proc/s info
+	  print $2 > FILENAME".proc.dat";
+#	  n = "";
+	}
+      if (n == "cswch/s")
+	{ # This is the context switches per second info
+	  print $2 > FILENAME".ctxsw.dat";
+#	  n = "";
+	}
+      if (n == "tps")
+	{ # This is the disk info
+	  print $2 > FILENAME".disk.tps.dat"; # total transfers per second
+	  print $3 > FILENAME".disk.rtps.dat"; # read requests per second
+	  print $4 > FILENAME".disk.wtps.dat"; # write requests per second
+	  print $5 > FILENAME".disk.brdps.dat"; # block reads per second
+	  print $6 > FILENAME".disk.bwrps.dat"; # block writes per second
+#	  n = "";
+	}
+      if (n == "kbmemfree")
+	{ # This is the mem info
+	  print $2 > FILENAME".mem.kbmemfree.dat"; # Amount of free memory available in kilobytes.
+	  print $3 > FILENAME".mem.kbmemused.dat"; # Amount of used memory in kilobytes. This does not take into account memory used by the kernel itself.
+	  print $4 > FILENAME".mem.memused.dat"; # Percentage of used memory.
+#         It appears the kbmemshrd has been removed from the sysstat output - ntolia
+#	  print $X > FILENAME".mem.kbmemshrd.dat"; # Amount of memory shared by the system in kilobytes.  Always zero with 2.4 kernels.
+#	  print $5 > FILENAME".mem.kbbuffers.dat"; # Amount of memory used as buffers by the kernel in kilobytes.
+	  print $6 > FILENAME".mem.kbcached.dat"; # Amount of memory used to cache data by the kernel in kilobytes.
+#	  print $7 > FILENAME".mem.kbswpfree.dat"; # Amount of free swap space in kilobytes.
+#	  print $8 > FILENAME".mem.kbswpused.dat"; # Amount of used swap space in kilobytes.
+	  print $9 > FILENAME".mem.swpused.dat"; # Percentage of used swap space.
+#	  n = "";
+ 	}
+      if (n == "totsck")
+	{ # This is the socket info
+	  print $2 > FILENAME".sock.totsck.dat"; # Total number of used sockets.
+	  print $3 > FILENAME".sock.tcpsck.dat"; # Number of TCP sockets currently in use.
+#	  print $4 > FILENAME".sock.udpsck.dat"; # Number of UDP sockets currently in use.
+#	  print $5 > FILENAME".sock.rawsck.dat"; # Number of RAW sockets currently in use.
+#	  print $6 > FILENAME".sock.ip-frag.dat"; # Number of IP fragments currently in use.
+#	  n = "";
+ 	}
+    }
+}
+END {
+  print " '" FILENAME "' done.";
+}

samples/BlitzBasic/HalfAndDouble.bb

@@ -0,0 +1,147 @@
+
+Local bk = CreateBank(8)
+PokeFloat bk, 0, -1
+Print Bin(PeekInt(bk, 0))
+Print %1000000000000000
+Print Bin(1 Shl 31)
+Print $1f
+Print $ff
+Print $1f + (127 - 15)
+Print Hex(%01111111100000000000000000000000)
+Print Hex(~%11111111100000000000000000000000)
+
+Print Bin(FloatToHalf(-2.5))
+Print HalfToFloat(FloatToHalf(-200000000000.0))
+
+Print Bin(FToI(-2.5))
+
+WaitKey
+End
+
+
+; Half-precision (16-bit) arithmetic library
+;============================================
+
+Global Half_CBank_
+
+Function FToI(f#)
+	If Half_CBank_ = 0 Then Half_CBank_ = CreateBank(4)
+	PokeFloat Half_CBank_, 0, f
+	Return PeekInt(Half_CBank_, 0)
+End Function
+
+Function HalfToFloat#(h)
+	Local signBit, exponent, fraction, fBits
+	
+	signBit = (h And 32768) <> 0
+	exponent = (h And %0111110000000000) Shr 10
+	fraction = (h And %0000001111111111)
+	
+	If exponent = $1F Then exponent = $FF : ElseIf exponent Then exponent = (exponent - 15) + 127
+	fBits = (signBit Shl 31) Or (exponent Shl 23) Or (fraction Shl 13)
+	
+	If Half_CBank_ = 0 Then Half_CBank_ = CreateBank(4)
+	PokeInt Half_CBank_, 0, fBits
+	Return PeekFloat(Half_CBank_, 0)
+End Function
+
+Function FloatToHalf(f#)
+	Local signBit, exponent, fraction, fBits
+	
+	If Half_CBank_ = 0 Then Half_CBank_ = CreateBank(4)
+	PokeFloat Half_CBank_, 0, f
+	fBits = PeekInt(Half_CBank_, 0)
+	
+	signBit = (fBits And (1 Shl 31)) <> 0
+	exponent = (fBits And $7F800000) Shr 23
+	fraction = fBits And $007FFFFF
+	
+	If exponent
+		exponent = exponent - 127
+		If Abs(exponent) > $1F
+			If exponent <> ($FF - 127) Then fraction = 0
+			exponent = $1F * Sgn(exponent)
+		Else
+			exponent = exponent + 15
+		EndIf
+		exponent = exponent And %11111
+	EndIf
+	fraction = fraction Shr 13
+	
+	Return (signBit Shl 15) Or (exponent Shl 10) Or fraction
+End Function
+
+Function HalfAdd(l, r)
+	
+End Function
+
+Function HalfSub(l, r)
+	
+End Function
+
+Function HalfMul(l, r)
+	
+End Function
+
+Function HalfDiv(l, r)
+	
+End Function
+
+Function HalfLT(l, r)
+	
+End Function
+
+Function HalfGT(l, r)
+	
+End Function
+
+
+; Double-precision (64-bit) arithmetic library)
+;===============================================
+
+Global DoubleOut[1], Double_CBank_
+
+Function DoubleToFloat#(d[1])
+	
+End Function
+
+Function FloatToDouble(f#)
+	
+End Function
+
+Function IntToDouble(i)
+	
+End Function
+
+Function SefToDouble(s, e, f)
+	
+End Function
+
+Function DoubleAdd(l, r)
+	
+End Function
+
+Function DoubleSub(l, r)
+	
+End Function
+
+Function DoubleMul(l, r)
+	
+End Function
+
+Function DoubleDiv(l, r)
+	
+End Function
+
+Function DoubleLT(l, r)
+	
+End Function
+
+Function DoubleGT(l, r)
+	
+End Function
+
+
+;~IDEal Editor Parameters:
+;~F#1A#20#2F
+;~C#Blitz3D