Twixt - Clojure Web Application Asset Pipeline

Build Status

Twixt is an extensible asset pipeline for use in Clojure web applications. It is designed to complement an application built using Ring and related libraries, such as Compojure. Twixt provides content transformation (such as Less to CSS), support for efficient immutable resources, and best-of-breed exception reporting.

Twixt includes a very readable HTML exception report page, which displays:

  • The entire stack of nested exceptions, top to bottom

  • The stack trace for only the root exception

  • Demangled namespace and function names for Clojure stack frames: io.aviso.twixt/new-twixt/reify/middleware/fn instead of io.aviso.twixt$new_twixt$reify__954$middleware__959$fn__960.invoke().

  • The contents of the Ring request map

  • All JVM system properties

Twixt draws inspiration from Apache Tapestry and Dieter.

Twixt currently supports:

  • CoffeeScript to JavaScript compilation (using Rhino) - including source maps

  • Jade to HTML compilation (using jade4j)

  • Less to CSS compilation (using less4j)

  • File system caching of compiled content

  • Automatic GZip compression

Twixt is available under the terms of the Apache Sofware License 2.0.

Twixt is available from the Clojars artifact repository as io.aviso:twixt. Follow these instructions to configure the dependency in your build tool.

API Documentation

Usage

Twixt serves resources located on the classpath, in the META-INF/assets/ folder. The contents of this folder is accessible, by default, via the URL /assets/.

By design, assets are segregated from the rest of your code. This prevents a malicious client from directly accessing your code or configuration files. Anything outside the META-INF/assets/ folder is inaccessible via Twixt.

Twixt maps file extensions to MIME types; it will then transform certain MIME types; for example .coffee files are compiled to JavaScript.

(ns example.app
  (:use compojure.core
        ring.adapter.jetty
        [io.aviso.twixt.startup :as startup]))

;;; Use Compojure to map routes to handlers
(defroutes app ...)

;;; Create twixt wrappers and handler to handle /asset/ URLs, in development mode
(def app-handler
  (startup/wrap-with-twixt app true)

;;; Use the Ring Jetty adapter to serve your site
(run-jetty app-handler)
Note
 Twixt changes its behavior in a number of ways in development mode (as opposed to the normal production mode). For example, Less compilation will pretty-print the generated HTML markup in development mode. In production mode, the markup omits all unnecessary white space.

The Twixt middleware intercepts requests for the /assets/ URI that map to actual files; non-matching requests, or requests for assets that do not exist, are delegated down to the wrapped handlers.

In development mode, Twixt will write compiled files to the file system (you can configure where if you like). On a restart of the application, it will use those cached files if the source files have not changed. This is important, as compilation of some resources, such as CoffeeScript, can take several seconds (due to the speed, or lack thereof, of the Rhino JavaScript engine).

The Twixt API includes alternate functions for constructing both the Ring middleware, and Twixt’s own asset pipeline; this allows you to add new features, or eliminate unwanted features. Please reference the code to see how to configure Twixt options, assemble the Twixt asset pipeline, and finally, provide the necessary Ring middleware.

A question of URLs

At its core, Twixt is a set of Ring middleware that maps certain URL patterns to matching files on the classpath, and does some transformations along the way.

Currently, the mapping is very straightforward: the path /assets/123abc/css/style.less is mapped to resource META-INF/assets/css/style.less which is read into memory and transformed from Less to CSS. Embedded in the middle of the URL is the content checksum for the file (123abc).

Note
 Placing this information directly into the URI is called fingerprinting.

In your application, assets will change during development, or between production deployments. The URIs provided to the client agent (the web browser) is a resource; on the web, resources are immutable, even though in your workspace, files change all the time. The checksum in the URI is based on the actual content of the file; whenever the underlying content changes, then a new checksum, new URI, and therefore, new resource will be referenced by the URI.

Twixt sets headers to indicate a far-future expiration date for the resource; the upshot of which is that, once the resource for an asset is downloaded to the client browser, the browser will not ask for it again. This is great for performance.

The checksum has a optional "z" prefix; this indicates a GZip compressed resource. Twixt detects if the client agent supports GZip compression (via the Accept-Encodings header). In addition, compression is only applied to specific content types, such as "text/html" and "application/edn". Most binary formats, including fonts and images, are already compressed.

Because of this, when referencing assets inside your templates, you must pass paths (relative to META-INF/assets) through Twixt to get URIs that will work in the browser:

(defhtml index
  [{context :twixt :as request}]
  (html
    (doctype :html5
    [:html
      [:head
        [:title "My Clojure App"]
        (include-css (get-asset-uri context "css/style.less"))
        ...

The parameter to defhtml is the Ring request map; Twixt has injected middleware that provides the Twixt context under the :twixt key.

get-asset-uri is defined in the io.aviso.twixt namespace.

Twixt must do all necessary compilations and other transformations, to arrive at final content for which a checksum can be computed. Although this can slow the initial HTML render, it is also good because any exceptions, such as compilation errors, will occur immediately, rather than when the asset’s content is later retrieved by the client.

When a client requests an asset that exists, but supplies an incorrect checksum, Twixt will respond with a 301 redirect HTTP response, directing the client to the correct resource (with the correct checksum). This is an extremely unlikely scenario that would involve a running client in a race with a redeployed application.

Stacks

In development, you often want to have many small source files downloaded individually to the browser. This is simpler to debug, and faster … a change to one file will be a small recompile of just that file.

In production, it’s a different story; you want the client to make as few requests to the server as possible.

This can be accomplished using stacks.

Stacks allow you to group together related files of the same type into a single asset. Commonly, this is used to aggregate JavaScript or CSS.

In development mode, you will see the individual files of the stack; in production mode, the stack is represented by a single URI which maps to the aggregated content of all the files in the stack.

A stack file is written in EDN. Each stack file contains a :content-type key, and a :components key. Stack files have a .stack extension.

{:content-type "text/css"
 :components "bootstrap3/bootstrap.css"
             "app.less"
             "ie-fixes.less"}

When using stacks, you will want a slight tweak to your page template:

(defhtml index
  [{context :twixt :as request}]
  (html
    (doctype :html5
    [:html
      [:head
        [:title "My Clojure App"]
        (apply include-css (get-asset-uris context "css/app-css.stack"))
        ...

Since get-asset-uris will return a collection of URIs (unlike get-asset-uri which always returns just one), we must change include-css to apply include-css.

This template will work in development (get-asset-uris returning several URIs) and in production (just a single URI).

It is possible for stacks to include other stacks as components.

Stack components are included not imported; if a component asset is listed more than once, its content will be aggregated more than once.

Direct asset URLs

Sometimes it is not possible to determine the full asset URL ahead of time; a common example would be a client-side framework, such as AngularJS that wants to load HTML templates dynamically, at runtime. It will know the path to the asset, but will not know the checksum.

In this case, an optional Ring middleware can be used: wrap-with-asset-redirector.

This middleware identifies requests that match existing assets and responds with a 302 redirect to the proper asset URL. For example, the asset stored as META-INF/assets/blueant/cla.html can be accessed as /blueant/cla.html, and will be sent a redirect to /assets/123abc/blueant/cla.html.

Configuring Twixt

Twixt’s configuration is used to determine where to locate asset resources on the classpath, and what folder to serve them under. It also maps file name extensions to MIME types, and configures the file system cache.

The default options:

(def default-options
  {:path-prefix          "/assets/"
   :content-types        mime/default-mime-types
   :twixt-template       {}
   :content-transformers {}
   :compressable         #{"text/*" "application/edn" "application/json"}
   :cache-folder         (System/getProperty "twixt.cache-dir" (System/getProperty "java.io.tmpdir"))

You can override :path-prefix to change the root URL for assets; / is an acceptable value.

The :content-types key maps file extensions to MIME types.

The :content-transformers key is a map of content type keys (as strings, such as "text/coffeescript") to a transformation function; The CoffeeScript, Jade, and Less compilers operate by adding entries to :content-types and :content-transformers+.

The :compressable key is a set used to identify which content types are compressable; note the use of the /* suffix to indicate that all text content types are compressable. Anything not explicitly compressable is considered non-compressable.

The :twixt-template key is a map that provides default values for the :twixt request key. This is often used to provide information to specific content transformers.

Caching

It is desirable to have Twixt be able to serve-up files quickly, especially in production. However, that is counter-balanced by the need to ensure the correct content is served.

Development Mode

Twixt will cache the results of compilations to the file system; the cache persists between executions. This means that on restart, the application will normally start right up, since the compiled files will be accessed from the file system cache.

Whenever a source file changes, the corresponding compiled file is rebuilt (and then the file system cache is updated). This is great for development, as you will frequently be changing your source files.

Twixt is smart about dependencies; for example, a Less file may @import another Less file; the compiled asset will have dependencies on both files; if either changes, Twixt will re-compile the sources into a new asset, with a new asset URI.

Twixt doesn’t bother to cache the GZip compressed versions of assets to the file system; it is relatively quick to rebuild the compressed byte stream. There’s an in-memory cache of the compressed assets, but each request includes checks to see if the compiled output itself must be updated.

You may need to manually clear out the file system cache after upgrading to a new version of Twixt, or any other configuration change that can affect the compiled output.

Production Mode

In production mode, Twixt starts from a clean slate; it does not use a file system cache. However, all assets are cached in memory; Twixt also caches the compressed versions of assets, to save the cost of repeatedly compressing them on the fly.

In production mode, there are no checks to see if the in-memory cache is valid; if a source file is changed, it is assumed that the entire application will be re-deployed and re-started.

Jade Notes

twixt helper

Twixt places a helper object, twixt, into scope for your templates. twixt supplies a single method, uri. You can pass the uri method a relative path, or an absolute path (starting with a slash).

img(src=twixt.uri("logo.png"))
Warning
 When the path is relative, it is evaluated relative to the main Jade asset (an explicitly not relative to any include -ed Jade sources).

This will output a fully qualified asset URI:

<img src="/assets/8ee745bf/logo.png">

Defining your own helpers

It is possible to define your own helper objects.

Helper objects are defined inside the Twixt context under keys :jade :helpers. This is a map of string keys to creator functions.

Each creator function is passed the main Jade asset, and the context. It uses this to initialize and return a helper object. A new set of helper objects is created for each individual Jade compilation.

Generally, you will want to define a protocol, then use reify. For example, this is the implementation of the twixt helper:

(defprotocol TwixtHelper
  "A Jade4J helper object that is used to allow a template to resolve asset URIs."
  (uri
    [this path]
    "Used to obtain the URI for a given path. The path may be relative to the currently compiling
asset, or may be absoluate (with a leading slash). Throws an exception if the asset it not found."))

(defn- create-twixt-helper
  [asset context]
  (reify TwixtHelper
    (uri [_ path]
      (twixt/get-asset-uri context (complete-path asset path)))))
Note
 Any asset URI will cause the asset in question to be added as a dependency of the main Jade template. This means that changing the referenced asset will cause the Jade template to be re-compiled. This makes sense: changing an image file will change the URI for the image file, which means that the Jade output should also change.

Creator functions can be added to the Twixt context using Ring middleware:

    (handler (assoc-in request [:twixt :jade :helpers "adrotation"] create-ad-rotation-helper))

However, more frequently, you will just add to the Twixt options in your application’s startup code:

  (assoc-in twixt/default-options [:twixt-template :jade :helpers "adrotation"] create-ad-rotation-helper))

This :twixt-template key is used to create the :twixt Ring request key.

Defining your own variables

Variables are much the same as helpers, with two differences:

  • The key is :variables (under :jade, in the Twixt context)

  • The value is the exact object to expose to the template

You can expose Clojure functions as variables if you wish; the Jade template should use func.invoke() to call the function.

Helper / Variable pitfalls

The main issue with helpers and variables relates to cache invalidation. Twixt bases cache invalidation entirely on the contents of the underlying files.

There is currently an ambiguity that comes into play when the referenced asset is a compressable file type (e.g., not an image file). This can cause the Jade compiler to generate a compressed URI that, for a different request and client, will not be useful.

Future Plans

The goal is to achieve at least parity with Apache Tapestry, plus some additional features specific to Clojure. This means:

  • E-Tags support

  • ClojureScript compilation

  • JavaScript minimization via Google Closure

  • CSS Minification

  • RequireJS support and AMD modules

  • Break out the the Less, Jade, CoffeeScript, and exception reporting support into a-la-carte modules

  • "Warm up" the cache at startup (in production)

Stability

Alpha: Many features are not yet implemented and the code is likely to change in many ways going forward … but still very useful!

A note about feedback

Feedback is very important to me; I often find Clojure just a bit frustrating, because if there is an error in your code, it can be a bit of a challenge to track the problem backwards from the failure to the offending code. Part of this is inherent in functional programming, part of it is related to lazy evaluation, and part is the trade-off between a typed and untyped language.

In any case, it is very important to me that when thing go wrong, you are provided with a detailed description of the failure. Twixt has a mechanism for tracking the operations it is attempting, to give you insight into what exactly failed if there is an error. For example, (from the test suite):

ERROR [       qtp2166970-29] io.aviso.twixt.coffee-script An exception has occurred:
ERROR [       qtp2166970-29] io.aviso.twixt.coffee-script [  1] - Invoking handler (that throws exceptions)
ERROR [       qtp2166970-29] io.aviso.twixt.coffee-script [  2] - Accessing asset `invalid-coffeescript.coffee'
ERROR [       qtp2166970-29] io.aviso.twixt.coffee-script [  3] - Compiling `META-INF/assets/invalid-coffeescript.coffee' to JavaScript
ERROR [       qtp2166970-29] io.aviso.twixt.coffee-script META-INF/assets/invalid-coffeescript.coffee:6:1: error: unexpected INDENT
      argument: dep2
^^^^^^
java.lang.RuntimeException: META-INF/assets/invalid-coffeescript.coffee:6:1: error: unexpected INDENT
      argument: dep2
^^^^^^
   ....

In other words, when there’s a failure, Twixt can tell you the steps that led up the failure, which is 90% of solving the problem in the first place.

Twixt’s exception report captures all of this and presents it as readable HTML. The exception report page also does a decent job of de-mangling Java class names to Clojure namespaces and function names.

How does Twixt differ from Dieter?

On the application I was building, I had a requirement to deploy as a JAR; Dieter expects all the assets to be on the filesystem; I spent some time attempting to hack the Dieter code to allow resources on the classpath as well. When that proved unsuccessful, I decided to build out something a bit more ambitious, that would support the features that have accumulated in Tapestry over the last few years.

Twixt also embraces system as transient state, meaning nothing is stored statically.

Twixt will grow further apart from Dieter as the more advanced pieces are put into place.