June 8, 2014

modular and versioned GLSL

^{a WebGL demo of glsl-lut, for efficient color transforms on the GPU. source code here}

Contents #

Intro
glslify
Example Modules
The Awesomeness of NPM
Unit Testing
Extras
As a Source Transform

Intro #

NPM and Node are brilliant tools for pushing JavaScript and web development forward. They have drastically changed the landscape; introducing the concept of small and composable modules, semantic versioning, and automatic dependency management. Gone are the days of “copy-pasting” snippets, fear of new versions, and manually maintaining vendor paths. Installing a dependency has never been easier:

npm install querystring --save

But now it’s starting to make some shifts in other areas; namely, game and graphics programming with GLSL. For the uninitiated, Browserify is a neat tool which statically analyzes your code for require() statements (Node’s take on import), and bundles it together in a way that the browser can understand. It includes “transforms” which can run on your source before the bundle step, such as inlining the contents of a readFileSync as a UTF-8 string in your source.

glslify #

So, where am I going with this? Well, Chris Dickinson, Hugh Kennedy and some other creative minds have been working on glslify and a related collection of modules. With these tools, we can statically analyze GLSL programs, transform their source code, use require() and export() to modularize shaders, and much more.

Typically you might use glslify as a Browserify or Webpack source transform, but the modules can also be composed to create a build step for a non-JavaScript projects (e.g. iOS and desktop OpenGL apps). The tool also includes a generic command-line interface for transpiling shaders.

To demonstrate:

uniform vec2 texCoord0;

//"import" our random function
#pragma glslify: random = require('glsl-random')

void main() {
    //quick pseudo-random 2D noise
    float n = random(texCoord0); 
    gl_FragColor = vec4(vec3(n), 1.0);
}

The above code will inline the function from glsl-random. No more copy-pasting from a random blog or StackOverflow answer.

Installing Dependencies #

For non-JavaScript devs, you first will need to install Node (which comes with NPM). Then install glslify globally, as below. If you are running into sudo issues, see here.

npm install glslify -g

Now, cd to the directory where you’re going to store all your shaders, and start installing some snippets:

npm install glsl-random --save

This will download the glsl-random function and store it in a folder called node_modules. You should add this folder to your .gitignore to avoid it being versioned. We then use the --save option to mark that snippet as a dependency, since our code won’t work without it.

The next step is to initialize the folder as a package; this will add a package.json to keep track of our dependencies and their version numbers. Enter the following, it will prompt for details which you can optionally fill in:

npm init

Now, when somebody checks out your code, they can just run npm install on the package and it will grab all the GLSL snippets you’ve saved as dependencies.

Compiling The Shader #

Let’s make a new directory called build, then compile the shaders:

glslify frag.glsl > build/frag.glsl

And if you check the built file, you’ll see it’s inlined the necessary functions. Awesome!

#define GLSLIFY 1

uniform vec2 texCoord0;
highp float a_x_random(vec2 co) {
  highp float a = 12.9898;
  highp float b = 78.233;
  highp float c = 43758.5453;
  highp float dt = dot(co.xy, vec2(a, b));
  highp float sn = mod(dt, 3.14);
  return fract(sin(sn) * c);
}
void main() {
  float n = a_x_random(texCoord0);
  gl_FragColor = vec4(vec3(n), 1.0);
}

Relative Requires #

Sometimes you need a project-specific function, but you might want to use it across multiple shaders. This isn’t something worth publishing on NPM, so instead you can require it using relative paths. Say you’ve got lib/foo.glsl:

vec4 foo(float n) {
    return n*n*n;
}

#pragma glslify: export(foo)

Now you can “require” it into your main program using ./ to denote a relative path:

#pragma glslify: foo = require('./lib/foo')

Example Modules #

There are a growing number of modules on glslify including:
http://stack.gl/packages

glsl-noise texture-free perlin/simplex/classic noise
glsl-specular-blinn-phong and other specular / diffuse BRDFs
glsl-easings Robert Penner’s easing equations
glsl-perturb-normal normal mapping in a fragment shader
glsl-film-grain high-level film grain utility
glsl-cornell-box a raymarched Cornell Box
and many more

The Awesomeness of NPM #

One of the real reasons this is awesome is because we get semantic versioning. When we did npm install --save, it marked down the latest version number of glsl-random. Now, say the author (me) finds a small bug, or a way to improve/optimize the function. I can make the new changes backward-compatible, and publish a new “patch” or “minor” version. This means that the next time users npm install glsl-random, it will grab that latest version with those fixes. And, most importantly, it won’t grab any new versions that are backward-incompatible, since those would be published with a “major” version number.

The random noise “one-liner” is actually a good example. A lot of people copy-paste it from blogs or StackOverflow, only later to find out that it doesn’t work. Rather than blindly copy-pasting, it’s much safer to depend on a repository which has its own issue tracking, semantic versioning, and can be maintained and improved over time.

Unit Testing #

This is another key feature. Test-driven development is particularly easy and useful for small modules, and we see it a lot in NPM. But unit testing GLSL is a bit trickier, and for the most part nobody does it. But when you start composing shaders using many npm dependencies, it’s nice to ensure that these will work in your target devices, and provide the expected output. Unit tests give users more confidence in the code, especially as it changes across versions.

There have been some past attempts to unit test GLSL, and hopefully we can learn from them. glslify is still fairly young; unit testing is not yet solved, but definitely within reach.

One such example is the unit tests in glsl-hsl2rgb. These could even be run in a testing platform like SauceLabs or Testling-ci to determine whether the shader will fail on any browsers or devices.

Support

Extras #

Other than modules, glslify also provides an interface to easily construct new source transforms in your GLSL shaders. For example:

glslify-hex transforms hex like #ff0000 into vec3(1.0, 0.0, 0.0)
glslify-import an import directive for organizing shader chunks
glslify-live experimental live shader re-compilation

As a Source Transform #

For JavaScript developers, the typical means of consuming glslify-capable modules is with the source transform. With browserify, your build step might look like this:

browserify index.js -t glslify > bundle.js

Alternatively, you can specify it as a transform in your package.json.

"browserify": {
    "transform": [
      "glslify"
    ]
  }

Now when you run your code through browserify, it will analyze it for glslify statements, and compile it as needed into your JS file.

var glslify = require('glslify')
var vert = glslify('./vert.glsl')
var frag = glslify('./frag.glsl')

You can see a practical example of using glslify in JS here:

https://github.com/stackgl/glsl-hash-blur/blob/master/demo.js

The resulting demo:

http://stack.gl/glsl-hash-blur/

You can also use it alongside ThreeJS or your favourite engine, like in this example:

https://github.com/mattdesl/three-glslify-example/

253

Kudos

253