Static is the new cool.

27.10.2015
Diego Caponera

Lately we have been rebrushing our homepage here at kreuzwerker, and we are quite happy with that! I am mostly happy with how things are going under the hood, because basically nothing is happening.

Just a bunch of .html files is being served, with some CloudFront topping for speeding things up (and for pleasing our AWS guru Joern!).

Static website generation from a determined data set (mostly Markdown documents) is nothing new, and has been implemented over all possible platform and programming languages. But trust me when I tell you, that no system was completely able to fullfil our needs, because most of them are very good for blog management, but not flexible enough to handle a more complex data graph.

On top of that, we needed good localisation support for document translation at URL level, another feature that at least at implementation time I could not find.

What then? Well, we are developers, we implemented our custom solution!

Et voilá, Waffel!

Static site generation done tasty.

Waffel is my tiny library, whose work is still in progress, able to convert a set of .markdown/YAML front-mattered documents to a hypertext, via a schema.yml configuration file plus a set of Nunjucks HTML templates.

Let our data structure be something like:

/data
├── images
│   ├── blog
│   │   ├── ...
│   └── users
│       ├── andreas-profous.jpg
│       ├── ...
│       └── ulrike-koenig.jpg
├── pages
│   ├── de
│   │   ├── about.md
│   │   └── ...
│   ├── en
│   │   ├── about.md
│   │   └── ...
│   └── pl
├── partners
│   ├── de
│   │   ├── amazon-web-services.md
│   │   └── ...
│   └── en
│       ├── amazon-web-services.md
│       └── ...
├── posts
│   ├── Create-train-and-test-your-devops-skills-with-the-simian-army.md
│   ├── ...
│   └── why-clojure-rocks-2.md
├── ...
└── users
    ├── de
    │   ├── andreas-profous.md
    │   ├── ...
    │   └── ulrike-koenig.md
    └── en
        ├── andreas-profous.md
        ├── ...
        └── ulrike-koenig.md

The structure is then defined as:

config:
  dateFormat:             YYYY.MM.DD
  google_analytics:       UA-XXXXXXX-1
  ...
structure:
  home:
    template: home
    url: /
  about:
    template: about
    url: /about
  team:
    collection: users
    pages:
      single:
        template: team/member
        url:      /about/team/:slug
        priority: 0.6
        filter:
          published: true        
  ...
  blog:
    collection: posts
    pages:
      index:
        template: blog/index
        url:      /blog
        paginate: 10
        priority: 0.5
        sort:
          field: date
          order: desc
      tags:
        template: blog/tag
        url:      /blog/tag/:tag
        groupBy:  tags
        paginate: 10
        changefreq: weekly
        priority: 0.7
        sort:
          field: date
          order: desc
      single:
        template: blog/post
        url:      /blog/posts/:slug
        priority: 0.8
...

Never mind the library DSL for a second, and focus on the team definition: it iterates over the users collection (namely from /data/users/**/*.md), passing every entry to the team/member template, and saving the output to /about/team/{{user.slug}}.html.

Same thing happens for all other sections.

In what does Waffel differ from the other zillion generators?

First, it is still on an early development stage, being thus sadly almost undocumented/untested yet. Among the positive differences:

It is a library, more than a CLI tool.

You just need to require('Waffel'), create an instance, and enjoy the full control on what you are doing, be it via Gulp/Grunt/Brunch, or your custom solution.

It handles multiple languages like a charm.

In the data structure above, you noticed the /de and /en subfolders of the /user folder; if we instantiate Waffel like this:

wfl = new Waffel
  languages:          ['de', 'en']
  defaultLanguage:    'de'

it will be aware that data is stored in both English and German, and that the default language will be the latter. English pages will be prefixed with /en, and the Queen will be happy.

It does not care about js/css concateminification.

Why do I find this a good thing? Everybody has his or her own workflow in order to deal with asset delivery, and I think that it should not be responsibility of a generator, to care about.

For instance I have always been a big fan of Brunch, so I created a skeleton that takes care of everything. But it would be easy to write a simple Grulpfile that does same stuff!

You can have preliminary build steps.

Being Waffel a simple library, that's trivial: you can do stuff before, then pass the data to the generator and do stuff with it, for instance:

var Waffel = require('Waffel')
var DataProvider = require('./lib/DataProvider')

// We do the Waffel stuff here.
var port = 3333
var wfl = new Waffel({
  domain:   "http://localhost:" + port,
  serverConfig: {
    port:       port,
    path:       'public',
    indexPath:  'public/404.html'
  }
})

// DataProvider is some custom library.
// Let's assume that the async call to provider.fetch returns an array of some quotes by Bill Hicks:
var provider = new DataProvider({
  url: http://path.to/some/service,
  query: { author: 'Bill Hicks' }
})

// once both the provider and Waffel are done, let's do stuff!
Promise.all([provider.fetch, wfl.init])
  .then(function(results){
    wfl.generate({
      data: {
        quotes: results[0]
      }
    })
  })

This way you may either skip or complement the local filesystem read, and generate your document from an arbitrary dataset, which I find pretty useful.

What if I like clicking on my mouse better than scripting?

First of all, you have all my respect. That's why we worked on an Electron app for managing your precious /datafolder, looking like this at the moment:

The /data/posts folder in all its glory.

And see this very blog post being written live!

A single entry being edited.

Worth noting: Waffel is completely agnostic about the way you manage data. In fact it does not care at all, as long as files belong to a format that it can understand.

Cool, but how does your editing tool understand how to parse and display the fields? Is there any black magic happening somewhere?

No, you have to place a simple schema configuration file in the root of the /data folder, that is the only price to pay. As the app is in its early lifecycle, documentation is still lacking, so take this as a mere (hopefully self-explaining) preview:

users:
  label:  Team
  icon:   fa-user
  description: 'Our beautiful team.'
  labelPattern: "#{name}"
  defaultView: grid
  languages: ['de', 'en', 'pl']
  fields:
    picture:
      type: image
      label: Photo
      filterable: false
      align:  center
      validation:
        required: false
    name:
      type: text
      label: Full Name
      sort:
        main: true
        sortable: true
    slug:
      type: slug
      label: Slug
      filterable: false
      display:
        table:  false
    email:
      type: email
      label: E-Mail
      sort:
        sortable: true
    published:
      type: boolean
      label: Published
      filterable: false
      align: center
  ...
...
posts:
  label:  Posts
  icon:   fa-file-text-o
  description: What we love to talk about.
  labelPattern: "<strong>#{author}</strong><br>#{title}"
  detailLabelPattern: "#{title}"
  main: true
  assetPath: /images/blog
  fields:
    cover:
      type: image
      label: Cover
      filterable: false
      align:  center
      validation:
        required: false
    title:
      type: text
      label: Title
      sort:
        sortable: true
    slug:
      type: slug
      label: Slug
      filterable: false
      display:
        table:  false
    author:
      type: select
      relation:
        type: collection
        from: users
      label: Author
      sort:
        sortable: true
    tags:
      type: tags
      label: Tags
    date:
      type: date
      label: Date
      sort:
        sortable: true
        main: true
        order: desc
    published:
      type: boolean
      label: Published
      filterable: false
      align: center

Every collection, in this case users and posts, has many fields of different type (text, slug, email, boolean, image, ...), and some basic relationships as well (see selectfrom users as the author field of the posts). Without going any deeper, this constraints are used by the app in order to build the forms and the list tables, and to validate user input.

At the moment, you have to write the schema by hand. In the future, we plan to have a simple graphic tool to set it up, and possibly a basic inference mechanism that learns the schema from already existing data. We'll keep you posted!

Cool, but what about more people willing to collaborate, if files are stored on one single machine?

Let me think, is there any way to edit a distributed set of files, tracking the name of who is editing what? Damn, if I could ever recall...

Putting content under version control was the natural choice of course!

We thought that granting the app built in support for that, would have been of great use for non command line fellows:

The CVS sidebar.

Ok, what about going live?

We are hosting the repository on Github, so we have a Travis task that on every new tagged commit takes care to generate the website, and to sync the output to a S3 bucket. As already mentioned, CloudFront sits on the very top of everything granting us super fast performance.

What about preview and testing? Every untagged commit to the master branch is deployed to a staging bucket instead, that is accessible by the team only.


Lesson learned: if you are not editing your whole data suite frequently, and you are the only deterministic source of change (i.e. no arbitrary Internet user is affecting your website state over time - see comments, placing orders, rating items...), it is completely acceptable to store it in static files!

We plan to release both Waffel, the generator tool, and Vaneella, the content manager app, as soon as they reach a stable implementation status. In the meantime, we hope you enjoyed the read and you consider going static!

Diego Caponera
comments powered by Disqus