Static is the new cool.

Lately we have been rebrushing our homepage here at kreuzwerker, and we are quite happy with that! I am mostly happy with how things are going under the hood, because basically nothing is happening.

27.10.2015

Diego Caponera

Tags

Lately we have been rebrushing our homepage here at kreuzwerker, and we are quite happy with that! I am mostly happy with how things are going under the hood, because basically nothing is happening.

Just a bunch of .html files is being served, with some CloudFront topping for speeding things up (and for pleasing our AWS guru Joern!).

Static website generation from a determined data set (mostly Markdown documents) is nothing new, and has been implemented over all possible platform and programming languages. But trust me when I tell you, that no system was completely able to fullfil our needs, because most of them are very good for blog management, but not flexible enough to handle a more complex data graph.

On top of that, we needed good localisation support for document translation at URL level, another feature that at least at implementation time I could not find.

What then? Well, we are developers, we implemented our custom solution!

Et voilá, Waffel!

Static site generation done tasty.

Waffel is my tiny library, whose work is still in progress, able to convert a set of .markdown/YAML front-mattered documents to a hypertext, via a schema.yml configuration file plus a set of Nunjucks HTML templates.

Let our data structure be something like:

/data├── images│   ├── blog│   │   ├── ...│   └── users│       ├── andreas-profous.jpg│       ├── ...│       └── ulrike-koenig.jpg├── pages│   ├── de│   │   ├── about.md│   │   └── ...│   ├── en│   │   ├── about.md│   │   └── ...│   └── pl├── partners│   ├── de│   │   ├── amazon-web-services.md│   │   └── ...│   └── en│       ├── amazon-web-services.md│       └── ...├── posts│   ├── Create-train-and-test-your-devops-skills-with-the-simian-army.md│   ├── ...│   └── why-clojure-rocks-2.md├── ...└── users    ├── de    │   ├── andreas-profous.md    │   ├── ...    │   └── ulrike-koenig.md    └── en        ├── andreas-profous.md        ├── ...        └── ulrike-koenig.md

The structure is then defined as:

config:  dateFormat:             YYYY.MM.DD  google_analytics:       UA-XXXXXXX-1  ...structure:  home:    template: home    url: /  about:    template: about    url: /about  team:    collection: users    pages:      single:        template: team/member        url:      /about/team/:slug        priority: 0.6        filter:          published: true          ...  blog:    collection: posts    pages:      index:        template: blog/index        url:      /blog        paginate: 10        priority: 0.5        sort:          field: date          order: desc      tags:        template: blog/tag        url:      /blog/tag/:tag        groupBy:  tags        paginate: 10        changefreq: weekly        priority: 0.7        sort:          field: date          order: desc      single:        template: blog/post        url:      /blog/posts/:slug        priority: 0.8...

Never mind the library DSL for a second, and focus on the team definition: it iterates over the users collection (namely from /data/users/**/*.md), passing every entry to the team/member template, and saving the output to /about/team/{{user.slug}}.html.

Same thing happens for all other sections.

In what does Waffel differ from the other zillion generators?

First, it is still on an early development stage, being thus sadly almost undocumented/untested yet. Among the positive differences:

It is a library, more than a CLI tool.

You just need to require('Waffel'), create an instance, and enjoy the full control on what you are doing, be it via Gulp/Grunt/Brunch, or your custom solution.

It handles multiple languages like a charm.

In the data structure above, you noticed the /de and /en subfolders of the /user folder; if we instantiate Waffel like this:

wfl = new Waffel  languages:          ['de', 'en']  defaultLanguage:    'de'

it will be aware that data is stored in both English and German, and that the default language will be the latter. English pages will be prefixed with /en, and the Queen will be happy.

It does not care about js/css concateminification.

Why do I find this a good thing? Everybody has his or her own workflow in order to deal with asset delivery, and I think that it should not be responsibility of a generator, to care about.

For instance I have always been a big fan of Brunch, so I created a skeleton that takes care of everything. But it would be easy to write a simple Grulpfile that does same stuff!

You can have preliminary build steps.

Being Waffel a simple library, that’s trivial: you can do stuff before, then pass the data to the generator and do stuff with it, for instance:

var Waffel = require('Waffel')var DataProvider = require('./lib/DataProvider')// We do the Waffel stuff here.var port = 3333var wfl = new Waffel({  domain:   "http://localhost:" + port,  serverConfig: {    port:       port,    path:       'public',    indexPath:  'public/404.html'  }})// DataProvider is some custom library.// Let's assume that the async call to provider.fetch returns an array of some quotes by Bill Hicks:var provider = new DataProvider({  url: http://path.to/some/service,  query: { author: 'Bill Hicks' }})// once both the provider and Waffel are done, let's do stuff!Promise.all([provider.fetch, wfl.init])  .then(function(results){    wfl.generate({      data: {        quotes: results[0]      }    })  })

This way you may either skip or complement the local filesystem read, and generate your document from an arbitrary dataset, which I find pretty useful.

What if I like clicking on my mouse better than scripting?

First of all, you have all my respect. That’s why we worked on an Electron app for managing your precious /datafolder, looking like this at the moment:

vaneella list

And see this very blog post being written live!

vaneella detail

Worth noting: Waffel is completely agnostic about the way you manage data. In fact it does not care at all, as long as files belong to a format that it can understand.

Cool, but how does your editing tool understand how to parse and display the fields? Is there any black magic happening somewhere?

No, you have to place a simple schema configuration file in the root of the /data folder, that is the only price to pay. As the app is in its early lifecycle, documentation is still lacking, so take this as a mere (hopefully self-explaining) preview:

users:  label:  Team  icon:   fa-user  description: 'Our beautiful team.'  labelPattern: "#{name}"  defaultView: grid  languages: ['de', 'en', 'pl']  fields:    picture:      type: image      label: Photo      filterable: false      align:  center      validation:        required: false    name:      type: text      label: Full Name      sort:        main: true        sortable: true    slug:      type: slug      label: Slug      filterable: false      display:        table:  false    email:      type: email      label: E-Mail      sort:        sortable: true    published:      type: boolean      label: Published      filterable: false      align: center  ......posts:  label:  Posts  icon:   fa-file-text-o  description: What we love to talk about.  labelPattern: "<strong>#{author}</strong><br>#{title}"  detailLabelPattern: "#{title}"  main: true  assetPath: /images/blog  fields:    cover:      type: image      label: Cover      filterable: false      align:  center      validation:        required: false    title:      type: text      label: Title      sort:        sortable: true    slug:      type: slug      label: Slug      filterable: false      display:        table:  false    author:      type: select      relation:        type: collection        from: users      label: Author      sort:        sortable: true    tags:      type: tags      label: Tags    date:      type: date      label: Date      sort:        sortable: true        main: true        order: desc    published:      type: boolean      label: Published      filterable: false      align: center

Every collection, in this case users and posts, has many fields of different type (text, slug, email, boolean, image, …), and some basic relationships as well (see selectfrom users as the author field of the posts). Without going any deeper, this constraints are used by the app in order to build the forms and the list tables, and to validate user input.

At the moment, you have to write the schema by hand. In the future, we plan to have a simple graphic tool to set it up, and possibly a basic inference mechanism that learns the schema from already existing data. We’ll keep you posted!

Cool, but what about more people willing to collaborate, if files are stored on one single machine?

Let me think, is there any way to edit a distributed set of files, tracking the name of who is editing what? Damn, if I could ever recall…

Putting content under version control was the natural choice of course!

We thought that granting the app built in support for that, would have been of great use for non command line fellows:

vaneella version control 1

Ok, what about going live?

We are hosting the repository on Github, so we have a Travis task that on every new tagged commit takes care to generate the website, and to sync the output to a S3 bucket. As already mentioned, CloudFront sits on the very top of everything granting us super fast performance.

What about preview and testing? Every untagged commit to the master branch is deployed to a staging bucket instead, that is accessible by the team only.


Lesson learned: if you are not** editing your whole data suite** frequently, and you are the only deterministic source of change (i.e. no arbitrary Internet user is affecting your website state over time - see comments, placing orders, rating items…), it is completely acceptable to store it in static files!

We plan to release both Waffel, the generator tool, and Vaneella, the content manager app, as soon as they reach a stable implementation status. In the meantime, we hope you enjoyed the read and you consider going static!