Lately we have been rebrushing our homepage here at kreuzwerker, and we are quite happy with that! I am mostly happy with how things are going under the hood, because basically nothing is happening.
Just a bunch of .html
files is being served, with some CloudFront topping for speeding things up (and for pleasing our AWS guru Joern!).
Static website generation from a determined data set (mostly Markdown documents) is nothing new, and has been implemented over all possible platform and programming languages. But trust me when I tell you, that no system was completely able to fullfil our needs, because most of them are very good for blog management, but not flexible enough to handle a more complex data graph.
On top of that, we needed good localisation support for document translation at URL level, another feature that at least at implementation time I could not find.
What then? Well, we are developers, we implemented our custom solution!
Et voilá, Waffel!
Waffel is my tiny library, whose work is still in progress, able to convert a set of .markdown
/YAML front-mattered documents to a hypertext, via a schema.yml
configuration file plus a set of Nunjucks HTML templates.
Let our data structure be something like:
/data
├── images
│ ├── blog
│ │ ├── ...
│ └── users
│ ├── andreas-profous.jpg
│ ├── ...
│ └── ulrike-koenig.jpg
├── pages
│ ├── de
│ │ ├── about.md
│ │ └── ...
│ ├── en
│ │ ├── about.md
│ │ └── ...
│ └── pl
├── partners
│ ├── de
│ │ ├── amazon-web-services.md
│ │ └── ...
│ └── en
│ ├── amazon-web-services.md
│ └── ...
├── posts
│ ├── Create-train-and-test-your-devops-skills-with-the-simian-army.md
│ ├── ...
│ └── why-clojure-rocks-2.md
├── ...
└── users
├── de
│ ├── andreas-profous.md
│ ├── ...
│ └── ulrike-koenig.md
└── en
├── andreas-profous.md
├── ...
└── ulrike-koenig.md
The structure is then defined as:
config:
dateFormat: YYYY.MM.DD
google_analytics: UA-XXXXXXX-1
...
structure:
home:
template: home
url: /
about:
template: about
url: /about
team:
collection: users
pages:
single:
template: team/member
url: /about/team/:slug
priority: 0.6
filter:
published: true
...
blog:
collection: posts
pages:
index:
template: blog/index
url: /blog
paginate: 10
priority: 0.5
sort:
field: date
order: desc
tags:
template: blog/tag
url: /blog/tag/:tag
groupBy: tags
paginate: 10
changefreq: weekly
priority: 0.7
sort:
field: date
order: desc
single:
template: blog/post
url: /blog/posts/:slug
priority: 0.8
...
Never mind the library DSL for a second, and focus on the team
definition: it iterates over the users
collection (namely from /data/users/**/*.md
), passing every entry to the team/member
template, and saving the output to /about/team/{{user.slug}}.html
.
Same thing happens for all other sections.
In what does Waffel differ from the other zillion generators?
First, it is still on an early development stage, being thus sadly almost undocumented/untested yet. Among the positive differences:
It is a library, more than a CLI tool.
You just need to require('Waffel')
, create an instance, and enjoy the full control on what you are doing, be it via Gulp/Grunt/Brunch, or your custom solution.
It handles multiple languages like a charm.
In the data structure above, you noticed the /de
and /en
subfolders of the /user
folder; if we instantiate Waffel like this:
wfl = new Waffel
languages: ['de', 'en']
defaultLanguage: 'de'
it will be aware that data is stored in both English and German, and that the default language will be the latter. English pages will be prefixed with /en
, and the Queen will be happy.
It does not care about js/css concateminification.
Why do I find this a good thing? Everybody has his or her own workflow in order to deal with asset delivery, and I think that it should not be responsibility of a generator, to care about.
For instance I have always been a big fan of Brunch, so I created a skeleton that takes care of everything. But it would be easy to write a simple Grulpfile
that does same stuff!
You can have preliminary build steps.
Being Waffel a simple library, that’s trivial: you can do stuff before, then pass the data to the generator and do stuff with it, for instance:
var Waffel = require('Waffel')
var DataProvider = require('./lib/DataProvider')
// We do the Waffel stuff here.
var port = 3333
var wfl = new Waffel({
domain: "http://localhost:" + port,
serverConfig: {
port: port,
path: 'public',
indexPath: 'public/404.html'
}
})
// DataProvider is some custom library.
// Let's assume that the async call to provider.fetch returns an array of some quotes by Bill Hicks:
var provider = new DataProvider({
url: http://path.to/some/service,
query: { author: 'Bill Hicks' }
})
// once both the provider and Waffel are done, let's do stuff!
Promise.all([provider.fetch, wfl.init])
.then(function(results){
wfl.generate({
data: {
quotes: results[0]
}
})
})
This way you may either skip or complement the local filesystem read, and generate your document from an arbitrary dataset, which I find pretty useful.
What if I like clicking on my mouse better than scripting?
First of all, you have all my respect. That’s why we worked on an Electron app for managing your precious /data
folder, looking like this at the moment:
And see this very blog post being written live!
Worth noting: Waffel is completely agnostic about the way you manage data. In fact it does not care at all, as long as files belong to a format that it can understand.
Cool, but how does your editing tool understand how to parse and display the fields? Is there any black magic happening somewhere?
No, you have to place a simple schema configuration file in the root of the /data
folder, that is the only price to pay. As the app is in its early lifecycle, documentation is still lacking, so take this as a mere (hopefully self-explaining) preview:
users:
label: Team
icon: fa-user
description: 'Our beautiful team.'
labelPattern: "#{name}"
defaultView: grid
languages: ['de', 'en', 'pl']
fields:
picture:
type: image
label: Photo
filterable: false
align: center
validation:
required: false
name:
type: text
label: Full Name
sort:
main: true
sortable: true
slug:
type: slug
label: Slug
filterable: false
display:
table: false
email:
type: email
label: E-Mail
sort:
sortable: true
published:
type: boolean
label: Published
filterable: false
align: center
...
...
posts:
label: Posts
icon: fa-file-text-o
description: What we love to talk about.
labelPattern: "<strong>#{author}</strong><br>#{title}"
detailLabelPattern: "#{title}"
main: true
assetPath: /images/blog
fields:
cover:
type: image
label: Cover
filterable: false
align: center
validation:
required: false
title:
type: text
label: Title
sort:
sortable: true
slug:
type: slug
label: Slug
filterable: false
display:
table: false
author:
type: select
relation:
type: collection
from: users
label: Author
sort:
sortable: true
tags:
type: tags
label: Tags
date:
type: date
label: Date
sort:
sortable: true
main: true
order: desc
published:
type: boolean
label: Published
filterable: false
align: center
Every collection, in this case users
and posts
, has many fields of different type (text
, slug
, email
, boolean
, image
, …), and some basic relationships as well (see select
from users
as the author
field of the posts). Without going any deeper, this constraints are used by the app in order to build the forms and the list tables, and to validate user input.
At the moment, you have to write the schema by hand. In the future, we plan to have a simple graphic tool to set it up, and possibly a basic inference mechanism that learns the schema from already existing data. We’ll keep you posted!
Cool, but what about more people willing to collaborate, if files are stored on one single machine?
Let me think, is there any way to edit a distributed set of files, tracking the name of who is editing what? Damn, if I could ever recall…
Putting content under version control was the natural choice of course!
We thought that granting the app built in support for that, would have been of great use for non command line fellows:
Ok, what about going live?
We are hosting the repository on Github, so we have a Travis task that on every new tagged commit takes care to generate the website, and to sync the output to a S3 bucket. As already mentioned, CloudFront sits on the very top of everything granting us super fast performance.
What about preview and testing? Every untagged commit to the master
branch is deployed to a staging bucket instead, that is accessible by the team only.
Lesson learned: if you are not** editing your whole data suite** frequently, and you are the only deterministic source of change (i.e. no arbitrary Internet user is affecting your website state over time - see comments, placing orders, rating items…), it is completely acceptable to store it in static files!
We plan to release both Waffel, the generator tool, and Vaneella, the content manager app, as soon as they reach a stable implementation status. In the meantime, we hope you enjoyed the read and you consider going static!