Automating GitHub release processes
In this section we explain how we automated the release process for two of our public GitHub projects written in Golang by leveraging GitHub actions, the awesome goreleaser and the missing package manager for Mac: brew.
Ok let’s gets started. We chose our two popular repositories awsu (Enhanced account switching for AWS, supports Yubikey as MFA source) and envplate (Docker-friendly trivial templating for configuration files using environment keys) as both tools are written in Golang and still uses travis-ci as CI. Furthermore, some parts of the release process we still do manually, such as uploading the assets to a GitHub release and generating the release notes and we wanted to have this automated as well.
The goals
We started off by defining our goals for the session:
- use the latest version of Golang for building, which is 1.17 at the time of writing this post. This implied
- bumping the version of Golang used internally
- and also removing deprecated package managers, such as godep and/or govendor.
- move from travis to GitHub actions
- provide a suitable config for goreleaser and use it for releasing. Fortunately there is already a GitHub action for this tool 🚀
- generate a homebrew formula pointing to a new release and update our homebrew-taps automatically 🍺 🤖
What we achieved
After we consolidated the knowledge of each person in the mentioned projects, the language Golang, and the technologies Docker and GitHub actions, we decided first to go bump both projects to the latest Golang 1.17. This went very smooth and straightforward. For the repository envplate we needed to update from the external package manager govendor. The next step, moving to GitHub Action was also very straightforward as we used our internal template as a base. Furthermore, we decided to use Goreleaser as well for the build step, to match the release step and to catch bugs on building the binaries very early in the CI process.
name: build
on:
push:
branches:
- master
pull_request:
types: ['opened', 'synchronize']
paths:
- '**.go'
- go.mod
- '.github/workflows/**'
jobs:
binaries:
runs-on: ubuntu-20.04
steps:
- name: Checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
- name: Set up Go
uses: actions/setup-go@v2
with:
go-version: 1.17
- name: Run GoReleaser
uses: goreleaser/goreleaser-action@v2
with:
version: latest
args: release --rm-dist --skip-publish --snapshot --skip-sign --debug
The most tricky part was awsu, as it needs libraries to be installed on the system. We realized that before the binaries were built on a Mac (with an Intel chip), one native and the other in a docker environment with a debian stretch image. As GitHub Actions run on an ubuntu runner, we decided to reproduce the build locally in this environment. Finally, we noticed that we it makes the most sense to build in a Mac environment and build the Linux binary manually and upload it later to the release. The caveat is that also the the awsu.rb brew formula has to be updated as well manually.
As a last step, for automatically updating our homebrew formulas, e.g. for awsu, we used a token from our technical user, the kreuzwerkerbot, which has access to all repositories 🎉
# typed: false
# frozen_string_literal: true
# This file was generated by GoReleaser. DO NOT EDIT.
class Awsu < Formula
desc "It provides a convenient integration of AWS virtual MFA devices into commandline based workflows."
homepage "https://github.com/kreuzwerker/awsu"
version "2.3.3"
depends_on :macos
on_macos do
if Hardware::CPU.arm?
url "https://there-is-no-release-yet-but-to-make-brew-not-fail-on-the-tap.com"
end
if Hardware::CPU.intel?
url "https://github.com/kreuzwerker/awsu/releases/download/v2.3.3/awsu_2.3.3_Darwin_x86_64.tar.gz"
sha256 "111c9c6934d4f745a82cf2ed04596487158df8648b28d0089aa99b815177668a"
end
end
on_linux do
if Hardware::CPU.intel?
url "https://github.com/kreuzwerker/awsu/releases/download/v2.3.3/awsu_2.3.3_Linux_x86_64.tar.gz"
sha256 "e8cfec1e9bc05a81fe9f15dee15f0b51af97da60c79340bcb1574cee84b63753"
end
end
def install
bin.install "awsu"
end
end
Next steps
- For envplate we will continue reviewing PRs and see which issues are still present.
- For awsu we still have an open issue to integrate the release of the linux/amd64 into the automatic release where we will leverage the usage of pre-built feature of goreleaser pro.
And this feature we implemented shortly after the Hacktoberfest, however it is very nice reusable detail for folks out there, we will incorporate it in this blog post. As mentioned we wanted to avoid updating the brew formula manually, so we decided to build the binaries for linux and mac in a single step and add them all to one release. Here is the final descriptions of the action:
name: goreleaser
on:
push:
tags:
- '*'
jobs:
binaries:
runs-on: macos-11
steps:
- name: Checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
- name: Set up Go
uses: actions/setup-go@v2
with:
go-version: 1.17
- name: Updating and upgrading brew
run: |
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
brew --version
- name: Install and start docker
# https://github.com/docker/for-mac/issues/2359#issuecomment-943131345
run: |
brew install --cask docker
sudo /Applications/Docker.app/Contents/MacOS/Docker --unattended --install-privileged-components
open -a /Applications/Docker.app --args --unattended --accept-license
echo "Waiting for docker to be up"
while ! /Applications/Docker.app/Contents/Resources/bin/docker info &>/dev/null; do echo -n "."; sleep 1; done
- name: Build linux binary
run: |
make build/awsu-linux-amd64
# as it is the format goreleaser expects. See goreleaser.yaml -> prebuilt -> path
cp build/awsu-linux-amd64 build/awsu_linux_amd64
- name: Run GoReleaser
uses: goreleaser/goreleaser-action@v2
with:
distribution: goreleaser-pro
version: latest
args: release --rm-dist
env:
GORELEASER_KEY: ${{ secrets.GORELEASER_KEY }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GORELEASER_TOKEN: ${{ secrets.GORELEASER_TOKEN }}
We accomplished our goal by 🎉
- running on a macos-11 GitHub actions runner (see line 10)
- installing Docker automatically, as it does not come out-of the box, as for example for ubuntu or windows runners (see line 29)
- and using the goreleaser-pro (see line 40) feature of prebuilt binaries
before:
hooks:
- go mod tidy
builds:
- id: "awsu-darwin"
env:
- CGO_ENABLED=1
goos:
- darwin
goarch:
- amd64
- id: "awsu-linux"
builder: prebuilt
goos:
- linux
goarch:
- amd64
prebuilt:
path: build/awsu_{{ .Os }}_{{ .Arch }}
Summary
With this project we not only learned more about building static Golang binaries that are linked to C libraries, but also how the devil can be in the details 😈 Also, we are happy that there are environments to build to Mac. Interteam-wise, we leveraged more and more the teamwork of DevOps Engineers and Fullstack developers, which we will further extend in the future. In order to make contribution and pull requests more clear and easier to follow, we also added templates for both.
Deploying Datadog Resources with CDK
We wanted to implement “something with CDK and Datadog” that we can directly apply in the base case in one of our projects. We are big fans of both CDK and Datadog and realized that it is not possible to create all Datadog resources as code. Thus: Perfect Match!
There were already two repositories on GitHub:
- https://github.com/blimmer/cdk-datadog-integration a CDK construct that implements just the AWS↔Datadog integration
- https://github.com/NomadBlacky/cdk-datadog-resources a collection of CDK constructs for various Datadog resources
The goal
We decided to go with the latter because a construct for the Datadog integration was missing and with this construct in place, we would be able to provision a complete infrastructure with Datadog monitoring in one go.
What we achieved
Before we could jump into coding we had to prepare our AWS account:
cdk-datadog-resources uses Datadogs Cloudformation extensions, therefore we had to import and activate these. We found the aws cli to be the least annoying tool to accomplish that:
aws cloudformation register-type \
--region "us-east-1" \
--type RESOURCE \
--type-name "Datadog::Integrations::AWS" \
--schema-handler-package "s3://datadog-cloudformation-resources/datadog-integrations-aws/datadog-integrations-aws-1.1.0.zip"
The implementation itself was straightforward. It is based on the fact that Cloudformation extensions are normal Cloudformation resources.
class DatadogIntegration {
constructor(scope: Construct, id: string, props: DatadogIntegrationProps) {
new CfnResource(scope, id, {
type: 'Datadog::Integrations::AWS',
properties: {
...
},
});
}
}
The tricky part was that we initially installed the current version of Datadog::Integrations::AWS. Our examples from the existing constructs used older versions and Datadog updated the schema for their extensions recently. After downgrading to an older version, we were able to provision a AWS↔ Datadog Integration. The PR was accepted by the owner and released.
Next steps
Currently, we do not pass the optional properties for service selection, instance filters, etc. to the underlying component. We also would have to update the documentation then. We may consider updating the existing resources to the current schemata for the future.
Summary
With this project we learned not only how to use custom Cloudformation extension with CDK, but also about the structure of these extensions. And for me as a new kreuzwerker, it was a great experience to join forces with my colleagues.