Back to blog

A smoke check at the end of every deploy

A short bash script now checks every public page after each deploy. On the day I added it, it failed a deploy over a capital B.

• 7 min read •
self-hosting testing automation bash

This site is a Rust program on a small rented server. When I deploy, a bash script on the server pulls the latest code, builds a release binary, swaps it in and restarts the service. Then it waits for an address called /health/ready to answer. That route runs one query against the database and replies ready or not ready. If it hasn't said ready within 30 seconds, the script puts the previous binary back.

That catches a crash on start, or a database the program can't reach. It says nothing about the pages. All of these would pass readiness: a template that errors, a stylesheet the layout links but the server can't find, a security header that went missing, or a footer still naming the old build. Security headers are the extra response lines that tell a browser to be stricter. Since the site went live in February, the only cover for that gap was running curl by hand after each deploy.

On 25 September I added a last step to the deploy, a second script called deploy/smoke.sh. A smoke check is a quick, shallow test for anything plainly broken. The core of this one is a small function called expect, which takes an address and what should come back: a status code such as 200, and optionally the start of a content type such as text/html. It asks curl for both and prints ok or FAIL. Most of the script is lines like these:

for path in / /projects /about /blog /guides /contact /privacy /cv /security /now /uses /login; do
  expect "$path" 200 text/html
done
expect /cv/download 200 application/pdf

After the fixed pages it reads the sitemap, the XML file that lists the site's pages for search engines, and checks every blog post and guide it finds there. So a new post gets covered without anyone editing the script. Other lines cover the feed, security.txt, robots.txt and the public PGP key. The script doesn't stop at the first problem. It counts failures as it goes and exits with an error at the end if there were any, which fails the deploy too.

The awkward part was my own visitor log. The site keeps a small record of visits that I read on an admin page, and it counts GET requests for public pages that come back as successful HTML. The first version of the check used plain GET. On my development database, each run added a dozen visits from 127.0.0.1, the machine visiting itself. That version never reached the live server.

So the pages go through HEAD instead, which asks for the headers without the body. The status code and content type both live in the headers, so expect has all it needs. The tracker only counts GET, so none of these requests reach the log. The page routes already answered HEAD, because I had added that earlier in September for uptime monitors.

Two checks do need a real page body, though. One wants the list of stylesheets and scripts the layout links, so it can request each of them. The other reads the footer. For both, the script asks for a page that doesn't exist. The 404 page uses the same layout as every other page, with the same links and the same footer. The tracker skips anything that isn't a success, so the script gets a full page and nothing lands in the log. I think this is the neatest part of the script.

There was a small curl trap on the way, because the obvious curl -X HEAD hung. -X only changes the method word curl sends. It doesn't change how curl behaves, so curl still sits waiting for a body that never comes. The curl manual says exactly this and points to --head instead. Every request also got a 10 second timeout, so a stuck one fails rather than holding up the deploy.

A couple of hours later I made /cv answer terminals differently. If you run curl https://ericjingryd.com/cv, you get the CV as plain text instead of a page of HTML. The server looks at the User-Agent header, and curl puts its own name there. The smoke check is curl too, so it would have been handed the text and failed its own HTML check on /cv. Now it sends smoke.sh as its User-Agent. One extra line asks for /cv as curl on purpose, so the text version gets checked as well:

AGENT=curl/8 expect /cv 200 text/plain

Then the footer. Every page has a small status bar at the bottom, and one part of it reads Build followed by the short commit hash the binary was built from. The deploy passes the hash it just built to the smoke check, and the check looks for it in the 404 page. If the old binary were somehow still the one answering, this is where it would show.

Getting that hash to the check needed a small bash change. The build step used to look like this:

BUILD_COMMIT=$(git rev-parse --short HEAD) cargo build --release --locked

Written that way, the variable only exists in cargo's environment, so once cargo finishes the script has nothing to pass on. Now it is set as an ordinary variable and exported before the build runs.

Later the same day, the footer check failed a deploy. The site was fine. I had changed the status bar labels to start with a capital letter, so build became Build. The script was still searching for the lowercase word. The label and the script sat in different files, and nothing tied them together. The fix was one letter in smoke.sh, and my roadmap now has a note that any change to that footer text needs the same change in the script.

When readiness fails, the deploy script restores the previous binary. When the smoke check fails, it doesn't. By then the new version is up, and on start it runs any new database migrations, the SQL files that change the database's layout. The old binary checks the migrations too. If the database has one it doesn't know about, it refuses to start. So a rollback at that point could swap a partly broken site for a dead one. The deploy exits with an error instead, and the fix goes forward in a new commit. I think that's the right call for a site this size, though it means the check tells me about a broken page after visitors can already reach it.

One quirk comes from the pair of braces the whole of deploy.sh sits inside. Bash reads a braced block completely before running any of it, which is what lets git pull replace the file halfway through a run without trouble. The catch is that the deploy which pulls a new deploy.sh still runs the old one, so the first deploy after I added the smoke check skipped it. The one after ran it.

What it can't do is look at a page. A stylesheet that loads fine but lacks a class a template needs still gets ok. It also checks only three security headers, and only on the home page. So my testing notes still keep a short list of things to look at by eye after a visual change.

You can also run it from any machine against the live site, and a short commit hash as the second argument adds the footer check:

bash deploy/smoke.sh https://ericjingryd.com

If you write one yourself, ShellCheck is worth running over it, and both of my deploy scripts pass it without a warning. Google's shell style guide is a good read too, on when a shell script is the right tool at all.

The next day I added the new /guides pages to its list. The deploy that put them live checked them along with everything else, and passed.

Sources and further reading: