A Love Letter to Plain Text
I have used Hugo, the blog engine this blog runs on top of, more and more lately for less and less typical use cases. Hopefully this post will inspire others in similar ways.
There was another post on twitter recently that inspired me to write this post. The point of that post was that when your blog is just a pile of textfiles generic Unix tools combine to make many things are trivial that wouldn’t be with a more traditional database backed system.
Some of the examples given in the post were using wc(1) to count words in a
post and aspell(1) to spellcheck posts. I’m all about that, but I have a tool
in my back pocket that allows more advanced analysis.
🔗 q: A Refresher
I posted about q before, but I’ll give a quick refresher here.
q is a 120 line perl script that parses all of my blog posts, puts the
metadata in an in-memory database, and thus allows querying the metadata with
SQL. Here’s an example invocation:
bin/q --sql 'SELECT COUNT(*) FROM _ WHERE tag = ?' perl
243
The _ thing is a view with the tags joined with the articles for brevity and
convenience. I use q very often for a chronological view of my posts in the
vim quickfix; see the original post for details on that. I want to discuss
some of the other doors that q has opened here.
By the way, the above invocation takes about 53 milliseconds to run, so q is
actually fast enough that it feels realtime and I use it in an autocompleter
(see the original post for details).
🔗 check-guids
For RSS feeds each post needs a unique id. By default the id is the url of the
post. Often you’ll notice that people will switch from http to https and their
RSS feed will list all posts ever as if they are new. For this reason I always
explicitly list an id. I used to have it match the post url but that led to
problems when they didn’t match. RSS actually assumes that the id is the url
unless you specify otherwise (with isPermaLink="false" in the id field.)
When I write a post I have a template that generates a fresh GUID each time. There was a time though when I would copy paste a prior post to build a new post. Once or twice I accidentally forgot to regenerate the GUID and RSS feeds never saw the new post. I resolved this by adding the following little script to detect any duplicate GUIDs and run it in my makefile before building the blog.
#!/bin/sh
duplicates="$(bin/q --sql \
'SELECT filename
FROM articles
WHERE guid IN (
SELECT guid
FROM articles
GROUP BY guid
HAVING COUNT(*) > 1
)' --formatter='" * $r{filename}"'
)"
if [ -n "$duplicates" ]; then
echo "Duplicate GUIDs found!\n$duplicates"
exit 1
fi🔗 Clog
I started roasting my own coffee a year ago and have been tracking details related to the roasts for about six months. Initially I tried to keep it simple and put the notes in a spreadsheet. The spreadsheet interface was terrible; it was incredibly slow and bloated, and hard to see a roast’s details at a glance. Someone maybe jokingly mentioned that I should make a blog. I liked the idea and my coffee log is working well. Big thanks to my wife for recommending “clog” as a name as opposed to my initial idea: “covfefe.”
One of the benefits of using Hugo is the ability to have arbitrary structured information for easy per roast data:
---
title: Ethiopia Grade 1 Dry Process Yirga Cheffe Dumerso
date: 2017-12-16T09:53:39
tags: [ ethiopia, grade-1, dry-process, yirga-cheffe, dumerso ]
guid: f98e617f-a62e-4b1b-9513-ccb42816ef73
total_roast: 12m2s
first_crack: 9m17s
start_weight: 1.2
end_weight: 1.016
weight_loss: 15
--- 🔗 check-math
Unfortunately, Hugo uses Go’s templating, which is, to put it nicely, simple.
So while it can do some math, it cannot calculate the weight loss above. I
resolved this issue by adding a check-math script that gets run
by the Makefile:
#!/bin/sh
errors="$(bin/q --sql \
"SELECT filename, weight_loss as rep,
round(100 * ( 100*start_weight - 100*end_weight ) / ( 100*start_weight ))
as calc
FROM articles
WHERE start_weight IS NOT NULL AND
round(100 * ( 100*start_weight - 100*end_weight ) / ( 100*start_weight)) !=
round(weight_loss)" \
--formatter '"$r{filename} says $r{rep}; should be $r{calc}"')"
if [ -n "$errors" ]; then
echo "$errors"
exit 1
fiIt’s a little weird because of annoying floating point issues, and honestly this script is a hack, but it was far easier than submitting a patch to add advanced math support to the templating engine (and likely have it rejected.)
🔗 Notes
For a long time I have tracked various facts, aspirations, and plans in a couple basic text files. I blogged about the format a while ago. I recently started taking a class on coursera and wished I could take notes as in depth as I take for coffee. I was inspired to convert my notes to a similarly plaintext page based format instead of a line based format.
🔗 public-build
In my notes I have data that must remain private, but I also have a non-trivial number of public pages that would be nice to see without any authentication; here’s an aspirational example. I wrote the following script to build a public view of the site:
#!/bin/sh
set -e
PATH="$PATH:$(pwd)/bin"
tmpdir="$(mktemp -d --tmpdir public-notes.XXXXXXXXXX)"
final_dir="$(pwd)/public/"
# 1. copy infrastructure to tmpdir
cp -r ./static/ ./layouts/ ./config.yaml "$tmpdir"
# 2. copy posts tagged public to tmpdir
mkdir -p "$tmpdir/content/posts"
q --sql 'SELECT filename FROM articles a WHERE
EXISTS (SELECT 1 FROM article_tag at WHERE a.guid = at.guid AND tag = ?)
AND NOT
EXISTS (SELECT 1 FROM article_tag at WHERE a.guid = at.guid AND tag = ?)
' public private |
xargs -n50 -I{} cp {} "$tmpdir/content/posts"
# 3. build
cd $tmpdir
hugo
check-private 'public/'
# 4. sync
rm -rf "$final_dir/"*
cp -r public/* "$final_dir/"
cd ..
rm -rf "$tmpdir"The use of q is pretty great here; as otherwise I’d be stuck at grepping for
words.
🔗 check-private
I am paranoid that there will be a bug in the above code on
accident, one way or another, so I wrote another script (check-private, run in
the above) that checks that a number of canary posts (posts tagged private, not
tagged public, tagged both) are not included in the built corpus:
#!/bin/sh
root="$1"
failed_build() {
echo "Canary got into the build ($1); something is wrong"
echo "Current build is at $tmpdir"
exit 1
}
test -e ${root}posts/private-canary &&
failed_build 'expected location, private'
git grep --no-index -q PRIVATECANARY &&
failed_build 'unexpected location, private'
test -e ${root}posts/untagged-canary &&
failed_build 'expected location, untagged'
git grep --no-index -q UNTAGGEDCANARY &&
failed_build 'unexpected location, untagged'
test -e ${root}posts/both-canary &&
failed_build 'expected location, public&private'
git grep --no-index -q BOTHCANARY &&
failed_build 'unexpected location, public&private'
exit 0The above runs before the build, before the commit, and before the push; so I think it’s sufficiently paranoid to keep my stuff safe.
🔗 plaintext
The original post mentioned spell checking, word counting, etc. I have enough large code blocks in my posts that I can’t sensibly send a whole file to one of those programs so I have a tool that will convert the data + markdown to plain text for this purpose, mostly. It’s slow (5.2s to convert my entire corpus,) and a little buggy but it’s a start. I may implement my own markdown parser specifically for improved speed. I haven’t yet because I hate implementing parsers. Here’s the code:
#!/usr/bin/env perl
use 5.20.1;
use warnings;
package MarkdownToText;
use Moo;
extends 'Markdent::Handler::HTMLStream::Fragment';
our $printing = 1;
sub start_code { $printing = 0 }
sub end_code { $printing = 1 }
# disabled events
sub code_block {}
sub preformatted {}
# disable html stuf
sub _stream_start_tag {}
sub _stream_end_tag {}
# disable encoding of html entities, only print when not in code block
sub _stream_text { shift->_output->print(shift) if $printing }
package main;
use autodie;
use Markdent::Parser;
die "usage: $0 \$mdwn [\$md2 \$md3 ...]\n"
unless @ARGV;
while (my $markdown = do { local $/; <<>> }) {
$markdown =~ s/---\n.*?---\n//s;
my $out = '';
open my $fh, '>', \$out;
my $parser = Markdent::Parser->new(
dialects => 'GitHub',
handler => MarkdownToText->new( output => $fh ),
);
$parser->parse( markdown => $markdown );
print $out;
}I’d like to build more on top of this plaintext tool (like detecting accidental duplicate words, grammatical mistakes, etc) but it’s too slow to be added to the build process, so I’m torn.
On top of that, the amount of words that aspell(1) calls misspelled is
crippling. There are some words that are made up for single posts, and I can
either globally whitelist the word, or I can change the post. Currently
counting all of the words in my corpus that aspell(1) complains about shows a
whopping 2,353 unique words. Joy.
I hope that this inspires others. These are all tools that could be written against a traditional blog engine, but would likely be slower and less flexible. On top of the above handmade features, there are all kinds of reasons to use a plaintext format: trivial version control, less dependence on the state of a pet server, trivial backups, relatively simple migrations, and more.
I am at a loss to recommend just the right book to learn more about these topics. I have some vague recommendations but I also think that the book I want to recommend just does not exist.
(The following includes affiliate links.)
I think every software engineer, especially newer ones, should read
The Pragmatic Programmer
Another book in the same vein, which I read many years ago, is
The Passionate Programmer
If you're interested in being notified when new posts are published, you can subscribe here; you'll get an email once a week at the most.