Stateless Notes

I made my stateless, Raspberry Pi hosted notes service have an in-memory SQLite for more features and better performance.

As I alluded to last week I was able to effectively merge notes and zine over the past few days. Here’s a quick rundown of how they differed before:

  • notes: runs on a Raspberry Pi and renders markdown, synchronously loading files from Dropbox as part of rendering the page. Thanks to tailscale it’s basically always available to me.

  • zine: I’ve blogged about this before. Runs on my laptop and renders markdown to static files. Builds up an in-memory SQLite database both to allow slicing and dicing the notes but also to allow notes to use SQL to refer to other notes.

zine is great but since I don’t have it on a server it tends to only be accessible from my laptop. That’s a hassle.

Now I’ve fixed that. Here’s an overview of how the system works now:

  1. At startup the notes tool loads all (~250) of the notes from dropbox in parallel. This takes about 6 seconds on my Raspberry Pi.

  2. After that’s done it inserts the notes into an in-memory SQLite database. This takes less than 100 milliseconds.

  3. (Nearly) all interactions with the tool use the database, not dropbox. This means the latency is hidden and pages can be much richer, using the database to render extra information.

  4. To keep the database up to date, I have a longpoll against the Dropbox API. When files change I reload them from dropbox and modify the database appropriately.

  5. After the database changes are committed, I close a channel, which triggers a message to be sent to all open pages, which causes all open pages to be refreshed, so no browsers should show the old data.

I feel pretty good about it! It’s one or two orders of magnitude more responsive and I enjoy the richer pages.

I am a little nervous that this system is inherently brittle because it will hide errors behind a background task. I have a couple of ideas to make that less risky:

  1. Completely reload the database every now and then, and track “unexpected” changes, so I can debug issues.

  2. When things go wrong, increment a counter and render it within the UI. I am hoping this will help me notice when things are broken.

We’ll see how it goes.

(Affiliate links below.)

Recently Brendan Gregg’s Systems Performance got its second edition released. He wrote about it here. I am hoping to get a copy myself soon. I loved the first edition and think the second will be even more useful.

At the end of 2019 I read BPF Performance Tools. It was one of my favorite tech books I read in the past five years. Not only did I learn how to (almost) trivially see deeply inside of how my computer is working, but I learned how that works via the excellent detail Gregg added in each chapter. Amazing stuff.

Posted Sat, Jan 30, 2021

If you're interested in being notified when new posts are published, you can subscribe here; you'll get an email once a week at the most.