Investigation: Why Can't Perl Read From TMPDIR?

On Wednesday afternoon my esteemed colleague Mark Jason Dominus (who already blogged this very story, but from his perspective), showed me that he had run into a weird issue. Here was how it manifested:

$ export TMPDIR='/mnt/tmp'
$ env | grep TMPDIR
$ /usr/bin/perl -le 'print $ENV{TMPDIR}'

So to be clear, nothing was printed by Perl.

Posted Thu, Jun 30, 2016

Reap slow and bloated plack workers

As mentioned before at ZipRecruiter we are trying to scale our system. Here are a couple ways we are trying to ensure we maintain good performance:

  1. Add timeouts to everything
  2. Have as many workers as possible

Posted Wed, Jun 29, 2016

AWS Retirement Notification Bot

If you use AWS a lot you will be familiar with the “AWS Retirement Notification” emails. At ZipRecruiter, when we send our many emails, we spin up tens of servers in the middle of the night. There was a period for a week or two where I’d wake up to one or two notifications each morning. Thankfully those servers are totally ephemeral. By the time anyone even noticed the notification the server was completely gone. Before I go further, here’s an example of the beginning of that email (the rest is static:)

Posted Wed, Jun 22, 2016

Vim: Goto File

Vim has an awesome feature that I think is not shown off enough. It’s pretty easy to use and configure, but thankfully many languages have a sensible configuration out of the box.

Posted Tue, Jun 21, 2016

Staring into the Void

Monday of this week either Gmail or OfflineIMAP had a super rare transient bug and duplicated all of the emails in my inbox, twice. I had three copies of every email! It was annoying, but I figured it would be pretty easy to fix with a simple Perl script. I was right; here’s how I did it:

Posted Thu, Jun 16, 2016

Vim Session Workflow

Nearly a year ago I started using a new vim workflow leveraging sessions. I’m very pleased with it and would love to share it with anyone who is interested.

Posted Thu, Jun 9, 2016

DBI Caller Info

At ZipRecruiter we have a system for appending metadata to queries generated by DBIx::Class. About a month ago I posted about bolting timeouts onto MySQL and in the referenced code I mentioned parsing said metadata. We are depending on that metadata more and more to set accurate timeouts on certain page types.

Posted Wed, Jun 8, 2016

My Custom Keyboard

A few years ago I made my own keyboard, specifically an ErgoDox. I’ve been very pleased with it in general and I have finally decided to write about it.

Posted Sat, Jun 4, 2016


A big trend lately has been the rise of “serverless” software. I’m not sure I’m the best person to define that term, but my use of the term generally revolves around avoiding a virtual machine (or a real machine I guess.) I have a server on Linode that I’ve been slowly removing services from in an effort to get more “serverless.”

Posted Wed, Jun 1, 2016

Iterating over Chunks of a Diff in Vim

Every now and then at work I’ll make broad, sweeping changes in the codebase. The one I did recently was replacing all instances of print STDERR "foo\n" with warn "foo\n". There were about 160 instances in all that I changed. After discussing more with my boss, we discussed that instead of blindly replacing all those print statements with warns (which, for those who don’t know, are easier to intercept and log) we should just log to the right log level.

Posted Wed, May 25, 2016

OSCON 2016

ZipRecruiter, where I work, generously pays for each engineer to go at least one conference a year. I have gone to YAPC every year since 2009 and would not skip it, except my wife is pregnant with our second child and will be due much too close to this year’s YAPC (or should I say instead: The Perl Conference?) for me to go.

Posted Fri, May 20, 2016

Faster DBI Profiling

Nearly two months ago I blogged about how to do profiling with DBI, which of course was about the same time we did this at work.

Posted Wed, May 18, 2016

Setting up Let's Encrypt and Piwik

Late last week I decided that I wanted to set up Piwik on my blog. I’ll go into how to do that later in the post, but first I ran into a frustraing snag: I needed another TLS certificate. Normally I use StartSSL, because I’ve used them in the past, and I actually started to attempt to go down the path of getting another certificate through them this time, but I ran into technical difficulties that aren’t interesting enough to go into.

Posted Sat, May 14, 2016

Rage Inducing Bugs

I have run into a lot of bugs lately. Maybe it’s actually a normal amount, but these bugs, especially taken together, have caused me quite a bit of rage. Writing is an outlet for me and at the very least you can all enjoy the show, so here goes!

Posted Tue, May 10, 2016

Putting MySQL in Timeout

At work we are working hard to scale our service to serve more users and have fewer outages. Exciting times! One of the main problems we’ve had since I arrived is that MySQL 5.6 doesn’t really support query timeouts. It has stall timeouts, but if a query takes too long there’s not a great way to cancel it. I worked on resolving this a few months ago and was disapointed that I couldn’t seem to come up with a good solution that was simple enough to not scare me.

Posted Sun, May 8, 2016

A new Join Prune in DBIx::Class

At work a coworker and I recently went on a rampage cleaning up our git branches. Part of that means I need to clean up my own small pile of unmerged work. One of those branches is an unmerged change to our subclass of the DBIx::Class Storage Layer to add a new kind of join prune. If you didn’t know, good databases can avoid doing joins at all by looking at the query and seeing where (or if) the joined in table was used at all.

Posted Fri, Apr 29, 2016

Python: Taking the Good with the Bad

For the past few months I’ve been working on a side project using Python. I’ll post about that project some other time, but now that I’ve used Python a little bit I think I can more reasonably consider it (so not just “meaningful whitespace?!?“) It’s much too easy to write a bunch of stuff that is merely justification of the status quo (in my case that is the use of Perl.

Posted Thu, Apr 21, 2016

Humane Interfaces

In this post I just want to briefly discuss and demonstrate a humane user interface that I invented at work. At ZipRecruiter, where I work, we use a third party system called Each employee is given $20 in the form of 100 Zip Points at the beginning of each month. These points can be given to any other employee for any reason, and then redeemed for gift cards basically anywhere (Amazon, Starbucks, Steam, REI, and even as cash with Paypal, just to name a few.

Posted Sat, Apr 9, 2016

CloudFront Migration Update

When I migrated my blog to CloudFront I mentioned that I’d post about how it is going in late March. Well it’s late March now so here goes! First off, I switched from using the awscli tools and am using s3cmd because it does the smart thing and only syncs if the md5 checksum is different. Not only does this make a sync significantly faster, it also reduces PUTs which are a major part of the cost of this endeavour.

Posted Sat, Mar 26, 2016

DBI Logging and Profiling

I built some code to profile DBI usage.

Posted Thu, Mar 24, 2016

How to Enable ptrace in Docker 1.10

This is just a quick blog post about something I got working this morning. Docker currently adds some security to running containers by wrapping the containers in both AppArmor (or presumably SELinux on RedHat systems) and seccomp eBPF based syscall filters. This is awesome and turning either or both off is not recommended. Security is a good thing and learning to live with it will make you have a better time.

Posted Fri, Mar 18, 2016

When I Planned on Moving to Australia

Many of you do not know that I was born on the Gulf Coast of Mississippi. I lived there, with a brief intermission in Oconomowoc, Wisconsin, until I moved to Texas to go to college. That first year of school is rife with good memories; but there was a dark spot. Specifcally, Hurricane Katrina. Katrina was a big deal. To this day there are houses that are just gone, with nothing but a slab and a lot of weeds in their place.

Posted Sat, Mar 12, 2016

Weirdest Interview So Far

This is a pretty good story and I want you all to hear it. When I was graduating from college I interviewed with three companies. Two of them (MTSI and Rockwell Collins) offered me jobs. The other one, Empire Systems Inc., did not.

Posted Sun, Mar 6, 2016

Migrating My Blog from Linode to CloudFront

I am now hosting my blog on a serverless platform!

Posted Sat, Feb 20, 2016


While CGI is a fairly well established, if aging, protocol, UCSPI seems fairly obscure. I suspect that UCSPI may see a resurgence as finally with systemd projects will have a reason to support running in such a mode. But here I go, burying the lede. CGI Refresher Just as a way of illustrating by example, I think that I should explain (hopefully only by way of reminder) how CGI works.

Posted Wed, Feb 10, 2016


I’ve really enjoyed writing Rust, lately. I posted yesterday about what I’m doing with it. In the meantime here are some immediate reactions to writing Rust: Documentation The documentation is pretty good. It could be better, like if every single method had an example included, but it could be a lot worse. And the fact that a lot (though not all for some reason) of the documentation has links to the related source is really handy.

Posted Tue, Feb 9, 2016

Announcing cgid

This post is an announcement of cgid. Over the past week I developed a small UCSPI based single-file CGI server. The usage is very simple, due to the nature of the tool. Here’s a quick example of how I use it: #!/bin/nosh tcp-socket-listen 6000 tcp-socket-accept --no-delay cgid www/cgi-bin/my-cgi-script If you don’t know anything about UCSPI, this will look like nonsense to you. I have a post that I’ll publish later this week about UCSPI, so you can wait for that, or you can search for it and find lots of documents about it already.

Posted Mon, Feb 8, 2016

Handy Rust Macros

I’ve been writing some Rust lately and have been surprised at the dearth of examples that show up when I search for what seems obvious. Anyway, I wrote a couple macros that I’ve found very handy. The first seems like it should almost be core: macro_rules! warn { ($fmt:expr) => ((writeln!(io::stderr(), $fmt)).unwrap()); ($fmt:expr, $($arg:tt)*) => ((writeln!(io::stderr(), $fmt, $($arg)*)).unwrap()); } // Examples: warn!("This goes to standard error"); warn!("Connected to host: {}", hostname); This allows you to trivially write to standard error, and it panics if it fails to write to standard error.

Posted Sat, Feb 6, 2016

Checking sudoers with visudo in SaltStack

At work we are migrating our server deployment setup to use SaltStack. One of the things we do at deploy time is generate a sudoers file, but as one of our engineers found out, if you do not verify the contents of the sudoers file before deploying it you will be in a world of hurt. Salt actually has a pretty good built in tool for this, but it’s very poorly documented.

Posted Thu, Jan 14, 2016

Pong for Pico-8

I originally wrote this for the Pico-8 Fanzine but it was sadly not accepted. I still had a lot of fun writing in a totally different style than usual. Imagine the following has been printed out, scanned, and reprinted maybe five times. Pico-8 is a “fantasy console.” It’s a reimagined 8-bit console sorta like the Commadore 64 but with Lua as the primary language instead of BASIC. It’s very fun to play with and I think anyone interested in making games would do well to get it, even if it’s nothing like real life games.

Posted Wed, Dec 23, 2015

Farewell, CPAN Contest

In August I write about being tired of The CPAN Contest. I decided recently that once I hit 200 releases I’d stop and put my efforts elsewhere. I am not giving up on CPAN or Perl; but I do not think timeboxed releases are best for individuals. Though I am very pleased to be able to write, test, and document a new CPAN module over the course of a couple hours.

Posted Wed, Dec 16, 2015

PID Namespaces in Linux

One of the tools I wrote shortly after joining ZipRecruiter is for managing a Selenium test harness. It’s interesting because there are a lot of constraints related to total capacity of the host, desired speed of the test suite, and desired correctness of the codebase. Anyway one of the major issues that I found was if I stopped a test prematurely (with Ctrl-C, which sends a SIGINT) I’d end up with a bunch of orphaned workers.

Posted Wed, Nov 25, 2015

Dream On Dreamer

I can’t speak for others, but I was pretty inspired as a teenager. What I’d do is read random stuff throughout the week, then listen to some kind of music or watch a movie on Friday, and do my best to stay up all night and use what I’d learned to make something new. For the most part, as a teenager, I failed. As with most teenagers, I was pretty much worthless.

Posted Sat, Nov 21, 2015

How I Integrated my blink(1) with PulseAudio

(Could also be phrased: “what does frew’s red light mean?”) At work I wear some noise cancelling ear buds. I do this because just twenty feet behind me there is a one hundred person sales team who sometimes claps, ring gongs, and is just generally loud. I also like to work to music and it helps me focus. My other coworkers all use large headphones, so they are used to being able to see at a glance if a given individual is listening to music.

Posted Tue, Nov 17, 2015

Fast CLI Tools: gmail

I have been using commandline tools to interact email for quite a while now. Basically there were two reasons: I wanted to use GnuPG gmail’s web interface became too slow The former should be obvious; attempting to have secure communications in the context of a web browser is laughable. The latter often surprises people. I think that if you pay a little more attention you’ll notice that gmail is clearly slower than local options.

Posted Sun, Nov 1, 2015

The Bad, The Good, and The Cloud

The Bad This weekend I was working on a little project that involved manipulating a fairly large (1.8 gB compressed, 17 gB uncompressed) 7zip archive. I don’t have 17 gB to uncompress to on my laptop and a significant amount of the archive was uninteresting to me. I thought it would be sortav fun and worthwhile to remove the files that are not needed, so I ran a man 7zr and started reading.

Posted Mon, Sep 7, 2015

The Chains of Get-It-Done

It’s 2AM. My wife and infant son are in bed. He’s teething so most of the interaction I have with her involves yelling over him and tense enjoyment of brief respites gifted by small cold things. I’m awake for two reasons. The first is that I enjoy the time I get alone. I tend to take advantage of this time by programming, watching film, or playing video games. Tonight I was especially excited because I got the PICO-8 fanzine and a bunch of new information regarding Linux Tracing was just published.

Posted Sat, Aug 22, 2015

CPAN Patch Workflow II

A couple of weeks ago I wrote an article about my CPAN Patch Workflow, but mentioned that I couldn’t use it with older projects that do not use Github for patches. This was due to my git configuration being subtly different from Yanick’s. Basically when I was running git send-email, I was being prompted for some input, notably the password to send email, as well as a confirmation dialog. I spent a few hours writing up some patches to Git::CPAN::Patch and resolved all of the issues I was running into, and the changes were released the other day!

Posted Tue, Aug 11, 2015

Perl ❤ Kickstarter

Today my boss, knowing that I am interested in weird modern cooking, sent me a link to the Imperial Spherificator, which lets you make whatever kind of caviar you want, like mint or coffee liqueur or Tabasco. I want to make some Worcestershire and soy sauce! Anyway, when he showed it to me there were no available “VERY EARLY BIRD” (or other limited variants) left. But to him there was one available, which is crazy because higher levels had been paid for.

Posted Mon, Aug 3, 2015

CPAN Patch workflow

I just wanted to write up a quick note on my workflow for CPAN patches, because I’m so pleased with it. I use three tools: Git::CPAN::Patch git-hub fugitive.vim When I first see a module that needs some love, like today I saw that Gazelle needed some POD reformatting, first I clone it (using Git::CPAN::Patch): git cpan clone Gazelle Then I fork it, so that I can make a pull request (using git-hub), make a branch, and set it up to track my fork.

Posted Thu, Jul 30, 2015

Index Hints in MySQL with DBIx::Class

I started at ZipRecruiter nearly two weeks ago and I finally feel like I’m finding my stride. It’s nice! Anyway, this post is mostly because I am positive that a lot of people need this and it’s difficult to make into an actual component. Sometimes in life one finds oneself using a “database” called MySQL. In order to make this “database” perform, one must sometimes hint at which indices to use or not use.

Posted Wed, Jun 24, 2015

AWS: Not just a place to run your VMs

I’m on my way out at Micro Technology Services, Inc.. Because of that my current boss has wisely taken me out of the loop of normal day-to-day programming so that my replacement can get plenty of experience before I’m truly “on vacation” as we say. With that in mind I’ve been tasked with stuff that will be interesting to me, valuable to the company, but if something goes sideways it won’t be a big deal if it never ever happens at all.

Posted Sat, May 23, 2015

Converting from SQL Server to Postgres

One of the things that I’ve been working towards for a long time at my current job (which I’m on the way out of) is to have the project work 100% on Linux. The main thing holding it back from that is that it depends on SQL Server. Now of course DBD::ODBC runs on Linux and even nowadays Microsoft distributes their Native Client for Linux. But our project is turnkey and runs on one physical machine, so the database is included.

Posted Tue, May 5, 2015


This has been a pretty big week for me; On Tuesday we listed our house to be sold! On Wednesday night I got what I thought was indigestion, and on Thursday had an appendectomy! Just today, Saturday, I think we have sold the house (pending all required legal grace periods of course.) This all pales in comparison to the really big news this week: this server got hacked. Saturday morning I noticd that Freenode was blocking my IRC client because I was on some blocklist (DroneBL, to be specific.

Posted Sat, May 2, 2015

Announcing DBIx::Class::Candy::ResultSet

Hello all! I just released DBIx::Class::Candy 0.003000, which comes with DBIx::Class::Candy::ResultSet. This should completely resolve the issues I mentioned in my previous post. This is how I use it: package Lynx::SMS::Schema::Candy::ResultSet; use strict; use warnings; use parent 'DBIx::Class::Candy::ResultSet'; sub base { 'Lynx::SMS::Schema::ResultSet' } 1;package Lynx::SMS::Schema::ResultSet::MessageChild; use Lynx::SMS::Schema::Candy::ResultSet; ... 1; If anyone runs into any issues let me know. Sorry that perl didn’t use c3 from day one!

Posted Tue, Apr 14, 2015

MRO's and you; how the distinction between C3 and DFS changed my life

Recently I fixed DBIx::Class::Helpers so that each helper would have a base class. This is actually something that ribasushi had been sorta hounding me to do for years but I could never figure out the case where it mattered. The reason I finally made the change was because a user ran into an issue that fixing the base class actually resolved. Unfortunately I neither documented what it was nor wrote a test.

Posted Tue, Apr 14, 2015

Docker Simplifications: Bugtowne City

I have a fairly complex docker container that I run on all of my machines. I would like to simplify it in a number of ways and for some reason I decided that it would be interesting to start on that project last night. The simplifications that I want to do are as follows: Have all three daemons log to /proc/1/fd/1; this would remove one of my volumes and let me view logs with just docker logs offlineimap

Posted Fri, Apr 3, 2015

Faster DBIC Schemata

Last week I did a talk for the Milwaukee Perl Mongers and this week I did it again for the Los Angeles Perl Mongers. I will do it one more time for DFW Perl Mongers soon, hopefully, if only to get the best version recorded. In the talk somehow near the end I discussed with Steve Nolte how to make a large DBIC schema faster. The best way to solve this is to make Core DBIC lazy load it’s results, but that has historically been hard to get working.

Posted Fri, Mar 27, 2015

Fear and Loathing in SQL-92

Like the tortoise I’ve been slowly but surely working on getting our application working on both SQL Server 2005+ and Postgres 9.4+. The latter is a new addition, hence the “latest and greatest” version. For the most part I’ve been surprised at how easy it has been. Both servers support using " as the identifier quote, which is all that I have to change in the majority of queries. For some dumb reason (there is a real reason, but it is dumb) most things use [, ] for the quotes in SQL Server.

Posted Wed, Mar 4, 2015


This is just a quick post to update you all on a nice new helper. Recently at my work we hired a new programmer and I’ve been showing him the ropes. I noticed him running into the age old confusion of treating a ResultSet like a Result, so I took note and decided to make a helper to give specific error messages when the user makes that mistake. If you plan on hiring new people ever, or you are a mere human yourself, why not add DBIx::Class::Helper::ResultSet::Errors to your base ResultSet?

Posted Fri, Feb 20, 2015