Technical

Building a grafana Debian package

I had reason to re-build a custom grafana Debian package a few weeks back.  It was a little bit tricky to figure out the right steps, and I wasn't able to find this process documented anywhere else, so here it is (for my future benefit as much as anyone else's).  This is brief, and is intended to be an outline/guide only.  Your mileage may vary

Read more →

Fixing (one case of) AWS EFS timeouts/stalls

AWS Elastic File System (EFS) is an NFS compatible network-accessible shared storage system.  It allows you to outsource the problem of HA network storage, which is highly attractive in some circumstances.  But, there are some sharp edges, which we discovered at work.

Read more →

Fun with SCSI tapes on Linux

I inherited an LTO-4 SCSI-attached tape drive recently (yes, old-school, I know, it's mostly just for fun), and have been fiddling with it to do some tape-based offsite backups (to complement my other offsite backups).  In doing so, I learned a bunch of things about the SCSI protocol (including its many many tendrils and offshoots), and about handy tools for interacting with SCSI devices.  Some of these things may be useful to others, so I'm putting them here.  Enjoy!

Read more →

Modern (2017/2018) CSS + Javascript guides

Despite being in IT and doing things in and around web-development for 20 years, my focus has been on System Administration for a long time.  I 'know' HTML, Javascript, and CSS at a perfunctory level and can do all sorts of fun (but typically ugly) things in them, but modern 'front-end' development has changed a lot over the last 5-10 years, and I haven't had the time (nor inclination, to be frank) to keep up.

Read more →

My First WebExtension - CookieMaster

I finally completed my first WebExtension today.  It's for controlling what domains may set cookies, in the style of the old CookieMonster add-on for Firefox that didn't get ported to WebExtensions. 

Read more →

Prometheus, Grafana, rates, and statistical kerfluffery

I love performance graphs and monitoring software; graphs are pretty, and there's nothing quite like the feeling of using a graph to identify precisely the cause of a technical problem. It means, however, that every few years I end up delving deep into some aspect of them to figure out why my graphs don't look 'right'. A few years ago it was Cacti + RRD losing information we thought we were keeping (pro-tip: consolidating to the maximum value of an average seen over a given period, rather than keeping the maximum of all maximums seen, is probably not what you want). This is the store of my latest battle with Prometheus and Grafana; there are quite a few different moving parts involved, and the story is an interesting one. Come on a journey with me (Spoiler: I win in the end).

Read more →

VarnishNCSA LogFormat hackery

Varnishncsa is the logging component of the Varnish web cache software (https://www.varnish-cache.org/). Sadly it doesn't care for reading a config file, so the only way to configure the format of the log lines is on the command line. The trouble with that is spaces and quotes. Sure, on Debian/Ubuntu you have /etc/default/varnishncsa, and you can set "$DAEMONOPTS" to add -F . But because this log format will typically contain spaces, the quoting is exactingly particular, and as far as I've been able to determine, by the time it's been set through $DAEMONOPTS, there's either no way to get the quoting right, or it's too complex for a simple sysadmin like me to figure out (and according to my web searching, many smarter people have tried that approach and failed).

Read more →