Or, how I managed to avoid having to edit /etc/init.d/varnishncsa to set the log format

Let me tell you a story of woe, intermittent/random corruption, and confusion.

Background

We had reason to stand up a new VPN, from a new data center, to a VPC in AWS. For some semi-philsophical semi-technical reasons, this was not a "VPC VPN", but rather a GRE-over-IPSec tunnel to an EC2 instance inside the VPC, and it was the first of this kind we'd deployed (i.e. GRE over IPSec, to/from AWS).

Pre-problem 1: Racoon sucks.

Short and sweet today: There is always a point of failure, between your redundant, non-single-point-of-failure components You know, the single cable or switch that connects your VRRP firewalls, which on failure results in two machines that both think they're master. Or the RAID controller that connects to both disks in your RAID-1 mirror, which on failure takes out both disks (or worse, corrupts data on them).

Yes, every Perl programmer knows that you should 'use strict', but sometimes it's just easier not to. BUT YOU SHOULD ALWAYS DO IT ANYWAY. I just spent an hour debugging a bit of existing code where I added a bit of fork/waitpid code (copy/pasted from elsewhere) to implement concurrent child processing. And because 'use strict' wasn't on in (not my fault, the original code isn't mine), and I didn't add use POSIX ":sys_wait_h"; at the start, the WNOHANG constant wasn't defined. So perl just said "ok, I'll make that 0".