Short and sweet today: There is always a point of failure, between your redundant, non-single-point-of-failure components You know, the single cable or switch that connects your VRRP firewalls, which on failure results in two machines that both think they're master. Or the RAID controller that connects to both disks in your RAID-1 mirror, which on failure takes out both disks (or worse, corrupts data on them). Or that little Environmental Monitoring Unit in your lovely big SAN that, on failure, makes the redundant SAN controllers decide that they cannot and should not be serving any delicious data from your racks of redundant disks to your servers over the multi-path multi-switch SAN fabric. That last one? Yeah, saw that in production once. Removing all single points of failure is actually rather hard. I'm not saying you shouldn't try, but when you think you're done, look again. Look in the cracks between your components, and ask yourself what will happen if those cracks widen. It's kinda fun, in a "watch a scary movie" way.
Yes, every Perl programmer knows that you should 'use strict', but sometimes it's just easier not to. BUT YOU SHOULD ALWAYS DO IT ANYWAY. I just spent an hour debugging a bit of existing code where I added a bit of fork/waitpid code (copy/pasted from elsewhere) to implement concurrent child processing. And because 'use strict' wasn't on in (not my fault, the original code isn't mine), and I didn't add use POSIX ":sys_wait_h"; at the start, the WNOHANG constant wasn't defined. So perl just said "ok, I'll make that 0". Which means that my waitpid that was supposed to not hang, did indeed hang, so my concurrency code failed miserably to be concurrent. This makes me grumpy.
Following on from part 2 of this series, is the requirement that e-mail must leave the system it's generated on, unless it's going to a mailbox on the local system which is actively monitored (weekly minimum, preferably every week-day or maybe even several times a day). Typically that place will only be on your actual mail server. In general, system mail must go to a real IMAP/POP/Exchange/LotusNotes/Whatever mailbox server (MDA), where someone' mail client will present it to them.
Systems should be silent, unless they have something important to say that requires action on the part of the system administrator. In particular, notifcations (e.,g. e-mails, but potentially other mechanisms) should only be sent if they require action. If the action required is not completely obvious from the direct contents of the notification, then you need to add a link to external documentation detailing what needs to happen (e.g. a "trouble code", or a link to your wiki etc).
I've been a system administrator for nearly 10 years now, and there's a few things I've learned along the way. This is the first in a series of short posts sharing some of these things.
Documentation. Supposedly the bane of every programmer, system administrator, or techie person who'd rather be Doing It than Writing About It. Those same people are also the first to complain about any lack lustre documentation of technology they're trying to use, once again proving that irony is not dead. Therefore, I'm going to complain some more, because it might make me feel better, and because I think I've got something else to say along the way.
I've just finished and published my very first add-on for Mozilla Thunderbird: Filter SubFolders, a plugin that applies your mail filters to sub folders of an IMAP account, not just the inbox. It does so automatically when new mail is detected, and manually through a menu item.