I recently had to set up a DR (Disaster Recovery) capability for our Puppet Master (Puppet 4, Open Source version); until now, we'd run with just a single puppet master in a single geographical location. Certain events brought DR to the forefront of our minds and priority lists, and the task fell to me. Note that I'm not talking about resilient/redundant arrangements of multiple always-live servers as very well documented at https://docs.puppet.com/guides/scaling_multiple_masters.html, but rather a somewhat simpler approach more suited to our current needs. The solution was fairly simple in the end, and worth recording in case it is of value for someone else. The constraints:
System Administration, or more accurately the Operations side of IT, is at its heart a technically complex job. However, there are some soft skills that are important. Note that I'm using 'soft' in a non-derogatory sense. The raw technical aspects are 'hard' in that they have well-defined edges, and typically very clear right and wrong answers. The 'soft' aspects tend to be fuzzier, with softer edges and more nuance. One of these soft skills is Situational Awareness.
I've been a system administrator for nearly 10 years now, and there's a few things I've learned along the way. This is the first in a series of short posts sharing some of these things.
Systems should be silent, unless they have something important to say that requires action on the part of the system administrator. In particular, notifcations (e.,g. e-mails, but potentially other mechanisms) should only be sent if they require action. If the action required is not completely obvious from the direct contents of the notification, then you need to add a link to external documentation detailing what needs to happen (e.g. a "trouble code", or a link to your wiki etc).
Following on from part 2 of this series, is the requirement that e-mail must leave the system it's generated on, unless it's going to a mailbox on the local system which is actively monitored (weekly minimum, preferably every week-day or maybe even several times a day). Typically that place will only be on your actual mail server. In general, system mail must go to a real IMAP/POP/Exchange/LotusNotes/Whatever mailbox server (MDA), where someone' mail client will present it to them.
Short and sweet today: There is always a point of failure, between your redundant, non-single-point-of-failure components You know, the single cable or switch that connects your VRRP firewalls, which on failure results in two machines that both think they're master. Or the RAID controller that connects to both disks in your RAID-1 mirror, which on failure takes out both disks (or worse, corrupts data on them). Or that little Environmental Monitoring Unit in your lovely big SAN that, on failure, makes the redundant SAN controllers decide that they cannot and should not be serving any delicious data from your racks of redundant disks to your servers over the multi-path multi-switch SAN fabric. That last one? Yeah, saw that in production once. Removing all single points of failure is actually rather hard. I'm not saying you shouldn't try, but when you think you're done, look again. Look in the cracks between your components, and ask yourself what will happen if those cracks widen. It's kinda fun, in a "watch a scary movie" way.
At my work we often throw around the phrase WCPGW (What Could Possibly Go Wrong) in response to ill-advised or just plain crazy ideas. It's fun, and lets off some steam, but it occurred to me recently that there's a useful kernel of truth in it. Indeed, a good sysadmin is always asking this question; when designing systems, preparing to make a change, in the heat of an emergency, and in security design and response.
On starting a new job recently (well, last year when I started this entry), I have been reminded yet again just how long it takes an Operations person (Sysadmin, SRE, whatever you want to call us this year; I'm going to use 'Ops' for this post) up to speed, compared to a developer.