| Mon | Tue | Wed | Thu | Fri | Sat | Sun |
|---|---|---|---|---|---|---|
| 1 | 2 | |||||
| 3 | 4 | 5 | 6 | 7 | 8 | 9 |
| 10 | 11 | 12 | 13 | 14 | 15 | 16 |
| 17 | 18 | 19 | 20 | 21 | 22 | 23 |
| 24 | 25 | 26 | 27 | 28 | 29 | 30 |
This section:
Operations and other mysteries: the subtle business of making technology
work... a blog by Andrew Cowie.
The syndication links at top left will give you a feed for the blog as a whole.
blogs > andrew > operations
Wed, 30 Apr 2008
No crisis for me
I was chatting with Atul earlier today when he expressed his dismay that Slashdot was off the air and asked me if I could get there? No. Meanwhile, I realized I couldn’t get to SourceForge. This was proving problematic as I had just done a release of an Open Source project I’m the maintainer of, and was trying to update the project website. Guess announcing the release will have to wait a bit :(.
Oh well. “The internet must be broken somewhere”.
We then recalled that these are both services of OSDN. I recall a few years back talking with the system administrators there about crisis management in IT environments and using procedures to manage change. They said they didn’t need any help with their operations. “We’re all set”. Uh huh. People always say that when things aren’t going wrong.
—
Of course, it is now 3am in California. It actually doesn’t matter where your servers are — it’s always 3am when things like this go wrong. Personally, I take it as conclusive proof about the underlying nature of the universe that this sort of thing only happens when it is the middle of the night. You can’t possibly find a less pleasant time to force sysadmins to get out of bed and to go try and fix things. Frankly, I think that’s just Someone trying to tell sysadmins that they made a poor career choice, but you know, He (or She, or It, or They, take your pick) needs to have a good laugh too.
AfC
Fri, 19 Oct 2007
Too much typing
According to the BBC, apparently, some people get stressed by email:
It seems a backlash against the always-on workplace has truly begun.
It’s not too much email. It’s too much typing
I have long observed that people in the Open Source movement spend far too much time conversing with each other by IRC, instant messenger, or on widely subscribed email lists when they could just pick up the phone and actually talk to each other. It certainly wouldn’t take as long as all the line by line writing does.
The thing that’s lacking in point-to-point voice calls, of course, is the shared collaborative nature of online text channels and mailing lists where everyone can observe and jump in if they have something to contribute.
I think what we need, therefore, are standing always-up conference voice bridges to ride over and in parallel with the text channels. This way people could listen in casually and chime in when they have something to say.
This would, admittedly, be an entirely new practice for software developers. But it’s long been the way things are done in the operations world. Air traffic control, space mission control, military combat teams, securities trading, business critical IT changes events — all are run with a voice loop as a crucial and driving element co-ordinating activities and providing the focus for dealing with crisis. There are certainly a number of mannerisms and practises that are required to make it work that people in the business world have long taken for granted (ie if you’re on a conference call, you mute unless you’re talking, that sort of thing), but the etiquette for who is talking and who isn’t is usually pretty clear. We’ve learned how to make text channels effective; we can certainly work it out for voice ones.
Since the first thing we do when running a mission critical change event for a client is put voice loops in place, how to use radio circuits effectively is something I’m used to teaching. I do realize that getting the distributed global IT world to use it effectively would indeed take some doing, but nevertheless, it’s something I think we should pursue.
Here’s hoping for less typing and more communicating.
AfC
Fri, 13 Apr 2007
Get used to thinking for yourself
Bernard Golden writes:
Fifth, open source. This doesn’t mean occasionally considering it. And it definitely doesn’t mean evaluating it by the standards of how you’ve done things with proprietary software … People criticize open source because it doesn’t “deliver business value.” What they typically mean is that they’re used to letting the vendor do their job of deciding what their infrastructure should look like, then providing them a roadmap of their infrastructure development plans, and then pre-integrating the solution with the vendor’s favored software partners. So, naturally, when you look at open source, it fails to do that. No open source vendor is going to do a dog-and-pony show and then build your proof-of-concept [for you] to get you committed to their solution. Instead of asserting that open source doesn’t deliver business value, run an experiment. Find out for yourself what the costs of doing open source are. And besides, as open source economics eats away at the margins of proprietary vendors … they’ll do less of the legwork for you. So get used to thinking for yourself.
I really like this. For all the FOSS cheerleading I do, I don’t normally get embroiled in proprietary vs open source debates, and I hadn’t really thought of the point that Bernard makes here: one of the biggest reasons that people will resist change is because there are many who want their answers handed to them on a silver platter.
AfC
Wed, 28 Mar 2007
Deterministic thinking
We get used to thinking there is a “right” answer.
Most of business is built on that premise: if you add 2 and 3 you get 5. Too easy, right?
Much of programming is like that too. Indeed, the whole discipline of unit testing is built around the idea that given a constrained environment and controlled inputs we can say whether or not the output is correct.
But that’s not really the way things work in science and engineering.
In measuring the real world, it’s (2.0 ± .5) and (3.0 ± 15%, we hope). What does that add up to? Even figuring out what the error bars are is an exercise in judgement and approximation. Much of experimental science revolves around the evaluation of uncertainty in measurements.
It’s tough when you stare at something and start doubting whether you can believe what you’re looking at. Davyd Madeley, who works with seismic data, said something interesting the other day which got me thinking about these issues:
“Hopefully I will work out how to design a better debugging framework for datasets that are too big to
printf… I can’t trust any layer to be giving me the correct results”
The vogue CS way of looking at this would be to scream “testing”. Certainly, unit and functional testing are an essential part of ensuring the different pieces of your application do what they are supposed to do. That’s a given. But in dealing with large sets of real world data and in applying algorithms to try and find hidden meaning in that data, there is often no “right” answer. How do you test that then? And how do you differentiate between miscalculation, and the noise inherent in the quality of the data?
Little of what we do in modern day information technology conveys error in addition to scalar quantities, so it’s difficult to even know where to start. We tend to think very deterministically in IT.
What does this inflexibility mean for the broader issue of reliability in our systems?
It means we’re not prepared for things going wrong. We’re not prepared for uncertainty. We say to ourselves “oh, it’ll be fine once things get back to normal,” but we’re not owning up to the reality that there is no normal. Change, interruption, and crisis are the only equilibrium.
This question of flexibility comes up in the procedures work I do for clients who run business critical systems. On the one hand you try to create precise instructions that explicitly capture the necessary sequence of events. On the other hand, not only do you need flexibility of mind, you need flexibility of practice in order to adapt to the changing circumstances and uncertainty that defines high pressure operations environments. It’s error bars all over again. Where most people run into trouble is that their procedures don’t encompass the fact that they may run into trouble. The techniques to deal with this are not that hard to learn, but what makes it elusive is that this is about more than IT. It means shifting the way the organization solves problems. That’s a bigger challenge.
Trapped in our usual we-already-know-all-there-is-to-know mindset, we can both tell just how it’s going to work out, right? After all, we’re all thinking deterministically about it. And my oh my, won’t we be surprised when tomorrow doesn’t turn out to be a normal day like we were counting on. But if we find the courage to face up to the fact that there might not be a right answer, then perhaps we’ve started down the road to making systems, companies, and communities that can stand the test of time. Or at least the test of next Sunday.
AfC
Thu, 01 Jun 2006
In Memoriam
An ex-cadet was killed in Afghanistan a few weeks ago. 22458 Captain Nichola Goddard, RMC class of ‘02, was calling indirect fire in support of her combat team when her vehicle was hit.
In reading some of the tributes sent in by her comrades-in-arms and classmates, I came across this:
“Nichola was one of the few people who truly understood what she was doing in the army. It wasn’t just a job, nor was she playing soldier or in for a free degree. She had thought through the responsibilities and ramifications of taking that oath we all took. Too few cadets are truly prepared - intellectually, morally - for the real job of soldiering - but she was. She was someone who gave her whole self to everything she did, and could always be counted on to give a hundred percent, even when we were all out of gas. She was intelligent, compassionate, generous and energetic. Nichola was one of the few people who would do the right thing, always. And make sure it was done right.
We need more people like her in the world, which makes it all the sadder that the world has lost her.”
— 21982 Adrian Rawle
Would that we all have this said of us. People like this are ones to look up to. Even when they’re gone.
And that’s the whole point.
AfC
19809
RMC ‘95
Mon, 03 Apr 2006
The leaders and the led
The other day I was discussing the subtleties of leadership with a client. I was mentioning an opinion I formed during my early days at RMC that there were two kinds of people: those who did what needed to be done, and those who waited to be told what to do.
This in turn reminded me of a passage from one of my favourite books, C.S. Forester’s Lieutenant Hornblower (that’s pronounced ‘leftenant’ for all you Americans reading this). As I looked it over, I was reminded of how complex the human condition really is.
The scene is the quarterdeck of HMS Renown. Thinking over Hornblower’s adroit handling of his superiors, Lieutenant William Bush (also senior to Hornblower and from whose point of view the story is told) looks over at Hornblower and starts to reflect:
“Bush was learning something about personalities. He would never be able to reduce the results of his observations to a tabular system, and it would never occur to him to do so, but he could learn without doing so; his experience and observations would blend with his native wit to govern his judgements, even if he were too self-conscious to philosophise over them.
He was aware that naval officers (he knew almost nothing of mankind on land) could be divided into active individuals and passive individuals, into those eager for responsibility and action, and into those content to wait until action was forced on them. Before that he had learned the simpler lesson that officers could be divided in to the efficient and the blunderers, and also into the intelligent and the stupid — this last division was nearly the same as the one immediately preceding, but not quite. There were officers who could be counted on to act quickly and correctly in an emergency. And those who could not — again the dividing line did not quite coincide with the preceding. And there were officers with discretion and officers with none, patient officers and impatient ones, officers with strong nerves and officers with weak nerves. In certain cases Bush’s estimates had to contend with his prejudices — he was liable to be suspicious of brains and of originality of thought and of eagerness for activity, especially because in the absence of some of the other desirable qualities these things might be actual nuisances.
The final and most striking difference Bush had observed during ten years of continuous warfare was that between the leaders and the led, but that again was a difference of which Bush was conscious without being able to express it in words, and especially not in words as succinct or as definite as these; but he was acutely aware of the difference even though he was not able to bring himself to define it.”
— C.S. Forester, Lieutenant Hornblower (Little, Brown and Company, 1951; republished by Back Bay Books, 1998) page 75.
So which are you? To be sure we don’t live in the harsh starkness of the 18th Century Royal Navy. But the amazing thing about leadership is that it is not static. It can be learnt. When we’re fortunate to have good examples to follow, people we can look up to and situations which force us to go beyond what we thought we could do, that’s when we look within ourselves and become the people we most want to be.
AfC
Category Specific Feeds.
Use these links for an RSS or ATOM feed limited to this category and its descendants.
Technorati Profile
