Wed, 28 Mar 2007

Deterministic thinking

We get used to thinking there is a “right” answer.

Most of business is built on that premise: if you add 2 and 3 you get 5. Too easy, right?

Much of programming is like that too. Indeed, the whole discipline of unit testing is built around the idea that given a constrained environment and controlled inputs we can say whether or not the output is correct.

But that’s not really the way things work in science and engineering.

In measuring the real world, it’s (2.0 ± .5) and (3.0 ± 15%, we hope). What does that add up to? Even figuring out what the error bars are is an exercise in judgement and approximation. Much of experimental science revolves around the evaluation of uncertainty in measurements.

It’s tough when you stare at something and start doubting whether you can believe what you’re looking at. Davyd Madeley, who works with seismic data, said something interesting the other day which got me thinking about these issues:

“Hopefully I will work out how to design a better debugging framework for datasets that are too big to printf … I can’t trust any layer to be giving me the correct results”

The vogue CS way of looking at this would be to scream “testing”. Certainly, unit and functional testing are an essential part of ensuring the different pieces of your application do what they are supposed to do. That’s a given. But in dealing with large sets of real world data and in applying algorithms to try and find hidden meaning in that data, there is often no “right” answer. How do you test that then? And how do you differentiate between miscalculation, and the noise inherent in the quality of the data?

Little of what we do in modern day information technology conveys error in addition to scalar quantities, so it’s difficult to even know where to start. We tend to think very deterministically in IT.

What does this inflexibility mean for the broader issue of reliability in our systems?

It means we’re not prepared for things going wrong. We’re not prepared for uncertainty. We say to ourselves “oh, it’ll be fine once things get back to normal,” but we’re not owning up to the reality that there is no normal. Change, interruption, and crisis are the only equilibrium.

This question of flexibility comes up in the procedures work I do for clients who run business critical systems. On the one hand you try to create precise instructions that explicitly capture the necessary sequence of events. On the other hand, not only do you need flexibility of mind, you need flexibility of practice in order to adapt to the changing circumstances and uncertainty that defines high pressure operations environments. It’s error bars all over again. Where most people run into trouble is that their procedures don’t encompass the fact that they may run into trouble. The techniques to deal with this are not that hard to learn, but what makes it elusive is that this is about more than IT. It means shifting the way the organization solves problems. That’s a bigger challenge.

Trapped in our usual we-already-know-all-there-is-to-know mindset, we can both tell just how it’s going to work out, right? After all, we’re all thinking deterministically about it. And my oh my, won’t we be surprised when tomorrow doesn’t turn out to be a normal day like we were counting on. But if we find the courage to face up to the fact that there might not be a right answer, then perhaps we’ve started down the road to making systems, companies, and communities that can stand the test of time. Or at least the test of next Sunday.

AfC

deterministic-thinking, posted by Andrew Cowie in /andrew/operations/procedures at 09:14, 28 Mar 2007.

---

Category Specific Feeds. Use these links for an RSS or ATOM feed limited to this category and its descendants. Technorati Profile

---