I recently had the less-than-entertaining experience of a series of emails exchanged with a customer (manager level) on how to measure the “perceived user experience of throughput” through one of our nodes in their network. Since this node is one of many – very many – in the end customer’s data path, I responded with a link to the DSL Reports web site and its series of speed tests (http://www.dslreports.com/speedtest?flash=1 – I find them useful for comparison purposes). That, after all, is the throughput the user will perceive

He replied, most unhappily.

It seemed, after much email talk back-and-forth, that what he really wanted was a means to measure the per-user throughput via a MIB on this particular node. Since the node has up to 40 distributed processors, each of which is capable of supporting 65,000 (old hardware)/136,000 (new hardware) user sessions simultaneously, and creating/deleting them at a rate of 100/200 per second, we don’t track the per-session throughput for each session.

What I was able to provide him was a set of MIBs – which they were already collecting – which measured the instantaneous throughput in each direction and which counted the number of active sessions at any given time. From this, they can derive averages, but only averages. Those averages may or may not be representative of their customers’ actual throughput experience … their customers, for instance, could be quite bursty in their traffic, but on average not use very much at all. In addition, the “user experience” involves latency at a large number of other nodes in the path of the data flows.

What I’m afraid this manager is going to get is some specific data which conveys to him a picture that doesn’t necessarily match reality.

Data is not the same thing as information.

Information has contextual meaning. As an example, consider this blog post, which I found quite interesting: http://pajamasmedia.com/richardfernandez/2009/07/20/hockey-stick/ I, too, thought the graph displayed was something else, though the particular thing I expected it to be reflected my biases, and wasn’t the thing the author suggested most people would expect.

That’s an important thought – I had biases in what I thought the data was telling me. And – like anyone else – I have biases in how I ask a question.

What I look for in KPIs from my nodes reflects my biases as to what’s important. What I look for reflects my education, my experience, and my exposure to other peoples’ experiences, usually in the form of horror stories. As the saying goes, Your Mileage May Vary.

Read the original blog entry...