Why observability is your next tech buzzword

Observability gives us a vantage point into the "inner neurons and transports" happening inside any given application.
27 January 2020

Observability is about understanding ‘in-flight’ requests happening throughout live operations. Source: Shutterstock

Just when you thought Artificial Intelligence (AI), Machine Learning (ML) and Autonomous Computing (AC) were going to be the technology buzzwords of the year to come… it turns out you (and me, and all of us) were wrong.

With AI and its ML intelligence on the rapid ascent to provide us with autonomous controls so that we never need to worry about backend computing management, it felt like we knew what was ahead of us and we could let 2020 play out to be a more intelligent space of intelligence.

But the pundits, industry analysts and evangelists had different ideas… 2020 is going to be the year of observability.

What is IT observability?

In simple terms, tech observability is all about knowing what’s going on with data from the depths of the data lake (where we keep all our uncharted unstructured data) all the way up through infrastructure layers and onward until we can surface at the user interface level.

But technology observability is a step beyond monitoring, auditing, management and systems administration. We’ve already got all those things, so what else does it bring?

Wikipedia is comparatively eloquent on this topic and states that, “In control theory, observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs. The observability and controllability of a system are mathematical duals.”

In-flight service for IT

So this means that observability is really all about understanding more about the ‘in-flight’ requests that are happening throughout the live operations of the system in question. 

Some of this will be log file analytics (every piece of software that performs any single move in any form creates a log file) and some of it will be a higher-level understanding of how the cloud services feeding our apps are coping with bottlenecks and spikes in user demand.

We’ve said log files, which are often used a basis for security, although they were never engineered as a specific window into security… but that’s okay in this case, because observability involves a study of knowing what we know about our IT systems. It actually goes full circle NSA-style and involves knowing about known knowns, known unknowns and unknown unknowns

Cindy Sridharan’s seminal 2017 Medium commentary entitled Monitoring and Observability, sees her quote one user lamenting observability being akin to a ‘devops-ifying’ of what we used to know as plain old monitoring i.e. changing the name of something (in this case Dev-development and Ops-operations) that already exists and suggesting it now comes in a different shape.  

For Sridharan, there are four pillars of the observability:

  • Monitoring
  • Alerting/visualization
  • Distributed systems tracing infrastructure
  • Log aggregation/analytics

“Observability, according to this definition, is a superset of monitoring, providing certain benefits and insights that monitoring tools come a cropper at,” wrote Sridharan.

So that makes a lot of sense, observability is no granular subset of IT monitoring, it is a higher octane superset (i.e. a set formed from a collection of other sets) that uses newer intelligence aids including AI & ML to help form the observed vision of the system that is needed.

Is observability for sale?

So given this introductory city bus tour of our new term, do we know how to build it, where to buy it or how to engineer it into our current IT stacks? That’s actually the US$64,000 question.

The most prevalent source for observability tooling is likely to come from Application Performance Management (APM) specialists – and yes, we know we said that observability isn’t just monitoring. 

With most APM vendors now self-styling themselves as something grander (like ‘intelligence fabric visionaries’, or some other fanciful label), it might be tough to know whether or not we’re being sold observability solutions in the first place. 

The clue is to listen for the world itself in the salesperson’s patter… and then take a deep breath.

For IT observability to deliver on its promise it does need the human factor to kick in, i.e. we need systems to be ‘made observable’ so that we can gain a vantage point into the inner neurons and transports happening inside any given application, database, analytics engine or cloud service.

When we humans observe IT observability, then we’ll be able to be observant to observability observations… and if that doesn’t give you a healthy wariness for a buzzword about to slap you around like a perfect storm of ‘digital transformation’, then nothing will.