What is Cloud Native Observability v. Visibility and Why is it so Important?

In our last post “Introducing the MantisNet CVF: Solving Cloud Native, Serverless, Observability Challenges“ we provided an overview of the Containerized Visibility Fabric (CVF) capabilities and made some assertions about how revolutionary and disruptive cloud-native technologies are, the resulting opacity problems, as well the art of the possible insofar as enabling next-generation observability.

In this installment we’ll go into more detail as to what exactly we mean by cloud-native observability as well as the significance, deeper implications of it, interoperability and investment protection. So, let’s revisit containerization, and by extension, why cloud-native observability is so revolutionary.

Virtualization v. Containers

In containerized environments, there is no intervening hypervisor between the OS and the underlying infrastructure. Consequently, it is possible to access and use operating system (and kernel) services directly to examine the state and operation of the underlying infrastructure (bare metal servers), as well as services, the operating system and applications. This is profoundly important because in the cloud-native world – as previously discussed;

• network topology (physical and virtual) is hidden
• interfaces (network namespaces) are hidden
• data flows are hidden

...and to make matters even more challenging- resources are dynamically configured, provisioned, deployed and reused.

Stated more clearly; cloud-native, serverless, compute environments, while representing the most scalable and performant way to exploit cloud infrastructure, these technologies obfuscate, or make very difficult, the ability to observe, instrument and monitor the systems with legacy tools and technologies.

The Significance of Real-time, Cloud Native Observability

MantisNet CVF agents, deployed as microservices and containerized applications, uniquely exploit deep system and kernel level instrumentation, to instrument, access, collect and process the resulting telemetry - where the telemetry is generated (not a TAP or other location further up the network) - in such a way as to provide programmatic visibility and access to the underlying infrastructure enabling better, newer forms of observability and control. Having access down to the kernel for instrumentation provides an immutable data source for security and network or application performance management. These new forms of observability and control are manifested through deep system level access and interactive command and control communications via a distributed message bus architecture.

Cloud native observability with CVF provides access to the systems, services and instrumentation hooks down to the kernel. The CVF agent process and packages the resulting data and publishes it, continuously and in real-time, using a distributed message bus architecture where external analytic tools and/or AI/ML workflows can analyze the resulting streaming metadata telemetry and make command-control decisions and responses back to the source, providing interactive observability and control of network activities.

This approach provides a more reliable and scalable, cloud-native, way to dynamically inspect, extract and process detailed telemetry from the cloud-native infrastructure, to the kernel and data link layer, and serve it up continuously and in real-time via serialized metadata using a high-performance streaming messaging system (NATS, Kafka…). This enables deep visibility, flexibility and investment protection. Whereas legacy traffic acquisition and monitoring tools give you a tools interpretation of what is going on with your network based on activity, we’re giving you the telemetry data right from the generating source.

Why is observability so important?

There has been a lot written about Observability recently. According to Wikipedia's Observability page, 

“In control theory, observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs.”. “Simply put, observability is achieved when data is made available from within the system that you wish to monitor."

"Monitoring is the actual task of collecting and displaying this data.” 

In other words; observability provides richer, deeper and more useful knowledge and context for understanding what is happening within a system.

So, observability requires that both the quality and utility of data (telemetry) obtained from the systems is sufficient to understand the system under observation; not constrained by what information is merely available. Conventional monitoring and visibility solutions have traditionally been a good source of information; they only provide static snapshots (data structures logs, PCAP files, traces…) obtained from pre-defined, available sources, or captured from monitoring applications or network traffic. Furthermore, with these legacy approaches, modifying the format or contents of those existing, predefined forms of telemetry could entail rewriting the application, or in some cases (specifically with a packet broker), requiring entirely new hardware.

The Deeper Implications of MantisNet Cloud Native Observability

MantisNet takes cloud-native observability further. In cloud-native, serverless environments, the observational data needs to be continuous and adaptable as the systems evolve. The MantisNet CVF agents are sensor agent programs which are event driven and operate in secure confines of protected kernel space, meaning they are secure, resource efficient and have access to functions and services at the operating system (kernel) level and directly process the resulting telemetry up to the application layer. Additionally, and because CVF agents collect and process information where it is generated and can be loaded and launched on demand, CFV agent capabilities can be used how, when and where they are needed. The CVF is not just a sensor, although it can serve as a simple probe; capturing, filtering and replicating traffic, a form of a cloud-native TAP, but it is much more in terms of its utility, scalability, and flexibility as described in more detail on the MantisNet CVF page.

MantisNet CVF agents also makes data correlation in cloud-native environments much, much simpler. The ability to generate the correct telemetry when and where it is needed as well as provide the ability to correlate CVF telemetry with other data sources not only supports more powerful analytic capabilities, but also provides you the ability to work with other forms of observability tools, telemetry, or semi-structured data within your overall operations analysis and monitoring practice.

How do MantisNet solutions (specifically the CVF) compare to the latest observability tools?
The answer is, it depends on what you want to accomplish. If you want to monitor the general health and status of process and applications, there are a raft of exciting new system tools available, many participating with the Cloud Native Computing Foundation.

Transitioning to the Future: Interoperability and Investment Protection

Outside of the benefits of cloud-native observability, a significant proportion of applications are not greenfield, they are hybrid environments that use a combination of stand-alone and virtualized infrastructure, and require the capabilities and type of telemetry that the MantisNet CVF can deliver.

While pure-play cloud-native environments are growing, many organizations have significant commitments and investments in analytics and tools which can only utilize the more conventional data-at-rest sources (logs, files, NetFlow, IP-FIX, PCAP, DPI...). To support those environments, the MantisNet CVF offers compatibility with the ability to stream both metadata and legacy formats (PCAP, DPI files, NetFlow, IP-FIX...) into static file and data formats such that existing tools and applications (e.g. WireShark) can benefit from the richness, accuracy and immutability of CVF produced telemetry. Also, there are many applications for supporting regulatory compliance, legal, forensic, auditing, etc., that by definition require the resulting traffic capture and telemetry to be collected and secured for later retrieval and analysis. Here too, the MantisNet CVF can be configured to stream the captured traffic and telemetry into data repositories; static storage, data lakes to better meet those requirements.

MantisNet is uniquely focused on the value of networks and “data-in-motion”. We obsess on understanding the “fluid dynamics” of virtual and physical traffic flows and providing observability into the sources and movement of data within and across the infrastructure. Providing instrumentation to observe and analyze the sources of data, the flow of data as well as how it is consumed. Whereas other observability tools focus on the health and state of the infrastructure the MantisNet CVF platform enables users to examine the data.