Opsview

Opsview provides visibility how the system is functioning using dashboards. This section describes the those dashboards for monitoring the Streamlio Cloud.

Access to dashboards

You can access the opsview dashboards using streamlioCLI using the following command:

streamlio ui

For accessing the opsview dashboards, you need to be within the same VPC or your peered VPC.

System Overview Dashboards

Broadly, there are two different types of dashboards.

  • User dashboards

  • System dashboards

User dashboards

These dashboards show how the Streamlio Cloud is being used

System Overview

Overall summary page that gives a glance of what's happening in the cluster, with all the aggregated metrics such as:

  • Overall CPU / memory / disk usage

  • Health state of all the components

  • Backlog and messaging storage

  • Publish / dispatch rates

Topics metrics

This dashboard presents per-topic level metrics that includes the following:

  • Publish / dispatch rates for data

  • Number of producers/consumers/subscriptions active in the system

  • Backlog for each topic/subscription and how it grow over time

  • Amount of storage consumed

  • Storage write latency

  • Geo-replication rates and backlogs

Messaging metrics

Similar to topics dashboard, this dashboard presents per-namespace level metrics that includes the following:

  • Publish / dispatch rates for data

  • Number of producers/consumers/subscriptions active in the system

  • Backlog for each topic/subscription and how it grow over time

  • Amount of storage consumed

  • Storage write latency

  • Geo-replication rates and backlogs

Functions metrics

This dashboard provides a detailed view of functions usage and it includes:

  • Messages received and processed within each function

  • Number of instances for each function

  • Average and 99pct processing latencies

  • Exceptions in system or user code while running functions

  • User defined metrics. (e.g) - when reporting latency of downstream services from within the function code with context.recordMetric("my-metric", latencyValue)

System dashboards

These dashboards are focused on the system components of Streamlio Cloud.

Broker metrics

Monitor the usage of the individual Apache Pulsar brokers

  • Publish and delivery rate per broker

  • Topics / producers / consumers count

  • Message backlog and storage write latency

Bookie metrics

Monitor the usage of the individual Apache BookKeeper storage nodes

  • Add and read entry rates

  • Latency breakdown for add and reads

  • Number of segment and entries stored

  • DIsk usage

  • Internal queues and caches sizes

Proxy metrics

Monitor the usage of Pulsar proxy instances with infos such as:

  • Number of connections

  • Network traffic

  • HTTP requests

  • Service discovery requests

ZooKeeper metrics

Monitor the health of Apache ZooKeeper ensemble running with:

  • Requests rates and latency

  • Breakdown of requests types

  • Data set size

JVM metrics

For all the Streamlio Cloud components, provides insight on JVM usage such as:

  • CPU

  • Heap & Direct memory usage

  • Garbage Collection

Log metrics

Monitor the amount of info/warn/error logs for any of the components

Auto-Recovery metrics

Monitor the Apache BookKeeper Auto-Recovery process. The auto-recovery is used by BookKeeper to create additional copies of data after a node has failed.

  • Number of under-replicated ledgers

  • Auto-Recovery read rates

Containers metrics

Per container metrics of both Streamlio Cloud components as well as Pulsar Functions submitted by users.

  • CPU

  • Memory

  • Disk IO

  • Network

Host metrics

Information regarding the VMs underlying all the containers.

  • CPU

  • Memory

  • Disk usage and IO

  • Network

_____ Copyright 2019 Streamlio, Inc. Apache, Apache BookKeeper, Apache Pulsar and associated open source project names are trademarks of the Apache Software Foundation.