Monitoring

Monitor data streams and data processing

Dashboard

The dashboard (accessible by clicking on the Streamlio name in the upper left corner of the UI) provides an at-a-glance overview of what is in your environment.

Web interface dashboard

Many elements of this dashboard are clickable to drill down into more information about namespace, topics, functions, sources, sinks and more.

Monitoring Namespaces

Namespaces are a logical concept in Pulsar for organizing topics and topic hierarchies. They also are critical to supporting multi-tenancy in Pulsar. Apache Pulsar and Streamlio Cloud support using multiple namespaces to organize topics (for example by department or application). Streamlio Cloud Preview supports only a single namespace that was precreated for you when you created your account.

Namespaces dashboard

The Namespaces dashboard shows you a summary of what is happening in your namespace: how many topics it contains, the rate of data in and out of the topics in that namespace, and the storage space used for message retention across all of the topics in this namespace.

For more information about Namespaces, please see the Streamlio Cloud documentation and the Pulsar documentation.

Monitoring Topics

The Topics dashboard page provides summary information about the topics that exist in your environment, and it also allows you to create new topics.

Topics dashboard

In the Topics dashboard, you can see all of the topics that exist in your current environment as well as information about incoming and outgoing message and data rates for each topic. Backlog, also visible from this dashboard, is a count of the number of messages in the topic that have not yet been consumed.

To get more information about a specific topic, you can click on the topic name to get to the topic details view to see more detailed timeline chart of key metrics as well as summary information about producers to a topic and subscriptions attached to the topic.

Topic details view

For more information about topics, please see the Streamlio Cloud documentation and the Pulsar documentation.

Monitoring Functions

Pulsar Functions that have been deployed can be monitored from the Functions summary dashboard, accessed by clicking on "Functions" in the navigation bar or by clicking on the Functions panel of the home dashboard.

Functions summary dashboard

For more information about a specific function, you can click on the function name to reach the dashboard view for that function.

Function details dashboard

To see more details about the work being done by a running function, click on the function name. The function status dashboard shows information about the function including the following:

  • Rate in and rate out: the rate at which the function is consuming messages from the input topic(s) and publishing messages to the output topic

  • Processed: the number of messages / second that the function is processing

  • Backlog: the number of available messages in the input topic that have not yet been consumed by the function

  • Errors: the error rate for this function

  • Latency: the time from the invocation of the function to completion of the function for processing of individual messages

  • Input topic: information about the amount of data the function has processed from the input topic(s)

  • Output topic: information on the amount of data the function has published to the output topic

  • Instances: the number of workers deployed for executing this function

Monitoring Sources and Sinks

Data sources and sinks can be monitored in the Sources dashboard and Sinks dashboard, respectively.

Data Sources dashboard

This dashboard lists the sources or sinks that have been deployed along with the resources reserved for each.

Details about a specific data source or sink can be viewed by clicking on the name of that source or sink.

_____ Copyright 2019 Streamlio, Inc. Apache, Apache BookKeeper, Apache Pulsar and associated open source project names are trademarks of the Apache Software Foundation.