Tutorials

A step-by-step tutorial for trying out Streamlio Cloud Preview

To get started quickly with Streamlio Cloud, follow the short tutorial below. This tutorial walks you through creating a data source that ingests test data into a topic and filters the data using a Pulsar Function written in Python and emits the result into another topic.

Step 1 : Generating test data by creating a data source

The Data Generator data source provided with Streamlio Cloud Preview randomly generates personal data for a fictitious person. The data is in JSON format. Below is an example.

{
"address": {
"street": "Cove Lane",
"streetNumber": "33",
"apartmentNumber": "",
"postalCode": "50556",
"city": "San Francisco"
},
"firstName": "Trinity",
"middleName": "Katherine",
"lastName": "Sutton",
"email": "trinity.sutton@mail.com",
"username": "tsutton",
"password": "vEuLeaB1",
"sex": "FEMALE",
"telephoneNumber": "266-795-2647",
"dateOfBirth": 745086880253,
"age": 25,
"company": {
"name": "Vitae",
"domain": "vitae.biz",
"email": "contact@vitae.biz",
"vatIdentificationNumber": "37-0002067"
},
"companyEmail": "trinity.sutton@vitae.biz",
"nationalIdentityCardNumber": "229-26-7245",
"nationalIdentificationNumber": "",
"passportNumber": "RQE66HFBR"
}

To create the Data Generator source to ingest sample data into a topic, first start by clicking on "Sources" on the navigation bar located on the top left.

Afterwards, click on the "Create" button to create a new source.

Then click on "Random Data Generator".

Fill out the form to complete the configuration for the Data Generator source and click "Next".

Finally, click "Create Source" to complete creating the source.

Step 2 : Filtering data by creating a Python Pulsar Function

Next, submit a Python Pulsar Function to process the data. You can start by clicking on "Functions" in the navigation bar located on the top left.

Fill out the form to configure the Pulsar Function. You can leave most of fields in the form as it is. For the "Input Topics" field, be sure to enter or select the same topic as the one you entered for the Randaom Data Generator source. Click "Add Topic" to add the topic as an input topic for the function. Please also enter a output topic for the function.

Afterwards, in the text editor, please input the python code for your Python Pulsar Function. You can copy and paste the simple example below. This example simply filters out people that are not within the age of 18 and 35

import json
def process(input):
person = json.loads(input)
if person["age"] > 18 and person["age"] <= 35:
return input

Click "Create function" to finish creating the function

Step 3 : Viewing the filtered stream

You can check the logs of the Python Pulsar Function you submitted by first clicking "Functions" in the navigation bar on the top left of the page. The Pulsar Function you submitted should be listed on the page. Click on the function to take you to a page with details about the function. Under the "Instances" section towards the bottom right of the page, click the icon to take you to the logs of the instance.

The logs will looking some like the following. Also, if you print or log anything from the function, the output will be displayed here.

You can also tail the output topic of the function to look at results. First click on the output topic.

Then click on "Inspect" which will take you to a screen with the latest messages from the topic.

_____ Copyright 2019 Streamlio, Inc. Apache, Apache BookKeeper, Apache Pulsar and associated open source project names are trademarks of the Apache Software Foundation.