Getting Data In and Out

Connecting data streams into and out of Streamlio Cloud Preview

Streamlio Cloud uses Pulsar IO connectors to connect incoming data to Pulsar topics and stream data in Pulsar topics to external destinations. (For an overview of Pulsar IO and how it works you can check out this Streamlio blog post.)

First, a quick clarification on terminology:

  • Sources: incoming data streams that are ingested into Streamlio Cloud

  • Sinks: external destinations to which data streams in Pulsar can be sent

For each source and sink, a Pulsar IO connector is configured and deployed. You can create and use custom Pulsar IO connectors with the Streamlio Cloud Preview (see this blog post for information about how to do that), but to make it easy to try it out Streamlio Cloud Preview, the following sample connectors are already included:

  • Sources

    • Amazon S3

    • Amazon SQS

    • Apache Kafka

    • Twitter

    • RSS feeds

    • Random data generator

  • Sinks

    • Amazon S3

    • Amazon SNS

    • Amazon SQS

    • FTP

    • Apache Kafka

    • Redis

    • Random data generator logger

Note that with the exception of the the Streamlio random data generator, which operate in the Streamlio-provided environment, these connectors will need to be connected to an external source or sink to which you have access.

Creating a Data Source

Creating a data source sets up a connection that flows data from a specific source to a specific Pulsar topic. To start creating a new data source using one of the built-in connectors, click on the "Sources" tab in the navigation bar to get to the Sources page.

Step 1: Choose a Data Source Type

Click on the "Create" button on the Sources page and then select from one of the built-in data source connectors.

Choosing a data source type

Step 2: Set Configuration Parameters

Next you will need to provide the basic information for the service. That information includes:

  • Name: the name to use for this data source; note that this must be all lower case, without special characters

  • Topic: the Pulsar topic which will receive data from this data source; note that topic names must be a URL-safe string

Random data generator data source configuration

Once you complete that information and click "Next", you'll be asked to provide configuration information specific to the chosen data source type. The information you'll need to provide varies depending on the specific type of data source, but commonly includes keys, secrets, or tokens needed to connect to the data source.

Step 3: Create the Data Source

The last step is simply to click "Create Source" and you're ready to go.

You can now see the data source that you created by clicking on the "Sources" link in the navigation bar (or on the "Sources" panel in the home dashboard of the console).

Creating a Sink

The steps to create a data sink using one of the built-in connectors are similar to those for a data source above. To start, click on the "Sinks" link in the navigation bar.

Step 1: Choose a Data Source Type

Click on the "Create" button on the Sinks page and then select from one of the built-in sink types.

Choosing a sink type

Step 2: Set Configuration Parameters

Next you provide the basic information for this sink. That information includes:

  • Name: the name to use for this sink; note that this must be all lower case, without special characters

  • Topic: the Pulsar topic which will send data to this sink; note that topic names must be a URL-safe string

After you fill out that information and click "Next", you may be asked to provide configuration information specific to the chosen type of sink. The information you'll need to provide varies depending on the specific type of sink, but commonly includes keys, secrets, or tokens needed to connect to it.

Step 3: Create the Sink

Finally, click the "Create Sink" button to finish creating the connection to this data sink. You can verify that the sink has been created by clicking on the "Sinks" link in the top navigation bar.

List of deployed sinks

_____ Copyright 2019 Streamlio, Inc. Apache, Apache BookKeeper, Apache Pulsar and associated open source project names are trademarks of the Apache Software Foundation.