Overview

Configuring an Amazon Kinesis Data Stream allows you to export, process, and analyze Cordial event data as it's created. This empowers you to respond instantly instead of having to wait for your data to be collected, packaged, and delivered through one-off or recurring exports.

Once your Kinesis Stream is up and running, Cordial will place all contact activities (system events and custom-named events) onto the Kinesis Stream in real-time, as they're created.  

Clients may read (consume) the events directly from the stream with their own custom processes, or with the help of Amazon's Kinesis Data Firehose product—or using a combination of these methods.

• You can learn more about Amazon Kinesis here.

You must have an AWS account along with permissions to create a Kinesis Data Stream, IAM Policies, and IAM Users.

For many use cases, we recommend using Kinesis Data Firehose to help with consumption of the stream, particularly when the destination will be Redshift, S3, Amazon Elasticsearch Service, or Splunk.

Create a Kinesis Stream

You can find detailed instructions for creating and managing the stream on the AWS website.

1. If you are using the AWS Management Console to create the stream, you can go directly to the Console Home Page and click the Create Data Stream button.

2. Give the stream a name (e.g. cordial-events).

3. Enter the number of shards to use for the stream.

There are limits to the rate that a single shard can be written to/read from, so you should adjust this number based on your expected traffic levels and event types. You can resize the number of shards later.

4. After clicking the Create Stream button, you should see it in the list of streams in your account. Click the stream name in the list to view its details.

5. Make sure to save the Stream ARN, as you will need this value in the following step.

Create an IAM policy

Create an IAM policy that allows (just) writing to the stream.

1. Go to the Policies page on your AWS console.

2. Click the Create Policy button.

3. Click the JSON tab.

4. Enter the following JSON and replace ARN_from_step_1 with the Stream ARN value from the first step in this task.

{
	"Version": "2012-10-17",
	"Statement": [{
		"Effect": "Allow",
		"Action": [
			"kinesis:PutRecord",
			"kinesis:PutRecords",
			"kinesis:DescribeStream",
			"kinesis:ListShards",
			"kinesis:ListTagsForStream"
		],
		"Resource": "ARN_from_step_1"
	}]
}

5. Click the Review policy button, give the policy a name (e.g. cordial-kinesis-write-access), and click Create.

Create an IAM user

Create an IAM user and allow Cordial to write to the stream.

1. Go to the Users page in your AWS Management Console.

2. Click the Add User button.

3. Enter a user name (e.g. cordial-kinesis-writer).

4. Select the checkbox to give it programmatic access, then click the Next: Permissions button.

5. Select the Attach Existing Policies Directly category.

6. Search for and select the name of the policy you previously created.

7. Click the Next: Review button.

8. Click the Create user button.

9. You'll see a success message with options to view or download the access credentials for this user. You will need to supply these when enabling Kinesis integration in the Cordial Marketplace.

Complete configuration in the Cordial Marketplace

After obtaining all of your AWS access credentials, visit the Cordial Marketplace to enable Amazon Kinesis Integration with your Cordial account in order to write to your stream.

From this point, you simply need to supply the Kinesis Stream Name, AWS User's API Key, and AWS User's Secret Access Key when prompted. Once configured, Cordial events will begin appearing in your Kinesis Stream.

We also support an exclusion filter if you want to exclude certain events from being added to the stream (not recommended for most use cases).

Set up Kinesis Firehose

Depending on your use case, you may find it helpful to have Kinesis Firehose automatically consume the stream and copy the events to a particular destination.

1. Go to the Kinesis Firehose page in your AWS Management Console.

2. Click the Create Delivery Stream button.

3. Give the delivery stream a name (e.g. cordial-events-delivery).

4. For Source, choose the Kinesis Stream radio button.

5. In the Kinesis Stream drop-down, select the Kinesis Stream created for Cordial.

6. Click Next and follow the AWS wizard to finish setting up your Firehose delivery system.

Set up Firehose delivery system

The following steps will vary depending on whether or not you need to convert or transform the data and your selected delivery destination.

• Define destination: Choose the preferred destination.
• Send data to S3: From here your developers write code to consume the data from S3.
• Send data to Redshift: Send data to Redshift and configure any transformations and how you'd like the data to load. From here your developers map the data into Redshift. Developers/Marketers/BI Partners will work together to configure Redshift as a source for the BI tool (Domo, Looker, Tableau, etc.).
• Send data to database for storage (your own or other): From here your developers map the data into the database. Developers/Marketers/BI Partners would work together to configure the Database as a source for the BI tool (Domo, Looker, Tableau, etc.).

By default, Kinesis is a 24-hour stream but can be configured up to a seven-day stream. If you choose to use Firehose consumer, 24 hours is acceptable.

Data collection starts from the time of configuration. Historical data cannot be extracted using this method.