Configuring an Amazon Kinesis Stream

Configuring an Amazon Kinesis Data Stream allows you to export, process and analyze Cordial event data as it is being created, enabling you to respond instantly instead of having to wait for your data to be collected, packaged and delivered through one-off or recurring exports.

Learn more about Amazon Kinesis

Once your Kinesis Stream is up and running, Cordial will place all Contact Activities, (system events and custom named events) onto the Kinesis Stream in real-time, as they are created.  

Clients may read (consume) the events directly from the stream with their own custom processes, with the help of Amazon's Kinesis Data Firehose product, or both.

The following steps are needed to set up your Kinesis Stream:

1. Client creates a Kinesis Stream and allows Cordial to write to the stream.
2. Client enables the Amazon Kinesis Integration in the Cordial Marketplace.
3. Client begins consuming the stream.
   For many use cases, Cordial recommends using Kinesis Data Firehose to help with consumption of the stream, particularly when the destination will be Redshift, S3, Amazon Elasticsearch Service, or Splunk.

Prerequisites:

You must have an AWS account along with permissions to create a Kinesis Data Stream, IAM Policies, and IAM Users.

Create a Kinesis Stream

1. Detailed instructions to creating and managing the stream can be found on the AWS website.
2. At the time of this writing, if you are using the AWS Management Console to create the stream, you can go directly to the Console Home Page and click the Create Data Stream button.
3. Give the stream a name (e.g. cordial-events).
4. Enter the number of shards to use for the stream.
   Note: There are limits to the rate that a single shard can be written to/read from, so you should adjust this number based on your expected traffic levels and event types. It is possible to resize the number of shards later.
   
5. After clicking Create Stream button, you should see it (being) created in the list of streams in your account. Click the stream name in the list to view its details.
   
   Note the Stream ARN as you will need this value in the following Create Policy step.
   

Create an IAM Policy

Create an IAM policy that allows (just) writing to the stream.

1. Go to the Policies page on your AWS console.
2. Click Create Policy button.
3. Click the JSON tab.
4. Enter the following JSON and replace ARN_from_step_1 with the Stream ARN value from step 1.

   {
   	"Version": "2012-10-17",
   	"Statement": [{
   		"Effect": "Allow",
   		"Action": [
   			"kinesis:PutRecord",
   			"kinesis:PutRecords",
   			"kinesis:DescribeStream",
   			"kinesis:ListShards"
   		],
   		"Resource": "ARN_from_step_1"
   	}]
   }

   
5. Click the Review policy button, give the policy a name (e.g. cordial-kinesis-write-access) and click Create.

Create an IAM User

Create an IAM user and allow cordial to write to the stream.

1. Go to the Users page in your AWS Management Console
2. Click Add User button.
3. Enter a user name (e.g. cordial-kinesis-writer).
4. Select the checkbox to give it programmatic access, then click the Next: Permissions button.
   
5. Select the Attach Existing Policies Directly category.
6. Search for and select the name of the Policy created in step 2.
7. Click the Next: Review button.
   
8. Click Create user button.
9. You'll see a success message with options to view or download the access credentials for this user. You will need to supply these when enabling Kinesis integration in the Cordial Marketplace.

Complete Configuration in the Cordial Marketplace

After obtaining all of your AWS access credentials, visit the Cordial Marketplace to enable Amazon Kinesis Integration with your Cordial account in order to write to your stream.

From this point, you simply need to supply the Kinesis Stream Name, AWS User's API Key and AWS User's Secret Access Key when prompted. Once configured, Cordial events will begin appearing in your Kinesis Stream.

Note: We also support an exclusion filter if you would like to exclude certain events from being added to the stream (probably not recommended for most use cases).

Set Up Kinesis Firehose (optional)

Depending on the use case, you may find it helpful to have Kinesis Firehose automatically consume the stream and copy the events to a particular destination.

1. Go to the Kinesis Firehose page in your AWS Management Console.
2. Click the Create Delivery Stream button.
3. Give the delivery stream a name (e.g. cordial-events-delivery).
4. For Source, choose the Kinesis Stream radio button.
5. In the Kinesis Stream drop-down, select the Kinesis Stream created for Cordial.
6. Click Next, and follow the AWS wizard to finish setting up your Firehose delivery system.
   

Note: These steps will vary depending on whether or not you need to convert or transform the data and your selected delivery destination.

Define Destination

Choose the preferred destination.

Send data to S3

• From here your developers would write code to consume the data from S3.

Send data to Redshift

• Send data to Redshift and configure any transformations and how you'd like the data to load.
• From here your developers map the data into Redshift.
• Developers/Marketers/BI Partners will work together to configure Redshift as a source for the BI tool (Domo, Looker, Tableau, etc.).

Send data to your own or other database for storage

• From here your developers map the data into the database.
• Developers/Marketers/BI Partners would work together to configure the Database as a source for the BI tool (Domo, Looker, Tableau, etc.).

Things To Note

• By default, Kinesis is 24-hour stream but can be configured up to a 7-day stream. If you choose to use Firehose consumer, then 24 hours is acceptable.
• Data collection starts from the time of configuration. Historical data cannot be extracted using this method.