Amazon MSK IoT Ingest and Control

Estimated time to complete 10-15 minutes.

Overview

The Zilla Plus for Amazon MSK IoT Ingest and Control MQTT Broker lets clients publish messages and subscribe to topics that are proxied to Kafka topics in your Amazon MSK cluster.

Prerequisites

Before creating an IoT Ingest and Control MQTT Broker to your Kafka cluster, you will need the following:

a Kafka cluster configured for SASL/SCRAM authentication
a custom DNS wildcard domain, for this guide we will use *.example.aklivity.io
a TLS Server Certificate stored in Secrets Manager for your custom DNS wildcard domain

Info

Note the server certificate secret ARN as we will need to reference it from the IoT Ingest and Control CloudFormation template.

Make sure you have selected the desired region, ex: US East (N. Virginia) us-east-1.

If you need help creating a TLS Server Certificate you can follow the Create Server Certificate with LetsEncrypt guide. Use your own custom DNS wildcard domain in place of the example domain *.example.aklivity.io.

AWS services used

Service	Required	Usage	Quota
Secrets Manager	Yes	Startup only	Not reached
Certificate Manager	No Private key and certificate can be inline in Secrets Manager instead	Startup only	Not reached
Private Certificate Manager	No Private key and certificate can be inline in Secrets Manager instead	Startup only	Not reached

Default AWS Service Quotas are recommended.

Tips

Check out the Troubleshooting guide if you run into any issues.

Create the Amazon MSK cluster

This creates your MSK cluster in preparation for secure access via the internet.

An MSK cluster is needed for secure remote access via the internet. You can skip this step if you have already created an MSK cluster with equivalent configuration.

Follow the Create MSK Cluster guide to setup the a new MSK cluster. We will use the below resource names to reference the AWS resources needed in this guide.

Cluster Name: my-zilla-iot-msk-cluster
Access control methods: SASL/SCRAM authentication

With a running MSK Cluster you will need to create these topics. You can reference this AWS developer guide for help creating topics.

Topics:
- With cleanup policy "delete" to store MQTT messages:
```
mqtt-messages
```
- With cleanup policy "compact" to store MQTT retained messages:
```
mqtt-retained
```
- With cleanup policy "compact" to store MQTT sessions:
```
mqtt-sessions
```

Create a Secret with SASL/SCRAM authentication params

When the MSK cluster is created you will need to follow the Sign-in credentials authentication with AWS Secrets Manager to associate your AmazonMSK_* secret to your cluster. There will be a prompt on the cluster summary page to create a new secret or associate an existing one. For the remainder of this doc we will assume the following values for this secret:

Secret Type: Other type of secret

Value:

Plaintext JSON object:

{
  "username": "alice",
  "password": "alice-secret"
}

Encryption key: <Customer managed key>
Secret Name: AmazonMSK_access
Review and store the secret

Important

A secret created with the default AWS KMS key cannot be used with an Amazon MSK cluster.

Go to the newly created secret and under Resource permissions > select Edit Permissions. Add the below AWSKafkaResourcePolicy and replace the resource secret name with your SASL/SCRAM auth secret name.

{
  "Version" : "2012-10-17",
  "Statement" : [ {
    "Sid" : "AWSKafkaResourcePolicy",
    "Effect" : "Allow",
    "Principal" : {
      "Service" : "kafka.amazonaws.com"
    },
    "Action" : "secretsmanager:getSecretValue",
    "Resource" : "arn:aws:secretsmanager:*:*:secret:<SASL/SCRAM auth secret name>*"
  } ]
}

Zilla proxy AWS resources

We need to create the AWS resources necessary to allow access to the IoT Ingest and Control MQTT Broker from the internet, the Zilla proxy instances access to Secrets Manager secrets, and SSH access to the instances for troubleshooting.

Create the VPC

This creates your Zilla proxy VPC in preparation for access from an MQTT client.

Create a VPC plus other VPC resources with the below resource names.

Name tag auto-generation: my-zilla-iot-proxy
VPC endpoints: none
Create the VPC

Create the security group

This creates your Zilla proxy security group to allow MQTT clients and SSH access.

A VPC security group is needed for the Zilla proxies when they are launched.

Follow the Create Security Group wizard with the following parameters and defaults. This creates your Zilla proxy security group to allow MQTT clients and SSH access.

Check your selected region

Make sure you have selected the desired region, ex: US East (N. Virginia) us-east-1.

Name: my-zilla-iot-proxy-sg
VPC: my-zilla-iot-proxy-vpc
Description: MQTT clients and SSH access
Add Inbound Rule
- Type: CUSTOM TCP
- Port Range: 8883
- Source type: Anywhere-IPv4
Add Inbound Rule
- Type: SSH
- Source type: My IP
Add Outbound Rule (if not exists)
- Type: All traffic
- Destination: Anywhere-IPv4
Create the Security Group

Check your network settings

Your IP may be different when you SSH into the EC2 instance. VPNs and other networking infrastructure may cause the My IP inbound rule to fail. Instead, you can use one of the other ways AWS provides to execute commands in an EC2 instance.

Create the IAM security role

This creates an IAM security role to enable the required AWS services for the Zilla proxies.

Navigate to the Create role form and fill out the form with the following details:

Region: Global
Trusted Entity Type: AWS Service
Choose a use case: EC2

Add Permissions policies

AWSCertificateManagerReadOnly
AmazonSSMManagedInstanceCore

Role Name:
```
my-zilla-iot-role
```

Click Create role

Open the newly created role

Use the Add permissions > Create Inline Policy action from the dropdown

JSON Summary

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": [
        "arn:aws:secretsmanager:*:*:secret:<TLS certificate private key secret name>*",
        "arn:aws:secretsmanager:*:*:secret:<SASL/SCRAM auth secret name>*"
      ]
    }
  ]
}

If you used a different secret name.

The IAM role Inline policy uses the Secrets Manager ARN regex pattern arn:aws:secretsmanager:*:*:secret:<Secret Name>*. Make sure you replace the resources listed with the appropriate patterns.

Name
```
MyZillaIotProxySecretsManagerRead
```

The Zilla Plus for Amazon MSK) is available through the AWS Marketplace. You can skip this step if you have already subscribed to Zilla Plus for Amazon MSK via the AWS Marketplace.

To get started, visit the Proxy's Marketplace Product Page and Subscribe to the offering. You should now see Zilla Plus for Amazon MSK listed in your AWS Marketplace subscriptions.

Deploy the IoT Ingest and Control MQTT Broker

This initiates deployment of the Zilla Plus for Amazon MSK stack via CloudFormation.

Navigate to your AWS Marketplace subscriptions and select Zilla Plus for Amazon MSK to show the manage subscription page.

From the Agreement section > Actions menu > select Launch CloudFormation stack
Select the CloudFormation Template > IoT Ingest and Control fulfillment option
Make sure you have selected the desired region selected, such as us-east-1
Click Continue to Launch
- Choose the action Launch CloudFormation

Click Launch to complete the Create stack wizard with the following details:

Step 1. Create Stack

Prepare template: Template is ready
Specify template: Amazon S3 URL
- Amazon S3 URL: (auto-filled)

Step 2. Specify stack details

Stack name

my-zilla-iot-proxy

Parameters:

Network Configuration
- VPC: my-zilla-iot-proxy-vpc
- Subnets: my-zilla-iot-proxy-subnet-public-1a my-zilla-iot-proxy-subnet-public-1b
Amazon MSK Configuration
- Access Credentials Secret ARN: <my-zilla-iot-access-secret secret ARN> *1
- Bootstrap Servers: <my-zilla-iot-msk-cluster bootstrap servers>
- Kafka Topics:
  - messages: mqtt-messages
  - retained: mqtt-retained
  - sessions: mqtt-sessions
Zilla Plus Configuration
- Instance count: 2
- Instance type: t3.small *2
- Role: my-zilla-iot-role
- Security Groups: my-zilla-iot-proxy-sg
- Public Port: 8883
- Public Wildcard DNS: *.example.aklivity.io *3
- Public TLS Certificate Key: <TLS certificate private key secret ARN> *4
- Key pair for SSH access: my-key-pair *5
*Configuration Reference
1. This is the ARN for the secret created in the Create a Secret with SASL/SCRAM authentication params step of this guide.
2. Consider the network throughput characteristics of the AWS instance type as that will impact the upper bound on network performance.
3. Replace with your own custom wildcard DNS pattern mentioned in the Prerequisites of this guide.
4. This is the ARN of the created secret for the signed certificate's private key mentioned in the Prerequisites of this guide.
5. Follow the Create Key Pair guide to create a new key pair to access EC2 instances via SSH.

Step 3. Configure stack options: `(use defaults)`

Step 4. Review

Confirm the stack details are correct and Submit to start the CloudFormation deploy.

Info

When your Zilla proxy is ready, the CloudFormation console will show CREATE_COMPLETE for the newly created stack.

Configure Global DNS

This enables MQTT clients to connect to the Zilla proxy via your custom DNS wildcard domain.

When using a wildcard DNS name for your own domain, such as *.example.aklivity.io then the DNS entries are setup in your DNS provider.

Navigate to the CloudFormation console. Then select the my-zilla-iot-proxy stack to show the details.

In the stack Outputs tab, find the public DNS name of the NetworkLoadBalancer. You need to create a CNAME record mapping your public DNS wildcard pattern to the public DNS name of the Network Load Balancer.

Info

You might prefer to use an Elastic IP address for each NLB public subnet, providing DNS targets for your CNAME record that can remain stable even after restarting the stack.

For testing purposes you can edit your local /etc/hosts file instead of updating your DNS provider.

Verify MQTT Client Connectivity

We need to verify that an MQTT client can publish and subscribe to topics and ensure the messages get routed through your Kafka cluster.

Connect with an MQTT Client

This verifies MQTT client connectivity to your Kafka cluster via the IoT Ingest and Control MQTT Broker.

We can now verify that the MQTT client can successfully communicate with your Kafka cluster.

Warning

Replace these TLS server names accordingly for your own custom wildcard DNS pattern.

Using eclipse-mosquitto subscribe to the zilla topic.

docker run -it --rm eclipse-mosquitto \
mosquitto_sub --url mqtts://mqtt.example.aklivity.io/zilla

Publish to a topic

In a separate session, publish a message on the zilla topic.

docker run -it --rm eclipse-mosquitto \
mosquitto_pub --url mqtts://mqtt.example.aklivity.io/zilla --message 'Hello, world'

You should see the Hello, world message printed by the subscriber.

Verify the Zilla proxy Service

This checks that the services and networking were properly configured. These checks aren't necessary if the service is running as expected.

Navigate to the EC2 running instances dashboard.

Check your selected region

Make sure you have selected the desired region, ex: US East (N. Virginia) us-east-1.

Select either of the Zilla proxies launched by the CloudFormation template to show the details.

Info

They each have an IAM Role name starting with my-zilla-iot-role.

Find the Public IPv4 Address and then SSH into the instance.

ssh -i ~/.ssh/<key-pair.cer> ec2-user@<instance-public-ip-address>

After logging in via SSH, check the status of the zilla-plus system service.

Service is running

Verify that the zilla-plus service is active and logging output similar to that shown below.

systemctl status zilla-plus.service

zilla-plus.service - Zilla Plus
   Loaded: loaded (/etc/systemd/system/zilla-plus.service; enabled; vendor preset: disabled)
   Active: active (running) since...

Check Ports

Check for the active ports with netstat.

netstat -ntlp

tcp6    0    0 :::9092    :::*    LISTEN    1726/.zpm/image/bin

Check Zilla Logs

You can get an stdout dump of the zilla-plus.service using journalctl.

journalctl -e -u zilla-plus.service | tee -a /tmp/zilla.log

systemd[1]: Started zilla-plus.service - Zilla Plus.
...

Check Cloud Init Logs

All output from cloud-init is captured by default to /var/log/cloud-init-output.log. There shouldn't be any errors in this log.

cat /var/log/cloud-init-output.log

Cloud-init v. 22.2.2 running 'init'...

Conclusion

You have successfully deployed the Zilla Plus for Amazon MSK IoT Ingest and Control MQTT Broker. Instructions on how to Monitor and Upgrade your Zilla proxy can be found in the managing a cloudformation stack section.