AWS SQS
Description
The AWS SQS Source enables you to stream log data from an SQS topic as a Pipeline source via Amazon's Simple Queue Service (SQS).
Configure SQS
- Log in to the AWS console and navigate to Simple Queue Service.
- On the Amazon SQS landing page, click Create queue.
- For Type, make sure Standard is selected.
- Enter a Name for the service, for example
mezmo-pipeline
. - Leave all the other settings at the default, and at the bottom of the page click Create queue.
- On the Queue details page, select the Access policy tab.
- Click Edit for the Access policy, and enter the JSON definition at the end of these instructions.
- In the JSON definition, enter these values, then click Save.
- For
Resource
, enter the value forARN
in the Details section for SQS. - For
aws:SourceARN
, navigate to your S3 bucket, open the Properties tab, and enter the value forAmazon Resource Name (ARN).
- For
aws:SourceAccount,
enter the value for your AWS user account. You can find your Account ID by selecting your account name in the upper-right corner of the AWS console.
- For
SQS Access Policy JSON Definition
{
"Version": "2012-10-17",
"Id": "example-ID",
"Statement": [
{
"Sid": "example-statement-ID",
"Effect": "Allow",
"Principal": {
"Service": "s3.amazonaws.com"
},
"Action": [
"SQS:SendMessage"
],
"Resource": "SQS-queue-ARN",
"Condition": {
"ArnLike": {
"aws:SourceArn": "arn:aws:s3:*:*:awsexamplebucket1"
},
"StringEquals": {
"aws:SourceAccount": "bucket-owner-account-id"
}
}
}
]
}
AWS Access Key Required IAM Permissions
Name | Purpose |
---|---|
sqs:ReceiveMessage | Operation |
sqs:DeleteMessage | Operation |
Mezmo Source Configuration
- Select the Amazon SQS Source in your Mezmo Telemetry Pipeline and select Edit config.
- Navigate to your SQS queue and copy the URL in the Details section.
- In the Amazon SWS Source configuration, copy the URL into the SQS Queue URL field.
- Enter the information for your AWS Authentication, and the Region for your SQS Queue.
- Select the Compression method to use for your SQS event notifications. The
Compression
setting ofauto
will attempt to use theContent Encoding
,Content-Type
, andkey
suffix to determine which encoding is in use. If the compression setting cannot be determined, it will default tonone
.
Mezmo Configuration Options
Option | Description |
---|---|
SQS Queue URL | The URL of an AWS SQS queue configured to receive S3 bucket notifications for the S3 buckets where your source data is stored. |
AWS Authentication - Access Key ID | The access key ID for your AWS account. |
AWS Authentication - Secret Access Key | The secret access key for your AWS account. |
Compression | The compression format for the S3 objects. |
Region | The name of the region for the S3 bucket. |
Included Metadata
Field | Format | Description |
---|---|---|
file_name | String | This is the name of the file being pulled |
size | Number | Number of bytes to be pulled |
Was this page helpful?