Kloudless API Guide

# Service Specific Configuration

There are certain services that require configuration for system wide operation of uploading/downloading files, events, and more.

# Amazon S3 Event (Bucket) Notifications

To allow for event notifications for the S3 service, a service key must be configured for S3. Unlike the other Service Keys, this is not used for auth, but is used for creation and management of infrastructure needed to collect events from S3, such as regional SNS topics and SQS queues. The key can be added with the following command:

ke_manage_service_keys --action add --service s3 \
    --key "aws_access_key_id" --secret "aws_secret_access_key"

The AWS access keys used here should not be reused between different Kloudless Enterprise deployments or clusters as this will result in conflicts when trying to fetch events. The IAM credentials used must have the following permissions:

iam:GetUser
sqs:* (for ks3-eventq-* SQS queues)
sns:* (for ks3-event-* SNS topics)

Example Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1444682999999",
            "Effect": "Allow",
            "Action": [
                "iam:GetUser"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Stmt1444682767000",
            "Effect": "Allow",
            "Action": [
                "sqs:*"
            ],
            "Resource": [
                "arn:aws:sqs:*:*:ks3-eventq-*"
            ]
        },
        {
            "Sid": "Stmt1444682948000",
            "Effect": "Allow",
            "Action": [
                "sns:*"
            ],
            "Resource": [
                "arn:aws:sns:*:*:ks3-event-*"
            ]
        }
    ]
}

Kloudless subscribes to bucket notifications for an S3 account, which will be sent to your AWS SNS topic. Kloudless creates an SQS queue for each region S3 buckets are present in. Each bucket will be configured to send events to the SNS topic. Kloudless subscribes the SQS queue to the appropriate topic. Each queue’s events will be retrieved by Kloudless Enterprise, processed, and stored for access through the API. Any configured Webhooks will be notified once the events are available.

# Troubleshooting

If Events aren’t available at the events endpoint for an S3 account, grep the logs for the ID of the connected S3 account for further information. e.g.: ke_logs | grep 12345 on each node. This may yield valuable information such as error messages that occur while connecting to S3. If it is still unclear where the issue lies, the steps below may assist with identifying the cause.

Be sure to disconnect and reconnect the account if it was connected prior to S3 keys being added via ke_manage_service_keys. Confirm the following:

  • SNS topics for the connected accounts are visible in SNS for the region you’re modifying the bucket in.
    • For example, if data is being modified in an S3 bucket in ap-southeast-2 (https://console.aws.amazon.com/sns/home?region=ap-southeast-2), should show an SNS topic named ks3-event-topic-User_AWS_Account_Number. If the topics are not present, the AWS keys configured may not have the permission to create SNS topics in your account.
  • SQS queues for the connected accounts are visible in SQS for the region you're modifying the bucket in.
    • For example, if data is being modified in an S3 bucket in ap-southeast-2 (https://console.aws.amazon.com/sns/home?region=ap-southeast-2), should show an SQS queue named ks3-eventq-Kloudless_Account_ID-AWS_Key_ID. If the queues are not present, the AWS keys configured may not have the permission to create SQS queues in your account.
  • A subscription from the SQS queue to the SNS topic. This is found under attributes of the SNS topic.
  • Additionally, verify the IAM policy in the previous section is in place for the AWS Keys used.

If the queues are present, check if there are messages available on them. If there are queues with no messages, there could be an issue with configuring the S3 bucket's notifications. Click on the appropriate queue for the Kloudless Account in question and verify that the bucket you are modifying data in is listed under Permissions -> Conditions as one of the aws:SourceArn fields. If you have access to the S3 bucket, click on it and check its Properties -> Events to verify that an Event Notification entry exists. Click on the 'Edit' pencil icon next to it and verify that the Events list contains ObjectCreated (All) and ObjectRemoved (All), and the notification points to the right SNS topic you identified previously for that Kloudless Account.

Feel free to reach out to us at support@kloudless.com if you continue to require any assistance.

# Sending email with attachments

To send email with attachments through the Kloudless Email API, you must first configure a Kloudless storage account to function as the temporary attachment upload location. We recommend using an Amazon S3 account as your temporary upload location. This is the configuration we use, and it is well-tested on our cloud server.

Configure an S3 account for email attachment uploads and downloads with the following steps:

  1. Create an S3 bucket for email attachment uploads.
  2. Create an S3 bucket lifecycle rule to expire objects after 1 day to automatically remove attachments.
  3. Create an IAM User with the following permissions:
    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:GetBucketLocation",
            "Resource": "arn:aws:s3:::*"
        },
        {
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::your-bucket-name"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::your-bucket-name/*"
        }
    ]
}
  4. Retrieve the Kloudless account ID for the S3 account by connecting the account via the API Explorer using the credentials you just created.
  5. Retrieve the Kloudless folder ID for the S3 bucket with the following cURL command:
    curl -X POST 'https://<your-appliance>/v1/accounts/me/encode_raw_id/' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer [TOKEN]' \
    -d '{"raw_id": "your-bucket-name/", "api": "storage", "type": "folder"}'
    The Kloudless folder ID can also be found via the API Explorer. Use the /v1/accounts/{account_id}/storage/folders/{folder_id}/contents/ endpoint to list the contents of the root folder and find the Kloudless folder ID corresponding to the bucket name.
  6. In the Admin portal, update the Appliance-wide Configuration for storage.files.create.ephemeral with the following JSON string:
    {"account_id": <Kloudless Account ID>, "parent_id": <Kloudless Folder ID>}

← UI Tools →