Kloudless API Guide

# System Configuration Guide

# Introduction

Initially, the Kloudless appliance can be run completely self-contained, with a local database and object store for the services it runs. You can configure the appliance by including appropriate values in /data/kloudless.yml. Based on these values, we use SaltStack to enable and configure other services within the appliance that our application relies on. These values are propagated on first boot and by running ke_update_configuration.

For many deployments, we recommend configuring your deployment by customizing a complete kloudless.yml template from the Minimal Configuration section which follows, including configuring an external database.

Alternatively, the appliance can use a mountable drive, placed at /data, to store the database, kloudless.yml configuration file, and generated configuration files.

If preferred, the appliance can instead keep configuration and database state, on the main volume of the virtual machine (or container), but this option is not recommended due to the complication it adds to scaling, re-provisioning, and upgrading.

# Docker Specific Notes

If you already have a running container, the container should be entered through ke_shell. This will ensure that the various utility scripts are executing in the proper environment. For example, via docker exec:

docker exec -ti kenterprise ke_shell

# Minimal Configuration

Generate a configuration file template with this script, and fill in necessary values. The script creates unique private keys to protect your data at rest. You have the only copy of the keys. Keep backups and kept the keys secret.

./ke_config_skel.sh > kloudless.yml

Customize the resulting YAML file for your deployment. It will later be deployed either as an environment variable or on a removable volume.

Your license code can be found on the Enterprise License page.

# ./kloudless.yml - Example Instantiation
hostname: my-kloudless.company.com
license_code: XyzEnterpriseLicenseCodeXyz
...
# db:    # for persisting data outside the instance [host,port,user,password,name]
# redis: # for clustering Kloudless instances

# Applying Changes

Containers will apply configuration from the provided kloudless.yml file or KLOUDLESS_CONFIG environment variable on first boot.

If you're testing out configuration changes interactively, or updating a VM deployment, you can re-apply your latest configuration reflected in /data/kloudless.yml, with the following command:

sudo ke_update_configuration

# kloudless.yml: Configuration File

# Context

All of the default configuration options are shown in /etc/kloudless-defaults.yml. The modifiable configuration files are stored on the data disk in /data/ so that they are preserved across system upgrades.

There are two main configuration files that are used to generate the rest of the system configuration. The configuration scripts mentioned here modify values in /data/kloudless.system.yml. The user modifiable configuration file, which must be manually created, is /data/kloudless.yml. The different configuration values and scripts are described in the following sections. In order to apply the changes made to /data/kloudless.yml use the ke_update_configuration utility.

# Hostname

By default the appliance can be accessed using the IP address that is assigned when it boots. Once a DNS hostname is assigned to the instance (or cluster's load balancer) then that needs to be configured on the appliance in /data/kloudless.yml:

# ./kloudless.yml ... trimmed for brevity
hostname: "kloudless-api.example.com"

For versions >= 1.25.0, there is an optional internal_hostname configuration parameter. If set, internal_hostname will be accepted by the api portal and the developer portal will use it exclusively instead of hostname. For example

# ...
hostname: "kloudless-api.example.com"
internal_hostname: "klouless-devs.example.com"

In the above example, the API gateway will be reachable at kloudless-api.example.com but the developer portal will only be reachable at kloudless-devs.example.com.

# Database

In a majority of deployments we recommend configuring an external database.

PostgreSQL 9.4+ is required. There are two different configuration options available for the backing database.

By default the appliance is configured to use a local PostgreSQL database which stores its data on the configured data drive. This configuration is only recommended for testing and small deployments since the database is not backed up and is not highly available.

The recommended production configuration is to have an external PostgreSQL database. A user, password, and database should be created on the external PostgreSQL host (this should not be shared by any other applications). Once created, add the hostname, port number, database name, username, and password to the system's configuration file /data/kloudless.yml as shown:

# ./kloudless.yml ... trimmed for brevity
db:
    name: 'example'             # database name
    user: 'kloudless'           # database user name
    password: 'examplepassword' # database user password
    host: 'db.example.com'      # database host
    port: 5432                  # database port

Migrating data from an in-container or on-VM database to an external database is possible. Contact support for help with database migration.

# SSL/TLS

When using a hosted database servce, connecting via SSL/TLS is sometimes required. For certain services, our appliance detects the required settings. For example with Azure and IBM cloud we detect the correct settings based on the hostname. However other services may require custom configuration for the connection pooler. In order to connect to other services, custom configuration can be configured as follows (requires Kloudless Enterprise >1.32.6):

# ./kloudless.yml ... trimmed for brevity
db:
    name: 'example'             # database name
    user: 'kloudless'           # database user name
    password: 'examplepassword' # database user password
    host: 'db.example.com'      # database host
    port: 5432                  # database port
    # this snippet will be included in the `pgbouncer` section of
    # the pgbouncer.ini file.
    # Note: Do not set this for Azure or IBM Cloud as it is automatically
    #       configured appropriately.
    pgbouncer: |
        server_tls_sslmode = require
        server_tls_protocols = secure

The above configuration will force connection via SSL/TLS without verifying the server certificate. This should be sufficient for most use cases, but please contact support if it is not.

# Redis

Redis 2.8+ is required for process orchestration and as a task broker within both single and multiple node deployments. There are two configuration options available.

By default a local Redis server is used and this is only usable for single node deployments.

For multi-node deployments, an external Redis instance is required. This can be configured in the /data/kloudless.yml configuration file using a standard Redis URL. Please note that if the password contains any special characters, it must be url encoded before being used here. Also note, that in order to avoid key conflicts, a Kloudless deployment or cluster should have its own Redis database. Please assign a unique database number for a given appliance/cluster:

# ./kloudless.yml ... Other values omitted for brevity
redis: redis://:password@hostname:port/db_number

In order to achieve high performance, as well as ensure resiliency in the case of a large number of connections. The following configuration parameters should be set on the external Redis server:

maxclients 50000
timeout 300

# SSL/TLS

Accessing redis over ssl is possible using a rediss url. If a non-standard CA or certificate is required, it should be included in the configuration file. For example:

# ...
redis: rediss://:password@hostname:port/db_number?ssl_cert_reqs=cert_required&ssl_ca_certs=/usr/local/share/ca-certificates/redis.pem
pki:
#   ... other certs/keys omitted ...
    redis: |
        -----BEGIN CERTIFICATE-----
        ....
        -----END CERTIFICATE-----

The key of the certificated in the pki mapping corresponds to the file name that is specified in the ssl_ca_certs parameter.

# EventBridge

Kloudless requires an AWS IAM access key that provides access to manage and publish events to a custom event bus in an AWS account.

In the AWS IAM console, create a new user and attach the AmazonEventBridgeFullAccess policy to it. The confirmation page should provide the user's Access key ID and a Secret access key. Alternatively, you may use credentials belonging to an existing user provided the user or one of the user's roles or groups includes the AmazonEventBridgeFullAccess policy.

Now you can add the access keys to the system's configuration file as shown below:

# /data/kloudless.yml ... Other values omitted for brevity
eventbridge:
  key: ACCESS_KEY_ID
  secret: SECRET_ACCESS_KEY

# License Management

The license key is a file that ensures your deployment of Kloudless Enterprise is authorized by Kloudless.

# License Code

To have the appliance automatically download the license key, include the license_code YAML key in kloudless.yml with the appropriate "license code" value from the enterprise licenses page.

With this license_code configuration, the appliance will download the license key on first boot, and save it to the default location at /data/license.key.

# ./kloudless.yml ... trimmed for brevity
license_code: XEnterpriseLicenseCodeX

# Management Script for License

You can also view information about the currently configured license with:

sudo ke_manage_license_key show

The appliance will perform validation checks on the License Key as described here.

# Connection Security with SSL/TLS

Making requests over HTTPS to the appliance is recommended for security reasons. In cases where the appliance handles incoming webhooks, many upstream services require that webhook receivers use HTTPS with a valid certificate. There are two ways to use SSL/TLS when communicating with the appliance. The first is to have the appliance host the certificates and negotiate the secure connection on it's own. This is only recommended for single node deployments. The second is to have HTTPS be handled by an intermediate load balancer which then passes HTTP request through to one or more Kloudless Enterprise appliances. Using a load balancer allows multiple Kloudless Enterprise appliances to appear to the client as a single host.

If the appliance will serve HTTPS directly, you will need to copy the certificate and key to the server (for example, via scp). The certificate and key should be placed in the /data directory so that they will be preserved across upgrades. Once in place, SSL can be configured as follows:

# ./kloudless.yml ...
ssl:
    is_configured: true
    local: true
    cert: /data/certificate_file.crt
    key: /data/key_file.pem

If HTTPS is terminated at the load balancer, the following configuration is required:

# ./kloudless.yml ...
ssl:
    is_configured: true
    local: false

# Lets Encrypt for development

In development or testing environments, you can configure the appliance to get Lets Encrypt certificates with certbot. With containers built after 1.30.2, use the custom_cron key in kloudless.yml to specify a certbot command and configure the ssl section to rely on the certificates certbot will fetch, as follows.

Replace example.com (host.example.com, fixme@example.com) with the public hostname of the instance and an email for expiration alerts respectively:

custom_cron: |
  # m h    dom mon dow user  command
  0   */12 *   *   *   root  certbot certonly -n --webroot --webroot-path /var/www/html --agree-tos -d host.example.com -m fixme@example.com

and in the ssl section:

ssl:
    is_configured: true
    local: true
    cert: /etc/letsencrypt/live/host.example.com/fullchain.pem
    key: /etc/letsencrypt/live/host.example.com/privkey.pem

For production deployments, we recommend customers manage certificates directly for visibility. For multi-node deployments the load balancer would handle TLS termination.

# Service White-Labelling & OAuth Keys

Services that use OAuth such as Box and Dropbox require OAuth Client IDs and Secrets to be configured in Kloudless. These keys can be obtained from the relevant service’s developer portal and will then be used by Kloudless to prompt your users to grant access to their account to the associated app.

The keys configured and instructions for setting up Service Keys are accessible in the Developer Portal's Third-Party Services "Configuration" page for each Application, and in the Admin/Internals page for administrators and via the command-line.

# Service Visibility

By default, services that use OAuth are not visible during the authentication process unless default OAuth keys are configured. If you would prefer to display the option to authenticate the service regardless of whether a global default service key has been configured, you can make the service visible though the Admin Portal.

Note that if neither a default key nor an app-specific key is configured for that service, authentication will fail.

To ensure that a service is marked as visible, visit the Admin Portal. The link for doing that is structured like so: https://<your-appliance>/admin/apimodels/service/<service-type-name>/change/ and then restart the application processes/container to ensure that the visibility is propagated.

This configuration should be the same as in our Cloud Version, the only difference would be the different hostname configured for the redirect URIs.

# Per-Service Instructions

For information on how to create and configure keys for individual services, see the Custom keys section for each service in the Custom OAuth Keys page of your appliance's Kloudless Developer Portal.

This page enables Kloudless developers to configure custom OAuth keys for their individual applications. However, the instructions can also be used to create OAuth keys for use by all developers signed up on your Kloudless instance. Once they keys have been created, they can be configured as follows:

sudo ke_manage_service_keys --action add --service service \
--key "OAuth Consumer Key" \
--secret "OAuth Consumer Secret"

Use --help to see a full list of options. --admin must be specified when connecting separate OAuth keys intended to authenticate admin accounts.

This should be done for each of the services that you would like to use with your appliance. To see a list of available services, as well as other options for listing and removing existing service keys, use ke_manage_service_keys --help. Visit /applications/\*/credentials on your Kloudless instance for assistance with the procedure to configure custom OAuth Keys. For multi-node deployments, this only needs to be done on one node per cluster. The keys are securely stored in the database and can be accessed by any node in the cluster as needed.

Restarting the API process is necessary for the changes to take effect:

supervisorctl restart api

# Logs

# Nginx Log Format

As of 1.30.2, json log format is enabled by default. legacy is available for users wanting to continue using the previous format. List desired formats under logging: nginx_format: One or both may be enabled.

logging:
  nginx_format:
  - json
  - legacy

# Log Retention (optional)

The verbosity of the appliance logs as well as their retention policy can be configured via the logging section of /data/kloudless.yml. The following values can be set:

  • level: The logging level of the Kloudless Enterprise applications. Valid values (in order of decreasing verbosity):
    • debug (default)
    • info
    • warning
    • error
  • rotation: sub-section describing retention with the following sub-keys
    • period: How often to rotate logs, valid values are:
      • daily (default)
      • hourly
      • weekly
      • size <size>
        • Example: 1G. Valid units: k for KB, M for MB, and G for GB.
    • count: How many rotations to store (default: 7).

Once the changes have been made to the configuration file, they can be applied via sudo ke_update_configuration.

# Logging Export & Customization (optional)

The Kloudless Enterprise Appliance can be configured to send its syslog logs to a remote syslog server or external logging service (like Splunk or Logstash). This can be done by modifying logging value in the /data/kloudless.yml file. By default the logging key contains safe defaults that only allow local logging. As an example, a remote rsyslog server using the RELP protocol can be configured as follows:

# ./kloudless.yml ... other values trimmed for brevity ...
logging:
  protocol: relp
  host: logging.example.com
  port:  20514

The following protocols are supported:
TCP
UDP
RELP

In order to prevent data loss, the RELP protocol is recommended, however it is not supported everywhere so TCP and UDP are offered as alternatives. Once the changes have been made, they can be applied with sudo ke_update_configuration.

For more advanced set ups, we allow the end user to include custom rsyslog configuration snippets. The configuration snippet should be set as the value of the custom key in the logging section. For example to configure logging via TLS:

# ./kloudless.yml ...
logging:
  custom: |
    $DefaultNetstreamDriverCAFile /data/server-cert.pem
    $DefaultNetstreamDriver gtls # use gtls netstream driver
    $ActionSendStreamDriverMode 1 # require TLS
    $ActionSendStreamDriverAuthMode anon # server is unauthd
    *.* @@logs.example.com:10514

This particular example requires the additional system package rsyslog-gnutls which can be installed using the mechanism described in the Extra Packages section.

# Data Disk Configuration

# For VMs

Configuration of a data disk is recommended or necessary in the following deployment types:

  • In long-lived Virtual Machines (VMs), it persists configuration on disk in an easy-to-migrate format across updates.
  • In single-node configurations with no external database in-use, account data can be persisted in the internal database instead.

Configuration of a data disk is unnecessary for Docker containers. Please see the Docker Mount section below instead.

In order to preserve configuration information across system upgrades, it is required that an extra disk be configured. This step is not required for Docker, since the extra volume is automatically mounted. For other platforms, finding the relevant block device can be done with the following command:

lsblk

Use the name of the disk that doesn't have any partitions and is currently not mounted when running the following command (e.g. if sdb is the extra disk, /dev/sdb should be entered when prompted):

sudo ke_configure_data_disk

This utility will format the specified drive and copy over any existing configuration and data from the original /data/ directory. This also updates the Kloudless YAML configuration file as shown below:

# /data/kloudless.yml
data_drive: /dev/xvdb

# Docker Mount

For Docker, a volume mounted to /data will cause the appliance to symlink /var/log to the /data/logs subdirectory, to persist logs, and also handles persistent storage of configuration data.

docker run ... \
        --volume /host/dir/path:/data \
        docker.kloudless.com/prod:1.99.x

This is optional provided all keys have been configured within the Kloudless YAML configuration itself.

# Monitoring (optional)

The Kloudless Enterprise Appliance can be configured to send system and application-level statistics to a StatsD receiver or InfluxData stack. The InfluxData TICK stack is included in the appliance and is enabled by default for nodes with no remote Redis server configured. This means that all single Kloudless instances will automatically run the InfluxData stack. All data related to the TICK stack is persisted on the data disk to ensure that it is preserved across system upgrade. Metric data in the embedded stack is retained for two weeks. To toggle the TICK stack on a specific node, change the configuration in /data/kloudless.yml as follows:

# ./kloudless.yml ...
influxdb:
    enabled: false  # To ensure it is enabled, set to true

Nodes in a cluster should send metrics to a common external InfluxData server instead. To enable metrics tracking to an external InfluxDB server, the configuration below should be used instead:

# ./kloudless.yml ...
influxdb:
    url: http://influxdb.yourdomain.com:8086
    database: YOUR_DATABASE  # (default: telegraf)
    retention_policy: YOUR_RP # (default: null)
    username: YOUR_USER  # (default: null)
    password: YOUR_PASSWORD  # (default: null)
    prefix: YOUR_PREFIX # prefix added to all metrics (default: '')

The statsd sender and influxdb sender are mutually exclusive. The InfluxDB output is recommended since it provides more metadata (in the form of tags) and more robust querying for each metric. To enable sending metrics to an external StatsD server instead, the following configuration should be used:

# ./kloudless.yml ...
statsd:
    host: stats.YOURDOMAIN.com
    port: 8125
    prefix: kloudless # prefix added to all metrics (default: '')

To send metrics to an external service other than InfluxDB or StatsD, please contact support@kloudless.com to discuss options.

For more information on metrics available, review Metrics and Monitoring.

# Extra Packages (optional)

For some deployment situations, extra package might be required so that the Kloudless Appliance can integrate with existing infrastructure. In order to maintain extra installed packages a list should be maintained in /data/kloudless.yml as follows:

# ./kloudless.yml ...
extra_packages:
  - "tmux"
  - "rsyslog-gnutls"

After the list has been updated, the packages will be installed by running ke_update_configuration. Note: Removing packages from this list will not remove them from the system. Any packages installed via this method must be manually removed.

← Developer Portal →