# System Configuration Guide
- System Configuration Guide
- kloudless.yml: Configuration File
- License Management
- Connection Security with SSL/TLS
- Service White-Labelling & OAuth Keys
- Data Disk Configuration
- Monitoring (optional)
- Extra Packages (optional)
Initially, the Kloudless appliance can be run completely self-contained, with a
local database and object store for the services it runs. You can configure the
appliance by including appropriate values in
/data/kloudless.yml. Based on
these values, we use SaltStack to enable and configure other services within the
appliance that our application relies on. These values are propagated on first
boot and by running
For many deployments, we recommend configuring your deployment by customizing a
kloudless.yml template from the Minimal Configuration
section which follows, including
configuring an external database.
Alternatively, the appliance can use a mountable drive, placed at
store the database,
kloudless.yml configuration file, and generated
If preferred, the appliance can instead keep configuration and database state, on the main volume of the virtual machine (or container), but this option is not recommended due to the complication it adds to scaling, re-provisioning, and upgrading.
# Minimal Configuration
Generate a configuration file template with this script, and fill in necessary values. The script creates unique private keys to protect your data at rest. You have the only copy of the keys. Keep backups and kept the keys secret.
./ke_config_skel.sh > kloudless.yml
Customize the resulting YAML file for your deployment. It will later be deployed either as an environment variable or on a removable volume.
Your license code can be found on the Enterprise License page.
# ./kloudless.yml - Example Instantiation hostname: my-kloudless.company.com license_code: XyzEnterpriseLicenseCodeXyz ... # db: # for persisting data outside the instance [host,port,user,password,name] # redis: # for clustering Kloudless instances
# Applying Changes
Containers will apply configuration from the provided
kloudless.yml file or
KLOUDLESS_CONFIG environment variable on first boot.
If you're testing out configuration changes interactively, or updating a VM
deployment, you can re-apply your latest configuration reflected in
/data/kloudless.yml, with the following command:
# kloudless.yml: Configuration File
All of the default configuration options are shown in
/etc/kloudless-defaults.yml. The modifiable configuration files are stored on
the data disk in
/data/ so that they are preserved across system upgrades.
There are two main configuration files that are used to generate the rest of the
system configuration. The configuration scripts mentioned here modify values in
/data/kloudless.system.yml. The user modifiable configuration file, which must
be manually created, is
/data/kloudless.yml. The different configuration
values and scripts are described in the following sections. In order to apply
the changes made to
/data/kloudless.yml use the
By default the appliance can be accessed using the IP address that is assigned
when it boots. Once a DNS hostname is assigned to the instance (or cluster's
load balancer) then that needs to be configured on the appliance in
# ./kloudless.yml ... trimmed for brevity hostname: "kloudless-api.example.com"
For versions >= 1.25.0, there is an optional
parameter. If set,
internal_hostname will be accepted by the api portal and
the developer portal will use it exclusively instead of
In a majority of deployments we recommend configuring an external database.
PostgreSQL 9.4+ is required. There are two different configuration options available for the backing database.
By default the appliance is configured to use a local PostgreSQL database which stores its data on the configured data drive. This configuration is only recommended for testing and small deployments since the database is not backed up and is not highly available.
The recommended production configuration is to have an external PostgreSQL
database. A user, password, and database should be created on the external
PostgreSQL host (this should not be shared by any other applications). Once
created, add the hostname, port number, database name, username, and password to
the system's configuration file
/data/kloudless.yml as shown:
# ./kloudless.yml ... trimmed for brevity db: name: 'example' # database name user: 'kloudless' # database user name password: 'examplepassword' # database user password host: 'db.example.com' # database host port: 5432 # database port
Migrating data from an in-container or on-VM database to an external database is possible. Contact support for help with database migration.
Redis 2.8+ is required for process orchestration and as a task broker within both single and multiple node deployments. There are two configuration options available.
By default a local Redis server is used and this is only usable for single node deployments.
For multi-node deployments, an external Redis instance is required. This can be
configured in the
/data/kloudless.yml configuration file using a standard
Redis URL. Please note that if the password contains any special characters, it
must be url encoded before
being used here. Also note, that in order to avoid key conflicts, a Kloudless
deployment or cluster should have its own Redis database. Please assign a
unique database number for a given appliance/cluster:
# ./kloudless.yml ... Other values omitted for brevity redis: redis://:password@hostname:port/db_number
In order to achieve high performance, as well as ensure resiliency in the case of a large number of connections. The following configuration parameters should be set on the external Redis server:
maxclients 50000 timeout 300
Kloudless requires an AWS IAM access key that provides access to manage and publish events to a custom event bus in an AWS account.
In the AWS IAM console, create a new user and attach the AmazonEventBridgeFullAccess policy to it. The confirmation page should provide the user's Access key ID and a Secret access key. Alternatively, you may use credentials belonging to an existing user provided the user or one of the user's roles or groups includes the AmazonEventBridgeFullAccess policy.
Now you can add the access keys to the system's configuration file as shown below:
# /data/kloudless.yml ... Other values omitted for brevity eventbridge: key: ACCESS_KEY_ID secret: SECRET_ACCESS_KEY
# License Management
The license key is a file that ensures your deployment of Kloudless Enterprise is authorized by Kloudless. The license key can be managed in one of a few ways.
# License Code: Auto-fetch (preferred)
To have the appliance automatically download a license key, include the
license_code YAML key in kloudless.yml with the appropriate "license code"
value from the enterprise licenses
license_code configuration, the appliance will download the license
key on first boot, and save it to the default location at
# ./kloudless.yml ... trimmed for brevity license_code: XEnterpriseLicenseCodeX
# Non-Interactive (Docker) License Key Inclusion
There is a non-interactive means of configuring the license key that is
particularly useful when deploying the appliance via Docker containers. The
license key should be copied into the volume that corresponds to the
directory. Once in place, the following configuration value can be set in
/data/kloudless.yml either directly or via the
variable when starting the container:
# ./kloudless.yml ... trimmed for brevity license: /data/license-example.key
# Manual Management of License Keys
You can also manage license keys directly by downloading them from the
Kloudless Enterprise Portal and
ensuring that they are copied onto each instance at the path
/data/license.key (default). You can also have the appliance look for the
license key at a different path by specifying the path as the value of the
license key in
scp -i your_private_key ./license.key firstname.lastname@example.org:~/
You can also use
sftp or another compatible client.
Once the key file is on the server you can configure the application server to use it using the following command:
sudo ke_manage_license_key set ./license.key
This will put the key file in the proper place and allow the API gateway to function properly. You can also view information about a license using the following command:
sudo ke_manage_license_key show ./license.key
Without a license provided via the command line, it will look at the license currently set on the appliance.
The appliance will perform validation checks on the License Key as described here.
# Connection Security with SSL/TLS
Making requests over HTTPS to the appliance is recommended for security reasons. In cases where the appliance handles incoming webhooks, many upstream services require that webhook receivers use HTTPS with a valid certificate. There are two ways to use SSL/TLS when communicating with the appliance. The first is to have the appliance host the certificates and negotiate the secure connection on it's own. This is only recommended for single node deployments. The second is to have HTTPS be handled by an intermediate load balancer which then passes HTTP request through to one or more Kloudless Enterprise appliances. Using a load balancer allows multiple Kloudless Enterprise appliances to appear to the client as a single host.
If the appliance will serve HTTPS directly, you will need to copy the
certificate and key to the server (for example, via
scp). The certificate and
key should be placed in the /data directory so that they will be preserved
across upgrades. Once in place, SSL can be configured as follows:
# ./kloudless.yml ... ssl: is_configured: true local: true cert: /data/certificate_file.crt key: /data/key_file.pem
If HTTPS is terminated at the load balancer, the following configuration is required:
# ./kloudless.yml ... ssl: is_configured: true local: false
# Lets Encrypt for development
In development or testing environments, you can configure the appliance to get
Lets Encrypt certificates with certbot. With containers built after 1.30.2,
custom_cron key in kloudless.yml to specify a certbot command and
ssl section to rely on the certificates certbot will fetch, as
Replace example.com (host.example.com, email@example.com) with the public hostname of the instance and an email for expiration alerts respectively:
custom_cron: | # m h dom mon dow user command 0 */12 * * * root certbot certonly -n --webroot --webroot-path /var/www/html --agree-tos -d host.example.com -m firstname.lastname@example.org
and in the
ssl: is_configured: true local: true cert: /etc/letsencrypt/live/host.example.com/fullchain.pem key: /etc/letsencrypt/live/host.example.com/privkey.pem
For production deployments, we recommend customers manage certificates directly for visibility. For multi-node deployments the load balancer would handle TLS termination.
# Service White-Labelling & OAuth Keys
Services that use OAuth such as Box and Dropbox require OAuth Client IDs and Secrets to be configured in Kloudless. These keys can be obtained from the relevant service’s developer portal and will then be used by Kloudless to prompt your users to grant access to their account to the associated app.
The keys configured and instructions for setting up Service Keys are accessible in the Developer Portal's Third-Party Services "Configuration" page for each Application, and in the Admin/Internals page for administrators and via the command-line.
# Service Visibility
By default, services that use OAuth are not visible during the authentication process unless default OAuth keys are configured. If you would prefer to display the option to authenticate the service regardless of whether a global default service key has been configured, you can make the service visible though the Admin Portal.
Note that if neither a default key nor an app-specific key is configured for that service, authentication will fail.
To ensure that a service is marked as visible, visit the Admin Portal. The link
for doing that is structured like so:
and then restart the application processes/container to ensure that the
visibility is propagated.
This configuration should be the same as in our Cloud Version, the only difference would be the different hostname configured for the redirect URIs.
# Per-Service Instructions
For information on how to create and configure keys for individual services, see the Custom keys section for each service in the Custom OAuth Keys page of your appliance's Kloudless Developer Portal.
This page enables Kloudless developers to configure custom OAuth keys for their individual applications. However, the instructions can also be used to create OAuth keys for use by all developers signed up on your Kloudless instance. Once they keys have been created, they can be configured as follows:
sudo ke_manage_service_keys --action add --service service \ --key "OAuth Consumer Key" \ --secret "OAuth Consumer Secret"
--help to see a full list of options.
--admin must be specified when
connecting separate OAuth keys intended to authenticate admin accounts.
This should be done for each of the services that you would like to use with
your appliance. To see a list of available services, as well as other options
for listing and removing existing service keys, use
ke_manage_service_keys --help. Visit
/applications/\*/credentials on your Kloudless instance for
assistance with the procedure to configure custom OAuth Keys. For multi-node
deployments, this only needs to be done on one node per cluster. The keys are
securely stored in the database and can be accessed by any node in the cluster
Restarting the API process is necessary for the changes to take effect:
supervisorctl restart api
# Nginx Log Format
As of 1.30.2,
json log format is enabled by default.
legacy is available
for users wanting to continue using the previous format. List desired formats
nginx_format: One or both may be enabled.
logging: nginx_format: - json - legacy
# Log Retention (optional)
The verbosity of the appliance logs as well as their retention policy can be
configured via the
logging section of
/data/kloudless.yml. The following
values can be set:
level: The logging level of the Kloudless Enterprise applications. Valid values (in order of decreasing verbosity):
rotation: sub-section describing retention with the following sub-keys
period: How often to rotate logs, valid values are:
count: How many rotations to store (default: 7).
Once the changes have been made to the configuration file, they can be applied
# Logging Export & Customization (optional)
The Kloudless Enterprise Appliance can be configured to send its syslog logs to
a remote syslog server or external logging service (like Splunk or Logstash).
This can be done by modifying
logging value in the
file. By default the
logging key contains safe defaults that only allow local
logging. As an example, a remote rsyslog server using the RELP protocol can be
configured as follows:
# ./kloudless.yml ... other values trimmed for brevity ... logging: protocol: relp host: logging.example.com port: 20514 The following protocols are supported: TCP UDP RELP
In order to prevent data loss, the RELP protocol is recommended, however it is
not supported everywhere so TCP and UDP are offered as alternatives. Once the
changes have been made, they can be applied with
For more advanced set ups, we allow the end user to include custom
configuration snippets. The configuration snippet should be set as the value of
custom key in the logging section. For example to configure logging via
# ./kloudless.yml ... logging: custom: | $DefaultNetstreamDriverCAFile /data/server-cert.pem $DefaultNetstreamDriver gtls # use gtls netstream driver $ActionSendStreamDriverMode 1 # require TLS $ActionSendStreamDriverAuthMode anon # server is unauthd *.* @@logs.example.com:10514
This particular example requires the additional system package
which can be installed using the mechanism described in the Extra Packages
# Data Disk Configuration
# For VMs
Configuration of a data disk is recommended or necessary in the following deployment types:
- In long-lived Virtual Machines (VMs), it persists configuration on disk in an easy-to-migrate format across updates.
- In single-node configurations with no external database in-use, account data can be persisted in the internal database instead.
Configuration of a data disk is unnecessary for Docker containers. Please see the Docker Mount section below instead.
In order to preserve configuration information across system upgrades, it is required that an extra disk be configured. This step is not required for Docker, since the extra volume is automatically mounted. For other platforms, finding the relevant block device can be done with the following command:
Use the name of the disk that doesn't have any partitions and is currently not
mounted when running the following command (e.g. if
sdb is the extra disk,
/dev/sdb should be entered when prompted):
This utility will format the specified drive and copy over any existing
configuration and data from the original
/data/ directory. This also
updates the Kloudless YAML configuration file as shown below:
# /data/kloudless.yml data_drive: /dev/xvdb
# Docker Mount
For Docker, a volume mounted to
/data will cause the appliance to
/var/log to the
/data/logs subdirectory, to persist logs,
and also handles persistent storage of configuration data.
docker run ... \ --volume /host/dir/path:/data \ docker.kloudless.com/prod:1.99.x
This is optional provided all keys have been configured within the Kloudless YAML configuration itself.
# Monitoring (optional)
The Kloudless Enterprise Appliance can be configured to send system and
application-level statistics to a StatsD receiver or InfluxData stack. The
InfluxData TICK stack is included in the appliance and is enabled by default
for nodes with no remote Redis server configured. This means that all single
Kloudless instances will automatically run the InfluxData stack. All data
related to the TICK stack is persisted on the data disk to ensure that it is
preserved across system upgrade. Metric data in the embedded stack is retained
for two weeks. To toggle the TICK stack on a specific node, change the
/data/kloudless.yml as follows:
# ./kloudless.yml ... influxdb: enabled: false # To ensure it is enabled, set to true
Nodes in a cluster should send metrics to a common external InfluxData server instead. To enable metrics tracking to an external InfluxDB server, the configuration below should be used instead:
# ./kloudless.yml ... influxdb: url: http://influxdb.yourdomain.com:8086 database: YOUR_DATABASE # (default: telegraf) retention_policy: YOUR_RP # (default: null) username: YOUR_USER # (default: null) password: YOUR_PASSWORD # (default: null) prefix: YOUR_PREFIX # prefix added to all metrics (default: '')
statsd sender and
influxdb sender are mutually exclusive. The InfluxDB
output is recommended since it provides more metadata (in the form of tags) and
more robust querying for each metric. To enable sending metrics to an external
StatsD server instead, the following configuration should be used:
# ./kloudless.yml ... statsd: host: stats.YOURDOMAIN.com port: 8125 prefix: kloudless # prefix added to all metrics (default: '')
To send metrics to an external service other than InfluxDB or StatsD, please contact email@example.com to discuss options.
For more information on metrics available, review Metrics and Monitoring.
# Extra Packages (optional)
For some deployment situations, extra package might be required so that the
Kloudless Appliance can integrate with existing infrastructure. In order to
maintain extra installed packages a list should be maintained in
/data/kloudless.yml as follows:
# ./kloudless.yml ... extra_packages: - "tmux" - "rsyslog-gnutls"
After the list has been updated, the packages will be installed by running
ke_update_configuration. Note: Removing packages from this list will not
remove them from the system. Any packages installed via this method must be