# Introduction

Kloudless Activity Monitoring provides an Activity API that allows your app to react to changes in connected cloud accounts. Bring reactivity to your application by leveraging Kloudless' ability to trigger actions to add functionality and awareness into your application.

This section covers how to enable Activity Monitoring and several ways to implement it in your Kloudless application.

Click here to see our full Activity Monitoring Documentation.

# How to Enable Activity Monitoring

To enable the Kloudless app to begin performing activity collection, log in to your Kloudless account and navigate to the Activity Monitoring Configuration section. Check Track Activity to enable activity monitoring. When you enable activity monitoring, it only affects cloud accounts that connect to the app after enabling activity monitoring. Any accounts connected before enabling tracking need to be re-connected or updated. To create an activity monitoring Subscription for previously connected accounts, use the Create Subscription API endpoint with an empty request body.

# Common ways to use Kloudless Activity Monitoring

You’ve gone through the various steps to enable Activity Monitoring; now you want to leverage them in an application. How will you keep abreast of changes in the activity stream? Periodic polling for new cursors? Apply webhooks to notify the app when new activity is ready? Will this monitoring cover all connected accounts, or only specific accounts? Kloudless can be used to support all these needs via the Activity API endpoints as described below. You can check out our full docs here.

# Polling the Activity API Endpoint

The most straightforward way to leverage activity monitoring with Kloudless is to regularly poll the /activity endpoint, and monitor for changes to the cursor value.

Your application needs to keep track of the cursor attribute. If a cloud account has been connected to the application, your app also needs to be retrieve an initial cursor to begin monitoring. If no cursor value is present, you may use the retrieve a subscription endpoint to get the last cursor from the Subscription object's last_cursor attribute, or simply make a request without any cursor for activity data beginning since the account was connected. Here is an example of what a curl request to retrieve the subscription metadata might look like:

curl -X GET --header 'Authorization: Bearer TOKEN' \
    'https://api.kloudless.com/v1/accounts/me/subscriptions/default'

Once the application is tracking the cursor attribute it becomes possible to know when there is new activity that the application can retrieve.

curl -X GET --header 'Authorization: Bearer TOKEN' \
    'https://api.kloudless.com/v1/accounts/me/subscriptions/default/activity' \
    -G -d cursor='123456789'

Polling has some performance drawbacks as the application grows, so a more efficient way is to only poll when a webhook from Kloudless is received, as described below.

# Advanced ways to Monitor Activity

Kloudless also supports other methodologies that can be used to maintain awareness of connected accounts.

# Use Webhooks to Notify an Application

Kloudless allows you to create webhooks that notify the application when activity has occurred in the resource you are subscribed to. Kloudless webhooks are much more efficient than polling since Kloudless notifies your application when changes are present on an account. We highly recommend configuring webhooks for your Kloudless project for this reason. When that webhook fires, make a query for the latest activity on the cloud account. You can set up a webhook on the app's Activity Monitoring Configuration page.

webhook

# Time-based Monitoring (Admin accounts only)

Rather than trigger queries for new activity by polling the endpoint for new cursors, or triggering queries by webhook notification, you can also check for new activity within a specific time frame. This way, you can manage batches of the activity data. Currently, this approach requires admin cloud account access. Kloudless supports the following cloud services with this feature:

  • Box box
  • Sharepoint sharepoint
  • OneDrive for Business onedrivebiz
  • Dropbox dropbox
  • Google Drive gdrive

While using time-based activity monitoring does generate a cursor, the cursor is now used for pagination when working with a large quantity of activity. The query does not need the cursor to determine which activity results to return, because you are determining the time-frame. The time ranges to query can be managed by your application and incremented as you complete retrieving activity for past time intervals. Here is an example of an API request to retrieve activity using a start and end time:

curl -X GET --header 'Authorization: Bearer TOKEN' \
    'https://api.kloudless.com/v2/accounts/me/subscriptions/default/activity' \
    -d from='ISO-8601 string' \
    -d until='ISO-8601 string'

# Monitoring Individual Resources

It is also possible to monitor specific resources, like Google Shared Drives that a Google Drive account can access. You can set up the resources you want to monitor by using the monitored_resources attribute when creating a Subscription. For example, you may add up to 4 IDs of the Shared Drives you want to monitor to the monitored_resources attribute. You can also extend monitoring to shared Google Calendars using a similar method by adding the additional Calendar IDs to the monitored_resources attribute for the connected cloud account's Subscription. Contact us to have limits raised on the number of resources each of your application's accounts can monitor.

# Parallel Tracking

When working with a large quantity of activity, time-based activity monitoring can help break the large sets of data down into smaller subsets to then pass to other threads. You can more efficiently work through the activity results by spreading the load across threads. This ‘parallelization’ allows you to split the load. We recommend a few steps that can help the application most effectively process these loads and avoid some common issues.

When the application makes a query for a large number of activity results, check the modified attribute of the last activity object retrieved. If this is for an account that was just connected or has previously not had activity, you can use the time the account was connected or any time in the past.

Next, the application needs to calculate the difference between the current time (Kloudless uses an ISO-8601 string format) and the timestamp you found from the modified attribute mentioned above. The calculated value represents the time-period for which the application needs to collect activity results. You can then split this time range between all the workers you have, and set their start/end (from/until) times for each respective worker’s requests. Each worker can then paginate through the time range provided to it using the cursor to advance to the next set of activity results until there are no remaining activity results. When a query for time-based activity returns an empty result, there is no more activity to report.

Here is an example curl request that paginates through a time range using a cursor returned in an earlier response:

curl -X GET --header 'Authorization: Bearer TOKEN' \
    'https://api.kloudless.com/v2/accounts/me/subscriptions/default/activity/' \
    -d from='ISO-8601 string' \
    -d until='ISO-8601 string' \
    -d cursor='123456789' # Optional, for pagination of results

# Race conditions when parallelizing Activity Monitoring

When first gathering activity data for distribution to the application’s workers, it is best to use the timestamp of the oldest activity you've collected as the start of the time range for the next run as mentioned earlier. Even if you query activity for the past 5 minutes, there may only be activity for the first minute of that range available now if the upstream provider is delayed by 4 minutes. Therefore, the next run should use the last known activity's time as the start time when it tries to retrieve future activity.

Another common issue with parallelization can occur when older time slices complete first before your app has caught up. If an older time slice completes before newer time slices, it can lead to a race condition where you miss activity from other time range slices before moving on to a newer time slice. To ensure all activity results are collected, you can schedule the time slices to execute in reverse order so that you ensure all time slices complete before moving onto the next time range.