Kloudless Activity Monitoring provides an Activity API that allows your app to react to changes in connected cloud accounts. Bring reactivity to your application by leveraging Kloudless' ability to trigger actions to add functionality and awareness into your application.
This section covers how to enable Activity Monitoring and several ways to implement it in your Kloudless application.
# How to Enable Activity Monitoring
To enable the Kloudless app to begin performing activity collection, log in to your Kloudless account and navigate to the Activity Monitoring Configuration section. Check Track Activity to enable activity monitoring. When you enable activity monitoring, it only affects cloud accounts that connect to the app after enabling activity monitoring. Any accounts connected before enabling tracking need to be re-connected or updated. To create an activity monitoring Subscription for previously connected accounts, use the Create Subscription API endpoint with an empty request body.
# Common ways to use Kloudless Activity Monitoring
You’ve gone through the various steps to enable Activity Monitoring; now you want to leverage them in an application. How will you keep abreast of changes in the activity stream? Periodic polling for new cursors? Apply webhooks to notify the app when new activity is ready? Will this monitoring cover all connected accounts, or only specific accounts? Kloudless can be used to support all these needs via the Activity API endpoints as described below. You can check out our full docs here.
# Polling the Activity API Endpoint
The most straightforward way to leverage activity monitoring with Kloudless is
to regularly poll
and monitor for changes to the cursor value.
Your application needs to keep track of the
cursor attribute. If a cloud
account has been connected to the application, your app also needs to be
retrieve an initial cursor to begin monitoring. If no
cursor value is
present, you may use the
retrieve a subscription endpoint
to get the last cursor from the Subscription object's
or simply make a request without any cursor for activity data beginning since
the account was connected.
Here is an example of what a curl request to retrieve the subscription metadata
might look like:
curl -X GET --header 'Authorization: Bearer TOKEN' \ 'https://api.kloudless.com/v1/accounts/me/subscriptions/default'
Once the application is tracking the
cursor attribute it becomes possible to
know when there is new activity that the application can retrieve.
curl -X GET --header 'Authorization: Bearer TOKEN' \ 'https://api.kloudless.com/v1/accounts/me/subscriptions/default/activity' \ -G -d cursor='123456789'
Polling has some performance drawbacks as the application grows, so a more efficient way is to only poll when a webhook from Kloudless is received, as described below.
# Advanced ways to Monitor Activity
Kloudless also supports other methodologies that can be used to maintain awareness of connected accounts.
# Use Webhooks to Notify an Application
Kloudless allows you to create webhooks that notify the application when activity has occurred in the resource you are subscribed to. Kloudless webhooks are much more efficient than polling since Kloudless notifies your application when changes are present on an account. We highly recommend configuring webhooks for your Kloudless project for this reason. When that webhook fires, make a query for the latest activity on the cloud account. You can set up a webhook on the app's Activity Monitoring Configuration page.
# Time-based Monitoring (Admin accounts only)
Rather than trigger queries for new activity by polling the endpoint for new cursors, or triggering queries by webhook notification, you can also check for new activity within a specific time frame. This way, you can manage batches of the activity data. Currently, this approach requires admin cloud account access. Kloudless supports the following cloud services with this feature:
- OneDrive for Business
- Google Drive
While using time-based activity monitoring does generate a
cursor is now used for pagination when working with a large quantity of
activity. The query does not need the
cursor to determine which activity
results to return, because you are determining the time-frame. The time ranges
to query can be managed by your application and incremented as you complete
retrieving activity for past time intervals. Here is an example of an API
request to retrieve activity using a start and end time:
curl -X GET --header 'Authorization: Bearer TOKEN' \ 'https://api.kloudless.com/v2/accounts/me/subscriptions/default/activity' \ -d from='ISO-8601 string' \ -d until='ISO-8601 string'
# Monitoring Individual Resources
It is also possible to monitor specific resources, like Google Shared Drives
that a Google Drive account can access. You can set up the resources you want
to monitor by using the
monitored_resources attribute when creating a
Subscription. For example, you may add
up to 4 IDs of the Shared Drives you want to monitor to the
monitored_resources attribute. You can also extend monitoring
to shared Google Calendars using a similar method by adding the additional
Calendar IDs to the
monitored_resources attribute for the connected cloud
Contact us to have limits raised on the number
of resources each of your application's accounts can monitor.
# Parallel Tracking
When working with a large quantity of activity, time-based activity monitoring can help break the large sets of data down into smaller subsets to then pass to other threads. You can more efficiently work through the activity results by spreading the load across threads. This ‘parallelization’ allows you to split the load. We recommend a few steps that can help the application most effectively process these loads and avoid some common issues.
When the application makes a query for a large number of activity results,
modified attribute of the last activity object retrieved.
If this is for an account that was just connected or has
previously not had activity, you can use the time the account was connected or
any time in the past.
Next, the application needs to calculate the difference between the current
time (Kloudless uses an ISO-8601 string format) and the timestamp you found from
modified attribute mentioned above. The calculated value represents the
time-period for which the application needs to collect activity results. You
can then split this time range between all the workers you have, and set their
until) times for each respective worker’s requests. Each
worker can then paginate through the time range provided to it using the
cursor to advance to the next set of activity results until there are no
remaining activity results. When a query for time-based activity returns an
empty result, there is no more activity to report.
Here is an example curl request that paginates through a time range using a cursor returned in an earlier response:
curl -X GET --header 'Authorization: Bearer TOKEN' \ 'https://api.kloudless.com/v2/accounts/me/subscriptions/default/activity/' \ -d from='ISO-8601 string' \ -d until='ISO-8601 string' \ -d cursor='123456789' # Optional, for pagination of results
# Race conditions when parallelizing Activity Monitoring
When first gathering activity data for distribution to the application’s workers, it is best to use the timestamp of the oldest activity you've collected as the start of the time range for the next run as mentioned earlier. Even if you query activity for the past 5 minutes, there may only be activity for the first minute of that range available now if the upstream provider is delayed by 4 minutes. Therefore, the next run should use the last known activity's time as the start time when it tries to retrieve future activity.
Another common issue with parallelization can occur when older time slices complete first before your app has caught up. If an older time slice completes before newer time slices, it can lead to a race condition where you miss activity from other time range slices before moving on to a newer time slice. To ensure all activity results are collected, you can schedule the time slices to execute in reverse order so that you ensure all time slices complete before moving onto the next time range.