Certificate Batching

Prior to the certificate batching logic, when a defined-CA profile is published to a large number of devices, it can result in significant performance issues on Airwatch Cloud Connectors in the AirWatch environment.  This is due to the extra processing required to issue each device a certificate from the CA, as well as the normal processing required to push the profile to the device.  The performance issues generally lead to CPU spikes and high memory usage on ACC or DS (if no ACC is configured) servers because:

• Certificate generation is a CPU intensive task. Too many concurrent certificate generation requests spike the CPU usage.
• Certificates are stored in memory of the process (w3wp - devicesservices) so that they can be reused for the same device.

AirWatch 7.1+ has certificate batch logic to throttle the deployment of all defined-CA profile publish jobs. These profile publish jobs will be broken into multiple batches at the environment level and a next batch will only be executed if the previous batch has been completed. 

Certificate Batching Logic

The deployment of defined-CA certificate profiles will follow the steps outlined below:

• AirWatch Administrator publishes a profile that contained a defined CA certificate
• Install profile commands are queued for all assigned devices in the Device Command Queue in the AirWatch Database.  These commands are queued in the Held status.  Commands in the Held status will not be processed by devices.
• A ProfilePublishBatchJob is created to process these commands.
• The ProfilePublishJobScheduler selects a batch and batch size, based on the settings configured in the AirWatch Console (under Settings → Installation → Performance Tuning for On-Premises environments).
• A batch detail entity is inserted into the AirWatch DB; the mobilemanagement.CommandQueueHeldBatch_Update stored procedure is called to “release” a number of the Install Profile commands that are currently in the Held status.  The number released will be equal to or less than the overall batch size.
• Periodically, more batches of the Install Profile command will be released until all commands are active (or have successfully been issued to devices).  Three scenarios are outlined below that describe this step in more detail.  This time period is defined by the Profile Publish Batch Job in the AirWatch Scheduler (Settings → Admin → Scheduler for On-Premises environments).

In SaaS and on-premise environments, the default Batch Size is set to 50, and the Profile Publish Batch Job interval is set to 3 minutes.

Scenario 1: All devices are consistently active

• Devices=24000
• Batch size=200

When a profile is published, all 24000 commands are queued in the Held status (status = 7).  The scheduler released the first 200 commands based on the batch size.  The commands are sent to devices (through APNs/Firebase), and devices consume the commands and install the profile.  At the interval defined by the Profile Publish Batch Job, the system will view how many pending install commands are still active.  As all devices are currently active and check in regularly, all of the previously queued commands will be processed, and another batch of 200 will be released.  This process repeats until all 24,000 install commands are released.

Scenario 2: Devices frequently check in (within 30 minutes)

• Devices=24,000
• Batch size=200

During any particular iteration of the Profile Publish Batch Job as outlined above, if {X} number of commands are still in Pending Install state for a period of less than 30 minutes, then only ({batchSize} -{X}) commands will be released in the current iteration.  For example, if the previous batch released 200 commands, but at the time of the next batch only 50 commands have been processed by devices (so that 150 are still active), then only 50 commands will be released at the current batch.  At the time of this iteration, it will ensure that the “active” batch size returns to 200.

Scenario 3:  Devices periodically check in (within 12 hours)

• Devices=24,000
• Batch size=200

During any particular iteration of hte Profile Publish Batch Job, if {Y} number of commands are still in Pending Install state for a period of more than 30 minutes, then {batchSize} number of commands will be queued up to maximum limit of *{batchSize}10 commands in queue. This means that at a single point in time, the maximum of *{batchSize}10 Install Profile commands can be queued in Pending Install status.

Following Scenario 2 above, if a new batch iteration is occurring, and all active Pending Install commands are over 30 minutes old, then a full batch size will be released.  If there are {X} pending commands less than 30 minutes and {Y} pending commands greater than 30 minutes, then {batchSize} - {X} commands will be released, up until a total of *{batchSize}10 overall active commands are queued.

Scenario 4:  Devices do not check in for over 12 hours

• Devices=24,000
• Batch size=200

After a pending install command has been queued for over 12 hours, it no longer counts toward the *{batchSize}10 limit specified in Scenario 3.  The logic outlined in the first 3 Scenarios will continue, not counting any of these older commands.  These “older” commands remain in the command queue and are active, but no longer affect the batching logic.

Relevant Database Tables

• dbo.BatchJob - Profile Publish places an entry with “profileVersionID” 
• dbo.BatchJobDetail - Every Batch Job iteration will place an entry with the number of commands released as “Count”
• deviceCommandQueue.DeviceQueue - Stores commands (profile install commands in this case) for every device