Рабочие заметки по EMM

Добро пожаловать на базу знаний по продуктам управления рабочими местами пользователей!

История изменений

30.05.2024 15.09.2023 14.09.2023 01.09.2023 25.08.2023 24.08.2023 23.08.2023 21.08.2023 18.08.2023
  • Добавил статью по KSMM API
15.08.2023 08.02.2023 01.02.2023 31.01.2023 26.01.2023 25.01.2023 24.01.2023 20.01.2023
  • Сайт переехал на новую тему, с меньшим числом проблем.
18.02.2022
  • Опубликован основной набор содержимого по Airwatch / Workspace ONE UEM

11.12.2020 Первая версия сайта.

Subsections of Рабочие заметки по EMM

Chapter 1

[VMW] WS1 UEM

Статьи по установке, настройке и решению проблем с Workspace ONE UEM (бывший AirWatch).

Subsections of [VMW] WS1 UEM

AWCM and ECC Troubleshooting

General Rules

  1. Do not install Enterprise Service Connector/AirWatch Cloud Connector (ESC/ACC), until you are absolutely sure AWCM is working
  2. Do not install AWCM or ESC (ACC) in Global tenant
  3. Check there is a Device Root Certificate in the Organization Group, in which work is done. It is located in  \system configuration\system\advanced\device root certificate. If there is nothing, click Generate.

AWCM Installation

  1. When installing AWCM, DO NOT use the self signed SSL certificate, check the box for “custom SSL” which really means the public SSL Cert you put in IIS for Device Services. Notice that in the installation dialog box the two fields for password are NOT the same. One is for the SSL Cert you are importing and one is for the password to the Java Keystore.

  2. Make sure that REST API is enabled in the OG where you are enabling AWCM.

  3. Make sure that AWCM is enabled in the Site URL’s page. Also, put the correct information in the two fields. The External URL should NOT contain http:// or https://. The Internal Service URL should contain https:// instead of http:// and should have the port number after the URL and “/awcm” at the end. It should look like https://{url}:2001/awcm.

  4. Download and run the AWCM Secure Channel Certificate program from the Secure Channel Certificate page ON THE SERVER RUNNING AWCM.

DO NOT download the program onto another computer and copy it to the AWCM server!

Download and run this program “As Administrator”.

There is a possibility that you will receive an error message that the application can’t find the Java Folder, this can be a result of not running the program “As Administrator”.

  1. Browse to the AWCM Status page by going to https://{url}:2001/awcm/status. If this page doesn’t come up or if there is an SSL error stop and fix it before you go on. Check the SSL Certificate common name, it should match the name of the DS URL. If it says “Air Watch “ then you need to uninstall and reinstall AWCM, this time installing the correct SSL Certificate (see #1).

ESC/ACC WILL NOT WORK if you use the self signed certificate!

AWCM Status page MUST BE TRUSTED by AirWatch Console AND by ESC. Test by opening https://{url}:2001/awcm/status status page in browser - there MUST BE NO CERTIFICATE WARNING!

  1. Confirm that the awcm.truststore and the awcm.keystores are not corrupt and contain the correct certificates. Run the keytool application (see next section) and list the contents of both stores.

    • In awcm.keystore there should be 1 certificate and it should contain the SSL certificate for the site.
    • In the awcm.truststore there should be 2 or 3 certificates: one of them is Secure Channel Certificate.

    If the certificates do not exist in the stores then you may need to re-install AWCM. If the password is not accepted then the store may be corrupt and you will need to reinstall AWCM.

Java KeyStore

AWCM is a Java web application and stores its certificates in the Java Keystore as opposed to the Microsoft Certificate store. The Java Keystore and Java Truststore are located in the \airwatch\airwatch {version}\awcm\config folder.

There is a utility in Windows called “keytool”. With this utility you can view, add, and delete certificates from the Java Keystores.

⭐️ Password to awcm.truststore = “password

Password to awcm.keystore = password to the PFX certificate uploaded on installation of AWCM. DO NOT use password less than 6 characters! Or you will not be able to change certificate in awcm.keystore.

Example of keytool commands:

# List the certificates in the store:
keytool -list -v -keystore awcm.truststore

# Import a certificate into a store:
keytool -import -trustcacerts -file {cert file} -alias {common name} -keystore $JAVA_HOME/jre/lib/security/cacerts

Replace database of AWCM

  • Run the following command to replace SSL cert on AWCM servers:
keytool -importkeystore -srckeystore <new-pfx-cert-name>.pfx -srcstoretype pkcs12 -destkeystore awcm.keystore.new -deststoretype JKS
  • Once this has completed successfully, you will now see a new file named awcm.keystore.new in the config directory.
  • Stop the AWCM service.
  • Rename the awcm.keystore to awcm.keystore.old.
  • Rename the awcm.keystore.new to awcm.keystore.
  • Start the AWCM service.

Reinstall of Secure Channel Certificate

If ESC is installed, then uninstall it and delete all its’ folders before reinstall of Secure Channel Certificate.

Before reinstall of Secure Channel Certificate, you must delete the old certificate from the AWCM Java database. Delete a certificate in the store:

keytool -delete -alias "aw secure channel certificate - {url}" -keystore awcm.truststore

AWCM Logs

See General Article on AirWatch Logs.

Verify correct work of AWCM

Verify AWCM is listening on the configured port through netstat:

netstat -ano | findstr LISTENING | findstr {port number} 

Perform the following to make sure that AWCM is functioning accurately:

  • Confirm that there is a device root certificate in the relevant OG by navigating to Settings / System / Advanced / Device Root Certificate.
  • Make sure that REST API is enabled.
  • Make sure that AWCM is enabled in the URL page of the site.
    • The external URL must not contain http:// or https://.
    • Exact format of the URL: https://{url}:2001/awcm.
  • Browse to the AWCM Status page by selecting https://{url}:2001/awcm/status (You should see “OK”) and https://{url}:2001/awcm/statistics

❗️For clustering infrastructure on SaaS, browse over port 443 instead of 2001 (https://awcm118.awmdm.com/awcm/statistics) while testing the status page.

❗️For lor load balanced deployments:

  • Ensure that clients who are required to connect to AWCM are pointed to and are able to reach the endpoint on the load balancer. This means that if installation of AWCM is on the DS servers, then ensure that the requests for AWCM from the DS services are still accessing the load balancer so that they are subject to the set rules;
  • As per the Installation Guide, the preferred deployment for a customer using ESC/ACC with AWCM is to deploy multiple AWCM nodes in an active-passive configuration. This makes everything easier since persistence of connections doesn’t matter. There are no specific advantages with having two active nodes as the network load is not much while using only ESC/ACC.

Load Balancing AWCM - Persistence Rules with F5 LTM

See KB for details: https://kb.vmware.com/s/article/2960904

To deploy AWCM with multiple nodes behind a load balancer without clustering, you must account for persisting the connections to the AWCM servers. In the HTTP request that is sent to AWCM (from a device, the Device Services server, the Console Server, ACC, and so on), there is a cookie value called awcmsessionid, which is used to establish request level affinity to an AWCM node from a pool of nodes. You must configure your load balancer or proxy to parse the HTTP request for this value and use it for persistence. The persistence settings are only necessary for AWCM servers that are load-balanced in an active-active manner. The persistence settings will ensure that established connections are not dropped when the F5 switches from one AWCM server to the other to balance the load.

The iRule might vary based on a client’s existing configuration or best-practices, but the basics are straight forward:

  • Parse the HTTP request for the awcmsessionid cookie’s value
  • Set persistence with this value via the “persist carp” command.

For more background on this methodology, see the following F5 solution page for Overview of the CARP hash algorithm.

  • In Local Traffic → iRules : iRule List create an iRule to inspect the HTTP request for the value of the “awcmsessionid” cookie:
    when HTTP_REQUEST {

        if { [HTTP::cookie exists "awcmsessionid"] }{

            set awcm_session_id [string tolower [HTTP::cookie "awcmsessionid"]]

            persist carp $awcm_session_id

        }

    }
  • In Local Traffic → Profiles: Persistence create a persistence profile based on the default “hash” profile. All items should be default except the following:

    • Algorithm: “CARP”
    • iRule: created in Step 1.
  • Configure the Virtual server with the following settings:

    • Select an HTTP profile from the “HTTP Profile” drop-down list;
    • Select OneConnect profile = oneconnect (for HTTP request balancing, not just connection);
    • Apply the persistence profile under the Resources tab

Load Balancing AWCM - Persistence Rules with Citrix NetScaler

See details in an archived article - http://web.archive.org/web/20190506104002/https://www.citrix.com/blogs/2010/06/01/configuring-verifying-and-monitoring-persistence-on-netscaler/

Check the Persistence Rule

Check for persistence by hitting the URL [https://awcm url:port/awcm/statistics?awcmsessionid=abc123](https://awcm urlport) on ACC server and on machine from outside network. If the servers are different then customer needs to change the persistence rule on load balancer on their end.

Common Errors

AWCM Status Error - DNS name

AWCM not working - page https://<DS_URL>:2001/awcm/status unavailable. Error log seen:

2017-08-15 12:02:44,229 ERROR (nioEventLoopGroup-3-3) [com.airwatch.awcm.event.AWCMChannelConnectedEventHandler] - java.nio.channels.ClosedChannelException
java.util.concurrent.ExecutionException: java.nio.channels.ClosedChannelException
<...>
at io.netty.handler.ssl.SslHandler.channelInactive(...)(Unknown Source) [netty-all-4.0.43.Final.jar:4.0.43.Final]

Solution: DNS name of Device Services is registered on external proxy and not known to servers. Go to C:\Windows\System32\drivers\etc\hosts file on AirWatch Admin Console, and also on server with Enterprise Connector Service and add the EXTERNAL public DNS name (listed in public certificate) of AWCM binded to its’ internal IP.

AWCM Status Error - Cryptography

AWCM not working - page https://<DS_URL>:2001/awcm/status unavailable. Error log seen:

2017-08-15 14:49:41,044 ERROR (nioEventLoopGroup-3-7) [com.airwatch.awcm.event.AWCMChannelConnectedEventHandler] - javax.net.ssl.SSLHandshakeException: no cipher suites in common
java.util.concurrent.ExecutionException: javax.net.ssl.SSLHandshakeException: no cipher suites in common

Solution: AWCM was installed AFTER crypto algorithms were disabled in IIS hardening, and it cannot launch normally. Reinstall of AWCM needed.

AWCM SSL Certificate Error

Certificate error while browsing the AWCM status page

  1. Login to the AWCM server.

  2. Open a command prompt, navigate to the following directory (E:\airwatch\airwatch\AWCM\config) and run the following: keytool -list -v -keystore awcm.keystore

  3. Enter the password when prompted

  4. Export a new SSL certificate from a machine.

❗️Make sure that the full signing chain is exported (settings that you select when exporting the certificate) and that the password used to export is same as the one used for the current awcm.keystore.

If the passwords are not same, the import happens but an error message appears when AWCM starts and the status page does not load (as the pre-configured password will be incorrect and the AWCM app will not be able to open the keystore).

  • AWCM_BRIDGE_FILE_TRANSFER_TIMEOUT_IN_MINUTES : 15 2013-09-23 10:46:22,036
  • Error (main) [com.airwatch.awcm.ssl.AWCMSSLContext]:
java.security.UnrecoverableKeyException: Cannot recover key java.security.UnrecoverableKeyException: Cannot recover key
<...>
2013-09-23 10:46:22,036 ERROR (main) [com.airwatch.awcm.server.AWCMServer] - Error initializing server environment, exiting
  1. When the certificate is on the AWCM server (copy into the C:\airwatch\airwatch\AWCM\config directory), run the following command to replace SSL certificate:

keytool -importkeystore -srckeystore <new-pfx-cert-name>.pfx -srcstoretype pkcs12 -destkeystore awcm.keystore.new -deststoretype JKS

  1. Once this has completed successfully, you will now see a new file named awcm.keystore.new in the config directory. Stop the AWCM service.

  2. Rename the awcm.keystore to awcm.keystore.old.

  3. Rename the awcm.keystore.new to awcm.keystore.

  4. Start the AWCM service.

  5. Using a valid AWCM URL, try to access the page (https://{url}:2001/awcm/status) and if the status page loads, then check the certificate details. It should display the values for the newly uploaded certificate.

  • If the status page does not load, check the log files.
  • If rollback is required, rename the awcm.keystore to awcm.keystore.new.
  • Then rename awcm.keystore.old to awcm.keystore. Restart AWCM to restore the old settings.

AWCM and Admin Console trust error

ESC/ACC starts and generates no errors in log, also no errors in AWCM. But error in console while performing Test connection for ESC/ACC: Undefined Error; Please check server logs.

Reason: there is no trust between AWCM and AirWatch Admin Console

**Remedy:
**Import Intermediate and Root certificates for public PFX certificate in AWCM server and AirWatch Admin Console Server

ACC Errors

ACC must be able to reach AWCM:

  • Protocol HTTPS
  • Telnet from ACC to AWCM Server on the relevant port (usually 2001 for On-Premise installations and 443 for SaaS environments)
  • Also, verify by opening a browser on the ACC server, entering https://:2001/awcm/status and /awcm/statistics to ensure there is no certificate trust error.
  • For On-Premise installations: if using ACC with AWCM and there are multiple AWCM servers and they required to be load-balanced them, persistence needs to be configured - see the above section.

ACC must be able to reach the Console Server:

  • Protocol: HTTP or HTTPS
  • Telnet from ACC to the Console URL on the relevant port (usually 80 or 443)
  • Also, verify by opening a browser on the ACC server, entering http(s)://
  • If auto-update is enabled, ACC must be able to query AirWatch Console for updates

ACC must be able to reach the API

  • Protocol: HTTPS, port TCP443
  • Verify by navigating to the URL of your API server on the ACC server: https:///API/help
  • When the credentials screen appears, enter the credentials of a console admin and the API Developer page should appear.
  • ACC to API access is required for the proper functioning of the AirWatch Diagnostics service.

Connectivity Errors

  • If you see errors in the ACC logs indicating connections being closed/aborted/terminated, check if there is any network device in between the ACC and AWCM that would close or terminate idle connections. The outbound connection required for use by ACC must remain open at all times. Check the TCP session timeout on this network device in between and see if this can be increased to a value >2 minutes;
  • ACC sends what is known as an IDLE message, by default every 2 minutes. This IDLE message by ACC helps ACC register itself as a listener on AWCM so that AWCM knows that this ACC is ready to take requests;
  • If there are any network devices between ACC and AWCM that closes the connection between these components deeming the connection as an idle connection, it could cause issues with this ACC/AWCM connectivity.

401 Errors on ACC

ACC service does not start:

Error log contains error:

System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.
<...>

Reason: ESC/ACC service does not start because there is no trust between ESC/ACC and AWCM.

If this error is present after trying to hit Update/Check URL on the console, check the SSL certificate on the console and do the following: keytool -list -v -keystore "{AWCM install path}/awcm.truststore" > c:\test.txt

In the .txt file, search for the secure channel and it should match with the secure channel certificate in the console.

Remedy:

  • Generate new certificates for ESC/ACC and download the installer. Then, uninstall ACC and install the new ACC with the renewed certificates. Restart the AWCM service, if required.
  • Reinstall AWCM and download the installer from the console.

Globalization settings

WS1 UEM .NET Globalization settings

Issue

Usually, on-prem installation of WS1 UEM is doing with localized Windows servers (French, Russian, as an examples). With that, default settings in IIS can be set up incorrectly. This issue may prevent enrolling Windows devices (iOS or Android devices enroll well at the same time).

Solution

To fix this isseu you need to change .NET Globalization settings for Default Website level for all DS servers in your environment:

Parameter Value
Culture
Culture English (United States) (en-US)
UI Culture English (United States) (en-US)
Encoding
File Windows-1252

.NET-Globalization-Settings

Important Tools

General Tools

Verification Tool Description...
  • The pre-installation verification tool is an exe file that can be loaded onto the AirWatch app servers and will perform standard checks on the local machine, to the DB, and to the internet to ensure that the environment meets the VMware AirWatch Pre-Reqs.
  • Once installed, certain information is required, such as DB server and username/PW, and once entered takes you through the checks and presents the results in the application. Should it need to be shared there is an export option that creates an Excel file.
  • There is also additional functionality tests related LDAP, SMTP, Exchange, SSRS, and PKI

OS Specific Tools

iOS

MacOS

Android

  1. https://awagent.com/mobileenrollment/airwatchagent.apk
  2. https://discovery.awmdm.com/mobileenrollment/airwatchagent.apk

Description of tool…

Windows 10/11

Windows Phone (support deprecated)

Other Tools

SSL Checkers

EAS Troubleshooting Tools

Integration - EMail

Linked Articles

EMail Integration Schema

Subsections of Integration - EMail

Boxer and SCL Restrictions

What needs to be checked to restrict transfer of mail attachments and files transmission between Boxer and SCL for iOS and Android:

1.  To restrict transfer of documents in specific controlled apps, on Organization Group level: a. Allow DLP – SCL-step2 functionality

b.  Search for and add Boxer for iOS/Android – SCL-step3

  1. Next step for devices in Organization Group: a.  Create a new profile for all devices of a group – SLC-step4.

b.  Turn off “Allow documents from managed sources in unmanaged destinations”  – SCL-step5

1.  To restrict transfer of files from SCL into Boxer the file share, connected to SCL, has to be configured in the Security tab - SCL-step6:

a.  Allow Open in Email = OFF b.  Allow Opent in Third Party Apps = ON

2.  Check that in the security profile properties, Enable Composing Email = No (SCL-step7).

Boxer and SSL for PoC

Boxer has 2 types of accounts:

  • main – is being added on install, is usually configured with policies. Cannot be deleted.
  • additional – is added by users if this is allowed by policies. Can be configured manually.
Warning

(15.08.2017) For policies which are being distributed centrally for the main account there is no possibility to configure “Ignore SSL errors”. If in a PoC of Boxer you have to connect to a private/test mail server, this may pose a problem.

Variant 1 (recommended): server is signed by a cert, which was given by a inner CA (issued by =/= issued to)

Solution: on all devices in the test, you have to manually deploy the certificate chain in the trusted section:

iOShttp://longwhiteclouds.com/2013/01/03/installing-corporate-ca-certificates-on-iphone-or-ipad-for-use-with-vmware-view/

Androidhttps://support.google.com/nexus/answer/2844832?hl=en

Warning

In Android 7.0+, by default, apps don’t work with CA certificates that you add. But app developers can choose to let their apps work with manually added CA certificates. https://support.google.com/nexus/answer/2844832?hl=en

Variant 2: server is signed by a self-signed cert (issued by == issued to)

Solution: use the first account to distribute policies from any public EMail. For example, Office365 from our VMTestDrive

Note

Account can be only one for all devices - this account will not be used

  1. After Boxer installation connect to the account provided by policies (VMTestDrive)
  2. Choose Add Account (“Добавить аккаунт”) in Boxer left menu (IMG_1455.jpg)
  3. Insert an address (IMG_1456.jpg) and later choose Manual configuration (“Ручная настройка”). Type -> Exchange Server.
  4. See config in IGM_1457.jpg ang IGM_1458.jpg as an example of such a connection. It is important to choose SSL (Accept any certificates / “принимать любые сертификаты”).
Warning

After adding the account first time, Inbox may sync for a long time (>30min)

  1. Choose the default account (IMG_1459.jpg)

Boxer Copy-Paste

Boxer Copy&Paste restrictions in two ways

Unlike all other WS1 SDK-enabled Apps, Boxer has two different approaches to restrict Copy&Paste:


At the Assignment stage in App Policies, there is a Copy Paste setting. As a result, copy&paste functions will be denied in ANY directions

❗️Unrestricted personal mail accounts in Boxer still can be troublmakers in this case. Recommend to disable it

Copy&amp;Paste Total restrict


If you need to do more granular restriction you need to implement this on SDK profile

  • Recommended to set Native Boxer DLP capability to Unrestricted. It can be Restricted for potential more secure way, but SDK settings must be enabled after this settings
  • The Boxer App must be published with SDK-profile enable. We use the Default profile, but it should work with Custom SDK-profile as well
  • Before actual install Apps on devices, you need setup SDK-profile: Authentication Type =/= Disable; SSO must be enabled
  • In DLP section (Security Policies) you may enable Copy&Paste Into to get user possibility to copy from unmanaged messengers/notes/etc into Boxer emails

As a result, copy&paste functions will be denied only in the desired way: into or out from managed Apps Copy&amp;Paste Total unrestrict Enable SDK SSO enable Copy&amp;Paste In and Out

Boxer for Android Fingerprint

  • VMware Boxer v 4.5+ for Android 
  • AirWatch Console 9.0.5+

Following are the steps for fingerprint authentication on the Android Boxer app:

  • In the Apps SDK settings (Groups & Settings > All Settings > Apps > Settings and Polices > Security Polices), enable the Biometric Mode.
  • While deploying the Boxer app, enable the Application to use AirWatch SDK and Select the Global SDK for Android.
  • In the Email settings, enable the Application Configuration and enter “AppForceActivateSSO” (without the quotes) under Configuration Key and Value Type as Boolean and Configuration Type as True.
  • Make sure passcode is set as None 
  • Push the boxer app to the device and download it from the Play Store

Boxer via Fiddler

From time to time it may be useful to troubleshoot Boxer, Inbox, and native mail issues using Fiddler. Fiddler will let you log all of the traffic between the device and the email server/SEG, etc, end-point. This is useful in proving that communication is happening and that some errors are being generated externally from the device (401, 403, etc…).

Configure the Windows 10 computer (which will act as the proxy) and an example Android device.

  • Set your Windows 10 computer up as a mobile hotspot. On Windows 10, click on the start menu and type in “Mobile Hotspot”.
  • Select “Change mobile hotspot settings”

• Configure your hotsptot settings and turn on the hotspot.

  • Download and install Fiddler from https://www.telerik.com/fiddler

  • In Fiddler, go to Tools\Options make these changes. See HTTPS sub-option

  • Make these changes on the Connections sub-heading

  • You will need your Proxy servers IP address to enter on the mobile device for later. To get this hover over the “online” option and view the list of IP addresses. In this example, I’ll be using 10.84.145.96.

Android

  • On the mobile device, go to your wireless settings and long tap the right-hand side of the connection you want to connect to. This will by the wireless connection you set up on your computer. On Android, click on “Modify Network”.

  • Enter the proxy hostname and proxy port information. The port will be 8888 if you kept the default settings for Fiddler.

iOS

On iOS use the following process to set the proxy

How to configure your iPad/iPhone proxy settings

  1. Start the iPhone/iPad.
  2. Tap on the Settings app. …
  3. Tap on the Wi-Fi settings category. …
  4. You will now be at the Wi-Fi network settings screen for the connected network. … 5. Tap on the Manual button. …
  5. When you are done setting up your proxy server, tap on the Wi-Fi Networks button.

On the mobile device, go to the IP address of the proxy followed by :8888/fiddler. See the example below. Click on “FiddlerRoot Certificate” to download and install the certificate.

• In the Fiddler application, click on Tools\WinNet Options.

  • Click on “Lan settings” and uncheck the “Use a proxy server…” from the following window.

  • Click OK and OK. This will show a yellow bar on the Fiddler application indicating that it’s not collecting any traffic from the local computer in the logs. For your test, you will only want device traffic.

  • You can clear the current logs by doing a CTRL-A and selecting everything.

  • Now you can begin your replication testing.

ENS Basic Troubleshooting

Warning

Database for ENSv2 must be named “ENS”.

Alive Check

Cloud-hosted ENS: https://ens-eu.getboxer.com/api/ens https://ens-eu.getboxer.com/api/ens/alive

Steps:

  1. publickey request
    a. The device requests a public key to encrypt the account credentials with. It sends a hash of the email address as the userid. This helps identify the user and link together all user devices

  2. subscribe
    a. device sends encrypted creds, user id (server created) and device apns token so ENS server has all the necessary pieces to subscribe and get notifications of new emails

  3. push subscription
    a. ENS discovers endpoint based on creds and subscribes to exchange using a webhook link that contains the encrypted credentials ie: ens.airwatch.com/notify?id=&creds=<Base64(RSAEncrypted(username:password))>

  4. new email notification
    a. Exchange sends notification of changes to the provided url.
    b. ENS extracts and decrypt creds and prepares call to fetch email

  5. email fetch
    a. ENS performs a fetch for the email

  6. push email
    a. ENS finds user devices with the user id and pushes email details to CNS for delivery to all user devices

On-prem ENS: https://_ENS_HOST/MailNotifications/api/ens https://_ENS_HOST/MailNotifications/api/ens/alive

Registration Interaction

The following diagram shows in more detail a registration/new email interaction between the client, ENSv2 and the exchange server. This diagram shows in more detail how we can use credentials without keeping them saved inside the ENSv2 environment.

ENSv2 Database Details - see Article

Locale

Warning

ENS may cause errors in case of date mismatch Ensure that server, DB, etc have US English locate!

Boxer - Add Assingment

Network

ENSLINKAddress On-Premise, should point to the externally accessible hostname pointed to ENS service. A support ticket has to be made with VMware Airwatch to request API token (internally the support reaches out to Boxer product manager and requests API key).

MS Exchange Server

User agent is configured for ENSv2 on MS Exchange Server CAS role. User agent must have access to receive data from MS Exchange, or ENS will not be able to receive PUSH notifications.

Troubleshooting

ENSLINKAddress for On-Premise installation should point it correctly to the customer’s externally accessible hostname pointed to ENS service.

Autodiscovery errors showing on logs.

  • Make sure that the EWSUrl key is configured in the console with a correct value for the EWS url for their exchange environments.
    Test it out by opening it on the browser and making sure you are prompted for credentials.
    Tip

    Alternatively they can just turn ON autodiscovery on their environment.

Authentication errors (401s)

  • Check what type of authentication is enabled in EWS? Make sure it has parity with whatever they are using for ActiveSync (Basic, OAuth, CBA) as Boxer will be the one to pass whatever type of credentials it has to ENS to use against EWS.
  • Verify the EWSUrl is correct and resolves to Exchange environment and test the credentials used by navigating to the EWS URL and testing them there.
  • Run “Outlook Connectivity Tests” on https://testconnectivity.microsoft.com
    Tip

    Try the Exchange Server or Office 365 tabs accordingly

ENSv2 not sending notifications but no logs found on help portal.

  • Verify the ENSAPIToken key is correct in the console configuration.

  • Verify the ENSLinkAddress key is correct in the console configuration

    Tip

    Try appending the “alive” endpoint for the environment and make sure it responds.

  • Verify the EWSUrl is correctly configured with a valid EWS value.

  • Verify that ENSv2 servers have inbound access to their EWS environment (firewall may be blocking access, they need to open the corresponding IPs)

  • Verify that EWS can send outbound traffic to the corresponding ENSv2 domain (https://.getboxer.com/api/ens)

General FAQ

  • **How are credentials or authentications tokens handled?
    **
    • Although the client does share the credentials/tokens with the ENSv2 environment upon registration, they are not kept (saved anywhere) by AirWatch servers. Rather, the Exchange server passes them back to AirWatch, encrypted, as part of a notification it sends whenever a new email is available. From that notification (Exchange -> ENSv2), ENS decrypts the credentials and uses them to make any requests necessary to the Exchange server. After performing any necessary requests, the credentials are once again discarded.
Warning

Bug in MS Exchange detected: Exchange returns the email information even though the user is not the owner. This results in the notification payload being created for the wrong user and ultimately another user seeing the notification. 

With ENSv2 1.2+, a new service object is created for each EWS request. This will prevent the application from making a request to the EWS endpoint with different credentials.

Patch for Exchange needed from Microsoft on this, since this is unexpected Exchange behavior.

  • If credentials are not persisted, is there any data persistent at all by ENS? How is it secured?
    • There is a secure database that keeps a list of devices and a list of public private key pairs used to unencrypt the credentials when they come from Exchange;
    • Logs are also kept to aide in debugging issues and monitoring the system. These don’t contain any customer’s private information and access to them is also tightly secured via account permissions.
  • What data is transmitted through the ENS server without being persisted? How is it secured?
    • User credentials (encrypted with RSA encryption)
    • Email subject and sender (sent via HTTPS)
    • All communication is done via HTTPS
  • What additional cloud services does ENS depend on?
    • AWS Simple Notification Service (SNS) for push notification handling.
    • Apple Push Notification Service (APNS) as it is the only way to pass notifications to Apple devices.
    • AWS Relational database service (RDS) for data persistence.
  • What is the user agent used by ENSv2 when sending requests to Exchange?
    • MailNotificationService/v2 (ExchangeServicesClient/15.00.0913.015+ (will change as new libraries from Microsoft are released)
  • What email folders does ENSv2 monitor for incoming messages and actions?
    • Currently, ENSv2 only monitors each user’s Inbox folder.

Load Balancing ENSv2

For HA, it is recommended to load balance several ENS web servers as needed following the Hardware Requirements. All web servers should point to the same database server as this will be their shared source of state for each of the clients.

Since the ENS web application itself is stateless there are no requirements to configure any session handling (stickiness) in the loadbalancer so a straightforward configuration should suffice.

Integration - Microsoft IRM-RMS

RMS features in Boxer

According to Boxer User Guide for iOS 4.5.1, Boxer User Guide for Android 4.5.0

Main features

  • Edit
  • Reply
  • Reply All
  • Forward
  • Copy-Paste
  • Modify recipients
  • Extract
  • Print
  • Export
  • Content Expiry Date

Other Features

  • Press and hold an email message to copy and paste it into the application.
  • You cannot copy data from the Boxer application and paste anywhere outside the application. However, you can copy data from outside the application and paste into the Boxer application.
  • If your email message has contact number details, tap hold on the number to immediately dial it.
  • If restricted by your administrator, attachments may open through the VMware Content Locker and other AirWatch approved apps. Hyperlinks may open only through the VMware Browser.
  • If configured by your administrator, you can preview emails and their attachments within Boxer (See Boxer supported files’ types).
  • On the attachment preview screen, the Share icon will be unavailable. When tapped on Share icon, you are presented with a toast message “Disabled by your admin”.
  • After performing an action on an email while viewing it, you can have Boxer either advance to the next message, the previous message, or return to the conversation list. This setting can be configured from Mail settings (navigate to Settings > Mail > More mail settings > Auto Advance).

RMS Attachments

Boxer does not open RMS-restricted attachments - it transmits them to Content Locker. To use Content Locker on iOS device, the following has to be done:

  1. Root certificate must be placed on device

  2. In new iOS version the root Trust has to be ACTIVATED in a special menu option

  3. In order for Content Locker to access RMS attachments, it must be registered on the ADFS server with this command:

Add-AdfsClient -Name "<App name>" -ClientId "<ID name>" -RedirectUri "<RedirectUri>"

Example: Client ID for VMware Content Locker for iOS is e9fcfce0-a20b-4d34-b580-909332723090

Tip

Client ID of application can be found in ADFS logs: every error Content Locker gives while trying to read a RMS-secured attachment is followed by its’ current Client ID.

Powershell for MEM

Linked Articles

EMail Architectures

Common Powershell Commands

Initializing a Session

This command is used for AirWatch to initialize a session. The two parameters required as the $creds and the PowerShell endpoint.

> $cred = Get-Credential

> $session = New-PSSession ConfigurationName Microsoft.Exchange -ConnectionUri https://<mailserver>/powershell Credential  
$cred Authentication Basic AllowRedirection

> Import-PSSession $session

Look at a user’s basic mailbox information

This command pulls basic information about a mailbox using an email address as the identity.

> Get-CASMailbox identity userguy | fl

Viewing a user’s list of devices

This command will list each device partnered with the CasMailbox.

> Get-ActiveSyncDevice mailbox userguy | fl (2010)  
> Get-MobileDevice mailbox userguy | fl (2013+)  

Additional device information

WS1 UEM does not pull from this listing, however, you can find some additional details (ex: when the device last synced) from this table.

> Get-ActiveSyncDeviceStatistics mailbox userguy | fl

Setting ActiveSync Devices to Allowed/Blocked

This is the form of a cmdlet used to issue an Allow/Block command to Exchange. This will insert “DeviceIDX” into the appropriate list.

> Set-CasMailbox identity userguy ActiveSyncAllowedDeviceIDs @{Add = DeviceId1} 
> Set-CasMailbox identity userguy ActiveSyncBlockedDeviceIDs @{Add = DeviceId2}

Selecting specific information or exporting data

This command is helpful when comparing AirWatch data to Exchange data.

> Get-ActiveSyncDevice ResultSize Unlimited | Select-Object  
DeviceID, DistinguishedName, DeviceType | Export-CSV  
ASD_selection.csv

WS1 UEM with Office 365

Disable the native access in O365 -> redirect to WS1 UEM First-time access will be denied, PowerShell command will be sent to O365 to whitelist the device, 2-3min later the email will flow

Set WS1 UEM as IDP to control other ways of accessing (Exchange Web Access, OWA etc)

This lacks some features (encrypt attachments, strip attachments etc), but can be mitigated using Boxer Needs ESC between Cloud AW and On-Prem Exchange

AW-PS Service Account
Remote Shell access to the Exchange Server associated mailbox on the server to issue remote commands

Required PowerShell roles: Mail Recipients Organization Client Access Recipient Policies Settings –> Email –> Email Settings Configure - Direct

Features:

  • Configure email over-the-air
  • Block unmanaged devices
  • Discover existing unmanaged devices
  • Require device encryption
  • Prevent compromised devices
  • Block mail client, user, device model or OS
  • Integrate or revoke certificates

Subsections of Powershell for MEM

Allowlist and Blacklist

ADMIN ACTIONS (BLOCKLIST)

Actions from Email dashboard:

  1. User selects device and clicks allowlist/ blocklist action
  2. Meg Queue Service sendsends allowlist/ blocklist powershell command to Exchange server appropriately 3. Meg Queue Service updates database to show your device status on email dashboard

Webconsole Log

Blocklist event:

After admin click Blocklist action for device, webconsole receives blocklist event for processing. Log prints device properties as described below:

MEMConfig - Email Settings used Device Count - Total number of devices blocklisted

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.MobileEmailBusiness.PeristDeviceAccessStateChanged Received device access state change. MEMConfig: 48 Device Count: 1 935c3f93-3f33-49fb-b6c9-a07d0bcc8619

Event written to Microsoft Messaging Queue:

Webconsole writes blocklist event to Microsoft Messaging Queue. MEG Queue will read the queue and will process event.

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.Util.DeviceStateChangeTypeMessageQueue.PersistGenericPayload Writing change device access state payload for LG '646' to queue: 'Name: AWSegCompliance, Protocol: Tcp, Address: .\Private$\, Encoding: Binary, QueueLocation: Local, BulkReadLimit: 1, ConnectionRetryCount: 3, ReadTimeout: -1, RetryInterval: 30, AutoCreate: False, ' 935c3f93-3f33-49fb-b6c9-a07d0bcc8619

MEG Que Log

Blocklist event received:

MEG Queue receives blocklist event for processing from webconsole. Device properties are printed in log identifying your device as shown below.

MemConfig Id - Email Settings used MEMDevice Id - Email Device Record Id number of devices - Total number of devices blocklisted EasDeviceIdentifier - Exchange Device ID User - Email user AccessLevel - Email access status Reasons - Reason for allow\block device Lg - Location Group ID Device Id - AirWatch device id

Debug AW.Meg.Queue.Service.Util.EndpointQueueManager._SplitByType Received 'DeviceAccessChangedPayload' message. MemConfig Id: '48', MEMDevice Id: '5744' 

Debug AW.Meg.Queue.Service.Util.EndpointQueueManager._ReceiveDeviceAccessChangedPayload Received 'DeviceAccessChangedPayload' message. MemConfigId: '48', number of devices: '1'.

Debug AW.Meg.Queue.Service.Processors.Office365DeviceAccessChangedProcessor.ProcessPayload Device access state changed. Device Id: '29'. Process ActiveSync command. Lg: '646', EasDeviceIdentifier: 'R4SQG79G556LPB3LFEVP66VO98', User: 'airwatchqa@airwatchpm.onmicrosoft.com', AccessLevel: 'Blocked', Reasons: Device is blacklisted: 29  

Debug WanderingWiFi.AirWatch.BusinessImpl.EnterpriseIntegrationHelper.SetBulkEasDeviceAccess BulkDeviceRequest - MemConfigId: 48, IsRunCompliance: False.

Powershell Admin details:

MEG Queue loads Powershell admin account details.

Debug WanderingWiFi.AirWatch.BusinessImpl.EnterpriseIntegrationHelper.DoSetBulkEasDeviceAccess Loading Exchange settings for MEMConfig: 48  

Debug WanderingWiFi.AirWatch.BusinessImpl.EnterpriseConfigurationProvider.LoadExchangeServiceConfiguration Loading exchange settings. MEMConfig: 48

Powershell Session creation:

MEG Queue creates powershell session to execute powershell command.

Debug AirWatch.CloudConnector.Common.PowerShell.SessionHelper.InitializeSession Initializing PowerShell session for Microsoft.Exchange @ PowerShell endpoint https://ps.outlook.com/powershell using Authentication type: Basic,  
User: airwatchadmin@airwatchpm.onmicrosoft.com, using service credentials: False, ViewEntireForest enabled: False  

Debug AirWatch.CloudConnector.Common.PowerShell.SessionHelper.InitializeSession Creating session for Microsoft.Exchange @ PowerShell endpoint https://ps.outlook.com/powershell using Authentication type: Basic, User: airwatchadmin@airwatchpm.onmicrosoft.com, using service credentials: False, ViewEntireForest enabled: False

Blocklisting device: MEQ Queue sends powershell block command to exchange so that email requests will be blocked.

Debug AirWatch.CloudConnector.Common.PowerShell.CommandHelper.SetActiveSyncDeviceIds Invoking command Set-CASMailbox -Identity 'airwatchqa@airwatchpm.onmicrosoft.com' -ActiveSyncBlockedDeviceIDs @{Add='R4SQG79G556LPB3LFEVP66VO98'} against the  
endpoint: SingleDeviceActionWithNormalPriority_airwatchadmin@airwatchpm.onmicrosoft.com@Microsoft.Exchange@https://ps.outlook.com/powershell

Powershell Session removal:

MEG Queue removes powershell session from memory.

Debug AirWatch.CloudConnector.Common.PowerShell.SessionHelper.InitializeSession PowerShell session successfully initialized for Microsoft.Exchange @ PowerShell endpoint https://ps.outlook.com/powershell using Authentication type: Basic,  
User: airwatchadmin@airwatchpm.onmicrosoft.com, using service credentials: False, ViewEntireForest enabled: False  

Debug AirWatch.CloudConnector.Common.PowerShell.SessionHelper.RemoveSession Removing session for Microsoft.Exchange @ PowerShell endpoint https://ps.outlook.com/powershell using Authentication type: Basic, User: airwatchadmin@airwatchpm.onmicrosoft.com, using service credentials: False, ViewEntireForest enabled: False

Database Update:

MEM Device Activity is saved to database so that Email List view reflects change.

Debug WanderingWiFi.AirWatch.ProviderImpl.MobileEmail.MobileEmailDataHandler.SaveMemDeviceActivitiesForPSRunCompliance() MEM Device Activity Saved Successfully... 1

Log files

aw.meg.queue.serviceblacklist.log weblogfile-blacklist.txt

Mailbox sync

Sync Mailboxes flow:

  1. Sync Mailboxes action is processed by webcosole and sent to MEG Queue for processing.
  2. MEG Queue Service invokes powershell fucntion to retrive all mailboxes.
  3. It then invokes powershell fucntion to retrive all EAS devices.
  4. Mailboxes and devices and reconciled and MEG Queue saves EAS device data to AirWatch database.

WEB CONSOLE LOG

Sync Mailboxes Event processing:

After admin clicks Sync Mailboxes action, webconsole receives event for processing.
Webconsole writes Sync Mailboxes event to Microsoft Messaging Queue. MEG Queue will read the queue and will process event.

2017/03/10 14:42:29.637 MEMCON 3c749df0-c28d-409e-84ac-0a2d29cc5566 [0000068-0000000] (28) Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.Util.DeviceStateChangeTypeMessageQueue.PersistGenericPayload Writing sync eas mailbox payload for LG '627' to queue: 'Name: AWSegCompliance, Protocol: Tcp, Address: .\Private$\, Encoding: Binary, QueueLocation: Local, BulkReadLimit: 1, ConnectionRetryCount: 3, ReadTimeout: -1, RetryInterval: 30, AutoCreate: False, ' 9525ac55-84d8-4ae2-b8e1- b7b183f84afd

MEG QUE LOG

Sync Mailboxes event received:

MEG Queue receives Sync Mailboxes event for processing from webconsole.
Sync Mailboxes operation is initiated.

Debug AW.Meg.Queue.Service.Util.EndpointQueueManager._SplitByType Received 'SyncEasDevicesPayload' message. MemConfig Id: '30' Info AW.Meg.Queue.Service.Util.SyncAllMailboxesTask.PerformTask Sync Mailboxes task will initiate for MEMConfig '30'.

Retrieving Mailboxes:

MEG Queue prepares to retrieve Mailboxes. It prints filter if applicable.

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.MobileEmailBusiness.SyncEasDevices Retrieving CAS mailboxes. MEMConfig: 30  

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.MobileEmailBusiness.SyncEasDevices Sync Result Type : None; No Filter Provided.

Powershell Admin details:

MEG Queue loads Powershell admin account details. It also loads ACC details if any.

Debug WanderingWiFi.AirWatch.BusinessImpl.EnterpriseConfigurationProvider.LoadExchangeServiceConfiguration Loading exchange settings. MEMConfig: 30  

Debug WanderingWiFi.AirWatch.BusinessImpl.EnterpriseConfigurationProvider.LoadExchangeServiceConfiguration Getting service relay configuration. ACC location group overridden by MEM configuration: False. Location group:'627'

Powershell Session creation:

MEG Queue creates powershell session to execute powershell command.

Debug AirWatch.CloudConnector.Common.PowerShell.SessionHelper.InitializeSession Initializing PowerShell session for Microsoft.Exchange @ PowerShell endpoint https://mail-mem13.ssdevrd.com/powershell using Authentication type: Basic, User: mem13\svcPSTest, using service credentials: False, ViewEntireForest enabled: False  

Debug AirWatch.CloudConnector.Common.PowerShell.SessionHelper.InitializeSession Creating session for Microsoft.Exchange @ PowerShell endpoint https://mail-mem13.ssdevrd.com/powershell using Authentication type: Basic, User: mem13\svcPSTest, using service credentials: False, ViewEntireForest enabled: False

Debug AirWatch.CloudConnector.Common.PowerShell.SessionHelper.InitializeSession PowerShell session successfully initialized for Microsoft.Exchange @ PowerShell endpoint https://mail-mem13.ssdevrd.com/powershellusing Authentication type: Basic, User: mem13\svcPSTest, using service credentials: False, ViewEntireForest enabled: False

Sending powershell command to Retrieve Mailboxes:

MEQ Queue calls powershell function to Retrieve Mailboxes from exchange. Number of mailboxes retrieved are listed in log as shown.

Debug WanderingWiFi.AirWatch.BusinessImpl.EnterpriseIntegrationHelper.ListCasMailboxes Retrieving CAS mailbox list records 1 to 25000 at ExchangeService https://mail-mem13.ssdevrd.com/powershell.  

Debug AirWatch.CloudConnector.ExchangeServices.MailboxManagementService.ListCasMailboxes Retrieving CAS mailbox list records 1 to 25000.  
Debug AirWatch.CloudConnector.Common.PowerShell.CommandHelper.ListCasMailboxes Invoking command 'AW-Get-CASMailboxList'. Endpoint: 'BulkDeviceAction_mem13\svcPSTest@Microsoft.Exchange@https://mail-mem13.ssdevrd.com/powershell'  

Debug AirWatch.CloudConnector.ExchangeServices.MailboxManagementService.ListCasMailboxes Retrieved CAS mailbox list records 1 to 23 of 23.  

Debug WanderingWiFi.AirWatch.BusinessImpl.EnterpriseIntegrationHelper.ListCasMailboxes Retrieved CAS mailbox list records 1 to 23 of 23 at ExchangeService https://mail-mem13.ssdevrd.com/powershell.

Powershell Session removal:

MEG Queue removes powershell session from memory.

Debug AirWatch.CloudConnector.Common.PowerShell.SessionHelper.RemoveSession Removing session for Microsoft.Exchange @ PowerShell endpoint https://mail-mem13.ssdevrd.com/powershell using Authentication type: Basic, User: mem13\svcPSTest, using service credentials: False, ViewEntireForest enabled: False  

Debug AirWatch.CloudConnector.Common.PowerShell.SessionHelper.RemoveSession Session removed for Microsoft.Exchange @ PowerShell endpoint https://mail-mem13.ssdevrd.com/powershell using Authentication type: Basic, User: mem13\svcPSTest, using service credentials: False, ViewEntireForest enabled: False

Retrieving ActiveSync devices:

MEQ Queue calls powershell function to Retrieve Mailboxes from exchange. Number of devices retrieved are listed in log as shown.

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.MobileEmailBusiness.SyncEasDevices Retrieving ActiveSync devices. MEMConfig: 30  

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.MobileEmailBusiness.SyncEasDevices Sync Result Type : None; No Filter Provided.

Debug AirWatch.CloudConnector.Common.PowerShell.CommandHelper.ListActiveSyncDevices Invoking command 'AW-Get- ActiveSyncDeviceList'. Endpoint: 'BulkDeviceAction_mem13\svcPSTest@Microsoft.Exchange@https://mail-mem13.ssdevrd.com/powershell', PowerShellDeploymentType: '5'.

Debug AirWatch.CloudConnector.ExchangeServices.MailboxManagementService.ListActiveSyncDevices Retrieved ActiveSync device list records 1 to 217 of 217.  
Debug WanderingWiFi.AirWatch.BusinessImpl.EnterpriseIntegrationHelper.ListActiveSyncDevices Retrieved ActiveSync device list records 1 to 217 of 217 at ExchangeService https://mail-mem13.ssdevrd.com/powershell.

Reconciling Devices:

MEG Queue compare AirWatch MEM Devices with EAS devices retreived from exchange.
If EAS device retrived from exchange matches with one of AirWatch device, MEG Queue will update AirWatch MEM Device with latest status. Otherwise new unmanged device record is created.

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.MobileEmailBusiness.SyncEasDevices Reconciling EAS devices with known devices in AirWatch. MEMConfig: 30

Updating Managed Device:

Below statement shows that there is 1 matched device after reconcilation process.

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.MobileEmailBusiness.ReconcileEasDevices Updating '1' known managed EAS devices. MEMConfig: '30'

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.MobileEmailBusiness.UpdateManagedDevices Finding managed devices from payloads.  
Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.MobileEmailBusiness.UpdateManagedDevices Finding unmanaged devices from payloads.

Debug WanderingWiFi.AirWatch.ProviderImpl.MobileEmail.MobileEmailDataHandler.SaveMemDevices() Save MEM Device  

Debug WanderingWiFi.AirWatch.ProviderImpl.MobileEmail.MobileEmailDataHandler.SaveMemDevices() Save MemDevice for ActiveSyncPayload Instance: MEMDeviceId: 5614, LocationGroupID: 627, MEMConfigId: 30, IsManaged: True, EasDeviceIdentifier: 0EANTNG9L56LL8FLD1SFVFNK64, DeviceId: 19, SyncAllowed: True  
, EasDeviceType: iPod, EasDeviceUserAgent: Apple-iPod5C1/1307.36, EasmailboxIdentity: MEM13.ORG/Users/MEM2, EasMailboxDisplayName: MEM2, EmailAddress: MEM2@mem13.ssdevrd.com  
, UserName: MEM2, Command: Reconciled Access State, GatewayHostName:  
, CreateNewUnmanaged: False, UpdateManaged: True, EmailClient: , TimeOfRequest: Friday, March 10, 2017  
, Allowed Reason: AllowedByDefault

Debug WanderingWiFi.AirWatch.ProviderImpl.MobileEmail.MobileEmailDataHandler.SaveMemDevices() MEM Devices Saved Successfully Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.MobileEmailBusiness.UpdateManagedDevices Updated managed devices. '1' Updated successfully.

Updating MemDeviceActivity:

MEG Queue updates MemDeviceActivity record with appropriate status.

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.MobileEmailBusiness.UpdateMEMDeviceActivity ++UpdateMEMDeviceActivity Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.MobileEmailBusiness.UpdateMEMDeviceActivity  
ActiveSyncPayload Instance: MEMDeviceId: 5614, LocationGroupID: 627, MEMConfigId: 30, IsManaged: True, EasDeviceIdentifier: 0EANTNG9L56LL8FLD1SFVFNK64, DeviceId: 19, SyncAllowed: True, EasDeviceType: iPod, EasDeviceUserAgent: Apple-iPod5C1/1307.36, EasmailboxIdentity: MEM13.ORG/Users/MEM2, EasMailboxDisplayName: MEM2, EmailAddress: MEM2@mem13.ssdevrd.com

UserName: MEM2, Command: Reconciled Access State, GatewayHostName:  
CreateNewUnmanaged: False, UpdateManaged: True, EmailClient: , TimeOfRequest: Friday, March 10, 2017
Allowed Reason: AllowedByDefault

Debug WanderingWiFi.AirWatch.ProviderImpl.MobileEmail.MobileEmailDataHandler.SaveMEMDeviceActivities() Saving MEM Device Activity  

Debug WanderingWiFi.AirWatch.ProviderImpl.MobileEmail.MobileEmailDataHandler.SaveMEMDeviceActivities() MEM Device Activity Saved Successfully... 1

Updating MemDeviceConfig:

MEG Queue updates MemDeviceConfig record with appropriate status.

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.MobileEmailBusiness.UpdateMEMDeviceConfig MemDeviceConfig for Managed Device: MemDeviceId: 5614, MemConfigId: 30  

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.MobileEmailBusiness.UpdateMEMDeviceConfig Total Mem Device Config Records Count: 1

Debug WanderingWiFi.AirWatch.ProviderImpl.MobileEmail.MobileEmailDataHandler.SaveMemDeviceConfig ++SaveMemDeviceConfig

Creating unmanaged devices:

If EAS device retrived from exchange do not match with one of AirWatch device, MEG Queue will create unmanaged AirWatch MEM Device.

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.MobileEmailBusiness.ReconcileEasDevices Updating '216' discovered unmanaged EAS devices. MEMConfig: '30'  

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.MobileEmailBusiness.UpdateDeviceStatus Updating access state for '216' devices in database.

Creating unmanaged MemDevice:

MEG Queue creates unmanaged MEMDevice records as shown below.

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.MobileEmailBusiness.CreateUnmanagedMEMDevices ++CreateUnmanagedMEMDevices

Debug WanderingWiFi.AirWatch.ProviderImpl.MobileEmail.MobileEmailDataHandler.SaveMemDevices() Save MemDevice for ActiveSyncPayload Instance: MEMDeviceId: 0, LocationGroupID: 627, MEMConfigId: 30, IsManaged: False, EasDeviceIdentifier: boxer1485882972781, DeviceId: 0, SyncAllowed: True  
, EasDeviceType: Android, EasDeviceUserAgent: AirWatch Boxer (Nexus 6P; Android 6.0.1) Version 4.1.0.12/352, EasmailboxIdentity: MEM13.ORG/Users/TBurgess, EasMailboxDisplayName: TBurgess, EmailAddress: TBurgess@mem13.ssdevrd.com

, UserName: TBurgess, Command: Discovered EAS Device, GatewayHostName:  
, CreateNewUnmanaged: True, UpdateManaged: False, EmailClient: , TimeOfRequest: Friday, March 10, 2017 , Allowed Reason: AllowedByDefault

Creating unmanaged MEMDeviceActivity:

MEG Queue creates unmanaged MEMDeviceActivity records as shown below.

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.MobileEmailBusiness.UpdateMEMDeviceActivity ++UpdateMEMDeviceActivity Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.MobileEmailBusiness.UpdateMEMDeviceActivity  
ActiveSyncPayload Instance: MEMDeviceId: 5748, LocationGroupID: 627, MEMConfigId: 30, IsManaged: False, EasDeviceIdentifier: boxer1485882972781, DeviceId: 0, SyncAllowed: True

, EasDeviceType: Android, EasDeviceUserAgent: AirWatch Boxer (Nexus 6P; Android 6.0.1) Version 4.1.0.12/352, EasmailboxIdentity: MEM13.ORG/Users/TBurgess, EasMailboxDisplayName: TBurgess, EmailAddress: TBurgess@mem13.ssdevrd.com  
, UserName: TBurgess, Command: Discovered EAS Device, GatewayHostName:  
, CreateNewUnmanaged: True, UpdateManaged: False, EmailClient: , TimeOfRequest: Friday, March 10, 2017

, Allowed Reason: AllowedByDefault

Debug WanderingWiFi.AirWatch.ProviderImpl.MobileEmail.MobileEmailDataHandler.SaveMEMDeviceActivities() Saving MEM Device Activity  

Debug WanderingWiFi.AirWatch.ProviderImpl.MobileEmail.MobileEmailDataHandler.SaveMEMDeviceActivities() MEM Device Activity Saved Successfully... 100

Debug WanderingWiFi.AirWatch.ProviderImpl.MobileEmail.MobileEmailDataHandler.SaveMEMDeviceActivities() MEM Device Activity Saved Successfully... 200  

Debug WanderingWiFi.AirWatch.ProviderImpl.MobileEmail.MobileEmailDataHandler.SaveMEMDeviceActivities() MEM Device Activity Saved Successfully... 216

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.MobileEmailBusiness.UpdateMEMDeviceActivity Saved device activities. '216' Saved successfully.

Creating unmanaged MEMDeviceConfig:

MEG Queue creates MEMDeviceConfig records for unmanaged devices as shown below.

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.MobileEmailBusiness.UpdateMEMDeviceConfig Collecting Managed MEM Device Config Records.  

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.MobileEmailBusiness.UpdateMEMDeviceConfig Collecting Unmanaged MEM Device Config Records.  

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.MobileEmailBusiness.UpdateMEMDeviceConfig MemDeviceConfig for Unmanaged Device: MemDeviceId: 5748, MemConfigId: 30

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.MobileEmailBusiness.UpdateMEMDeviceConfig MemDeviceConfig for Unmanaged Device: MemDeviceId: 5762, MemConfigId: 30

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.MobileEmailBusiness.UpdateMEMDeviceConfig Total Mem Device Config Records Count: 216

Discovered Mail Clients:

MEQ Queue collects all mail client names and saves in AirWatch database. These mailclients are presented to user for selection in Mail Client policy for configuring policy rules.

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmailGateway.ActiveSyncBusiness Finding new mail clients.  

Debug WanderingWiFi.AirWatch.ProviderImpl.MobileEmailGateway.MailClientDataHandler Look for mail client list in cache.  

Debug WanderingWiFi.AirWatch.ProviderImpl.MobileEmailGateway.MailClientDataHandler Mail client list not found in cache. Loading from database.  
Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmailGateway.ActiveSyncBusiness Found '17' new mail clients.  

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmailGateway.ActiveSyncBusiness Saving new mail clients.  

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmailGateway.ActiveSyncBusiness Save mail client successful for LG : '627'  

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmailGateway.ActiveSyncBusiness Saved new mail clients. '1' Saved successfully. '0' had errors.

Discovered User Accounts:

MEQ Queue collects all User Accounts and saves in AirWatch database.
These User Accounts are presented to user for selection in User policy for configuring policy rules.

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmailGateway.ActiveSyncBusiness Finding new email account users.  

Debug WanderingWiFi.AirWatch.ProviderImpl.MobileEmailGateway.ActiveSyncDataHandler.LoadAccountUserNames Look for account user list in cache.  

Debug WanderingWiFi.AirWatch.ProviderImpl.MobileEmailGateway.ActiveSyncDataHandler.LoadAccountUserNames Account user list not found in cache. Loading from database.  

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmailGateway.ActiveSyncBusiness Found '2' new email account users.  

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmailGateway.ActiveSyncBusiness Saving new email account users.  

Debug WanderingWiFi.AirWatch.ProviderImpl.MobileEmailGateway.ActiveSyncDataHandler.BulkSaveAccountUserName Saving Account User(s)...  

Debug WanderingWiFi.AirWatch.ProviderImpl.MobileEmailGateway.ActiveSyncDataHandler.BulkSaveAccountUserName Saved Account User(s), index: 2  

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmailGateway.ActiveSyncBusiness Saved new email account user for location group: '627'  

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmailGateway.ActiveSyncBusiness Saved new email account users. '1' Saved successfully. '0' had errors.

Finished:

Info AW.Meg.Queue.Service.Util.SyncAllMailboxesTask.PerformTask Sync Mailboxes task finished for MEMConfig '30'.

Log Files

aw.meg.queue.service-syncmailboxes.log

Run Compliance

WEBCONSOLE LOG

Run Compliance Event processing:

After admin clicks Run Compliance action, webconsole receives event for processing.
Webconsole writes Sync Mailboxes event to Microsoft Messaging Queue. MEG Queue will read the queue and will process event.

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.MobileEmailBusiness.PersistDeviceStateChangeByLg Received state change event. LG: 627 Type: PolicyPublish 6e76d060-02d6-4515-ab6d-1f90ff41ec1b  

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.Util.DeviceStateChangeTypeMessageQueue.PersistGenericPayload Writing device state change payload for LG '627' to queue: 'Name: AWSegCompliance, Protocol: Tcp, Address: .\Private$\, Encoding: Binary, QueueLocation: Local, BulkReadLimit: 1, ConnectionRetryCount: 3, ReadTimeout: -1, RetryInterval: 30, AutoCreate: False, ' 6e76d060-02d6-4515-ab6d-1f90ff41ec1b

MEG QUE LOG

Run Compliance event received:

MEG Queue receives Run Compliance event for processing from webconsole. Run Compliance operation is initiated.

Debug AW.Meg.Queue.Service.Util.EndpointQueueManager.Process A message is added to the queue for endpoint 'https://mail- mem13.ssdevrd.com/powershell-.-mem13\svcPSTest', for MemConfig Id: '30'.  

Debug AW.Meg.Queue.Service.Util.EndpointQueueManager._SplitByType Received 'DeviceStateChangePayload' message. MemConfig Id: '30', Device Id: '', StateChangeType: 'PolicyPublish'  

Debug AW.Meg.Queue.Service.Processors.Office365DeviceStateChangedProcessor.DoProcess Processing 'PolicyPublish' event for LG '627', Device Id: '', MEM Config Id: '30'

Powershell Admin details:

MEG Queue loads Powershell admin account details. It also loads ACC details if any.

Debug WanderingWiFi.AirWatch.BusinessImpl.EnterpriseConfigurationProvider.LoadExchangeServiceConfiguration Getting service relay configuration. ACC location group overridden by MEM configuration: False. Location group:'627'  

Debug WanderingWiFi.AirWatch.BusinessImpl.EnterpriseConfigurationProvider.LoadExchangeServiceConfiguration Loaded exchange settings. MEMConfig: 30

Debug WanderingWiFi.AirWatch.BusinessImpl.EnterpriseIntegrationHelper.TestExchangeConnection Load Exchange service configuration for location group 627

Policy Evaluation:

MEG Que evaluates policy and determine all devices that needs access state change (allow or block). Below log shows example of device evaluated to be blocked by user policy.

Debug AW.Meg.Queue.Service.Util.RunComplianceTask.ProcessUpdateDevicePolicies Total 1 policies retrieved from database for the process.  

Debug AW.Meg.Queue.Service.Util.RunComplianceTask.ProcessUpdateDevicePolicies Total 1 mailboxes were found with known devices. Debug AW.Meg.Queue.Service.Util.RunComplianceTask.ProcessUpdateDevicePolicies Known Policy: EasDeviceIdentifier 0EANTNG9L56LL8FLD1SFVFNK64, MemDeviceId: 5614, Allowed: False, MailboxIdentity: , LastCommand: Mail Server Update, DeviceAccessStateReason: , LastMailAccessAllowed: True

Debug AW.Meg.Queue.Service.Util.RunComplianceTask.ProcessUpdateDevicePolicies Evaluated device. DeviceId: '19', EasDeviceIdentifier: '0EANTNG9L56LL8FLD1SFVFNK64', Allowed: 'False', Reason(s): Account user mem2 is blocked

Powershell Session creation:

MEG Queue creates powershell session to execute powershell command.

Debug AirWatch.CloudConnector.Common.PowerShell.SessionHelper.InitializeSession Initializing PowerShell session for Microsoft.Exchange @ PowerShell endpoint https://mail-mem13.ssdevrd.com/powershell using Authentication type: Basic, User: mem13\svcPSTest, using service credentials: False, ViewEntireForest enabled: False  

Debug AirWatch.CloudConnector.Common.PowerShell.SessionHelper.InitializeSession PowerShell session successfully initialized for Microsoft.Exchange @ PowerShell endpoint https://mail-mem13.ssdevrd.com/powershellusing Authentication type: Basic, User: mem13\svcPSTest, using service credentials: False, ViewEntireForest enabled: False

Blacklisting device:

MEQ Queue sends powershell block command to exchange so that email requests will be blocked.

Debug AirWatch.CloudConnector.Common.PowerShell.CommandHelper.SetActiveSyncDeviceIds Invoking command Set-CASMailbox -Identity 'mem2@mem13.ssdevrd.com' -ActiveSyncBlockedDeviceIDs @{Add='0EANTNG9L56LL8FLD1SFVFNK64'} against the endpoint: BulkDeviceAction_mem13\svcPSTest@Microsoft.Exchange@https://mail-mem13.ssdevrd.com/powershell

Powershell Session removal:

MEG Queue removes powershell session from memory.

Debug AirWatch.CloudConnector.Common.PowerShell.SessionHelper.RemoveSession Removing session for Microsoft.Exchange @ PowerShell endpoint https://mail-mem13.ssdevrd.com/powershell using Authentication type: Basic, User: mem13\svcPSTest, using service credentials: False, ViewEntireForest enabled: False  

Debug AirWatch.CloudConnector.Common.PowerShell.SessionHelper.RemoveSession Session removed for Microsoft.Exchange @ PowerShell endpoint https://mail-mem13.ssdevrd.com/powershell using Authentication type: Basic, User: mem13\svcPSTest, using service credentials: False, ViewEntireForest enabled: False

Updating AirWatch Database:

After successfully blocking device, AirWatch database is updated with current status.

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.MobileEmailBusiness.UpdateMemDeviceActivitiesForPSRunCompliance ++UpdateMemDeviceActivitiesForPSRunCompliance  

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.MobileEmailBusiness.UpdateMemDeviceActivitiesForPSRunCompliance Total 1 MEMDeviceActivity records will be updated.

Debug WanderingWiFi.AirWatch.ProviderImpl.MobileEmail.MobileEmailDataHandler.SaveMemDeviceActivitiesForPSRunCompliance()

2 of 3 19.07.2022, 13:11

VMWare Workspace ONE MEM Team - PowerShell - Run Compliance https://onevmw.sharepoint.com/teams/VMWareAirWatchMEM/SitePage...

++SaveMemDeviceActivitiesForPSRunCompliance  
Debug WanderingWiFi.AirWatch.ProviderImpl.MobileEmail.MobileEmailDataHandler.SaveMemDeviceActivitiesForPSRunCompliance() MEM Device Activity Saved Successfully... 1  

Debug WanderingWiFi.AirWatch.ProviderImpl.MobileEmail.MobileEmailDataHandler.SaveMemDeviceActivitiesForPSRunCompliance() --SaveMemDeviceActivitiesForPSRunCompliance  

Debug WanderingWiFi.AirWatch.BusinessImpl.MobileEmail.MobileEmailBusiness.UpdateMemDeviceActivitiesForPSRunCompliance --UpdateMemDeviceActivitiesForPSRunCompliance

Finished:

Info AW.Meg.Queue.Service.Util.SyncAllMailboxesTask.PerformTask Run compliance task finished for MEMConfig '30'.

Log files

aw.meg.queue.service-runcompliance.log weblogfile-runcompliance.txt

SEG on Windows Proxy

SEG Guide - VMware AirWatch SEG Guide

Prevent usage of Native EMail on enrolled devices

Problem: Users can gain access to Exchange ActiveSync from uncontrolled devices and mail clients on them. Usage of SEG solves the problem of uncontrolled devices access.

You can enforce using Boxer/Inbox by creating an email compliance policy from the AirWatch console:

Email> Compliance Polices > General Email Policies > Mail Client

SEG as MS Exchange OWA Proxy

Warning

Article is for OLD separate SEG. NOT about SEG on UAG.

You can restrict mobile traffic to seg.company.com by installing IP and Domain restrictions on the IIS on the Exchange server, and then enable IP filtering to deny everyone but the SEG on the ActiveSync endpoint on IIS. This will ensure all enrolled mobile devices will access email through SEG. You can also implement email policies to ensure that unmanaged devices do not access the SEG.

AirWatch cannot block access to OWA for unenrolled mobile devices since SEG does not manage OWA. The only way to do so would be checking through the AD for unenrolled users and preventing them from webmail access from there.

Note

OWA traffic can be routed through the SEG however it will act as a simple pass through.

Warning

The OWA through SEG & proxying Webmail through SEG is not a supported setup as it could lead to a single point of failure for email access.

SEG Java Keystore

Warning

Article is for OLD separate SEG. NOT about SEG on UAG.

Default password for SEG Java Key store = changeit

SEG on Windows Java Memory

Warning

Article is for OLD separate SEG. NOT about SEG on UAG.

Zulu is the new Java Corporate middleware.

Resolution

  • Upgrade to latest version of SEG (2.18+);
  • Set the max heap size to 5Gb;
  • Use Shenandoah as the garbage collection method.

Follow these steps to apply the settings:

  1. Stop SEG service;
  2. Go to SEG install directory and edit file SecureEmailGateway-2.18/service/conf/segServiceWrapper.conf
  3. Update max heap to 5Gb, look for “Xmx” and update the property to: 

wrapper.java.additional.3=-Xmx5120m

  1. Use Shenandoah GC, look for “#wrapper.java.additional.38” and in the next line add: 

wrapper.java.additional.39=-XX:+UseShenandoahGC
wrapper.java.additional.40=-Xlog:gc=debug:file=tmp/gc-%p-%t.log:time,level,tags:filecount=10,filesize=50m*

  1. Save file and start the SEG service. 
  • Observe the system resources once this change is placed + enable GC logs in the above settings.

If no issue is seen, then remove the GC logging setting by following these steps: 

  1. Stop SEG service. 
  2. Go to SEG install directory and edit file SecureEmailGateway-2.18/service/conf/segServiceWrapper.conf
  3. Remove line: 

wrapper.java.additional.40=-Xlog:gc=debug:file=tmp/gc-%p-%t.log:time,level,tags:filecount=10,filesize=50m*

  1. Save and start SEG service.

SEG Clustering

If multiple SEG servers are load balanced, single policy broadcast messages apply to only one SEG. This includes the messages sent from the AirWatch Console to SEG upon enrollment, compliance violation, or correction. Use Delta Sync with a refresh interval of ten minutes to facilitate newly enrolled or compliant devices. These devices experience a waiting period of maximum ten minutes before email begins to sync. Benefits of this approach include:

  •  Updated policies from the same API source for all SEG servers. 
  • Smaller performance impact on API server.  
  • Reduced implementation or maintenance complexity compared to the SEG clustering model.   
  • Fewer failure points as each SEG is responsible for its own policy sets.
  •  Improved user experience. 
     

 SEG Clustering is also available to facilitate the sharing of single policy updates to all nodes of a SEG cluster.

SEG TLSv1

Please go through the following instructions in order to enable TLSv1.0 on SEG V2:

  • Go to SEG installation directory -> {SEG_DIRECTORTY}/service/conf
  • Edit file conf
  • Look for following properties:

Property 1

wrapper.java.additional.9=-Djdk.tls.disabledAlgorithms=MD5\, RC4\, TLSv1\, SSLv2Hello\, SSLv3\, DSA\, DESede\, DES\, 3DES\, DES40_CBC\, RC4_40\, MD5withRSA\, DH\, 3DES_EDE_CBC\, DHE\, DH keySize < 1024\, EC keySize < 224

set this property as: (Removing TLSv1 from the disabled list):

wrapper.java.additional.9=-Djdk.tls.disabledAlgorithms=MD5\, RC4\, SSLv2Hello\, SSLv3\, DSA\, DESede\, DES\, 3DES\, DES40_CBC\, RC4_40\, MD5withRSA\, DH\, 3DES_EDE_CBC\, DHE\, DH keySize < 1024\, EC keySize < 224

Property 2

wrapper.java.additional.12=-Dhttps.protocols=TLSv1.1\,TLSv1.2

set this property as:

wrapper.java.additional.12=-Dhttps.protocols= TLSv1\,TLSv1.1\,TLSv1.2

  • Restart SEG Service.

Integration - LDAP

Linked Articles

List of LDAP Queries used in Directory Searches (Active Directory and NON-Active Directory)

Search Group

(&(objectClass=group)(|(CN=*{inputName}*)(distinguishedName={inputName})))  

Sync Group

(&(objectClass=group)(|(objectGUID={ExternalID1})(objectGUID={ExternalID2}))) {ExternalID}
  • Hex Value if the object identifier has GUID value - String Value if the object identifier has string value

Search User

(&(objectCategory=person)(sAMAccountName={InputUserName}))  

Sync User

(&(objectCategory=person)(sAMAccountName=*)(|(objectGUID={ExternalID1})(objectGUID={ExternalID2}))  

Search Group member

(&(objectCategory=person)(sAMAccountName=*)(memberOf={GroupDN})) (&(objectClass=Group)(memberOf={GroupDN})) -- For recursive member search {GroupDN} - Group's distinguishedName value  

Add Missing Users

(&(objectCategory=person)(sAMAccountName=*)(|(distinguishedName={UserDN1})(distinguishedName={UserDN2})))

A sample how LDAP query works on the customer envirionment.

–sample—

ldapsearch -h gvx0lsami01q.company.com -p 389 -b O=COMPANY cn=EU_MDM_AirwatchEnabled,ou=groups,O=COMPANY member

Output:

C:\tools>ldapsearch -h gvx0lsami01q.company.com -p 389 -b O=COMPANY cn=EU_MDM_AirwatchEnabled member  
cn=EU_MDM_AirwatchEnabled,ou=groups,O=COMPANY  
member=secAuthority=Default  
member=cn=emmatest01,ou=eu,O=COMPANY  
member=cn=emmatest02,ou=eu,O=COMPANY  
member=cn=emmatest03,ou=eu,O=COMPANY  
member=cn=emmatest04,ou=eu,O=COMPANY
ldapsearch -h gvx0lsami01q.company.com -p 389 -b cn=emmatest01,ou=eu,O=COMPANY objectclass=* cn sn mail uid uniqueidentifier  
Alternatief evt: ldapsearch -h gvx0lsami01q.company.com -p 389 -b cn=emmatest01,ou=eu,O=COMPANY uid=* cn sn mail uid uniqueidentifier

Output:

C:\tools>ldapsearch -h gvx0lsami01q.company.com -p 389 -b cn=emmatest01,ou=eu,O=COMPANY objectclass=* cn sn mail uid uniqueidentifier  
cn=emmatest01,ou=eu,O=COMPANY  
cn=emmatest01  
cn=John Doe  
sn=emmatest01  
uid=emmatest01  
uniqueidentifier=5510012309  
mail=rr@company.com
C:\tools>ldapsearch -h gvx0lsami01q.company.com -p 389 -b cn=emmatest02,ou=eu,O=COMPANY objectclass=* cn sn mail uid uniqueidentifier  
cn=emmatest02,ou=eu,O=COMPANY  
cn=emmatest02  
cn=Adam Smith  
sn=emmatest02  
uid=emmatest02  
uniqueidentifier=5510012319  
mail=ph@company.com

Subsections of Integration - LDAP

IBM ISAM

Notes on integration:

  • Change the attribute where Distinguished Name in the AW Console is ‘distinguishedName’ to ’entryDN’ (see attached screenshot)
  • This is Accessible from the AW Console from Groups & Settings > All Settings > System > Enterprise Integration > Directory Services

Microsoft Active Directory

Note

VMware vSphere & Microsoft LDAP Channel Binding & Signing patch

Due to CVE-2017-8563 potential exploit, Microsoft is changing behavior of AD to accept connections only using TLS.

See external links:

Test if LDAP server listens to ports TCP636 and TCP3269:

nc -v LDAP-SERVER-IP 636
nc -v LDAP-SERVER-IP 3269

Integration

Go to Configuration > System Configuration > System > Enterprise Integration  > Directory Services

  • Directory Type: Directory platform being utilized
  • Server: Address of the directory services server
  • Port: TCP Port for communication with directory services
  • Protocol Version: Version of LDAP being used
  • Bind Authentication Type: Protocol used to authenticate the LDAP session (GSS-NEGOTIATE recommended, gives auto-choice of Kerberos/NTLM)

Advanced config

  • Search Subdomains: Search subdomains by LDAP chase referrals
  • Connection/Request Timeout: Timeout setting per connection/request in seconds
  • Search without Base DN: Enable to search without sending a base DN
  • Use Recursive OID at Enrollment/Group Sync : Only supported by AD; used to obtain group membership during enrollment and not rely on the scheduler to run
  • Object Identifier Data Type: Set the object ID type to string or binary. AD is binary by default while most other LDAPs are strings.
    Note

    In AirWatch 7.3+, the AD group structure itself is stored in the sync table, and if the user is shown to be a member of a nested group whose External ID is already stored in the table, the group membership will be reflected. The only flow unaccounted for is the group structure itself changes between scheduler iterations, in which at most there will be up to a 12 hour delay from when the user enrolls and is associated with the group after the group structure has changed.

Directory users mapping

  • From Directory Services, navigate to User

  • Specify the Base DN AirWatch uses to find users in the directory

  • For User Search Filter, enter the parameters used to associate AirWatch user accounts with AD/LDAP accounts (&(objectCategory=person)(sAMAccountName={EnrollmentUser}))

  • Select Show Advanced to display additional options

  • Enable Auto Merge to consolidate changes made in the directory with AirWatch automatically

  • Use Attributes to assign directory services information to the correct AirWatch fields

  • Click Save

    Note

    All members of the Group or Organizational Unit from the directory are first synced into the dbo.UserGroupEnrollmentUserMapSync table by running the below LDAP query (for AD).

  • Another query is run for all of the DN’s in the sync table to pull the actual user information from the directory: (&(objectCategory=person)(sAMAccountName=*)(|(distinguishedName={UserDN1})(distinguishedName={UserDN2})))

  • When we sync User attributes, we query the directory for the users based on their ExternalID:
    (&(objectCategory=person)(sAMAccountName=*)(|(objectGUID={ExternalID1})(objectGUID={ExternalID2})))

  • The User Group Membership Sync is the process by which we sync up the group membership for users from the directory. This will happen during a scheduler iteration if the Auto Sync option is enabled on the User Group and can be performed manually by clicking the Sync button on the User Group in the List View: (&(objectClass=group)(|(objectGUID={ExternalID1}) (objectGUID={ExternalID2}))

Sync interval

There are three primary scheduler jobs that sync up the group membership and user attributes:

  1. LDAP Sync – the LDAP Sync job will sync the User Group membership.
  2. Sync Directory Users – this job will sync up user attributes.
  3. Sync Admin Users – this job will sync up admin attributes.
Note

member vs. memberOf

Every object in AD has certain attributes like phone number, name, etc. In AD, membership to a group is determined by both the member and memberOf attribute, which is not the case in other directory types. Some directories only have member or only memberOf. The difference is key, memberOf is the attribute on the actual “member” or “user” object that says “I am a member of XYZ group”, this value is usually the Distinguished Name (DN) of the group the object is a member of. The Member attribute is on the “container” or “group”, and says “I have XYZ as a member”, which is also usually the DN of their members. Directories that use the Member relation will have groups with a list of Member attributes of all the users that are members of that group. Directories that use the MemberOf attribute will have users that list the groups they are members of on the actual user object.

Auto sync

In production all LDAP based scheduler jobs are set to fire every 12 hours. It is not recommended to lower these values for On-Premise or Dedicated SaaS customers as setting the scheduler interval too low may cause performance issues in versions older than 7.1. If a customer requires a lower interval, it is recommended to run at least one sync, and pull the LastSyncTimeInMinutes column from the dbo.LDAPDefinition table to determine how long it takes to sync the Organization Group.

Note

All issues that arise during a scheduler iteration will appear in the scheduler logs

Manual sync

The Add Missing Users, User Group Membership Sync, and Sync User Attributes processes can all be triggered manually by clicking a button in the console.

Note

Manual user attribute sync will only sync enrollment users, not administrator attributes

Warning

If any issues arise during one of these processes when they are triggered manually, the BulkProcessingServiceLogFile.txt in the Services folder will contain the backend information. If it appears there is a UI issue with the buttons, the WebLogFile.txt file in the WebConsole folder will contain the information needed to troubleshoot.

User enrollment

The mobileManagement.EnrollmentUser table contains information on all of the enrollment users in the environment:

select ExternalID, SecurityTypeID, * from mobileManagement.EnrollmentUser EU
join dbo.LocationGroup LG
on LG.LocationGroupID = EU.LocationGroupID
where LG.Name = 'ams'
  • ExternalID – the ExternalID column contains a hashed value of the attribute configured for Object Identifier. This value is used to match the AirWatch user with the customer’s directory user. If for whatever reason this value is null or incorrect, the AirWatch user will not sync.
  • SecurityTypeID – this column determines the type of user. 1 denotes a directory user, 2 denotes a basic user, and 3 denotes an authentication proxy user.
  • LocationGroupID – the Organization Group ID where the user is imported. Note that all directory users will always reside at the same level Directory Services is configured, even if imported or added at a child.
  • LDAPDefinitionID – the ID of the LDAP Definition the user is associated with.

dbo.LDAPDefinition table contains all of the configuration information from the Directory Services for an Organization Group

select LD.UserSearchFilter, LD. * from dbo.LDAPDefinition LD
join dbo.LocationGroup LG
on LG.LocationGroupID = EU.LocationGroupID
where LG.Name = 'ams'
  • LastSyncDurationInMinutes – this column contains the time it took to sync the entire Organization Group in minutes.
  • LastSyncedOn – last date the Organization Group synced with the directory.
  • MemberPageSize – the MemberPageSize value can be configured, but should not exceed 5000 if the customer is using EIS. This value determines the chunk size of information being sent back and forth between ACC\EIS
  • IsSortControlSupported – determines if the directory type supports sorting results at the directory server before the response is sent.

The dbo.UserGroupEnrollmentUserMapSync table will contain only the ExternalIDs of both the users and the groups that are members of the AirWatch group that was added

select MAP. * from dbo.UserGroupEnrollment(Core)UserMapSync MAP
join dbo.UserGroup UG
on UG.UserGroupSyncID = MAP.UserGroupSyncID
where UG.FriendlyName = 'ams'

The dbo.UserGroup table contains information about the configured User Groups in AirWatch.

select UG. * from dbo.UserGroup UG
join dbo.LocationGroup LG
on UG.RootLocationGroupID = LG.LocationGroupID
where LG.Name = 'ams'

The dbo.UserGroupSync table contains syncing information for directory User and Admin groups. It provides a few more columns of information that contain the settings you have configured per group in the console.

select UGS. * from dbo.UserGroupSync UGS
join dbo.LocationGroup LG
on UGS.RootLocationGroupID = LG.LocationGroupID
where LG.Name = 'ams'
Note

A user’s primary group in Active Directory cannot be added to AirWatch, as the primary group has no memberOf attribute on the user object. This is an AD limitation.

Query Troubleshooting

Tip

LDAP Admin is the LDAP browser most commonly used internally. An LDAP browser is an excellent way to troubleshoot certain queries and determine which attributes should be configured in Directory Services. The tool can be download at http://www.ldapadmin.org/

With some issues, AirWatch is not able to find certain users and/or groups. This is often due to an incorrect filter or lack of permissions in the directory. If we are unable to find the user and group objects with the LDAP Admin tool using the same settings from the console, we will be unable to find them using AirWatch. It will be necessary to test queries during troubleshooting. To run a custom query, click the magnifying glass icon in the toolbar, select the Custom tab in the window that appears, type the Base DN in the Path field, and type the query in the Filter field.

Connection Troubleshooting

Test connection failures are usually due to one of two error codes, either 49 or 81. An 81 error code indicates the console cannot find the directory server, which can happen if the hostname was entered incorrectly, ACC\EIS is not functioning properly, the directory server is firewalled, or there is no route to the directory server from the console server.

When an administrator encounters a 49 error it is important to note that this error is generated by the directory server, not AirWatch. In 99% of cases this is because the bind authentication type is not supported, or the account and passwords are incorrect. To verify that the console is not sending a bad username or password, SSL must be turned off and the authentication type must be set to basic so the bind request can be sniffed off the network in plaintext. Use Wireshark!

Warning

“System.DirectoryServices.Protocols.LdapException: Error code:81”

Tip
  • LDAP error 81 = The LDAP library can’t contact the LDAP server
  • AirWatch is not able to communicate with the LDAP server. Verify that Airwatch services are enabled in AirWatch Cloud Connector. If not, the AirWatch Cloud Connector may require re-installation after enabling.

Optimization

Warning

Optimization of slow enrollment into AD

Groups & Settings > All Settings > Enterprise Integration > Directory Services, Advanced section

Use Recursive OID At Enrollment = Disable

Common LDAP Queries

Search Group: (&(objectClass=group)(|(CN={inputName})(distinguishedName={inputName})))

Sync Group: (&(objectClass=group)(|(objectGUID={ExternalID1})(objectGUID={ExternalID2})))

Search User: (&(objectCategory=person)(sAMAccountName={InputUserName}))

Sync User: (&(objectCategory=person)(sAMAccountName=*)(|(objectGUID={ExternalID1})(objectGUID={ExternalID2})))

Add Missing Users: (&(objectCategory=person)(sAMAccountName=*)(|(distinguishedName={UserDN1})(distinguishedName={UserDN2})))

Novell eDirectory

Server Settings

User Settings

Advanced User Settings

Group Settings

Advanced Group Settings

Integration - Microsoft CA

DCOM preferred architecture for PoC

  • Windows Server for Certificate Authority role should be Datacenter/Enterprise Edition
  • The Certificate Authority should be in Enterprise Mode (AD integration)
  • The ACC/ESC connector should be in the same Active Directory Domain with appropriate CA
  • Check network ports! They are different for DCOM and SCEP

WS1 UEM Integration with MS CA Schema

Network Requirements

  • TCP Port 135: Microsoft DCOM Service Control Manager.
  • TCP Ports 1025 - 5000: Default ports DCOM processes.
  • TCP Ports 49152 - 65535: Dynamic Ports.

Configuration

Step 1 - Install Microsoft CA

Installation of MS CA Role...

Step 2a - Configure Microsoft CA, Basic

Warning

The following steps are applicable only if the Security Department allows a service account to have access to their CA

Add a Service Account on the CA

  1. Launch the Certification Authority Console from the Administrative Tools in Windows.
  2. In the left pane, select (+) to expand the CA directory.
  3. Right-click the name of the CA and select Properties. The CA Properties dialog box displays.
  4. Click the Security tab.
  5. Click Add. The Select Users, Computers, Service Accounts, or Groups dialog box displays.
  6. Click within the Enter the object names to select field and type the name of the service account (e.g., Ima Service).
  7. Click OK. The CA Properties dialog box displays.
  8. Select the service account you added in the previous step (e.g., Ima Service) from the Group or user names list.
  9. Select the Read, the Issue and Manage Certificates, and the Request Certificates checkboxes to assign permissions to the service account. Click OK.
Info

If Security Department does not allow the above, propose to use a separate child CA, and do the above configuration on it. Separate CA can be turned off in case of problems.

Warning

CA server must be in Enterprise Mode of operation. To install CA in Enterprise mode, Enterprise Admin account is needed. CA in Enterprise Mode is installed on forest level! This means Domain Controllers will have links to this CA, even if it is installed in a subdomain.

Step 2b - Configure Microsoft CA, Enroll On Behalf Of (EOBO)

Tip

Security Advantage!

  • The following steps are applicable if the Security Department DOES NOT allow a service account to have access to their CA;
  • As result of EOBO configuration, users will request certificates by themselves. If service account gets disabled, users will still be able to request certificates.

CA Step 0: Enable LDAP Referrals

Info

This action is only needed in multi-domain environment!

Run the following commands on the CA. This configuration is needed on ADCS CA since we are requesting certificates on behalf of some other user using service account.
This feature is only supported on Windows 2008 R2 Enterprise and later. See the link below for context and details: https://technet.microsoft.com/en-us/library/ff955842(v=ws.10).aspx

Use CMD to restart certificate services and enable cross-forest LDAP Referrals:

net stop certsvc
certutil -setreg policy\EditFlags +EDITF_ENABLELDAPREFERRALS
net start certsvc

CA Step 1: Create the Restricted Enrolment Agent Certificate Template

  1. Open the Certificate Authority (CA).
  2. Expand the CA Name, Right click Certificate Templates, and select Manage.
  3. Right click the Enrollment Agent (Computer) template and select Duplicate Template (Do not choose Enrollment Agent user cert!). Name it per your preference.
  4. Select Windows Server 2008+ Enterprise.
  5. On the Request Handling tab, select Allow Private Key to be Exported.
  6. In the Subject Name tab, make sure Build from this Active Directory Information is activated and Subject Name format is set to Fully distinguished name. Click OK.
  7. Navigate back to the CA, right click Certificate Templates, select New, and select Certificate Template to Issue.
  8. Select the duplicate copy of the template created in the previous step. Click OK.

CA Step 2 - Enroll a computer for the Signer Certificate

Substep A: Generate a new Restricted Enrollment Agent Signer Certificate
Tip

The following actions in this step can be done on any server that can connects to the Certificate Authority. Hint: do it on the server with ESC/ACC connector.

  1. Open MMC.
    Warning

    Use the same service user account to open MMC and import certificate, as the one used later for transmitting EOBO certificates. Using other user account, including admin accounts, will break the certificate request schema!

  2. Click File and select Add/Remove Snap in.
  3. Select Certificates.
  4. Select Computer Account.
  5. Select Local Computer and select Finish. Click OK.
  6. Expand Certificates (Local Computer), double click Personal, right click Certificates, select All Tasks, and select Request New Certificate. Click Next.
  7. Select Active Directory Enrollment Policy and select Next.
  8. Check the duplicate template created in earlier steps and select Enroll.
  9. Once completed, select Finish.
    Warning

    Service user account, which is used by AirWatch to create a certificate request, must be a domain account and have enough permissions to access Windows certificate store. Standard user will not work. Local admin rights on the computer is a simple solution.

Troubleshooting ESC service rights...
Substep B: Configure the issued certificate
  1. Once the certificate has been issued, right click it and select All Tasks followed by Manage Private Keys.
  2. Click Add.
  3. Type Network Service and select Check Names. Once added, select OK twice.
Substep C: Export the Certificate

Note

If the certificate needs to be installed on multiple Device Services servers or AirWatch Cloud Connector servers, export with the private key. If not, skip to exporting just the public key.

Export the public key to .cer file Only the public key needs to be exported for upload to the console:

  1. Right click the issued certificate, select All Tasks followed by Export.
  2. Select No, do not export the private key, select Next.
  3. Select DER encoded binary X.509 (.CER), select Next.
  4. Select a destination for the exported certificate and select Next. Click Finish.
Substep D: Import the certificate for Device Services and ESC/ACC servers
  1. Open MMC.
  2. Click File and select Add/Remove Snap in.
  3. Select Certificates.
  4. Select Computer Account and select Next.
  5. Select Local Computer and select Finish. Click OK.
  6. Expand Certificates (Local Computer) and select Personal. Right click Certificates, select All Tasks and select Import
  7. Select the .cer file exported in previous steps and select Next.
  8. Ensure Place all certificate in the following store is set to Personal and select Next. Click Finish.

CA Step 3: Add a User Certificate Template on the CA

  1. Open the CA (certsrv) window.
  2. In the left pane, select (+) to expand the CA directory.
  3. Right-click the Certificate Template folder and select Manage. The Certificate Templates Console window displays.
  4. Select the desired template (User) under Template Display Name, and right-click Duplicate Template. The Duplicate Template dialog box displays. AirWatch will use the duplicate certificate template. The template you choose depends on the function being configured in AirWatch.
    Tip

    For Wi-Fi, VPN, or Exchange Active Sync (EAS) client authentication select User template.

  5. Select the Windows Server that represents the oldest enterprise version being used within the domain to ensure backward compatibility of the certificate that was issued.
  6. Click OK. The Properties of New Template dialog box displays.

CA Step 4: Configure Certificate Template Properties

  1. Click the General tab.
  2. Type the name of the template displayed to users in the Template display name field. The Template name field auto-fills with the template display name without spaces.

You may use this default value or enter a new template name if desired. The template name may not contain spaces. Make note of the template name. You will need to enter this information in AirWatch. You will enter the Template name you just configured with no spaces in the AirWatch Console in the Issuing Template field within the Configuring the Certificate Template screen.

  1. Select the desired length of time for the certificate to be active from the Validity period entry field/drop-down menu.
  2. Click Apply.
  3. Click the Request Handling tab.
  4. Select the appropriate client authentication method from the Purpose: drop-down menu. This selection might be based on the application of the certificate being issued, although for general purpose client authentication, select Signature and Encryption.
  5. Select the Allow private key to be exported checkbox. For a certificate to be installed on an iOS device, this checkbox MUST be selected.
  6. Click Apply.
  7. Select the Subject Name tab.
  8. Select Supply in the request or Build from this Active Directory
Note

Selecting Supply in the request means the certificate fields will be generated by AirWatch console. Doing this will give AirWatch admin control over the text in certificate request.

Selecting Build from this Active Directory allows to write something simple in the AirWatch console Request Template fields (DN). CA admin will control over the text in certificate request. Do this if multiple fields are required for customer to configure EMail etc., and add these:

  • Include e-mail name in subject name
  • Include this information in alternate subjject name: E-Mail name, User principal name (UPN), etc.

11 (optional). If Enrollment agent template is used, select the Issuance Requirements tab and select This number of authorized signatures = 1. Under the Application policy drop-down field, select Certificate Request Agent and select Apply.

Enable the Template for Certificate Authentication
  1. Click the Extensions tab.
  2. Select Application Policies from the Extensions included in this template: field. This allows you to add client authentication.
  3. Click Edit. The Edit Application Policies Extension dialog box displays.
  4. Click Add. The Add Application Policy dialog box displays.
  5. Select Client Authentication from the Application policies: field.
  6. Click OK. The Properties of New Template dialog box displays.
Provide the AD Service Account Permissions to Request a Certificate
  1. Click the Security tab.
  2. Click Add. The Select Users, Computers, Service Accounts or Groups dialog box displays. This allows you to add the service account configured in Active Directory to request a certificate.
  3. Enter the name of the AirWatch service account in the Enter the object names to select field.
  4. Click OK. The Properties of New Template dialog box displays.
  5. Select the service account you created previously - for AirWatch on the CA, from the Group or user names: field.
  6. Select the Enroll checkbox under Permissions for CertTemplate ServiceAccount. Click OK.
Enable the Certificate Template on the CA
  1. Navigate to the Certificate Authority Console.
  2. Click (+) to expand the CA directory.
  3. Click Certificate Templates folder.
  4. Right-click and select New > Certificate Template to Issue. The Enable Certificates Templates dialog box displays.
  5. Select the name of the certificate template (e.g., Mobile User) that you previously created in Creating a Name for the Certificate Template.
  6. Click OK.
  1. Open the Certificate Authority (CA).
  2. Expand the CA Name, Right click Certificate Templates, and select Manage.
  3. Choose the new Enrollment Agent (Computer) template created for AirWatch
  4. Open the Superseded Templates tab.
  5. Click Add..
  6. Choose the created User Template, and add it to the list. Click OK.

Step 3 - Configure the AirWatch console

Configure the CA

  1. Login to the AirWatch Console as a user with AirWatch Administrator privileges, at minimum.
  2. Navigate to System > Enterprise Integration > Certificate Authorities.
  3. Click Add.
  4. Select Microsoft ADCS from the Authority Type drop-down menu. You need to select this option prior to populating other fields in the dialog so applicable fields and options display.
  5. Enter the following details about the CA in the remaining fields:
    • Enter a name for the CA in the Certificate Authority field. This is how the CA will be displayed within the AirWatch Console.
    • Enter a brief Description for the new CA.
    • Select ADCS radio button in the Protocol section.
      Note

      If you select SCEP, note that there are different fields and selections available not covered in this guide

    • Enter the host name of the CA server in the Server Hostname field (FQDN or IP)
    • Enter the actual CA Name in the Authority Name field. This is the name of the CA to which the ADCS endpoint is connected
      Tip

      Authority Name can be found by launching the Certification Authority application on the CA server - the inner name of the CA.

    • Select the radio button that reflects the type of service account in the Authentication section. Service Account causes the device user to enter credentials. Self-Service Portal authenticates the device without the user having to enter their credentials.
    • Enter the service account Domain\Username and Password. This is the username and password of the ADCS service account which has sufficient access to allow AirWatch to request and issue certificates.

Note

If Enrollment Agent is used:

  • In Additional Options list choose Restricted Enrollment Agent.
  • Upload the public key file (.cer) exported in previous steps.
6. Click Save.

Configure the Certificate Template

For Enrollment Agent, data supplied by AD (in CA config)
  1. Select the Request Templates tab. Click Add.
  2. Complete the certificate template information:. + Name: a friendly name for the new Request Template. This name is used by the AirWatch Console + Description (optional): a brief Description for the new certificate template + Certificate Authority: choose the just created one from the certificate authority drop-down menu + Issuing Template: the name that you configured in CA in Configuring Certificate Template Properties in the Template name field. Make sure you enter the name with no spaces. AirWatch automatically places “certificatetemplate:” prefix afterwards. This is normal. Do not enter the word “certificatetemplate:” yourself! + Requester Name: put something simple here, since request attribs are handled at the CA level. Example: {EmailDomain} or {EnrollmentUser}
  3. Click Save.
For direct User template, data supplied in request
  1. Select the Request Templates tab. Click Add.
  2. Complete the certificate template information. + Name: a friendly name for the new Request Template. This name is used by the AirWatch Console + Description (optional): a brief Description for the new certificate template + Certificate Authority: choose the just created one from the certificate authority drop-down menu + Issuing Template: the name that you configured in CA in Configuring Certificate Template Properties in the Template name field. Make sure you enter the name with no spaces. AirWatch automatically places “certificatetemplate:” prefix afterwards. This is normal. Do not enter the word “certificatetemplate:” yourself! + Subject Name: put specific fields, which will be in the certificate. Example syntax for multi-line Subject Name field:
CN={EnrollmentUser},E={EmailAddress},OU={UserDistinguishedName}

Private Key Length: choose a value (This is typically 2048 and should match the setting on the certificate template that is being used by DCOM) + Private Key Type: +Signing, +Encryption (This should match the setting on the certificate template that is being used by DCOM) + San Type: enter fields for Subject Alternative Name. Email AddressUser Principal Name, and DNS Name are supported by ADCS Templates by default, and AirWatch recommends that you use them.

Example of fields to match CA config: User Principal Name = {UserPrincipalName} Email Address = {EmailAddress}

- Select the **Automatic Certificate Renewal** checkbox to have certificates using this template automatically renewed prior to their expiration date. If enabled, specify the Auto Renewal Period in days.

- Select the **Enable Certificate Revocation** checkbox to have certificates automatically revoked when applicable devices are unenrolled or deleted, or if the applicable profile is removed.
Note

If you are making use of the Enable Certificate Revocation feature, go to Devices & Users > General > Advanced and set the number of hours in the Certificate Revocation Grace Period field. This is the amount of time in hours after the discovery that a required certificate is missing from a device that the system will wait before actually revoking the certificate.This will help to NOT identify certificate missing on device because of big Wi-Fi latency or network issues.

- Select the **Force Key Generation on Device** checkbox to generate public and private key pair on the device which improves CA performance and security.
  1. Click Save.

Test the certificates

  • Create a new device profile

Profile screen 1

  • Configure EMail settings in profile to use the Certificate (automatically named “certificate #1”)

Profile screen 2

  • Configure Wi-Fi settings in profile to use Certificate (automatically named “certificate #1”)

Profile screen 3

Check the profile on iPhone/Android: you should see the Certificate profile applied, then certificate gets issued. Check the customized fields: Subject Name and Subject Alternative Name.

Subsections of Integration - Microsoft CA

SCEP and DCOM

SCEP

SCEP vs DCOM: SCEP does NOT support certificate revokation, unlike DCOM, so DCOM integration is the preferred method.

SCEP Workflow

Manual Revocation

Manual Renewal

EOBO

Question: We trying to integrate AirWatch with CA authority (ADCS). Instead of creating the certificate for the user that enrolled his device, every certificate is created for the service account we used in the integration. Any config we missed?

Answer: This is expected behavior. If you need the certificate created for the user object you need to leverage Enrollement On Behalf Of(Eobo). Just be aware that you lose flexibility with the cert template and values you can use.

This is done in the template settings in Airwatch. The template in ADCS is configured with Subject Name = “supply in request”. So whatever you set up for the template in Airwatch will be requested from the CA. Including SN (can be username/email/serial number and so on) and SAN (again all kinds of values available in AW)

That is also the main difference to EOBO where you configure SN/SAN in the ADCS template and can only use attributes from the user object in AD.

Internal Apps

Articles in section

Apps Upload

Automatic application upload can be made using the following CURL command via blob file:

curl -k -v -X POST -H 'Authorization: Basic ***' -H 'aw-tenant-code: ***'
-H 'Content-Type: application/octet-stream' -H 'Accept: application/json;version=1'
-H 'Expect: 100-continue' --data-binary '@./TestApp2.ipa'
'https://airwatch.testserver.ru/API/v1/mam/blobs/uploadblob?fileName=TestApp2.ipa&organizationGroupId=571'

Important moments: -H ‘Expect: 100-continue’ MUST be sent, because else the server answers with code 100, and curl does not know what to do;

-H ‘Content-Type: application/octet-stream’ MUST be specified, because in the original text we have some JSON or similar.

As result we receive the needed:

Value {"uuid":"7182fd4c-7661-4753-b4d3-fd9353ad4c6a","Value":856}

We take this value and do begininstall

curl -k -v -X
POST -H "Authorization: Basic ***" -H "aw-tenant-code: ***"
-H "Accept: application/json" -H "Content-Type: application/json" -d
'{"ApplicationName":"TestApp20", "AutoUpdateVersion":true, "BlobId":856,
"PushMode":"Auto", "DeviceType":"Apple", "LocationGroupId":571,
"SupportedModels":{"Model":[{"ApplicationId":139, "ModelId":2,
"ModelName":"iPad"}]}}' "https://airwatch.testserver.ru/API/v1/mam/apps/internal/begininstall"
Note

What is Blobid? Not fully understood. Tried different numbers..

As an answer we receive:

{"ApplicationName":"TestApp20","BundleId":"ru.testserver.TestApp3",
"AppVersion":"2.54.0","Platform":2,"IsReimbursable":false,"ApplicationSource":0,
"LocationGroupId":571,"OrganizationGroupUuid":"24e7ceee-d233-4697-9fb9-78e32ee25462",
"PushMode":0,"AppRank":0,"AssignedDeviceCount":0,"InstalledDeviceCount":0,
"NotInstalledDeviceCount":0,"AutoUpdateVersion":false,"EnableProvisioning":false,
"IsDependencyFile":false,"ContentGatewayId":0,"Id":{"Value":141},"Uuid":"dde99cc2-228d-4886-8fc8-c28b82683c36"}

Subsections of Internal Apps

Apps Deployment

AirWatch has 2,5 mechanisms to deploy apps:

  1. Apps & Books → Internal. You can only deploy 3 versions of an application called “Alpha”, “Beta”, “Production”;
  2. Devices → **Staging & Provisioning →**Product List View → Add Product. Add Manifest → Install Application. Difference between this method and the one below is not clear, except slightly easier config;
  3. Devices → Staging & Provisioning → Components → Files/Actions. Add Files/Actions → Add Manifest → Install Unmanaged Application.

I recommend the 3rd method as the most descriptive and stable if several versions of an application are needed in one Organization Group.

Things to check in application deployment

Application ID

Check if there is a modification of the Application ID while uploading the application in the Console via Staging & Provisioning → Components → Applications or check while uploading the Application, is it fetching the Application ID and populating the same name in the Application ID field.

Application Metadata

Once the app gets uploaded, AirWatch SQL writes down its’ metadata to SQL. Parsing the data:

SELECT a.name, a.versionhash, * FROM INTERROGATOR.APPLICATIONLIST AL
JOIN interrogator.Application A ON AL.ApplicationID=A.ApplicationID
WHERE DEVICEID in (1473) and isinstalled=1

See more on SQL querying apps

Warning

On AirWatch 9.2.3 version and lower, metadata provided by some build systems like Gradle 3 is incorrectly processed. Specifically a VersionHash is needed in the correct place for the app to see it.

Application Deployment Logs

  • Device receives command to install application
  • Device is directed to app destination for download via Manifest.plist file - so search for “manifest” in the logs:

  • Device is redirected to Manifest.plist URL
  • Device locates Blobhandler.pblob URL:

  • Device is redirected Blobhandler.pblob URL
  • Application download begins
Tip

You can copy the blob link (see picture) manually in the browser to check that download starts.

macOS Apps Deployment

External link:

You are able to deploy apps on Mac OSX devices either from Apps & Books or from Product & Provisioning method. If you have a pkg or dmg file, then you could simply use Apps & Books. However, if you have multiple files to be executed or scripts to run then you could use Product & Provisioning. In AirWatch 9.3+, all macOS application file types (.dmg, .pkg, .mpkg, .app) can be managed through the Internal Applications section. This new framework leverages the popular Mac Admin community tool, Munki.

Provisioning with Munki

Enabling Software Management

  • Navigate to Settings > Devices & Users > Apple > Apple macOS > Software Management
  • Enable Software Management , there is a check in place to verify if File Storage is enabled at this page. If there is no File Storage, the admin will be requested to enable File Storage.
  • On-Prem deployments will need to enable CDN.

Before uploading macOS File to AirWatch

All primary macOS software file types will now be uploaded through Internal Applications (.pkg, .dmg, .mpkg). A .pkg file can be a Bootstrap Package or can be managed through full lifecycle management (this feature). In order to configure Advanced Management options for macOS software and for effective desired state management, which is achieved through the integrated Open-Source Munki library in the AirWatch Agent, a metadata file must be generated for the file separately before uploading it to the AirWatch Console. Munki’s deployment logic is built on the concept of pkginfo files, which are xml/plist files that contain metadata, configuration information, and more for a given application file. AirWatch requires this pkginfo file along with the application file to manage the deployment in the Console.

The pkginfo file is genetrated with VMware AirWatch Admin Assistant (see tools page for download). This is a GUI wrapper for a Munki command-line utility and is used to generate the pkginfo metadata file for a given application file.

Generating a pkginfo metadata file using VMware AirWatch Admin Assistant

  1. Download and install the Admin Assistant tool to a macOS device or VM.

  2. Open the Assistant. The Assistant dialog will ask for the application installer files to parse.

  3. Upload an application installer file by dragging & dropping a .pkg, .dmg, .mpkg, .app file into the labeled field, or browse the local files on the machine in order to find an installer file.

    Note

    If the file is .app, it will be converted into a .dmg for deployment

  4. Once the file is selected and uploaded, the Assistant will automatically start parsing process the process. Additional files can be added during this time if needed.

  5. Once the parsing is complete, the tool will prompt to reveal the parsed metadata files in Finder. Store the metadata files in a local folder where they can be easily retrieved during the Software Distribution procedure.

Generating a pkginfo metadata file using Autopkg

There are multiple ways to obtain the metadata/pkginfo file aside from using the Admin Assistant. One of them is to use Autopkg and its’ GUI tool Autopkgr.

Uploading a macOS Application to the AW Console:

  1. Navigate to Apps & Books > Applications > Native > Internal
  2. Click Add Application
  3. Upload app/software file (.pkg, .mpkg, .dmg)
  4. Continue to the next screen and Upload the pkginfo .plist file
  5. Continue to customize the app deployment. The proceeding screens will display any available configurations present in the pkginfo file (for example, a pkginfo from AutoPkg may contain an install_check script). Additional deployment configurations can be defined, and will be merged with the existing configurations
    • If the pkginfo file has one or more key and configuration that are not exposed in the UI, they will not be affected. This feature is important as it allows administrators to upload pkginfo files with keys that would be supported by the Munki client but would not be configurable in the UI. Any changes in the UI will be merged with the existing keys.

Pre/Post Install Scripts

A common reason to repackage software is to perform additional configuration tasks or to install additional items. A technique to avoid repackaging is to add pre-install scripts and/or post-install scripts to the configuration of an item. These scripts can take care of some of the tasks which previously may have required repackaging to implement.

An Exit Code of 0 will define the script as successfully executed in order to allow Munki to continue.

Note

Failure of the pre-install script will abort the installation attempt. Failure of the post-install script will log errors, but the installation will be considered complete.

Note

SCRIPT LOG echo statements will be logged to /Library/Application Support/AirWatch/Data/Munki/Managed Installs/Logs/ManagedSoftwareUpdate.log

Uninstall Methods

There are multiple options available for uninstallation of software. The appropriate option will be selected by default by the VMware Admin Assistant tool based on the software’s file type. If needed options to override the default values are available in the AirWatch Console.

Remove Copied Items
Used primarily for software installed from a .dmg

  • Pulls from items_to_copy array[dicts] in the pkginfo file
    • All file paths in this array will be deleted
  • Future Console release will show the paths in the items_to_copy array in the UI

Remove App

  • Pulls from installs array[dicts] in the pkginfo file
    • All file paths in this array will be deleted
  • Future Console release will show the paths in the installs array in the UI

Remove Packages 
Used primarily for software installed from a .pkg

  • Uses receipts and analyzes the packages to remove
    • Tries to determine what files were installed via the Bom file
    • Deletes receipt
  • Will only remove if package is not associated with any other files or programs
  • Future Console releases will show the receipts that Munki check for in the UI

Uninstall Script 
Can be used for any installer type

  • Written in shell script
  • Used to perform custom uninstall operations if needed
    • If the Admin has a customized deployment of an app, they will need to also write a corresponding uninstall script to remove their custom configurations

Install or Uninstall Verification

With some software, the Admin needs to configure what exactly defines a Successful Install or Uninstall. Munki allows this through setting an Install or Uninstall Check Script

Install Check Script - If present, this script is executed to determine if an item needs to be installed. A return code of 0 means install is needed; any other return code causes install to be skipped.

Uninstall Check Script - If present, this script is executed to determine if an item needs to be uninstalled. A return code of 0 means uninstall is needed; any other return code causes uninstall to be skipped.

Conditions

Conditions can defined on a per-application level so that they are evaluated prior to the download and installation of a software.

Warning

Custom conditions will be created in a later version of AirWatch.

Built-in Conditions Currently available built-in attributes for conditional comparison:

AttributeTypeDescriptionExample Comparison
hostnamestringHostnamehostname == "Lobby iMac"
archstringProcessor architecture. e.g. 'powerpc', 'i386', 'x86_64'.arch == "x86_64"
os_versstringFull OS Version e.g. "10.7.2"os_vers BEGINSWITH "10.7"
os_vers_majorintegerMajor OS Version e.g. '10'os_vers_major == 10
os_vers_minorintegerMinor OS Version e.g. '7'os_vers_minor == 7
os_vers_patchintegerPoint release version e.g. '2'os_vers_patch >= 2
machine_modelstring'Macmini1,1', 'iMac4,1', 'MacBookPro8,2'machine_model == "iMac4,1"
machine_typestring'laptop' or 'desktop'machine_type == "laptop"
ipv4_addressarray of stringsThis contains current IPv4 addresses for all interfaces.ANY ipv4_address CONTAINS '192.168.161.'
munki_versionstringFull version of the installed munkitoolsmunki_version LIKE '*0.8.3*'
serial_numberstringMachine serial numberserial_number == "W9999999U2P"
dateUTC date stringDate and time. Note the special syntax required to cast a string into an NSDate object.date > CAST("2013-01-02T00:00:00Z", "NSDate")

Example:

machine_type == “laptop” AND os_vers BEGINSWITH “10.7”

date > CAST(“2016-03-02T00:00:00Z”, “NSDate”)

Dates in conditions:

The date string must be written in UTC format, this format is interpreted as a local date/time. The condition date > CAST("2013-01-02T00:00:00Z", "NSDate") is True if the local time is after midnight local time on 02 Jan 2013.

Literal types in comparisons

  • Strings are delimited by either single or double-quotes: os_vers BEGINSWITH "10.7"

  • Integers have no quotes: os_vers_major == 10

  • Booleans are indicated as TRUE or FALSE (and have no quotes, or they’d be strings!): some_custom_condition == TRUE

  • Dates are possible, but they must be cast from ISO 8601 strings: date > CAST("2013-01-02T00:00:00Z", "NSDate")

Updates

Updates can be managed similarly to the other platforms in the AirWatch Console. If a new version of the file needs to be added, perform the following:

  1. Navigate to Apps & Books > Native
  2. Click on the App that requires an update. Clicking the app will navigate to the Details View page.
  3. In the top right side, click “Add Version
  4. Upload the new installer for the new app version
  5. Upload the new pkginfo file for the new version
  6. Make any additional changes and then save the configuration

Troubleshooting

The AirWatch Console will show report macOS app installation data from a device in several  locations:

  • Apps & Books > Applications > Native > Internal. Click onto the Application to drill into Application Details > Devices Tab. The grid in this tab will display installation statuses for each device.
  • Devices & Users > Devices > List View. Click on a device to drill into Device Details > Troubleshooting Tab. The grid on this tab will show activity on the device and provides filtering options to show information relating to Software Distribution.

Munki Logs can also be directly accessed on the device: /Library/Application\ Support/AirWatch/Data/Munki/Managed\ Installs/Logs/ManagedSoftwareUpdate.log

tail -n 20 -F /Library/Application\ Support/AirWatch/Data/Munki/Managed\ Installs/Logs/ManagedSoftwareUpdate.log

Munki Known Issues

  • Once software has been published and installed on a few devices, if any of the configuration options such as “install script”, “uninstall script” etc.. are changed, devices which already have the application installed successfully will not receive the updated options as these can only be updated when an “InstallApplication” command. This is the only command that can update the locally cached PKGInfo files on the device. In order to update the scripts, administrators will have to manually select all of the devices which have the software installed and repush the command. Repushing the command will update the respective cached PKGinfo file with the latest information.
    Note: Repushing the command will NOT reinstall the application unless there is a change in the software version. Munki is has mechanisms in place to not reinstall software when the application is already installed.
  • If a package has a dependency on another package or software then an administrator will have to manually provide a required key in the pkginfo of the package with a dependency. Providing the key in the pkginfo is the only way for Munki to recognize if there are any dependencies, and determine the sequence of installation steps for such dependent packages. Unless otherwise specified, packages are installed in random order. Such package installation will fail if a package dependency is not met during the installation. 
    ExampleNetbeans requires that Java be present and installed on the machine. Netbeans will not implicitly install Java while installing the Netbeans software package. Even if administrators attempt to install Netbeans without utilizing M****unki or using standalone Munki, the installation will fail and will prompt the administrator to first install Java before installing Netbeans.
  • For packages which have the same receipts array with same version even on an upgraded package, Munki will not be able to detect that a new package is an upgraded package. The reason is that the receipt array has the same version as the previous package. The impact of duplicated receipts arrays is that the packages will not be installed on User machines as Munki cannot determine that a new version needs to be installed.
    Note: The best solution for packages with duplicate versions in the Receipts Array is to use an Autopkg Recipe, which adds an installs array in the pkginfo. This method allows Munki to detect upgraded packages; alternatively an administrator can manually add an installs array into the pkginfo.

Example: Wireshark is an example of a package which has identical receipts which contain the same version, thus Munki cannot detect the when upgraded packages are deployed.

Legacy methods (AirWatch 9.2 & older)

Apps & Books

  • Go to Groups & Settings > All Settings > Devices and Users > Apple > MAC OS > Software Distribution.
  • Enable Software distribution.
  • Now go to Apps and Books > Native > Internal > Add Application.
  • Upload the .pkg or .dmg file.
  • Once uploaded, click on Continue. You will get an option to download VMware AirWatch Admin Assistance tool.
  • Download the tool.
  • Upload the .pkg or .dmg file there.
  • You will see that it will open the package into .dmg file, .plist file and .png file.
  • Go back to the Console and upload the .plist file for metadata.
  • Click on upload and you will see the application description.
  • You will get an option to upload the image. Upload .png file there.
  • Now assign the application to the required Smart Group.
  • Click on Save and Publish.

Product provisioning

It allows you to create, through AirWatch, products containing profiles, applications, and files/actions (depending on the platform you use). These products follow a set of rules, schedules, and dependencies as guidelines for ensuring your devices remain up to date with the content they need.

  • Navigate to Devices > Staging & Provisioning > Components > Files/Actions
  • Click on Add > General
  • Navigate to Files > Add Files and enter the specific path
  • Navigate to Manifest. Under Install Manifest, click Add Action. Choose the action type Install and provide the specific path. 
  • Under Uninstall Manifest, click Add Action and set the action type as Install, again providing the specific path. 
  • Under Uninstall Manifest, click on Add Action and choose the action type Uninstall, and provide the exact application name with the extension as ‘.app’.
  • Select Save
  • Navigate to Staging & Provisioning > Product List View and select Add Product.
  • Under General, verify the details and assignment group are filled in.
  • Navigate to the Manifest tab and click on Add. Choose the created component after selecting Install/Action.

With this configuration, the application will first be downloaded to the Downloads folder on the Mac. You can then install the application, which will be available in the Applications folder. If you want only to uninstall the application which was previously installed, you may choose the Delete Files option while creating files and actions.

Recipes

Script recipies

TextWrangler Post Install Script Implement a post-install script that copies the command-line tools from the TextWrangler bundle to their intended locations.

After uploading the appropriate files to AirWatch, pre/post install scripts can be configured for the app in the Scripts tab of the Application Details. Simply paste the script into the appropriate field and AirWatch will format it to be used by Munki.

App recipes

Adobe Reader DC...
Apple Enterprise Connect...
McAfee Endpoint Protection...
Microsoft Skype For Business...
Palo Alto GlobalProtect...
RSA SecureID...
Sophos Antivirus...
Symantec Removal Tool...

Manifest recipies

Quick fix for root user login bug (https://techcrunch.com/2017/11/28/astonishing-os-x-bug-lets-anyone-log-into-a-high-sierra-machine/ )

Enable root user with random password

Below are steps to fix the issue via  Product Provisioning -

A. Create a File

  1. Go to Devices > Staging & Provisioning > Components > Files /Actions.
    2.  Add Files / Actions > macOS
  2. Manifest > Install Manifest > Run > Command Line and Arguments to run /usr/bin/dscl . -passwd /Users/root $(/usr/bin/openssl rand -base64 20)
  3. Click Save

B. Create a Product to include above File

  1. Go to Devices > Staging & Provisioning > Products List View  > Add Product > macOS
    2.  Click on Manifest > Add Manifest > Actions to Perform > Install Files / Actions > Files/Actions (Enter the saved file).
  2. Save & Activate the product

Unsigned application launch bypassing Security & Privacy

Use above recipe with command Manifest > Install Manifest > Run > Command Line and Arguments to run:

sudo xattr -rd com.apple.quarantine /Applications/@cursor # where '/Applications/@cursor' is the target app

Large-Scale Performance

Large-Scale Performance Tweaks

For 75000+ devices (iOS, Android & Windows Desktops)

Linked Articles

Attached Documents (pdf)

Test Report 1811 - 75k devices Test Report 1907 - 1.2mln devices Test Report 1907 (500k devices) Test Strategy 1902 UI Test 1902 Sizing Topology - 1mln devices

SQL

See SQL Database page for base tweaks on MS SQL.

  • Add 1 tempdb file per core (20 tempdb files for 20 core)

Each tempdb data file with below parameters: - Initial Size: 16000 MB - Autogrowth: 512 MB

USE [master];
GO
ALTER DATABASE [tempdb] ADD FILE (NAME = N'Temp1', FILENAME = N'H:\TempDBs\Temp1.ndf', SIZE = 8GB , FILEGROWTH = 512);
ALTER DATABASE [tempdb] ADD FILE (NAME = N'Temp2', FILENAME = N'H:\TempDBs\Temp2.ndf', SIZE = 8GB , FILEGROWTH = 512);
ALTER DATABASE [tempdb] ADD FILE (NAME = N'Temp3', FILENAME = N'H:\TempDBs\Temp3.ndf', SIZE = 8GB , FILEGROWTH = 512);
GO
  • Add below Trace flags in Startup Parameters for “mssqlserver” service as per Microsoft recommendations:
    T174 - https://support.microsoft.com/en-us/help/3026083/fix-sos-cachestorespinlock-contention-on-ad-hoc-sql-server-plan-cache 
    T834 - https://support.microsoft.com/en-us/help/920093/tuning-options-for-sqlserver-when-running-in-high-performance-workloads
    T3247 - https://support.microsoft.com/en-us/help/3216543/workloads-that-utilizemany-frequent-short-transactions-in-sql-server
  • Set “Lock Pages in Memory” privilege for the service account
    Open Windows Local Group Policy Editor
    Open Computer Configuration > Windows Settings > Local Policies > User Rights Assignment, find Lock Pages in Memory
  • Disable “Named Pipes” and Enable “TCP/IP” network protocol
    Open SQL Server Configuration Manager tool
    Open SQL Server Network Configuration > Protocols for MSSQLSERVER, set Named Pipes to Disabled, set TCP/IP to Enabled
  • Increase Maximum Worker threads in server properties to “7500”
    Open Server Properties from SSMS and go to Processors tab, find Maximum Worker threads field
  • Set “Max Degree Of Parallelism” to 2 and “Cost threshold for Parallelism” to 50
    Open Server Properties from SSMS and go to Advanced tab, Parallelism section in the right pane
  • Enable “Received Side scale” setting for Network Adapter on SQL server
    Run DEVMGMT.msc from CMD
    Expand Network adapters, right click on your adapter and select Properties
    Select the Advanced tab and find Receive Side Scaling. Set this to Enabled
  • Set “Delayed Durability” under database properties = “Forced” to reduce WriteLog waits
    Open Database Properties from SSMS and go to Options
  • Update “Minimum and Maximum Server Memory” allocation in Server Properties
    Open Memory from SSMS → Server Memory Options section
    • Minimum Server Memory (in MB): 256000
    • Maximum Server Memory (in MB): 1500000

Performance Tweaks

Example Architecture

  • Update Workspace ONE UEM application server to latest version

In WS1 UEM Console go to Settings > Installation > Performance Tuning:

  • Set Certificate Profile Publish Frequency: 100
  • Set Apple Profile Installation Batch Size: 300
  • Set iOS Device Invites Per Second: 30
  • Set FastLaneMessageRateMultiple: 1.5
  • Run Real-Time Compliance is set to Disabled

Check Windows Sample Frequency (for managing Windows Desktops):

  • Go to Settings > Devices & Users > Windows > Windows Desktop > App Settings
  • Go to Settings > Devices & Users > Windows > Windows Desktop > Windows Sample Schedule
  • Override the default value in Systemcode table for FastLaneMessageRateMultiple to 1.5 by updating Systemcodeoverride table in DB.
  • Add the below config to AW.Send.Messaging.Service.exe.config file in Services folder of all console servers under appSettings:
<add key="WnsMessageProcessingRate" value="10" />
<add key="WnsMessageThrottlingEnabled" value="true" />

Subsections of Large-Scale Performance

Certificate Batching

Note

This logic is only applicable for profiles that contain certificates issues through a defined Certificate Authority (CA) in the AirWatch Console.

Prior to the certificate batching logic, when a defined-CA profile is published to a large number of devices, it can result in significant performance issues on Airwatch Cloud Connectors in the AirWatch environment.  This is due to the extra processing required to issue each device a certificate from the CA, as well as the normal processing required to push the profile to the device.  The performance issues generally lead to CPU spikes and high memory usage on ACC or DS (if no ACC is configured) servers because:

  • Certificate generation is a CPU intensive task. Too many concurrent certificate generation requests spike the CPU usage.
  • Certificates are stored in memory of the process (w3wp - devicesservices) so that they can be reused for the same device.

AirWatch 7.1+ has certificate batch logic to throttle the deployment of all defined-CA profile publish jobs. These profile publish jobs will be broken into multiple batches at the environment level and a next batch will only be executed if the previous batch has been completed. 

Certificate Batching Logic

The deployment of defined-CA certificate profiles will follow the steps outlined below:

  • AirWatch Administrator publishes a profile that contained a defined CA certificate
  • Install profile commands are queued for all assigned devices in the Device Command Queue in the AirWatch Database.  These commands are queued in the Held status.  Commands in the Held status will not be processed by devices.
  • A ProfilePublishBatchJob is created to process these commands.
  • The ProfilePublishJobScheduler selects a batch and batch size, based on the settings configured in the AirWatch Console (under Settings → Installation → Performance Tuning for On-Premises environments).
  • A batch detail entity is inserted into the AirWatch DB; the mobilemanagement.CommandQueueHeldBatch_Update stored procedure is called to “release” a number of the Install Profile commands that are currently in the Held status.  The number released will be equal to or less than the overall batch size.
  • Periodically, more batches of the Install Profile command will be released until all commands are active (or have successfully been issued to devices).  Three scenarios are outlined below that describe this step in more detail.  This time period is defined by the Profile Publish Batch Job in the AirWatch Scheduler (Settings → Admin → Scheduler for On-Premises environments).

In SaaS and on-premise environments, the default Batch Size is set to 50, and the Profile Publish Batch Job interval is set to 3 minutes.

Scenario 1: All devices are consistently active

  • Devices=24000
  • Batch size=200

When a profile is published, all 24000 commands are queued in the Held status (status = 7).  The scheduler released the first 200 commands based on the batch size.  The commands are sent to devices (through APNs/Firebase), and devices consume the commands and install the profile.  At the interval defined by the Profile Publish Batch Job, the system will view how many pending install commands are still active.  As all devices are currently active and check in regularly, all of the previously queued commands will be processed, and another batch of 200 will be released.  This process repeats until all 24,000 install commands are released.

Scenario 2: Devices frequently check in (within 30 minutes)

  • Devices=24,000
  • Batch size=200

During any particular iteration of the Profile Publish Batch Job as outlined above, if {X} number of commands are still in Pending Install state for a period of less than 30 minutes, then only ({batchSize} -{X}) commands will be released in the current iteration.  For example, if the previous batch released 200 commands, but at the time of the next batch only 50 commands have been processed by devices (so that 150 are still active), then only 50 commands will be released at the current batch.  At the time of this iteration, it will ensure that the “active” batch size returns to 200.

Scenario 3:  Devices periodically check in (within 12 hours)

  • Devices=24,000
  • Batch size=200

During any particular iteration of hte Profile Publish Batch Job, if {Y} number of commands are still in Pending Install state for a period of more than 30 minutes, then {batchSize} number of commands will be queued up to maximum limit of *{batchSize}10 commands in queue. This means that at a single point in time, the maximum of *{batchSize}10 Install Profile commands can be queued in Pending Install status.

Following Scenario 2 above, if a new batch iteration is occurring, and all active Pending Install commands are over 30 minutes old, then a full batch size will be released.  If there are {X} pending commands less than 30 minutes and {Y} pending commands greater than 30 minutes, then {batchSize} - {X} commands will be released, up until a total of *{batchSize}10 overall active commands are queued.

Scenario 4:  Devices do not check in for over 12 hours

  • Devices=24,000
  • Batch size=200

After a pending install command has been queued for over 12 hours, it no longer counts toward the *{batchSize}10 limit specified in Scenario 3.  The logic outlined in the first 3 Scenarios will continue, not counting any of these older commands.  These “older” commands remain in the command queue and are active, but no longer affect the batching logic.

Relevant Database Tables

  • dbo.BatchJob - Profile Publish places an entry with “profileVersionID” 
  • dbo.BatchJobDetail - Every Batch Job iteration will place an entry with the number of commands released as “Count”
  • deviceCommandQueue.DeviceQueue - Stores commands (profile install commands in this case) for every device

Large-Scale Performance

WS1 UEM Services/Applications

API Workflow
Batch Processing
Entity Change Queue Monitor
EventLog Processor Service
Integration Service
MEG Service
Messaging Service
Outbound Queue Monitor Service
Smart Group Service
Agent Builder Service
Background Processor Service
Compliance Service
Content Delivery
Device Scheduler
Directory Sync Service
GEM Inventory Service
Policy Engine
SMS Service
Google Play Service
Data Platform Service
Interrogator Queue Service
Provisioning Package Service
MSMQ

PerfMon Object Perfmon Counter
Process “% Privileged Time”, “% Processor Time”, “% User Time”, “Elapsed Time”, “Handle Count”, “IO Read Bytes/sec”, “IO Read Operations/sec”, “IO Write Bytes/sec”, “IO Write Operations/sec”, “Private Bytes”, “Thread Count”, “Virtual Bytes”, “Working Set”, “Working Set - Private”
.NET CLR Exceptions “# of Exceps Thrown”, “# of Exceps Thrown / Sec”, “# of Filters / Sec”, “# of Finallys / Sec”, “Throw to Catch Depth / Sec”
.NET CLR LocksAndThreads “# of current logical Threads”,"# of current physical Threads","# of current recognized threads", “# of total recognized threads”,“Contention Rate / Sec”,“Current Queue Length”,“Queue Length / sec”, “Queue Length Peak”,“rate of recognized threads / sec”,“Total # of Contentions”
.NET CLR Memory “% Time in GC”,"# Bytes in all Heaps","# Gen 0 Collections","# Gen 1 Collections","# Gen 2 Collections", “# Induced GC”,“Allocated Bytes/sec”,“Finalization Survivors”,“Gen 0 heap size”,“Gen 1 heap size”, “Gen 2 heap size”,“Large Object Heap size”,"# of Pinned Objects","# GC Handles","# of Sink Blocks in use", “# Total committed Bytes”,"# Total reserved Bytes",“Finalization Survivors”,“Gen 0 Promoted Bytes/Sec”, “Gen 1 Promoted Bytes/Sec”,“Large Object Heap size”,“Process ID”,“Promoted Finalization-Memory from Gen 0”, “Promoted Memory from Gen 0”,“Promoted Memory from Gen 1”
.NET Data Provider for SqlServer “NumberOfActiveConnections”, “NumberOfFreeConnections”, “NumberOfPooledConnections”, “NumberOfReclaimedConnections”, “SoftConnectsPerSecond”, “SoftDisconnectsPerSecond”
.NET Memory Cache 4.0 “Cache Entries”, “Cache Hit Ratio”, “Cache Hits”, “Cache Misses”, “Cache Trims”, “Cache Turnover Rate”
MSMQ Service “Total bytes in all queues”, “Total messages in all queues”

IIS Web Server Performance Counters

PerfMon Object Perfmon Counter
Web Service “Service Uptime”, “Current Connections”, “Bytes Sent/sec”, “Total Bytes Sent”, “Bytes Received/sec”, “Total Bytes Received”, “Bytes Total/sec”, “Total Bytes Transferred”, “Get Requests/sec”,“Total Get Requests”, “Post Requests/sec”,“Total Post Requests”, “Put Requests/sec”,“Total Put Requests”, “Delete Requests/sec”,“Total Delete Requests”, “Anonymous Users/sec”, “NonAnonymous Users/sec”, “Not Found Errors/sec”, “Locked Errors/sec”, “Total Method Requests”, “Total Method Requests/sec”
Web Service Cache “Current Files Cached”, “Active Flushed Entries”, “Total Files Cached”, “Total Flushed Files”, “File Cache Hits”, “File Cache Misses”, “File Cache Hits %”, “File Cache Flushes”, “Current File Cache Memory Usage”, “Maximum File Cache Memory Usage”, “Current URIs Cached”, “Total URIs Cached”, “Total Flushed URIs”, “URI Cache Hits”, “URI Cache Misses”, “URI Cache Hits %”, “URI Cache Flushes”, “Current Metadata Cached”, “Total Metadata Cached”, “Total Flushed Metadata”, “Metadata Cache Hits”, “Metadata Cache Misses”, “Metadata Cache Hits %”, “Metadata Cache Flushes”, “Kernel: Current URIs Cached”, “Kernel: Total URIs Cached”, “Kernel: Total Flushed URIs”, “Kernel: URI Cache Hits”, “Kernel: Uri Cache Hits/sec”, “Kernel: URI Cache Misses”, “Kernel: URI Cache Hits %”, “Kernel: URI Cache Flushes”
ASP.NET “Application Restarts”, “Requests Queued”, “Requests Rejected”
ASP.NET Applications “Requests/Sec”, “Errors Total”, “Cache Total Entries”, “Cache Total Turnover Rate”, “Cache Total Hits”, “Cache Total Misses”, “Cache Total Hit Ratio”, “Pipeline Instance Count”, “Requests/Sec”, “Requests In Application Queue”, “Request Execution Time”, “Request Wait Time”, “Requests Executing”
HTTP Service Request Queues “ArrivalRate”, “CurrentQueueSize”,“RejectedRequests”

Logs

Linked Articles

Collecting AirWatch Services Logs (On-Premise)

AirWatch Enterprise Systems Connector (ESC) / Cloud Connector (ACC)

To verbose the ACC log, perform the following:

  • Open Windows Explorer on the ACC server, and browse to the C:\AirWatch\CloudConnector\folder
  • Note the presence of two folders: Bank1 and Bank2. Every time the Cloud Connector software is updated, the update is applied to the inactive bank folder. The updated bank folder then becomes the active bank folder.
  • Open each Bank folder and sort the file list by date modified. Compare the most recent date modified in each file. The current bank file has the most recent date modified.
  • Within the current bank folder (C:\AirWatch\CloudConnector\Bank#), open the CloudConnector.exe.config file and change the level value in the from error to verbose and save the file.
  • After reproducing the error, open Windows Explorer on the ACC server and browser to the C:\AirWatch\Logs directory. Copy the appropriate log to a new location for use in support/troubleshooting.
  • Be sure to change the loggingConfiguration level value from verbose to error and save the file to prevent unnecessary impact to the ACC server.

AirWatch API Services (API)

To verbose the API Service Log, perform the following:

  • On the server running API services, open Windows Explorer and browse to C:\AirWatch\AirWatch #.#\Websites\AirWatchApi. Note: You can determine the API server by browsing to Groups & Settings > All Settings > System > Advanced > Site URL’s.
  • Open the web.config file, and look for the loggingConfiguration key.
  • Change the value for level from error to verbose and save the web.config file.
  • Restart IIS services.
  • Reproduce your issue and then copy the log from C:\AirWatch\Logs\AirWatchAPI\webserviceapi.log.
  • Change the value for level from verbose back to error and save the web.config file.
  • Restart IIS Services.

AirWatch Cloud Messaging (AWCM)

To verbose the AWCM logs, please perform the following steps:

  1. Open the logback.xml file. The path to access the file:\AirWatch\AirWatch x.x\AWCM\config\logback.xml.
  2. Search for the following:
  3. Change the state from error to debug.
  4. Save the file and restart the AWCM services.
Note

Once the issue is reproduced, return logging level back to info and restart the AWCM services. Or the AWCM disk may overflow with logs.

Folder = AWCM Log name = Awcm.log Contains information on AWCM such as status, history, properties, and additional sub-services.

Log name = AWCMservice.log Contains log information on AWCM Java service wrapper.

ACC Logs

Use these steps below to verbose ACC logs:

  •  On the ACC server navigate to *\AirWatch\AirWatch #.#\CloudConnector\Bank#*

  •  #.# will be the AirWatch version you are using, if there are multiple choose the most recent.

  •  ACC utilizes two distinct banks: one active and one is used for installation of automatic updates. If you are unsure of which bank is active, make changes to the CloudConnector.exe.config file in each bank. If one bank is empty or does not have the file, it is not the active bank.

  •  Edit the CloudConnector.exe.config lines:

    • level = “Verbose”

    • tracingEnabled = “true”

      <loggingConfiguration filePath="..\..\Logs\CloudConnector\CloudConnector.log" tracingEnabled="true" level="Verbose" logFileRollSize="10240" maxArchivedFiles="20"/>
      
  •  Locate the log by looking at the filePath attribute from the line above. The path is included below as well \AirWatch\Logs\CloudConnector\CloudConnector.log

  • ACC doesn’t need to be restarted to pick up the logging level configuration change.

Console Services (CS)

To enable verbose logging for console and scheduler services, please perform the following steps:

  • Log in to the AirWatch console in question.
  • With the Global organization group selected, browse to Groups & Settings > All Settings > Admin > Diagnostics > Logging.
  • Change the logging level for the services in question to verbose and click Save.
    • Admin Console
    • Self-Service Portal
    • API
    • Scheduled Services (such as Inventory, Workflow, and Monitor services)
    • Reproduce your error, then open Windows Explorer and browser to C:\AirWatch\Logs_Service Folder_ and look for the latest log.
    • Change the Device Services logging level back to Error. This prevents logging from impacting system performance.

Device Services – Targeted

Depending on the version of AirWatch, it is possible to collect verbose logs for an individual device without having to verbose the logs for all devices. This is particularly helpful when troubleshooting a single device in a large production deployment. To do this, perform the following steps:

  • From within the AirWatch console device list view, click on your device to take you to the device details page.
  • Click More > Targeted Logging. If necessary, click the Continue Targeted Logging File Path to ensure the logging path is configured.
  • From the targeted logging page, click Create New Log and select the timeframe you want the logs to collect. Click Start.  At any point, you can click Stop Logging to stop log collection for the device (such as after you have reproduced the issue).
  • Once the tests are completed, go to the appropriate server and look for the TargetedLogging folder.
  • Inside the folder is a zip file with the current date and time. Unzip the file to view the files.

Device Services – General

When you wish to verbose device services logging for all devices, perform the following:

  • Log in to the AirWatch console in question.
  • With the Global organization group selected, browser to Groups & Settings > All Settings > Admin > Diagnostics > Logging.
  • Change the Device Services logging level to verbose and click Save.
  • Reproduce your error, then open Windows Explorer and browser to C:\AirWatch\Logs\DeviceServices\ and look for the latest log.
  • Change the Devices Services logging level back to Error. This prevents logging from impacting system performance.

SEG Console, Setup, and Integration Logs

To verbose the SEG logs for console/setup/integration, please perform the following steps:

  • Open Windows Explorer on the SEG server and browser to C:\AirWatch\Logs
  • Note the following Folders and change the appropriate config log level from error to verbose:
    • Services – Contains the AW.EAS.IntegrationService.log file which details communications between the AirWatch API server and SEG server. Note: This log is verbosed by changing the level value in the key of the AW.ES.IntegrationService.Exe.config file in the C:\AirWatch\AirWatch #.#\AW.Eas.IntegrationService folder.
    • SEG Setup – Contains the AW.EAS.Setup.log file which details activity related to the http://localhost/SEGSetup website. Note: This log is verbosed by changing the level value in the key of the web.config file in the C:\AirWatch\AirWatch #.#\AW.Eas.Setup folder.
    • SEG Console – Contains the AW.EAS.Web.log file which details activity related to the http://localhost/SEGConsole website. Note: This log is verbosed by changing the level value in the key of the web.config file in the C:\AirWatch\AirWatch #.#\AW.Eas.web folder.
    • Before reproducing your issue, making the necessary change to the LoggingConfiguration key for the service in question.
    • After reproducing the error, open Windows Explorer on the SEG server and browse to the appropriate subfolder in the C:\AirWatch\Logs\ directory. Copy the appropriate log to a new location for use in support/troubleshooting.
    • Be sure to change the loggingConfiguration level value (currently verbose) in the appropriate configuration file back to error to prevent unnecessary impact to the SEG server.

SEG Exchange ActiveSync (EAS) Listener Logs

To verbose the SEG EAS Listener logs, please perform the following steps:

  • On the SEG server, open an Internet browser and navigate to http://localhost/SEGSetup
  • In the “Log Level” drop down, select verbose and click Save. Note: This changes the level value in the key of the web.config file in the C:\AirWatch\AirWatch #.#\AW.Eas.Web.Listener folder.
  • Copy the AW.Eas.Web.Listener.log to a new location for use in support/troubleshooting.
  • In your Internet browser, change the “Log Level” drop down back to error and click Save.

FTP Relay Server (Rugged Management)

Logging for the relay server is saved in the following location:

  • Browse to the C:\AirWatch\Logs\Service folder.
  • The logging for the Relay Server is saved in the ContentDeliveryService.log file.

Collecting logs from the Admin Console

If you do not have immediate access to the on-premise servers to access the logs, you can retrieve SEG/ACC logs directly from the console from the following page:

  • Navigate to System > Admin > Diagnostics > System Health. Click on the installed service you wish to pull the logs from.
  • In the pop-up box now displayed, click on the “Acquire Logs,” for the required service, from the four-button menu on the right.
  • Now the “Download” button is activated and you can click on it to download and view logs remotely.

Note: The System Health dashboard will be populated only if you have any of the services (ACC/SEG) already installed and running.

EMail Notification Service (ENS)

ENSv2 is a Windows Service ‘AWSubscription’.  Like other AirWatch services, relevant logs can be found in the path {Installation Path}/Logs, and the logging level can be configured by editing parameters to traceEnabled=“true” Level=“Verbose” in the app config ({Installation Path}\Config\WebSites\Web.config) file located in the installation folder

ENSv2 Errors are in ENS.log and ReSubscriptionMechanism.log

Note

The service must be restarted for logging changes to take effect.

When set to verbose, you will be able to identify log messages pertaining to both new subscriptions being created, as well as any device compliance state-changes being identified.  For example, if a device becomes compromised and is then marked as non-compliant.  In the logs, a message indicating that a device’s access state is True indicates that the device is allowed, whereas False means the device is blocked.

Subsections of Logs

Boxer for Android Logs

Boxer for Android Architecture

Boxer communicates with a few different systems to provide these services.
The flow of the Android Boxer log file looks like this:

  • Boxer reaches out to the console to get the profile information;
  • Boxer reaches out to the console to get any S/MIME certificates and client authentication certificates available from the console;
  • Boxer reaches out to the Exchange ActiveSync(EAS) email endpoint to make the options, provision, foldersync, and ping request;
  • Boxer reaches out to the Email Notification S ervice (ENS) and provides the Exchange Web Services (EWS) credential information that was used for ActiveSync so it can subscrib e to email notifications.
graph TD CS[Console Server] --> Boxer; Boxer --> CS; Exchange --> Boxer; Boxer --> Exchange; Boxer --> ENS; ENS --> Boxer;
  • Boxer logs are in GMT. That means the entries there are 5 hours ahead of EST time. In addition, the logs are in 24 hour time so you will need to subtract 12 if you want am/pm.
  • The very first line in the Boxer logs has the last date of the logs in the file. For example 2019-01-08T21:28:04.539Z - [-] - ###--HEADER--###
  • Boxer logs can persist in the app for up to 3 days, so they can be pulled for review even if the issue hasnt occurred recently, but at least within the 3 day period. This is useful for intermittent types of issues, or where there is a delay in an affected user reporting the issue to their techsupport.

Boxer Communication Logs

Here is an example of what you might see in the logs followed by notes (in bold) that detail what the log entry tracks:

Boxer log sample...

Common Ways to Search Through the Logs

On the console where you configure your Boxer profile, you will have places where you can enter profile information like Account Name, Domain, User, etc… However, in the Boxer logs, when you are looking for the corresponding data there, the wording might be slightly different. You can use the mapping information below to see what this would say in the Boxer logs. The value on the left is what you will see in the console. The value after “=” is what you will see in the Boxer logs.

Account Name = DESCRIPTION  
Exchange ActiveSync Host = SERVER_ADDRESS  
Domain = USER_DOMAIN  
User = USERNAME  
Email Address = EMAIL  
Email Signature = SIGNATURE  
Authentication Type = ACCOUNT_AUTHENTICATION_TYPE | 0 basic, 1 certificate, 2 both Copy Paste = ALLOW_COPY_PASTE  
Screenshots = ALLOW_SCREEN_CAPTURE  
Allow email widget = ALLOW_EMAIL_WIDGET  
Allow calendar widget = ALLOW_CALENDAR_WIDGET  
Hyperlinks = POLICY_ALLOW_OPEN_IN  
Sharing = DOCUMENT_SHARING_RESTRICTION  
Caller ID = ALLOW_CALLER_ID  
Personal Accounts = ALLOW_OTHER_ACCOUNTS  
Personal Contacts = ALLOW_LOCAL_CONTACTS

Boxer reaches out to the console to get the profile information

2019-02-07T15:21:55.345Z I [18459-BoxerWorker-5] - App initialization step complete: 1

2019-02-07T15:21:55.578Z E [18459-main] - FLF.setSelectedAccount(null) called! Destroying existing loader.

2019-02-07T15:21:55.589Z I [18459-BoxerWorker-3] - Waiting for app restrictions

This entry is the best example we have of when Boxer reaches out to the console. You can use Fiddler to further determine when Boxer is reaching out to the console. In the Boxer logs, you can’t directly see this. The other place you can see this is in the ADB device logs. The line here that says “waiting for app restrictions” just indicates that Boxer is waiting to read from the database. Boxer reaches out to the console to get any S/MIME certificates and client authentication certificates available from the console.

2019-02-07T15:21:56.256Z I [18459-IntentService[AirWatchAccountSetupService]] - Fetching S/MIME signing certificate

2019-02-07T15:21:58.021Z I [18459-AsyncTask #2] - Certificate being fetched for : AccountAuthenticationCertificateId
2019-02-07T15:21:59.013Z I [18459-AsyncTask #2] - Certificate fetch successful 
2019-02-07T15:21:59.755Z I [18459-Thread-7] - TrackingKeyManager: requesting a client cert alias for 66.170.96.7 
2019-02-07T15:22:05.776Z I [18459-Thread-9] - Registering socket factory for certificate alias [AW-be6fad91fe2b4291b1d392c9eabf90be] 
2019-02-07T15:22:05.846Z D [18459-Thread-9] - Found cert chain: [ [0] Version: 3
SerialNumber: 468315651040541492877707779630191635557515624
IssuerDN: DC=local,DC=milkyway,CN=milkyway-SUN-CA Start Date: Thu Feb 07 10:11:47 EST 2019  
Final Date: Sat Mar 09 10:11:47 EST 2019  
SubjectDN: CN=lwilcox
Public Key: RSA Public Key
modulus: ccbf28975eafd49dedfcfc43f9d66a309d36c4c636fbece2bf9faa87fe6544c8e025c3da260eb4bc28096f665a231202c5cf53dbb5f1d08ceac4d9af90f0605ee2951e6239576749d39d17e8d411b39c03e6a68e155083866a69de610853ab83d149c0228b6f0b0d67d4e9e1093ce8a3316a563c48d91fcc000bae01c150644113a527d611d83f7303825a58a4382c1818d8fd56b2064ef495c709e4ae4366e7793e29cd3b376e9660028269c8233ba8cd9a42e57dc2f07087d25fc6ce536d3bcbc8fabe2f6c408ddb2fefd8b80fdc029ba16237b48d26072d0f88f9e982ab37720883ff7bfb35bc409637fb314c00cbc1cb262e7406732c45c400ca45d9357b
public exponent: 10001

Boxer reaches out to the Exchange ActiveSync (EAS) email endpoint to make the options, provision, foldersync, and ping request.

Making the Options Request

Boxer Options Request...

Perform a FolderSync

Boxer FolderSync...

Perform a provision request

Boxer provision request...

Perform another FolderSync, this time it works. Start email sync.

Boxer another FolderSync...

Ping example

2019-02-07T15:40:12.911Z I [18459-PingTask-lwilcox@labmail.airwlab.com] - Ping task starting for 1  
2019-02-07T15:40:12.912Z I [18459-PingTask-lwilcox@labmail.airwlab.com] - Exchange ping starting  
2019-02-07T15:43:00.125Z I [18459-PingTask-lwilcox@labmail.airwlab.com] - Changes found in: 8  
2019-02-07T15:43:00.126Z I [18459-PingTask-lwilcox@labmail.airwlab.com] - Ping found changed folders for account 1  
2019-02-07T15:43:00.137Z I [18459-PingTask-lwilcox@labmail.airwlab.com] - requestSync EasOperation requestSyncForMailboxes Account {name=lwilcox@labmail.airwlab.com, type=com.boxer.exchange}, Bundle[{__mailboxCount__=1, force=true, expedited=true, __mailboxId0__=5, PING_ERROR_COUNT=3}] 2019-02-07T15:43:00.139Z I [18459-PingTask-lwilcox@labmail.airwlab.com] - Exchange ping finished with result 2

2019-02-07T15:43:00.200Z I [18459-SyncAdapterThread-8 ] - Email sync for account lwilcox@labmail.airwlab.com, with extras Bundle[{ignore_settings=true, __mailboxCount__=1, force=true, expedited=true, ignore_backoff=true, __mailboxId0__=5, PING_ERROR_COUNT=3}] 2019-02-07T15:43:00.316Z I [18459-SyncAdapterThread-8 ] - Starting sync command

Boxer reaches out to the Email Notification Service (ENS) and provides the Exchange Web Services (EWS) credential information that was used for ActiveSync so it can subscribe to email notifications.

2019-02-07T15:23:08.315Z I [18459-AsyncTask #4 ] - Ens registration for account (id=1) is successful! 
2019-02-07T15:43:30.041Z I [18459-main ] - Sync triggered from distance

You can use Notepad++ to search the logs using the following search terms.

“error” - This is a very general way to look for errors. You will have a lot of false/positives when using this search criteria.

“exception “ - You can use this to search the file for exceptions that are generated when the application runs into an error.

“Current Network” - This will show you what network connection the mobile device was connected to 4g, wi-fi, etc…

“distance” - This shows you each time the ENS server has reached out to the mobile device to wake it up, send a notification, and trigger a sync • “transitioning” - Use this search term to tell when the application is transitioning from background to foreground and vice versa.

“requestPing” - This will show you when the device reaches out to the OS to use the Android SyncAdaptor which it uses to request the ping. The response to this request will be “Email sync for account”.

“Email sync for account” - This will be a response from the OS for a command “requestPing” or “requestSync “. Take note of how long between the request and the reply from the OS. If it’s a long time, it’s likely being throttled by the OS due to battery optimization or a third party product.

“Changes found in: “ - is the response to a ping request. This tells us if the ping found any changes in the inbox. If it did, we will now do a “requestSync " to bring those emails or changes into Boxer.

“requestSync” - This will show you when the device reaches out to the OS to use the Android SyncAdaptor which it uses to request email sync. The response to this request will be “Email sync for account”.

“Q-AppInitialization “ - This shows you when the application is starting up. You will see this when you force close and reopen the application or when you start the application after a crash.

“PingTask “ - shows you all activities related to pings

“SyncAdapterThread “ - shows you all activities related to sync operations.

“Email sync for account “ - The response from the OS after a “requestSync” or “RequestPing” operation.

“ens” - Search the log files for ens errors and settings.

Collecting Boxer Logs When You Can’t Get Into Android Boxer

Depending on what model phone you have, you can follow one of the processes below to collect Boxer logs in the event that you can’t get into Boxer due to an error. You may have a different model device than the ones listed, but the process will be very similar.

Samsung S9+

• Go to “Settings” on the phone followed by “Apps” • Select “Boxer”
• Select “Mobile Data”
• Select “View App settings” • Select “Send logs” • Use one of the other mechanisms to send the logs (either by copying the logs out or another email client).

On Motorola X Pure

• Go to “Settings” on the phone followed by “Apps” • Select “Boxer” • Select “Data Usage”
• Select “App settings”
• Select “Send logs” • Use one of the other mechanisms to send the logs (either by copying the logs out or another email client).

Device-side logs collection

Collecting Device-Side Logs

macOS Devices

With version 2.x of the macOS agent, you can now collect logs via the agent by performing the following steps:

  • Open the agent by right-clicking the agent icon from the menu bar, and then clicking Preferences
  • From the Status screen, click on Diagnostics
  • Click the button to Send Logs to Administrator
    • The agent will gather logs and zip them into an email for you automatically

To manually gather logs on older versions of the AirWatch agent, please perform the following:

  • Enable Debug Logging in the Mac OS X Terminal:
    • $ sudo defaults write /Library/Preferences/com.apple.MCXDebug debugOutput -2
    • $ sudo defaults write /Library/Preferences/com.apple.MCXDebug collateLogs 1
    • $ sudo touch /var/db/MDM_EnableDebug
    • Open the Console application from the Launchpad. Select All Messages and then click Clear Display to clear out old logs.
    • Reproduce the issue on Mac.
    • Click File > Save a Copy As. and save a copy of the logs to be sent to AirWatch.
    • Disable the Debug logging in the Mac OS X Terminal:
      • $ sudo rm –rf /var/db/MDM_EnableDebug
      • $ sudo defaults delete /Library/Preferences/com.apple.MCXDebug debugOutput
      • $ sudo defaults delete /Library/Preferences/com.apple.MCXDebug collateLogs

iOS Devices (iPhone, iPad, Apple TV)

To gather logs from iOS devices (iOS 7.x and below), please perform the following steps:

  • Install iPhone Configuration Utility (iPCU) on your workstation.
  • With iPCU started, connect the iOS device to your computer via the USB cable.
  • On the left-hand side of iPCU, select the iOS device where you wish to collect logs.
  • Click on the Console tab at the top right-hand corner.
  • Reproduce the issue with your iOS devices.
  • Click the Save Console As button to save the text in the console to a file.

To gather logs from iOS devices (iOS 8.0+), please perform the following steps:

  • Install xCode 6 on your OS X Device. Note: You cannot gather iOS 8 logs from a Windows-based computer. You must use a Mac OS X computer.
  • With xCode started, connect the iOS device to your computer via the USB cable.
  • From within xCode, click on the Window menu and click Devices.
  • Select your iOS device from the left hand side, then select the up arrow at the bottom corner of the right hand side.
  • Reproduce the issue with your iOS devices.
  • Save the contents of the activity log to a file.

Android

To gather logs using Console, please see the following documentation.  

To gather logs using ADB, please perform the following steps (see the following page for details):

  • Download and set up the Android SDK per the SDK documentation.
  • Open Windows Explorer and browse to the \platform-tools folder. Ensure you see the adb.exe file.
  • Open a CMD window and navigate to the platform-tools folder. Or, in Windows 7, navigate back to the SDK folder, then Shift + Right-Click on the platform-tools folder, and select Open Command Window Here.
  • Ensure USB Debugging is enabled on your Android device (from the Developer Settings menu). For more information on how to enable the Developer Settings menu, browse to http://www.androidcentral.com/how-enable-developer-settings-android-42
  • In your Notification Center, you may need to make sure the device is not connected in USB Media Device mode.
  • In the CMD window, type adb logcat –v long > androidlog.txt
  • On the Android device, recreate whatever error you are trying to log.
  • When complete, from within the CMD window use CTRL + C to end the logging.
  • Go back to the platform-tools folder and find the log file (androidlog.txt) that you just created.

Windows 7/8/8.1/10/11

To gather logs, please perform the following steps:

  • Click on Start > Run, type eventtvwr.msc and click OK. On Windows 8/8.1, from the start menu you can simply start typing Event and select the View Event Logs item returned from universal search.
  • Expand Event Viewer (Local) > Windows Logs and select the Application log.
  • You can filter logs by Event ID or Source if desired.
  • To export for support, click on either Save All Events As or Save Selected Events to export the log entries as an *.evtx file which can be sent to support.
  • You can also find logs in the following location: \AgentUI\Logs
    • AwclClient.log - AWCM-related Issues
    • AWProcessCommands.log - Issues with sending commands to the device
    • NativeEnrollment.log - Issues with Enrollment
    • TaskScheduler.log - Issues with samples sent to console

Windows Phone 8.1 (deprecated)

To gather logs, please perform the following steps:

  • Ensure you have Visual Studios 2013 Update 3 installed. If not, perform the following:
    • From your Windows Laptop of VM, browse to Windows Phone SDK Archives
    • Download the Visual Studio Express 2013 for Windows and Install it.
    • From within Visual Studios, click on Tools > Windows Phone 8.1 > Developer Unlock. Follow the prompts to unlock your Windows Phone 8.1 device.
    • From within Visual Studios, click on Tools > Windows Phone 8.1 > Developer Power tools.
    • Select Device from the Select Device dropdown, then click Connect. If prompted, click Install to install the Phone Tools Update Pack.
    • Select the Performance Recorder tab, then check the Enterprise Management option under the Extras profile category.
    • Click the Start button in the Developer Power Tools window to start a log.
    • Run your scenarios and re-create the issue you’re experiencing.
    • Click the Stop button in the Developer Power Tools window to stop logging and save the ETW to a local location.
    • You will need to download the Windows Performance Analyzer to view the logs. This can be found in the Windows Performance Toolkit included in the Windows Assessment & Deployment Toolkit (ADK) and Windows Software Development Kit (SDK).

Windows Performance Toolkit

  • Open the Windows Performance Analyzer and Open the ETL file.
  • In the Graph Explorer window, expand System Activity and view the Generic Events window.
  • Double-click the graphic bars in the Generic Events window to display an Analysis window.
  • In the Analysis window, click Open View Editor to show a Generic Events View Editor window.
  • In the Generic Events View Editor window, ensure the Message box is checked and click Apply:
    • The Message field in the analysis window provides the MDM specific log message under various providers.
    • Microsoft-WindowsPhone-Enrollment-API-Provider – ETW logs for MDM Enrollment and MDM Client Cert Renew Process.
    • Microsoft-WindowsPhone-SCEP-Provider – SCEP Cert enrollment logging
    • Microsoft-WindowsPhone-CmCspVpnPlus – VPN Configuration logging

Windows Mobile Devices with Agent 5.x

All log settings are configured in the log_config.cfg file in the \Program Files\AirWatch directory on the device. The file will resemble the following:

[*]trace_level=5

max_file_size_kb=256

files_to_keep=2

log_file_path=\Program Files\AirWatch\Logs

use_local_time=false

[aw_setup]trace_level=5

max_file_size_kb=256

files_to_keep=2

log_file_path=\

use_local_time=false

[awregisterdevice.exe]
trace_level=3

max_file_size_kb=256

files_to_keep=2

log_file_path=\Program Files\AirWatch\Logs

use_local_time=false

[awapplyprofile.exe]trace_level=5

max_file_size_kb=256

files_to_keep=2

log_file_path=\Program Files\AirWatch\Logs

use_local_time=false

[awremotecontrol.exe]trace_level=1

max_file_size_kb=256

files_to_keep=2

log_file_path=\Program Files\AirWatch\Logs

use_local_time=false

In general, the following notes apply to Windows Mobile device logging:

  • The logging level can be modified as a whole, or on an individual basis:
    • The asterisk configuration is the default config for all logs. Trace levels vary from 1 (basic) to 5 (verbose/debug).
    • Each individual section, which can be used to increase logging to override the default setting from the asterisk section.
    • The log files which are available can vary (based on configuration and OEM), but the following are the most common:
      • aw_setup - Provides logging information relating to the AWMasterSetup utility, which is responsible for initiating the agent install and uninstall process on a device. This is the only log file that is not located in the “\Program Files\AirWatch” directory and is instead located in the root of the file system.
      • awacmclient - Provides logging information relating to the AWCM client on the device
      • awapplicationmanager - Provides logging information relating to product provisioning
      • awprocesscommands - Provides logging information relating to the execution of MDM commands and installation of profiles
      • AWService - Provides information about the AWService.exe component, which is responsible for managing beacon and interrogator samples
      • awapplyprofile - Relates to installation of the agent settings xml file which occurs during the enrollment process
      • awregisterdevice - Provides information about the registering of the device that occurs during the enrollment process
      • awapplauncher - Provides information about the Application Launcher executable. This log will only be present if the App Launcher utility is assigned to and being used by a device.
      • fusionwlansetup - Provides information about configuring and setting up the Fusion WiFi driver on Motorola devices.

The general process for configuring log files is as follows:

  • Transfer the log file to your machine. This can be done through the file manager utility in device details or through remote management if a client has that configured.
  • Open the log file via a basic text editor such as notepad.
  • Edit the desired trace level to the needed value.
  • Save the log file.
  • Transfer the log file back down to the “Program Files\AirWatch” directory on the devices. This can be accomplished via file manager, remote manager, or product provisioning. To be safe, you may elect to first delete the old log_config.cfg file.
  • Restart AWService on the device once it has the updated log_config.cfg file. This can be accomplished by directly restarting the AWService through the “Restart AirWatch Agent” or the “Warm Boot” MDM commands that are available in the AirWatch Console.
  • Once the AWService has been restarted, the new logging configuration will take effect. Reproduce your issue and then repeat the steps to turn the logging back down on the device.

Collecting Service/Functionality Specific Logs

Product Provisioning

Review the AirWatch Agent Logs and look for the following items to help you troubleshoot what is occurring:

  • If the device is newly enrolled, you’ll see the following in the logs: A message from [AWProductHandler sendProductResponses] stating “Products: No products with results to be sent!”
  • A message from [AWEnhancedProductsHandler handleCommand:] stating “Got Products New Manifest”
    • Note: In the manifest will be a line entry called ProductID". You’ll want to save this for later on.
    • Depending on the number of products being installed, you may see an entry for each product that is required.
    • Messages from [AWAppDataManager readJobProduct:] looking to see if the product is downloaded to the local cache
    • Messages from [AWOSXUtils deleteFile:] where it attempts to delete any pre-existing plist file for the products.
    • Messages from [AWJob printJob] which show the sequence number assigned to the Product which will be installed.
    • From this point forward you can search the log by the sequence number assigned to the product install job:
      • Messages about the job being queued
      • Messages about the job being started.
        • The line will look like this: airwatchd[PID] : - [AWJobQueue doJob] [Line 98] THREAD: Current Job: where PID is the AirWatch Agent Process ID and the JobID is the Sequence Number assigned to the product.
        • You can get additional information about the product actions occurring by searching from that point forward for entries from the process ID!
  • Messages about any files being downloaded to the product cache
  • Messages about Job Status Change. You’ll want to search for a line ending in Job Status changed ========> :AWJobStatusFailed!" From that point, search up in the log for messages relating to the JobID and/or the ProductID (as found in the manifest). All these messages should be coming from the Process ID of the Airwatch agent that initially started the install.

Memory Dumping

Dumping memory for certain mobile app processes and analyzing the data files generated.

https://github.com/Nightbringer21/fridump https://www.frida.re/docs/installation/ https://github.com/frida/frida/releases

General plan

  • Perform key generation operations 
  • Dump the memory using fridump
  • Pull the boxer_generated_keys.txt file
  • Use the MemDumpAnalyzer.py script with -f flag,  providing the boxer_generated_keys.txt
  • Check the output and total finds

Detailed guide (for Android) to analyze AirWatch Boxer:

Tip

For information regarding iOS frida-server installation, please follow https://www.frida.re/docs/ios/.

1. Have a rooted/jailbroken device or an emulator running in your computer.

2. Install python if you haven’t already https://www.python.org/downloads/, this will allow us to run the scripts.

3. Open a new terminal and install frida, this will be the main framework that will be used by the underlying scripts

pip install frida

4. Once the frida is installed successfully, make sure to download the frida-server binary depending on the device type. The frida-server binary version and the frida version must match for proper dump.

The download links can be found in https://github.com/frida/frida/releases 

Once the correct device frida-server binary is downloaded, unarchive the compressed file. The frida-server binary file should look like “frida-server-10.7.5-android-x86_64” which will be different depending on the device architecture.

5. Put the frida-server binary in to the Android device (using ADB tool):

adb root # might be required
adb push frida-server /data/local/tmp/ # push the binary frida-server file into the device
adb shell "chmod 755 /data/local/tmp/frida-server" # make the binary file executable via permissions
adb shell "/data/local/tmp/frida-server &" #run

Check if frida-server is running properly, type the following in the terminal:

frida-ps -U

This command will show you all the process running inside the mobile device. 

6. At this point, we have frida-server as well as our application running on the device. It is time to download the fridump library that will allow us to dump the process memory in the device.

Clone the fridump Git repository:

git clone https://github.com/Nightbringer21/fridum

7. Now lets go ahead and try to dump the memory using fridump 

Provide flag -s so that at the end of the dump process, there will be a separate script which will capture all the strings in generated dump files.

python fridump.py -s com.example.name

Dump files will be located under fridump/dump/* strings.txt file is located under fridump/dump/strings.txt For generating dump files in another location, please provide flag -o with the full path.

8. At this point, you will be able to analyze all the strings that reside in memory in strings.txt. However what if you are looking for something that consists of set of random bytes? For this purpose, you will need a separate python script that will analyze the dump files. 

Go to the fridump repo and make sure you are at the root. 

Download the python script MemDumpAnalyzer.py and copy the script to fridump folder root. For this script to work properly, your dump files must be located at the original destination which is fridump/dump/*.

Attached Script

MemDumpAnalyzer Python Script

The script supports two functionality which are to search a string in dump files or a file with hex encoded strings to be searched in the file separed with new lines. The string option allows you to search normal strings while file option will let you search random byte in hex. This is particularly useful since our keys are randomly generated keys which fail during encoding. Here are some examples of usages.

-f : file path to search for -s : string to search for -h : help for usage

python MemDumpAnalyzer.py -h
python MemDumpAnalyzer.py -s boxer
python MemDumpAnalyzer.py -f boxer_generated_keys.txt

9. Generate hex encoded keys and store them in a file shown below. All the keys will have to be separated by new lines and have to hex encoded. The scripts supports file with upper and lower case or with ‘0x’ suffix in hex encoding.

58F75A01A1A5AE0F9B4C92AFF742468B751DD293517B6FEAE3A71BA126155922
35FD53F845FE5C01711002AF719F5DCBDFE8E45BE8611357B7D4AC44A28401DE
72F74A5B8D94DE8C3E2A6341DF09B0C414CCB7DB34FE367138015DB923505AE9
76E3368BE3EF445E18FD6E036F69258332E79F95B2D4A1CE9B2CBF0459E40C11
4D0309AFCB4CD47C2EC19F0B569ACEACD318FB0BDFB7C20A00570C1E2CA916A6
71A2AFC7C6B87211F0C2DD36245041B251AF92D66AEC2FD78E4FE2C2C8A48C7A
E7CBE67D09D7C4A9BB06FE5CF9E2B3ACC09BC30672DCD21B7A3156B20DEEB501
61C4425D8305C6859EB88565FC7F4CC220A3F1C8CEFE4F020C9B7B2EAC7CBDB3
80B5E111C27136FDDEDB6A3868F2E0FE5B15D4E55C03A72C07A6647CDBAE0054
7489346AE746E258775364CADC80B7E26751708B3688A210AAC36F99B4072E13
64D4BC15DBD0E0014DE7A6C568CB47160F037A95D9EE3D92FE6B8AC70AA99CD3
55291C59B69BDCF136DFB1995F174D7B3D2813189E575EB13DC41609E29F262C
DCD9C7FB65D31094CF3F75271722D8A23CA417AD6082921E5A78126A0ED10973
424C3550A53335FE1BBFE844E00EC51622D1D670BD22C6380664C94B6A50740D
EDFCDA5996A84300C14D627FE32392219E9B67D4F6D8515B757464C0FFE8C5CE
A0B20AABF91ACDF27C90DE8265042BB4282C245668F1C855C51C9638B38FFD12

10. To pull the file from the device to the current folder, use

adb pull data/data/com.boxer.email/files/boxer_generated_keys.tx

Screen Recording

Screen Recording

Windows 7/8/8.1/10/11

To document click-through steps on Windows machines, perform the following:

  • Click on Start > Run and type psr.exe to bring up the Problem Steps Recorder (or PSR, a built-in Windows utility).
  • Click on Start Record to begin capturing steps. Note: PSR captures screenshots of ALL monitors; no scoping.
  • Each Mouse-Click you make captures a screenshot. At any time during the session, click on Add Comment to provide more details about the screen, error, etc.
  • When finished, click Stop Record.
  • Choose where to Save the PSR file – it outputs a zip file containing a pre-compiled HTML (*.mhtml) file with all your screenshots and comments.

macOS

To document click-through steps on macOS machines, perform the following:

  • Launch QuickTime Player. You’ll find it in the Other folder within Launchpad.
  • From the QuickTime menu bar, click File > New Screen Recording. Click the red record button.
    • Optionally you may wish to select View > Float on Top before you start recording.
    • Optionally, you can select the upside-down triangle in the record screen to include audio recording during the screen capture for annotation.
    • Click the screen (or Click-Drag to select part of the screen) for recording.
    • When complete, click the Stop button that appears in the menu bar of the screen where you’re recording.
    • Click File > Save (or simply quit QuickTime) to be prompted with a location to save the screen capture. Note: Keep it moving when you record these; they create full-blown movies and the file gets large quickly.

REST API

### URI

Old API URI example: https://host/api/v1/mam/apps/public/{applicationid}/addsmartgroup/{smartgroupid}

New API URI example: https://host/api/mam/apps/public/{applicationid}/smartgroup/{smartgroupid}

Attached Documents

REST API SDK Guide Deprecated REST API update Main REST API Guide VMware AirWatch Peripheral API Guide VMware AirWatch API Programming Quick Start Guide

Articles in section

Preparation

You need a user account to run scripts in AirWatch REST API. Create a local account, take its’ credentials: domain\login:password, and go encode it at https://www.base64encode.org/

For example, 'lab\restguy:P@ssw0rd' ==> 'bGFiXHJlc3RndXk6UEBzc3cwcmQ='._

Turn API access ON for this account in the AirWatch console System -> Advanced -> API -> REST, we also should get an API Key for it.

Warning
  • AirWatch requires all requests to be made over SSL.

  • AirWatch limits the number of API requests that can be made during certain time periods. Rate limiting is done per Organization Group (OG) based on the API key used in the request.

    **Server Throttling** - The limit imposed on an Organization Group for a 1 minute interval.  
    **Daily Quota** - The limit imposed on an Organization Groupvfor a 24-hour interval.
    

Local REST API Help

REST API Help is built into the AirWatch API Server host, go to link: https:///api/help/

Examples of using REST API

Let’s take some device from those enrolled in AirWatch and check out the apps list on it.

import requests
consoleURL = 'https://airwatch.example.com'
b64EncodedAuth = 'bGFiXHJlc3RndXk6UEBzc3cwcmQ='
API_Key = 'S390lvIVeFRBSO4InP73MFG3YSFDNKL+BXKW7Wv5Wwo='
basic_headers={"Authorization": "Basic " + b64EncodedAuth, "aw-tenant-code": APItenantCode,"Accept": "application/json"}
 
appsTest = requests.get(consoleURL + "/api/mam/apps/internal/46", headers=basic_headers)
print(appsTest.json())

46 is the device ID inside AirWatch. How do you find the ID of a device? - simplest way to do this is open AirWatch console, device list page, and hover your mouse over any device name: you will see the ID in the browser URL status bar. Now, let’s see where the device has been for the last 2 days:

gps2days = requests.get(consoleURL + "/api/mdm/devices/46/gps?dayrange=2", headers=basic_headers)
print(gps2days.json())

If there are any GPS points recorded for the device, we will get a list of them here. How can we use this? For example, customers say they hate Bing Maps embedded in AirWatch console. So we can build a little portal for searching devices, embed Google Maps into it and use the coordinates list to draw points.

One step deeper: suppose we need to show a customer how GPS coordinates are being collected, but the device enrolled is fresh, did not catch any sattelites yet or has problems with GPS module. Since GPS data is actually stored and taken from the SQL database, we can insert some coordinates there manually and see them right away on the Bing map in the console. See article on inserting false GPS history for a device in SQL section

Last example in this article I got from a cool client. They give out corporate iPhones to their employees, and all of those devices are supposed to be supervized. Only supervized devices are to be enrolled in AirWatch and get corp data, no personal gadgets coming through! AirWatch does have a tag for Apple device status (is it Supervized/DEP/Education?), but uses it only for reporting - there is no Compliance rule or filtering around this currently (a ticket in Jira on the topic is promised to be closed in AirWatch 9.6+). So the client gets a list of all enrolled devices and filters them manually:

import json
devicelist = requests.get(consoleURL + '/api/mdm/devices/search', headers=basic_headers)
for i in devicelist.json()["Devices"]:
    if i["Platform"] == "Apple" and i["IsSupervised"] == False and i["EnrollmentStatus"] == "Enrolled":
        # Tag the toxic devices, or enterprise wipe them

By running this script every half-hour, all unsupervised devices are Enterprise Wiped shortly after they are enrolled.

Subsections of REST API

REST API Example on Python

Several typical functions of working with REST API:

  • Searching for devices in a tenant
  • Filtering devices by OS (Win/Linux)
  • Getting a list of tags for the tenant
  • Scanning a file with list of Notebook/Smartphone/Tablet serial keys, then checking if enrolled devices have such serials, and writing the result in a file
Tip

Default tags have ID 1-8. User-created tags have ID=10000+

  • Assigning devices to a chosen tag

Code is for Python version 3.7+, and is bundled into a dataclass as methods, with error-catching and logging support.

Link to code on GitHub.

'''Version 0.3
Created by Alexei Rybalko aka Aagern.
22.05.2021'''
 
import requests, json, base64, logging, sys
from dataclasses import dataclass
from pprint import pprint
 
logging.basicConfig(filename="DeviceControl.log", level=logging.DEBUG,
                    format="%(asctime)s:%(levelname)s:%(message)s")
 
@dataclass
class DeviceControl:
    API_URL: str = 'https://mdm.example.local'
    API_Key: str = 'oxGI6OORljw/Qsql1OFBycjHvzQzEXVXa/tyEcCfIPI='
    API_User: str = 'defaultDomain\\api_user:VMware1!'
    Serials_File: str = 'S.csv'
    Output_File: str = 'DeviceSerials.csv'   
     
    API_User_B64 = base64.b64encode(API_User.encode('utf-8'))  # convert to Base64
    API_User_UTF8 = API_User_B64.decode('utf-8')
    Platforms = {'Windows':12,'Linux':21}
    Header = {"Authorization": "Basic " + API_User_UTF8, "aw-tenant-code": API_Key, "Accept": "application/json"}
    Devices = {}
    Serials = []
     
    def getMDM(self,request):
        """Get data from MDM Server. Input: API request text. Output: response data in JSON."""
        try:
            dataMDM = requests.get(self.API_URL + request, headers=self.Header)
            dataMDM.raise_for_status()
            data = dict(dataMDM.json())
        except requests.exceptions.RequestException as e:
            logging.error(f'Get request failed with {e}')
            sys.exit(1)
        logging.debug(f'Data received from {self.API_URL}')
        return data
         
    def postMDM(self,request,data):
        """Post data to MDM Server. Input: API request text. Output: HTTP response status and details."""
        try:
            responseMDM = requests.post(self.API_URL + request, headers=self.Header, json=data) # Important to do json= here, rather than prepare data with json.dumps()
        except requests.exceptions.RequestException as e:
            logging.error(f'Post request failed with {e}')
            sys.exit(1)
        logging.debug(f'Data sent to {self.API_URL}')       
        return responseMDM
     
    def scanSerialsFile(self,file=Serials_File):
        """Input file with device serials. Output list of serials from file"""
        try:
            with open(file,encoding='utf-8') as f:
                for serial_line in f:
                    self.Serials.append(serial_line)
                    Serials[0] = Serials[0][1:] # Omitting special symbol at file start
                logging.debug(f'Serials file {file} processed.')
        except OSError as e:
            logging.error(f'File opening error with {e}')
            print('File not opened. Serials list empty')
            self.Serials = []
        return self.Serials
             
    def filterDevices(self,platform='Windows'):
        """
        Method filters out Devices by Platform.
        Inputs: Platform name.
        Outputs: File with serials, Dictionary of Device:[Serial,ID]
        """
        PlatformID = self.Platforms[platform]
        dataDict = self.getMDM('/api/mdm/devices/search')
        for device in dataDict['Devices']:
            if device['PlatformId']['Id']['Value'] == PlatformID:
                self.Devices[device['DeviceFriendlyName']] = [device['SerialNumber'] if device['SerialNumber'] else 'Not set',device['Id']['Value']]
        try:
            with open(self.Output_File,mode="wt",encoding='utf-8') as output_file:
                for device,serial in self.Devices.items():
                    print(f"{device},{serial},",file=output_file)
                    if serial in Serials: print("Registered\n",file=output_file)
                    else: print("NOT Registered\n",file=output_file)
                logging.debug(f'Output file {self.Output_File} written.')
        except OSError as e:
            logging.error(f'File opening error with {e}')
            print('File not written.')
            sys.exit(1)
        return self.Devices
     
    def getMDMTags(self,tenant='570'):
        """Get all tags in tenant. Input: tenant ID. Output: list of tags and their IDs."""
        tagsDict = self.getMDM(f'/api/system/groups/{tenant}/tags')
        for tag in tagsDict['Tags']:
            print(f"Name={tag['TagName']}\t\tID={tag['Id']}")
             
    def setTagDevices(self,tagID='10000',DeviceIDs=['3']):
        """Method assigns devices to tag. Input: tag id, devices ID list. Output: total  of assigned devices"""
        FuncDeviceIDs = { "BulkValues":{"Value":DeviceIDs}}
        response = self.postMDM(f'/api/mdm/tags/{tagID}/adddevices',FuncDeviceIDs)
        print(f'Status code: {response.status_code}, \n {response.text}')
 
# Examples of usage
 
Device = DeviceControl()
data = Device.filterDevices()
Device.getMDMTags()
Device.setTagDevices()

VPN

Articles in section

Troubleshooting Per-App-VPN

Check device profile has VPN section and Per-App-VPN enabled: View profile in XML, record VPN profile unique identifier (VPNUUID):

Verify the Application has “Use Per-App VPN” enabled: Verify on device: check-in command is delivered to device, device receives App Install command:

  • Verify on device: Device receives Managed Application Attributes command
  • Managed Application Attributes command links app with VPN profile

Verify on device: (on iOS) Settings → General → Profiles & Device Management → MDM Profile → Apps
Application displays VPN information under Device Management settings: ”App will use a VPN for all network access”

Device Services (DS) server (front-end communication with managed device) logs:

  • DeviceServicesLogs (C:\AirWatch → Logs → DeviceServices)

Search for:

  • ApplicationAttributes
  • App Bundle ID
  • VPNUUID

Troubleshooting Windows 10 Tunnel Client

Client is unable to connect to Tunnel Server

  • Check Network connection
  • Check DNS
  • Check Configuration of Tunnel Server
  • Check Firewall configuration - rules denying outbound session
  • Check Tunnel Server port with telnet/curl

No applications configured 

  • Add the application to the UEM Console 
    • Configure the Tunnel.
    • Add application details (Example: add chrome and Firefox ) and DTR rules from that added application to Block, Tunnel ,By Pass or Proxy and provide destinations ( like *company-site.com)
    • Set default rule action to the Tunnel.
    • Create a user VPN profile and publish it to the device.
  • Check for application to be whitelisted in DTR, if not then add it with proper spelling/format
  • Check the logs for registration status of application

No Traffic Rules configured

  • Check for addition of application in Device Traffic Rule configuration in Windows Registry
    • Open \HKLM\SOFTWARE\VMware, Inc.\VMware Tunnel
    • Open the DeviceTrafficRules file
    • Check for the application to be whitelisted

Mutli-Auth. failure or compliance failures

  • Device must be whitelisted in Tunnel Configuration in Registry
  • Check for Device to be compliant
  • Check for validation of certificates
    Go to Manage Compute Certificate → Trusted Root Certificate Authorities → Certificate → Check for Tunnel Server Authorized certificate

Whitelisted App’s traffic is not getting tunneled

The app’s executable may not be the one which is creating the connection.
Turn off the tunnel service, open the app in question and browse to an end-point. Run command netstat -aonb  to check what executable is connecting to the end point. If this executable is different than the whitelisted exe then use this exe instead.

Warning

DO NOT whitelist svchost.exe. This is the common service used for many functions in windows. This may lead to BSOD.

Unable to open internal website (in configured domain) from whitelisted application

NRPT may be corrupted. Stop the vmware tunnel service. This should ideally clear all NRPT entries. Now open NRPT- Edit Group Policy → Windows Settings → Name Resolution Policy. Check if there any entries left, if there are, then delete them.

Browsing experience via whitelisted app seems to be staggered. And unable to access configured domain websites

Turn on Debug logs for tunnel client. It is possible there is an issue with tunnel connectivity. Either tunnel client cannot reach tunnel server (there is an SSL error while trying to connect to server) or there is Multi-Auth Failure/ Whitelist failure for the device in server.

Subsections of VPN

IKEv2 on iOS

Example of device profile config

Warning

IKE2 cipher must be configured as well as CHILD cipher. Or else XML of config will be incorrect! Error example: ChildSecurityAssociationParameters does not specify keys ‘EncryptionAlgorithm’ and ‘IntegrityAlgorithm’ in the dictionary on the console XML.

Configure IKE as follows:

Configure Child as follows:

Warning

If configuring Always-On mode, the same thing applies to WIFI/Cellular radio buttons and IKE2/Child in this mode: both must be configured for correct XML to be formed!

Win10 Management

Articles in section

Clean PC

External links:

CleanPC is the ability to remotely execute a PC Refresh (via MDM) which users can do manually on their device by going to Settings > Update & Security > Recovery > Reset this PC > Get Started, then you are presented with Keep my Files or Remove Everything.

This best explains the differences between Retaining User Data and without Retaining User Data. Calling these CSPs will un-enroll your device. If you are using the AirWatch Agent this will also be removed when calling retaining user data option. When the AirWatch Agent is removed this will un-enroll your device.

Subsections of Win10 Management

AppLocker

External link: https://docs.microsoft.com/en-us/windows/configuration/lock-down-windows-10-to-specific-apps

AppLocker contains capabilities and extensions that allow you to create rules to allow or deny applications from running based on unique identities of files and to specify which users or groups can run those applications.

Using AppLocker, you can:

  • Control the following types of applications: executable files (.exe and .com), scripts (.js, .ps1, .vbs, .cmd, and .bat), Windows Installer files (.msi and .msp), DLL files (.dll and .ocx), and packaged apps (.appx).
  • Define rules based on file attributes derived from the digital signature, including the publisher, product name, file name, and file version. For example, you can create rules based on the publisher attribute that is persistent through updates, or you can create rules for a specific version of a file.
  • Assign a rule to a security group or an individual user.
  • Create exceptions to rules. For example, you can create a rule that allows all Windows processes to run except Registry Editor (Regedit.exe).
  • Use audit-only mode to deploy the policy and understand its impact before enforcing it.
  • Import and export rules. The import and export affects the entire policy. For example, if you export a policy, all of the rules from all of the rule collections are exported, including the enforcement settings for the rule collections. If you import a policy, all criteria in the existing policy are overwritten.
Note

AppLocker is only supported on Windows 10 Enterprise and Education SKUs when using GPOs, however, when configuring via MDM (AirWatch) all versions are supported

Create AppLocker Rule (Windows side)

Use a test Windows 10 device.

Creating the AppLocker Configuration File

  • Click on the Windows logo
  • Enter “group policy”
  • Click Edit group policy

AppLocker GPO

  • Go to Windows SettingsSecurity SettingsApplication Control Policies → AppLocker
  • Click Configure rule enforcement

Enforce Packaged App Rules

In this example we will block the Xbox application (.appx). If you wanted to block RegEdit then you would configure the Executable rules.

  • Check Configured under Package app Rules; Enforce rules option is default,
  • You should ALWAYS test the rules before applying them, to do so, change the “Enforce rules” option to “Audit mode”. Once you’ve confirmed that the policy you’ve created at the end of the lab is working as expected then you return and change this back to “Enforce rules”
  • Click Apply & OK

Create Default Rules

  • Click Packaged app Rules, to start configuring the rules.
  • Right click in the white space to the right of the window
  • Select Create Default Rules

Edit Default Rule

For blacklisting only a few apps, start with an Allow rule and add your blacklisting exceptions. If you want to only allow a few apps then convert the default to a Block and whitelist your exceptions.

  1. Right click on the default rule
  2. Click Properties

Exceptions

  1. Click Exceptions
  2. Click Add…

Packaged App Reference

  1. Select Use an installed packaged as a reference
  2. Click Add…

Select Packaged Application

  1. Using the scroll bar, scroll to the bottom
  2. Check the Xbox app with Package Name of Microsoft.XboxApp & click OK

Package Name

All of the package’s information is pre-populated. You can block the Xbox app based on the specific version, package name, or by the publisher. We want to block any version of the Xbox application.

Raise the lever from Package version to Package name & click OK

Confirm Exceptions

  • Click Apply & OK
  • Now is the time to test if the policy is functioning correctly, if you can no longer access the xbox application on the device then you know that the policy working as expected.
  • As long as you do not see any other issues with the current configuration, go back to the app locker settings referred to in section 1.2 and change the “Audit mode” option to “Enforce rules”

Export AppLocker Policy

  • Right click AppLocker
  • Click Export Policy…
  • Save Policy as XML

Clear Policy

Now that we have exported our policy, we want to remove it from our test device.

  • Right click AppLocker

Click Clear Policy & Yes & OK

Create AppLocker Profile (AirWatch side)

Creating the Application Control Profile

  • In the AirWatch console go to click Device → Profiles → Add Profile → Windows → Windows Desktop → Device
  • Enter a profile name and select a Smart Group for the Assigned Groups
  • Select Application Control at the bottom of the policy list
  • Check the Import Sample Device Configuration box & click Upload
  • Upload the XML file create in the previous steps
  • Save & Publish the profile

Verify Profile

You should now see your Block Xbox Application Control (AppLocker) profile.

Bat Scripts

Device -> Staging & Provisioning -> Components -> Files/Actions -> Add Files/Actions button -> Windows Desktop button.

  • Files tab - script file itself
  • Manifest - permissions to run script with, folder where script is downloaded and run from, and type of action. To circumvent a bug in AirWatch 9.1.1 a batch file can be run by using the “Install” command instead of the “Run” command…

Note though that the ECHO command is always suppressed on the endpoint and even the pause command is escaped.

The script is run using System, Admin or User permissions - defined in the script Manifest.

The script is inserted in the Product (Device -> Staging & Provisioning -> Product List View → Add Product) with deployment options like Compliance triggers or Schedule for installation.

Desktop Auto-Enroll

Tip

The MVC4 Package is required. A lightweight version of the MVC4 installer is included, however the full version (which can be baked into the image) can be downloaded from: https://www.microsoft.com/en-us/download/details.aspx?id=30683

In order for machines to register you need to ensure they have a proper serial number. While this is never an issue on physical machines virtual machines often need updates to get working.

For FUSION Machines:

  1. Before you start the VM navigate to the root folder for the VM. You’ll see a config file with a .vmx extension.
  2. If you insert the following two lines in the .vmx file, it will boot with a shorter 12 Char serial number. Without this you cannot use WS1 or any feature that relies on serial number.
  3. SMBIOS.useShortSerialNumber = “TRUE”
  4. SMBIOS.use12CharSerialNumber = “TRUE”

For VSphere Machines:

  1. In the vSphere Web Client, navigate to the vCenter Server instance.
  2. Select the Manage tab.
  3. Select Advanced Settings.
  4. Click Edit.
  5. Add the following two lines:
  6. SMBIOS.useShortSerialNumber = “TRUE”
  7. SMBIOS.use12CharSerialNumber = “TRUE”

Step 1: Copy the contents of the AirWatch folder (optionally just copy the AirWatch folder) to a location of your choice. My preferred location for the files/folders is C:\Installs\AirWatch.

Step 2: Create a staging user in the AirWatch console at the top Organization Group. Set the staging mode to: Single User, Advanced: Enroll on behalf of user. Record the username and password of this user.

Step 3: (Optional): Download the latest agent (you can use the download_latest_agent1.ps1 in \setupfiles) then copy that agent to the same folder as the localdevice.exe OR Registration.cs file. Rename the file to AirWatchAgent.msi (you may need to replace an existing file).

Step 4: Create an AirWatch Administrator account API Service Account in AirWatch with Console Administrator role. Using a Base 64 encoder get the encoded string using the format:

`username:password`

Copy the encoded string to be used later in the INI file.

Step 5: Ensure that you have a Rest API key generated in the AirWatch Console. Settings -> General -> Advanced -> API -> REST API

Step 6: Modify the localdevice.ini file to reflect the correct settings. ; represent comments in ini files.

#************************************#
# 		 INI SAMPLE FILE			 #
#************************************#
[Config]
Authorization=Basic %BASE_64_ENCODED_API_CREDENTIALS%
API_Key=%API_KEY%
API_Server=https://%API_SERVER_URL%/api
Enrollment_Server=%ENROLLMENT_SERVER_URL%
;LocationGroupID is Optional - can search by group id
LocationGroupID=%LOCATIONGROUP_ID% 
GroupID=%GROUP_ID%
AdminEmailAddress=%ADMIN_EMAIL_ADDRESS%
StagingUser=%STAGING_USERNAME%
StagingPassword=%STAGING_PASSWORD%

[SMTP]
UseSMTP=0
SMTPServer=%SMTPServer%
Sender=%SMTPSender%

[Staging]
AllowedStagingUsers=%UserAccount% 
;Deliniate multiple accounts using commas.  Use a period to represent local machines
;Azure Users 

[Debug]
EnableDebug=0
DebugUser=%DebugUserName%
;This section is for testing only.  Delete entire section when deploying.

Step 7: In the imaging software you will like to use, you will need to copy the software to the install path, and either have the scheduled task built OR have an instruction to install the scheduled task. The recommended approach is the \setupfiles\install_task_psonly.ps1

Step 8: On user login the device will:

  1. Register with AirWatch
  2. Enroll

Packet Sniffing

External link: https://techzone.vmware.com/troubleshooting-windows-10-vmware-workspace-one-operational-tutorial#968025

Use Fiddler. Fiddler is a free web debugging proxy server tool (local MitM-attack) which logs HTTP(S) (with decryption, using fake certificate) traffic to quickly obtain all network communications to and from the device.

Installation

  • Download and install Fiddler on Windows 10 client device

https://www.telerik.com/download/fiddler

  • Run Fiddler, click Cancel to disable warning

Configuration

  • Select WinConfig button

  • Choose No in “Orphaned Exemption Record Found” message window

  • In “AppContainer Loopback Exemption Utility” window, choose Exempt, then Save Changes, then close the window This setting captures UWP application traffic and setting on Windows 10. By default, Fiddler captures traffic only for Win32 app types.

  • Use Menu Tools → Options…

  • Check Decrypt HTTPS Traffic

  • Confirm all warnings: Yes, Yes, Yes, OK

  • Configure filters: most simple way is to only show traffic from specific hosts

  • Toggle Capture traffic in Menu File → Capture Traffic, OR use F12 hotkey

Traffic Inspection

  • Click Inspectors
  • Select Raw. Because most MDM/IDM communication is in SyncML format, for Windows 10, always select XML.
  • If inspecting HTTPS packets, they may be encoded, the click “Response body is encoded. Click to decode” message.

Enrollment Troubleshooting

The most important sessions which deal with enrollment are the Policy.aws and Enrollment.aws messages and the authentication traffic in them.

SCCM Enroll Check

#Compliance Script. For use in SCCM Compliance item as a discovery script.
#Checking first for Airwatch Enrollment
$val = (Get-ItemProperty -Path "HKLM:SOFTWARE\Microsoft\Provisioning\OMADM\Accounts\*" -ErrorAction SilentlyContinue).PSChildname
 
#Now checking whether enrollment is with a real user or the staging user
$path2 = "HKLM:\SOFTWARE\Microsoft\Enrollments\$val"
$val2 = (Get-ItemProperty -Path $PATH2 -ErrorAction SilentlyContinue).UPN
 
#This will be "Completed" if it is staged enrollment has completed but not yet flipped to final user
$staging = (get-itemproperty -path HKLM:\SOFTWARE\AIRWATCH\EnrollmentStatus -ErrorAction SilentlyContinue).status
$reassignment = (get-itemproperty -path HKLM:\SOFTWARE\AIRWATCH\Reassignment -ErrorAction SilentlyContinue).status
 
 
if ($staging -eq "Completed" -and $reassignment -eq $null)
{
    Write-Host "Non-Compliant"
}
Elseif ($val2 -like "*staging*" -or $val2 -eq $null)
{
    Write-Host "Non-Compliant"
}else
{
    write-host "Compliant"
 
}

Software Distribution

This feature is available through the Apps & Books section. The article also suggests ways to get the executable commands to enter for your Win32 applications in the AirWatch Console. It ends with steps to troubleshoot issues.

Validated Use Cases

AirWatch validated the success of the software distribution feature in the listed use cases. Review the list and see if your deployment is similar to the validated use cases.

  • Silent deployment of MSI applications
  • MSIs with multiple transforms, and the ability to deploy different transforms to different sets of users
  • 64 and 32 bit apps on 64 bit devices
  • Installers with registry validations and file checks after installation
  • Patch applied to an already deployed application
  • Application installation on system context and user context
  • A complete silent application installation
  • Application installation with dependencies
  • Packages with scripts that invoke multiple files (ZIP files that contain PowerShell scripts, EXE, and MST)
  • Installation of applications that require reboot
  • Applications with disk space, battery, and RAM checks
  • Uninstallation of installed applications

Application-specific templates

Internet Explorer

External link - Internet Explorer CSP Documentation

To deploy this sample, navigate to  Devices & User > Profile > Add > Windows > Desktop > Device > Custom Settings, then copy and paste the SyncML into the box and publish the profile.

  • Modify the values inside of the data tags.
  • Change the target of the policies to either device or user. Inside of you will want to change to either ./Device/ or ./User/ but be careful as some policies support User, Device, or Both.

Google Chrome

Deploy attached Chrome CSPs samples via AirWatch. To deploy navigate to Devices & User > Profile > Add > Windows > Desktop > Device > Custom Settings, then copy and paste the SyncML into the box and publish the profile.

  • Modify the values inside of the data tags.
  • Change the target of the policies to either device or user. Inside of you will want to change to either ./Device/ or ./User/ but be careful as some policies support User, Device, or Both, you can reference which are support by looking at the Chrome ADMX template.

Tips to Get Configurations

Review some ways to get the commands and criteria for the Win32 application. Enter the data in the AirWatch Console when you upload the Win32 application package.

Get the Install Command

Review a few ways to get install commands for Win32 applications.

Note: If an install command prompts for user interaction on the UI, then enter these commands with the User option in the Install Context option.

  • Call any script from the command-line that results in a successful installation of the Win32 application.
  • The MSI file has the install command pre-populated with silent parameters. You can edit and update these in the AirWatch Console.
  • If the EXE or ZIP file contains the MSI file of the Win32 application, use the msiexec command to install.

Get the Uninstall Command

Review some ways to get uninstall command for Win32 applications.

  • In a command-line session, use the /? or /help parameters to display supported actions. For example, Mysampleapp.exe /?.
  • Look at the HKEYs in the listed registries on the device.
    • HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Uninstall
    • HKLM\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\
    • HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Uninstall
    • HKCU\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\
  • If the EXE contains an underlying MSI, use the msiexec uninstall command. For example, msiexec /x <path_to_file>.

Get Detection Criteria

Use detection criteria to determine if the Win32 application is on devices. To get the detection criteria, install the application and identify the checks on the device.

  • Product ID check

    • Run the wmic command and use WMIC Product where name=”.
    • Look at the HKEYs in the listed registries on the device for the product ID.
      • HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Uninstall
      • HKLM\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\
      • HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Uninstall
      • HKCU\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\
  • File check

    • Look at the HKEYs listed for Product ID check to find the file criteria.
    • Look in the Program Files folder or the Program Files(X86) folder to find the file criteria.
  • Registry check 

  • Look at the HKEYs listed for Product ID check to find registries.

  • Look in HKEY_CLASSES_ROOT\Installer\Products.

Get Exit Codes

Use the environmental variable, %errorlevel%, to get exit codes. Use it in conjunction with built-in DOS commands like ECHO, IF, and SET to preserve the existing value of %errorlevel%.

  1. In a command-line session, run the install command for the Win32 application.
  2. Run ECHO %errorlevel%.
  3. The %errorlevel% variable returns the reboot exit code, if the Win32 application requires a reboot for installation.

Troubleshoot Software Distribution Issues

Win32 application installations involve the successful execution of multiple steps. If your application installation fails, follow the troubleshooting steps to find the issue.

Win32 Package Received Reported by App Deployment Agent

The App Deployment Agent on the user’s device handles Win32 application installations. The system deploys the agent to devices either upon enrollment or when it collects the latest App List sample from devices that are already enrolled.

The system holds the app-install commands in the queue until the agent reports back that the application installed.

Steps

Check the following components to see that the agent installed on your end-users’ devices.

  • In the AirWatch Console, check that the device successfully enrolled and syncs with the console.
  • Check the registry for the AW App Deployment Agent.
  1. Open a command-line session and run regedit. This opens the Registry Editor.
  2. In the Registry Editor, go to HKEY_LOCAL_MACHINE > SOFTWARE > Microsoft > EnterpriseDesktopAppManagement.
  3. Look for the AW App Deployment Agent. The correct status value for the AW App Deployment Agent is 70.
  • Check services on the device to ensure that the AW App Deployment Agent is running.
  • Check the registry for the AW MDM nodes.
  1. Open a command-line session and run regedit. This opens the Registry Editor.
  2. In the Registry Editor, go to HKEY_LOCAL_MACHINE > SOFTWARE > AirWatchMDM > AppDeploymentAgent.
  3. Find three nodes. If these three nodes are missing, then the device did not receive the Win32 application package.
  • App Manifest – This node contains information about the options set in the AirWatch Console on the Deployment Options tab.
  • Content Manifest – This node contains information about the options set in the AirWatch Console on the Files tab.
  • Queue – This node contains detailed logs about the installation of the application. You can view the logs to check the progress of the download of the application.

Win32 Application Installation Status

After the agent installs on devices, you can track the application installation to troubleshoot issues. The install status for the Win32 application displays the listed statuses. 

  1. Install command ready for device – The install command is queued on the device but the device has not checked in to the AirWatch system.
  2. Install command dispatched – The device checks in to the system and consumes the install command.
  3. Installing – The Win32 application is downloading and the installation is in progress on the device. 
  4. Installed – The installation is complete and the device sent an alert to the AirWatch Console.

Status Codes

Expand to see...

Steps

If the installation fails after status #2, Install command dispatched, take these steps to find the reason for the failure.

  1. In the AirWatch Console, validate the configurations for the Win32 application on the Deployment Options tab.
    1. Go to Apps & Books > List View > Internal and edit the Win32 application.
    2. Select Edit > Deployment Options tab.
    3. In the How To Install section, review the Install Context configurations for Device or User.
    4. Review the Admin Privileges setting.
    5. Review the Install Command setting.
    6. Side-load the application to the device to see if this actions triggers the install command.
  2. In the AirWatch Console, look at the Console Event Logs to find the reason for the failure in HUB > Reports & Analytics > Events > Console Events.
  3. Look for a failure reason on the device.
    1. On the device, open a command-line session and run regedit. This opens the Registry Editor.
    2. In the Registry Editor, go to HKEY_LOCAL_MACHINE > SOFTWARE > AirWatchMDM > AppDeploymentAgent.
    3. Look in the Queue node at the log field.
    4. If there is no Queue node, look for a node with the device or user SID. This value has the Win32 application product code. Select the product code to view the reason for installation failure.

Review the App Installer  Flow chart  for a depiction of how the device validates the pre and post installation checks.

SQL Queries for Win Apps

Check DB

  • Check DB command id for Profile Installation
SELECT * FROM deviceCommandQueue.DeviceQueueAudit WHERE deviceid='';

  • Copy transaction id from SQL DB ‘CBDFAD47-28F4-4D63-(A1D-91C3542840’ and search in syncml inside Device Services logs

  • Request Sent to Device 1.2DM/1.2721174828C828F0E444B325860CB5C3037Dhttps://migtest.ssdevrd.com/DeviceServices/Dm.svc/token/uq7Qncbdfad47-28f4-4d63-9a1d-91c3542840161ca07aea-16ea-4989-b793-0fe0d44d28ae./cimv2/MDM_AppInstallJob/MDM_AppInstallJob.JobID=%22WC_96%22/Exec=CreateJobchrtext/plainJobData=<AppInstallJob id=“96”><WebApplication PackageFullName=“webclicp” ActionType=“1” DeploymentOptions=“1” IsBundle=“false”><ContentURLList><ContentURL>https://google.com</ContentURL></ContentURLList><FrameworkDependencies/></WebApplication></AppInstallJob> 

  • Response Received from Device 1.2DM/1.2722https://migtest.ssdevrd.com/DeviceServices/Dm.svc/token/uq7Qn174828C828F0E444B325860CB5C3037D110SyncHdr20021cbdfad47-28f4-4d63-9a1d-91c354284016Atomic200311ca07aea-16ea-4989-b793-0fe0d44d28aeExec200 

  • DB Queries

-- GENERAL QUERIES
 
SELECT * FROM dbo.Device WHERE DeviceID = '';
SELECT * FROM dbo.DeviceMappingWindowsPhone WHERE DeviceID = '';
SELECT * FROM deviceCommandQueue.DeviceQueueCommand;
SELECT * FROM deviceCommandQueue.DeviceQueue WHERE DeviceId = '';
SELECT * FROM deviceCommandQueue.DeviceQueueAudit WHERE DeviceId = '';
SELECT * FROM deviceCommandQueue.DeviceQueue WHERE DeviceID IN (9275,9276);
SELECT * FROM deviceCommandQueue.DeviceQueueAudit WHERE DeviceID IN (9275,9276);
 
DELETE FROM deviceCommandQueue.DeviceQueue WHERE DeviceID IN (9275,9276);
DELETE FROM deviceCommandQueue.DeviceQueueAudit WHERE DeviceID IN (9275,9276);
 
-- PROFILE QUERIES
 
SELECT * FROM deviceProfile.DeviceProfile WHERE DeviceProfileID = '';
SELECT * FROM deviceProfile.DeviceProfileVersion WHERE DeviceProfileID = '';
SELECT * FROM deviceProfile.DevicePlatformSettingGroup WHERE DevicePlatformID IN (11,12);
SELECT * FROM deviceProfile.DevicePlatformSetting WHERE DevicePlatformSettingGroupID = '';
SELECT * FROM deviceProfile.DeviceProfileSettingValue WHERE DeviceProfileVersionID = '' AND DevicePlatformSettingID = '';
 
EXEC [deviceProfile].[DeviceProfile_Delete] '';
 
SELECT * FROM deviceProfile.DevicePlatformSetting WHERE DevicePlatformSettingGroupID = 417;
SELECT * FROM deviceProfile.DeviceProfileSettingValue WHERE DeviceProfileVersionID = 42866 AND DevicePlatformSettingID IN (40646,40647,40674,40675);
SELECT * FROM deviceProfile.DeviceProfile WHERE DeviceProfileID = 9407;
SELECT * FROM deviceProfile.DeviceProfileVersion WHERE DeviceProfileID = 9407;
 
-- APPLICATION SPECIFIC
 
SELECT * FROM device application;
SELECT awapptargetidentifier,* FROM deviceapplication.application WHERE DeviceTypeID=12 AND ISSystemApplication=1;
 
-- DELETE DEVICE
 
EXEC Device_delete '';
 
-- SCHEDULER
 
SELECT * FROM interrogator.Scheduler WHERE DeviceID = '';
 
-- PRODUCTS
 
SELECT * FROM [provisioningPolicy].[PolicyEngineQueue];
SELECT * FROM [provisioningPolicy].[DevicePolicy];
SELECT * FROM [provisioningPolicy].[DevicePolicyJob];
SELECT * FROM [provisioningPolicy].[DevicePolicyJobStatus];
SELECT * FROM [provisioningPolicy].[PolicyEngineAction];
 
-- PATCH MANAGEMENT
 
SELECT * FROM osupdate.RevisionAssignment;
SELECT * FROM smartGroup.AWEntitySmartGroupAssignmentMap WHERE RevisionAssignmentID IN (18);
SELECT * FROM smartGroup.AWEntitySmartGroupAssignmentMap_Audit WHERE RevisionAssignmentID IN (18);
SELECT * FROM osUpdate.UpdateDeviceAssignment;
 
DELETE FROM smartGroup.AWEntitySmartGroupAssignmentMap WHERE RevisionAssignmentID IN (18);
DELETE FROM smartGroup.AWEntitySmartGroupAssignmentMap_Audit WHERE RevisionAssignmentID IN (18);
DELETE FROM osupdate.RevisionAssignment;
DELETE FROM osUpdate.UpdateDeviceAssignment;
 
SELECT * FROM osUpdate.UpdateEula;
SELECT * FROM osUpdate.UpdateEulaAcceptance;
SELECT * FROM osUpdate.UpdateMetaDataUpdateEulaMap WHERE UpdateMetadataID = 21117;
SELECT * FROM osUpdate.UpdateMetadata WHERE UpdateMetadataID = 21117;
SELECT * FROM osUpdate.Revision WHERE RevisionID = 189518;
SELECT * FROM osUpdate.UpdateDescription WHERE UpdateMetadataID = 21117;
 
EXEC osUpdate.LoadRevisionAssignments  @LocationGroupID=3711 , @RevisionID = @RevisionID;
 
SELECT * FROM osUpdate.UpdateEULA;
SELECT * FROM osUpdate.UpdateEulaAcceptance WHERE LocationGroupID = 3711;
SELECT * FROM osUpdate.UpdateEulaLocationGroupMap;
SELECT * FROM osUpdate.UpdateMetaDataUpdateEulaMap;
SELECT * FROM osUpdate.UpdateMetadata WHERE RevisionID IN (794,799,800);
 
--INSERT INTO osUpdate.UpdateEULA([Language],DescriptionText,[Hash],MSEULAID) VALUES ('en','EULA','rKUV1NrYBUeMDv3uQ8dOD0LqUX4=','3c7d5240-94f7-4df7-9c8c-489b8cf2b58a');
--INSERT INTO osUpdate.UpdateEULA([Language],DescriptionText,[Hash],MSEULAID) VALUES ('en','EULA','rKUV2NrYBUeMDv3uQ8dOD0LqUX4=','3c7d5240-94f7-4df7-9c8c-489b8cf2b58b');
--INSERT INTO osUpdate.UpdateEULA([Language],DescriptionText,[Hash],MSEULAID) VALUES ('en','EULA','rKUV3NrYBUeMDv3uQ8dOD0LqUX4=','3c7d5240-94f7-4df7-9c8c-489b8cf2b58c');
--INSERT INTO osUpdate.UpdateMetaDataUpdateEulaMap(UpdateMetaDataID,UpdateEULAID) VALUES (36,10);
--INSERT INTO osUpdate.UpdateMetaDataUpdateEulaMap(UpdateMetaDataID,UpdateEULAID) VALUES (37,11);
 
DELETE FROM smartGroup.AWEntitySmartGroupAssignmentMap WHERE RevisionAssignmentID IN (30,31,32);
 
-- WINDOWS AUTO-ENROLLMENT
 
SELECT * FROM dbo.SystemCodeCategory WHERE SystemCodeCategoryID = 420;
SELECT * FROM dbo.SystemCodeGroup WHERE SystemCodeGroupID = 420;
SELECT * FROM dbo.SystemCode WHERE SystemCodeGroupID = 420;

Command ID with Command Name details

Command NameCommand ID
DeviceInformation3
DeviceLock4
AppList Sample5
CertificateListSample6
InstallProfile

7

ProfileList8
ProvisioningProfileList9
RemoveProfile10
SecurityInformation12
BreakMDM15
InstallApplication21
ManagedApplicationList23
RemoveApplication24
RemoteControl34
ChangePasscode40
InstallProvisioningPolicy60
RemoveProvisioningPolicy61
EnterpriseReset64
OSUpdates100
HealthAttestation104
HealthAttestationCertificate105
HealthAttestationStatus106
SelectiveAppListSample116
ApproveUpdate126
UnApproveUpdate127
RemoveUpdate130
SyncSensors

143

SyncML

Create a Complete Block of SyncML

A complete block of SyncML code consists of the following attributes:

  • Runs from<[characteristic]>to<[characteristic]>
  • Uses Add, Delete, Replace, or Exec as a characteristic.
  • Does not contain text before or after the characteristics.
  • May or may not remove all whitespace and linearize the code block to condense its size.
For Example:

<Replace><CmdID>2</CmdID><Item><Target> <LocURI>./Device/Vendor/MSFT/AssignedAccess/KioskModeApp</LocURI></Target> <Meta><Format xmlns="syncml:metinf">chr</Format></Meta><Data> {"Account":"standard","AUMID":"AirWatchLLC.AirWatchBrowser_htcwkw4rx2gx4!App"}</Data></Item></Replace>

Update or Delete Settings

Manually apply tags to update or delete settings using a Windows 10 custom settings profile.

  • To update settings, use the replace tag: <Replace> to </Replace>
  • To remove settings, use the delete tag: <Delete> to </Delete>
For Example: Remove Kiosk Assigned Access Setting

<Delete><CmdID>2</CmdID><Item><Target> <LocURI>./Device/Vendor/MSFT/AssignedAccess/KioskModeApp</LocURI></Target></Item></Delete>

Configure a Windows 10 Custom Settings Profile

Create a block of SyncML code:

  1. Go to VMware Code Sample Exchange.
  2. Find the correct sample.
  3. Copy the text.

Copy the Profile Settings from the Latest AirWatch Console

  1. Log into a version of the AirWatch console that supports the desired profile functionality.
  2. Configure and Save this payload to create a profile.
  3. Find the new profile in the list view: 1) Click its radio button. 2) Click the </>XML option. 3) Copy the SyncML that appears.
  4. Paste the SyncML into a text editor, and edit it: 1) Remove lines of text so that all the code falls between the tags: <[ Add, Delete, Replace, or Exec ]> to <[ Add, Delete, Replace, or Exec ]> 2) Optionally, remove the whitespace, and linearize the SyncML.
  5. Copy the formatted code.

Create New SyncML

  1. Go to the Configuration Service Provider (CSP) Reference.
  2. Access the newest Windows Insider features.
  3. Follow the site’s available guidelines to create the code sample.
  4. Copy the text.

Publish SyncML code:

  1. Navigate to Devices > Profiles > List View > Add > Add Profile > Windows > Windows Desktop.

  2. Refer to the LocURI to determine the profile’s context.

    • User Profile: Select if the LocURI begins with ./User/.
    • Device Profile: Select if the LocURI begins with ./Device/.
  3. Configure General settings to determine how the profile deploys and who receives it.

  4. Select the Custom Settings payload.

  5. Click Configure, and paste the complete block of SyncML code in the text box.

  6. Select Save & Publish.

Windows SSO

External Links:

Configuration

  • You must have a CA with template for SSO. Refer to MobileSSO for iOS for configuring internal AirWatch CA with SCEP protocol.
  • Check WS1 Access network ports.

Certificate Chain

  • From version of vIDM 1903+ Cert Auth Service port TCP7443 must be opened for vIDM Appliance (regardless of connector).
  • Also certificate chain and private key must be pasted into vIDM Appliance Admin Portal (accessed via port TCP8443).

Reference document - https://docs.vmware.com/en/VMware-Identity-Manager/19.03/vidm-install/GUID-B625A0BA-2991-4F46-9D41-A1BD8C4D8BE2.html

On AirWatch / Workspace ONE UEM side

  • If inner AirWatch CA/SCEP is used, go to Groups & Settings → All Settings → Enterprise Integration → Workspace ONE Access → Configuration, click Certificate → EXPORT button
  • Go to Devices → Profiles & Resources → Profiles → ADD → Add Profile → Windows → Windows Desktop → User Profile and make a User profile
Warning

There is also a Windows Desktop → Device Profile. Do NOT use it for SSO/Conditional Access, it will not work!

  • Give it a Name. Example: “Win10 SSO”. Select a group in Smart Groups. For example, choose all devices (World icon)

  • Go to SCEP tab/payload, and set:

    Credential Source: **AirWatch Certificate Authority
    **Certificate Template: **Certificate (Cloud Deployment)
    **Issuer: CN=<Issuer name in certificate, example = name of current Organization Group>

On vIDM / Workspace One Access side

  • Go to Identity & Access Management → Manage and select Authentication Methods
  • Choose Certificate (Cloud Deployment) authentication method (pencil icon)
  • Check the Enable Certificate Adapter, then Select File and upload the Certificate (*.cer) which you downloaded from the AirWatch CA/SCEP from the step above, or from ADCS Domain CA
  • Click Save
  • Go to Identity & Access Management →  Identity Providers, click on Built-in IF you are NOT using the ESC connector. If you are using the Connector, choose it in the list
  • Find Authentication Methods area and select Certificate (Cloud Deployment) check box, then click Save
  • Go to Identity & Access Management → Policies and select default_access_policy_set, select Edit
  • In Configuration tab, ALL RANGES, select Device Type = Windows 10, and in “**then the user may authenticate using “**choose Certificate (Cloud Deployment)
  • (Optional) In “if the preceding method fails or is not applicable, then” choose Password (Cloud Deployment)
  • (Optional) Select the (****+) ADD FALLBACK METHOD and in “If the preceding method fails or is not applicable, then” select Password ( Local Directory)

Windows 10 Device Tests and Checks

  • On enrolled Win10 device, open MMC, select Menu File → Add/Remove Snap-In…, select My User Account

  • Check Personal folder to see that the profile certificate was delivered

  • Use Hub to access vIDM/Workspace ONE Access portal.

Troubleshooting certificate issues

CertificateAuthAdapterBase function header for requesting the certificate from Windows:

//function names & var names obfuscated
 
protected X509Cert[] getCert(@Nonnull String tenantId,
                             @Nonnull HttpServletRequest request,
                             @Nonnull HttpServletResponse response,
                             @Nonnull Map<String, String> attribVal,
                             @Nullable Map<String, String> inputParam) throws AuthAdapterConfigException { ... }
 
 
X509Cert[] certs = getCert(tenantId, request, response, attribVal, inputParam);
        if (certs == null || certs.length == 0) {
            logger.info("No certificates were provided by the browser"); // --> horizon.log
            if (certs == null) {
                adapterResponse.setStatus(AuthnAdapterResponse.AuthnStatus.FAILURE);
                logger.info(logId + " authentication failure, no certificate provided"); // --> horizon.log

getCert method returns the certificate, received from client browser HTTP-request. If not, it logs errors in horizon.log file on WS1 Access/vIDM.

WS1 Launcher and Restrictions

Custom XML Profiles for Android

Custom XML for Workspace ONE Launcher - https://docs.vmware.com/en/VMware-Workspace-ONE-UEM/2209/Launcher_Publication/GUID-CustomXML.html?hWord=N4IghgNiBcIMYFcDOAXA9gWwAQA8MSwAcAnNAMwEsIBTLCMBAOzgAtriQBfIA

When using Ws1 Launcher for Android, it will block Phone app and incoming phone call notifications. In order to allow phone dialer app (example: com.samsung.android.dialer) a custom profile has to be added:

<characteristic type="com.airwatch.android.androidwork.launcher" uuid="568bc89d-1df8-4ce9-a041-e5a24acdb7ec">
<parm name="SkipCosuSetup" value="True"/>

Enable WiFi in Android Kiosk

By default, WiFi is blocked by kiosk mode. To unlock possibility for user to choose WiFi network do this:

  • Check the APP ID of “Settings” application on Android OS;
  • Add Settings app APP ID to AirWatch Launcher as Hidden Apps.

Custom XML Profiles for iOS

Custom Settings profile payload allows admins to enter their own XML into a profile and apply the profile to devices.

(For iOS 11.3+ and macOS 10.13.4+)

  • XML should contain the complete block of code as listed below, from to
  • Administrators should configure each setting from to as desired.
  • If certificates are required, then configure a Certificate payload within the profile and reference the Payload UUID in the Custom Settings payload.

iOS 11.3

Restrictions

<dict>
         <key>allowUSBRestrictedMode</key>
         <true/>
         <key>forceClassroomRequestPermissionToLeaveClasses</key>
         <true/>
         <key>forceDelayedSoftwareUpdates</key>
         <true/>
         <key>enforcedSoftwareUpdateDelay</key>
         <integer>30</integer>
         <key>PayloadDisplayName</key>
         <string>Restrictions</string>
         <key>PayloadDescription</key>
         <string>RestrictionSettings</string>
         <key>PayloadIdentifier</key>
         <string>7480b205-2e1c-40fe-bd59-b53db434652d.Restrictions</string>
         <key>PayloadOrganization</key>
         <string></string>
         <key>PayloadType</key>
         <string>com.apple.applicationaccess</string>
         <key>PayloadUUID</key>
         <string>99b5b40b-5683-4315-9ec2-f9e014a6XXXX</string>
         <key>PayloadVersion</key>
         <integer>1</integer>
 </dict>

Home Screen Layout

Note

Web clips should be configured and deployed using the current web clip payload UI

<dict>
    <key>Dock</key>
    <array>
        <dict>
            <key>Type</key>
            <string>WebClip</string>
            <key>URL</key>
            <string>https://google.com</string>
        </dict>
    </array>
    <key>Pages</key>
    <array>
        <array>
            <dict>
                <key>Type</key>
                <string>WebClip</string>
                <key>URL</key>
                <string>https://yahoo.com</string>
            </dict>
            <dict>
                <key>Type</key>
                <string>Folder</string>
                <key>DisplayName</key>
                <string>My Web Clip</string>
                <key>Pages</key>
                <array>
                    <array>
                        <dict>
                            <key>Type</key>
                            <string>WebClip</string>
                            <key>URL</key>
                            <string>https://www.vmware.com</string>
                        </dict>
                    </array>
                </array>
            </dict>
        </array>
    </array>
    <key>PayloadDisplayName</key>
    <string>Home Screen Layout</string>
    <key>PayloadDescription</key>
    <string>HomeScreenLayout</string>
    <key>PayloadIdentifier</key>
    <string>97213d06-b750-466b-8a89-782d8a406f86.Home Screen Layout</string>
    <key>PayloadOrganization</key>
    <string></string>
    <key>PayloadType</key>
    <string>com.apple.homescreenlayout</string>
    <key>PayloadUUID</key>
    <string>2fa8fe03-30fa-4189-aa00-ba752eabXXXX</string>
    <key>PayloadVersion</key>
    <integer>1</integer>
</dict

Enable Bluetooth Command

Unlike custom profiles, the payload content and UUID are not required for custom commands. This command will not take place if the Allow Bluetooth Settings Modification restriction is enforced.

<dict>
  <key>RequestType</key>
  <string>Settings</string>
  <key>Settings</key>
  <array>
    <dict>
      <key>Item</key>
      <string>Bluetooth</string>
      <key>Enabled</key>
      <true/>
    </dict>
  </array>
</dict>

macOS 10.13.4 Custom XML

Restrictions

<dict>
         <key>forceDelayedSoftwareUpdates</key>
         <true/>
         <key>enforcedSoftwareUpdateDelay</key>
         <integer>30</integer>
         <key>PayloadDisplayName</key>
         <string>Restrictions</string>
         <key>PayloadDescription</key>
         <string>RestrictionSettings</string>
         <key>PayloadIdentifier</key>
         <string>7480b205-2e1c-40fe-bd59-b53db434652d.Restrictions</string>
         <key>PayloadOrganization</key>
         <string></string>
         <key>PayloadType</key>
         <string>com.apple.applicationaccess</string>
         <key>PayloadUUID</key>
         <string>99b5b40b-5683-4315-9ec2-f9e014a6XXXX</string>
         <key>PayloadVersion</key>
         <integer>1</integer>
 </dict>

Autonomous Single App (ASA) Mode

<dict>
    <key>AllowedApplications</key>
    <array>
      <dict>
        <key>BundleIdentifier</key>
        <string>com.sample.app1</string>
        <key>TeamIdentifier</key>
        <string>ABCDEFG1HI</string>
      </dict>
      <dict>
        <key>BundleIdentifier</key>
        <string>com.sample.app2</string>
        <key>TeamIdentifier</key>
        <string>ABCDEFG1HI</string>
      </dict>
    </array>
    <key>PayloadDisplayName</key>
    <string>Autonomous Single App Mode</string>
    <key>PayloadDescription</key>
    <string>AutonomousSingleAppMode</string>
    <key>PayloadIdentifier</key>
    <string>7480b205-2e1c-40fe-bd59-b53db434652d.AutonomousSingleAppMode</string>
    <key>PayloadOrganization</key>
    <string></string>
    <key>PayloadType</key>
    <string>com.apple.asam</string>
    <key>PayloadUUID</key>
    <string>91b5e40b-5683-4376-9ec2-f9e214a6XXXX</string>
    <key>PayloadVersion</key>
    <integer>1</integer>
</dict
Warning
  • Can only be installed on User Approved MDM Enrolled devices. Must be installed as Device profile. Only one payload allowed per machine.
  • To be granted access, applications must be signed with the specified Bundle Identifier and Team Identifier using an Apple-issued production developer certificate. Applications must specify the com.apple.developer.assessment entitlement with a value of true.
  • The application’s bundle identifier. BundleIdentifier must be unique. If two dictionaries contain the same BundleIdentifier but different TeamIdentifiers, this will be considered a hard error and the payload will not be installed.

To check if the .app has the correct entitlement noted above:

codesign –d –entitlements - /Applications/Example.app

This will print out XML with the entitlements. It needs to have the com.apple.developer.assessment entitlement with a value of true.

To get the Bundle & Team Identifier for an .app:

codesign –dvvvv /Applications/Example.app

The Bundle Identifier will be in the ‘Identifier’ field. The Team Identifier will be a 10 character string in the ‘TeamIdentifier’ field.

Content Caching

<dict>
    <key>AllowPersonalCaching</key>
    <true/>
    <key>AllowSharedCaching</key>
    <true/>
    <key>AutoActivation</key>
    <true/>
    <key>CacheLimit</key>
    <integer>100000000</integer> <!--100 MB example-->
    <key>DataPath</key>
    <string>/Library/Application Support/Apple/AssetCache/Data</string>
    <key>DenyTetheredCaching</key>
    <false/>
    <key>ListenRanges</key>
    <array>
      <dict>
        <key>type</key>
        <string>IPV4</string>
        <key>first</key>
        <string>0.0.0.0</string>
        <key>last</key>
        <string>255.255.255.255</string>
      </dict>
    </array>
    <key>ListenRangesOnly</key>
    <false/>
    <key>ListenWithPeersAndParents</key>
    <true/>
    <key>LocalSubnetsOnly</key>
    <true/>
    <key>LogClientIdentity</key>
    <false/>
    <key>Parents</key>
    <array>
      <string>1.1.1.1</string>
      <string>2.2.2.2</string>
    </array>
    <key>ParentSelectionPolicy</key>
    
 <string>round-robin</string> <!-- Possible values are
round-robin, first-available, url-path-hash, random, and
sticky-available-->
    <key>PeerFilterRanges</key>
    <array>
      <dict>
        <key>type</key>
        <string>IPV4</string>
        <key>first</key>
        <string>0.0.0.0</string>
        <key>last</key>
        <string>255.255.255.255</string>
      </dict>
    </array>
    <key>PeerListenRanges</key>
    <array>
      <dict>
        <key>type</key>
        <string>IPV4</string>
        <key>first</key>
        <string>0.0.0.0</string>
        <key>last</key>
        <string>255.255.255.255</string>
      </dict>
    </array>
    <key>PeerLocalSubnetsOnly</key>
    <true/>
    <key>Port</key>
    <integer>0</integer>
    <key>PublicRanges</key>
    <array>
      <dict>
        <key>type</key>
        <string>IPV4</string>
        <key>first</key>
        <string>0.0.0.0</string>
        <key>last</key>
        <string>255.255.255.255</string>
      </dict>
    </array>
    <key>PayloadDisplayName</key>
    <string>Content Caching</string>
    <key>PayloadDescription</key>
    <string>ContentCaching</string>
    <key>PayloadIdentifier</key>
    <string>7480b205-2e1c-40fe-bd59-b53db434652d.ContentCaching</string>
    <key>PayloadOrganization</key>
    <string></string>
    <key>PayloadType</key>
    <string>com.apple.AssetCache.managed</string>
    <key>PayloadUUID</key>
    <string>98f5b40b-5683-2415-9ec2-f9e014a6XXXX</string>
    <key>PayloadVersion</key>
    <integer>1</integer>
</dict>

WS1 SQL Database

Articles in section

SQL Requirements

  • AirWatch 9.1 and later versions are supported on SQL Server 2016, SQL Server 2014, and SQL Server 2012.
  • Both Enterprise and Standard Editions are supported, Express Edition is NOT supported because it does not offer all of the features that are used by AirWatch.
  • It is recommended that the AirWatch databases are operating on 64-bit editions of Windows and using the 64-bit installation of SQL Server.

❗️ SQL Server Collation: AirWatch supports SQL_Latin1_General_CP1_CI_AS as the server AND database collation ONLY.

SQL Connections

MAX database connections are set to 150 per Application Pool and Service. Currently in production on each DS server there are :

4 Application Pools 8 Services
AirWatch Local AW Tunnel Queue Monitor Service
AirWatch API AW Entity Change Queue Monitor
AirWatch DS AW Interrogator Queue Monitor
AirWatch SSP AW Interrogator Server
AW Log Manager Queue Monitor
AW Master Queue Service
AW MEG Queue Service
AW Messaging Service

⭐️ Total = 12 * 150 max connections = 1800 per DS Server.

SQL Recommendations

VMware official recommendations

  • TempDB Configuration: the number of tempDB files must match the number of CPU cores when the core is less than or equal to 8 cores. Beyond 8 cores, the number of files must be the closest multiple of 4 that is less than or equal to the number of cores (e.g. 10 cores will need 8 tempDBs, 12 cores will need 12 tempDBs, 13 cores will need 12 tempDBs, 16 cores will need 16 tempDBs.) File size, growth rate, and the location need to be the same for all tempDB files.

    ⭐️ Microsoft SCCM best practice is to create no more than 8 temp DB files

  • Memory Allocation: Eighty percent of the server memory should be allocated to SQL. The remaining 20% must be freed up to run the OS.

    • Test = 16Gb
    • Production = 128GB In Properties of server in Memory tab set restrictions for memory usage of SQL Server:
    • min = 60%
    • max = 80%
  • Cost Threshold for Parallelism and Maximum Degree of Parallelism: Cost Threshold for Parallelism is the cost needed for a query to be qualified to use more than a single CPU thread. Maximum Degree of Parallelism is the maximum number of threads that can be used per query. The following are recommended values for these parameters:

    • Cost Threshold of Parallelism: 50
    • Max Degree of Parallelism: 2 and reduce to 1 in case of high server utilization.
  • Trace Flag: The following trace flags must be set to 1 at Global. 1117 (https://msdn.microsoft.com/en-us/library/ms188396.aspx) 1118 (https://msdn.microsoft.com/en-us/library/ms188396.aspx) 1236 (https://support.microsoft.com/en-us/kb/2926217) 8048 (https://blogs.msdn.microsoft.com/psssql/2015/03/02/running-sql-server-on-machines-with-more-than-8-cpus-per-numa-node-may-need-trace-flag-8048/) Set flags 1117, 1118, 1236, 8048 = 1

  • Hyperthreading: If the database is running on a physical server, hyperthreading must be disabled on the database to ensure best performance. If it is on a VM, then having hypertherading enabled on the ESX host will not have any performance impact, but hyperthreading must be disabled on the Windows host level;

  • Optimize for Ad hoc Workloads: Enable Optimize for Ad hoc Workloads under SQL server properties. This is recommended in order to free memory from the server. Refer to the following article for more information: https://msdn.microsoft.com/en-us/library/cc645587(v=sql.120).aspx;

  • Lock Escalation: Disable Lock Escalation for “interrogator.scheduler” table by running alter table interrogator.scheduler set (lock_escalation = {Disable}) This is recommended as the scheduler table has very high rate of updates/inserts. There is a high contention on this table with the use of GCM, and disabling lock escalation helps improve performance. However, the drawback is that more memory is consumed. Refer to the following article for more information: https://technet.microsoft.com/en-us/library/ms184286(v=sql.105).aspx.

Additional Microsoft recommendations for SQL DB for large installations

  • SQL Server Agent autostart: In SQL Server Configuration Manager properties of SQL Server Agent component put automatic start;

  • Disable Auto shrink of DB and journals: According to Microsoft Best Practices for SCCM, Auto Shrink of DB and journals should be turned OFF;

  • Journal size edit: According to Microsoft Best Practices for SCCM, initial size of transaction log should be 16Gb, growth by 512Mb;

  • Recovery Model type: Choose recovery model type - Full or Simple See page on SQL Recovery model choice and consequences

    Best practive by Microsoft - to set an SQL Alert on event “log size greater than 18Gb” and a Job to execute Shrink Log File.

Subsections of WS1 SQL Database

Admin Password Reset

Changing an administrator’s password

Changing an administrator’s password requires first getting the CoreUserID for that user, and then updating the password and password salt for that user in the database.

Getting the CoreUserID

Run the following SQL query to get the CoreUserID for the user.  Make sure to replace with the administrator’s username.  If more than one result are returned, make sure you identify the right entry.

Select * from CoreUser where UserName = '<USERNAME>';

Updating the password and password salt

The following query will update the administrator’s password.  Make sure to replace with the proper CoreUserID identified in the first section.  Have updating the password, log in with the account and reset the password to something else.

-- This Updates the password to 'a123b456c789'
UPDATE CoreUser SET Password = 'c2otxl1SURGxVibCX1K9IJvyizYl4ylnIIfXhwNtfe1iCuuVM8LNPK1oWWSwE3C3BB3AYxspGqrfXaVnryxjzw==' ,
PasswordSalt = '6eklCHP5ixaMT9RREfQbmi4Z2jc=' WHERE CoreUserID = <COREUSERID>
 
-- This Updates the password to 'Password123!'
UPDATE dbo.CoreUser SET Password = 'awhash4:IGdeYQ7eaHqyh3g6RelU0zkbHjoRW111/dg2xyqjiK0=:100000:K91XOTYtcw/20ZxDOUm7pe3DVLFA3XrzuUAYtWdl21M=' WHERE UserName = 'Administrator'

Account lock-out

Check IsLockedOut flag - it must be set to 0 for active/unlocked account.

UPDATE dbo.CoreUser SET IsLockedOut = 0 WHERE UserName = 'Administrator'
UPDATE dbo.CoreUser SET LastLoginAttempts = 0 WHERE UserName = 'Administrator'

APNs Troubleshooting

APNs incorrect renewal

  •  First you need to get the GroupID of the organization group that has APNs configured.  In Chrome, you can do this by right-clicking the organization group drop down as shown below and choosing Inspect Element.  In the HTML, search for data-current-lg and note the GroupID.
  • Perform the following SQL query in the database to retrieve information about the APNs certificate:
SELECT DISTINCT DC.CertificateID, DAGS.appleid, DC.SubjectName,
                DC.serialnumber, DC.certificate thumprint,
                DC.notBefore, DC.NotAfter
FROM
      dbo.Certificate (NOLOCK) AS DC
      JOIN
            dbo.ApnsGenerationStatus (NOLOCK) as DAGS
            ON DC.CertificateID = DAGS.IssuedCertID
WHERE
      DAGS.LocationGroupID = 10289
ORDER BY NotBefore ASC
  • Validate the results of this query.  In a correct renewal, both the appleid and SubjectName columns of the results should match between the latest certificate and the previous certificate.  You can tell which certificate is which based on the notBefore and notAfter columns, which match the validity dates of the certificate.
CertificateID appleid SubjectName serialnumber thumprint notBefore NotAfter
378672 userA@yourcompany.com C=US,CN=APSP:06efab8c-1d88-4fed-8c4d-a92f88ea9281,OID.0.9.2342.19200300.100.1.1=com.apple.mgmt.External.06efab8c-1d88-4fed-8c4d-a92f88ea9281 52DEAB4B44A18861 0x30820C9F0… 9/5/2014 9/5/2015
30874966 userA@yourcompany.com C=US, CN=APSP:06efab8c-1d88-4fed-8c4d-a92f88ea9281, OID.0.9.2342.19200300.100.1.1=com.apple.mgmt.External.06efab8c-1d88-4fed-8c4d-a92f88ea9281 0EA275019DF7D704 0x30820C975… 8/28/2015 8/27/2016
  • If the appleid field is different, then the administrator used an incorrect Apple ID when renewing the APNs certificate, and so the SubjectName will not match.  The administrator must use the same Apple ID when renewing APNs, or all devices will lose the ability to communicate.  If appleid matches but SubjectName is different, then the administrator used the correct Apple ID but chose the wrong certificate it Apple’s portal to renew.  They must go back to Apple’s portal and renew the correct certificate.
  • Note: All devices enrolled after the notBefore date of an incorrect APNs certificate must re-enroll when the certificate is corrected.  These devices will lose communication once the certificate is correct.
  • Note: If the administrator previously cleared out the APNs certificate (by selecting the Clear button in the APNs settings page and saving), they will lose communication with all devices.  In this event, a database backup is required to restore the tokens, and this process generally cannot be completed on SaaS environments.

APNs Database Queries

Below are a list of helpful SQL queries specifically when troubleshooting APNs. All of the queries below are SELECT statements, which allows us to view the information but not update it.

  1. This statement will display an overall view of the location group. This allows us to look at certain fields as well as the Location Group ID for the location where the APNs certificate is being uploaded. By inputting the name, you can receive the Group ID.
SELECT * FROM dbo.locationgroup lg (NOLOCK) WHERE lg.Name like '%%'
  1. This statement displays the APNs settings at and above the LG provided. We are looking for these first four fields and which have values that are not NULL. It should only be configured at one level and inherited below. You simply need to input the group ID found in the first query.
SELECT lgf.LGlvl, lg.MDMAppleApplicationID, lg.MDMAppleDeviceProfileID, lg.AppleMDMEnabled, lg.MDMSettingsInheritable,lg.*
FROM dbo.LocationGroup lg (NOLOCK)
 JOIN dbo.LocationGroupFlat lgf (NOLOCK)
  ON lgf.ParentLocationGroupID = lg.LocationGroupID
WHERE lgf.ChildLocationGroupID = ###
ORDER BY lgf.LGlvl DESC
  1. This statement shows the APNs settings at or below the LG provided. You simply need to input the group ID found in the first query. The first four columns after the level should be NULL at every level below the level where the cert was uploaded at. You can also view the levels of the tree in the first column. If you do see multiple levels with values not being NULL, we can look at the different APNs certs and the number of devices associated. This will require some update statements which a DB or T3 Member can assist with.
SELECT lgf.LGlvl, lg.MDMAppleApplicationID, lg.MDMAppleDeviceProfileID, lg.AppleMDMEnabled, lg.MDMSettingsInheritable,lg.*
FROM dbo.LocationGroup lg (NOLOCK)
 JOIN dbo.LocationGroupFlat lgf (NOLOCK)
  ON lgf.ChildLocationGroupID = lg.LocationGroupID
WHERE lgf.ParentLocationGroupID = ###
ORDER BY lgf.LGlvl ASC
  1. This statement will display all the APNs certificates in the environment. You simply need to input the group ID found in the first query. It is important to match this with the previous query so we know which cert is at each level. It is also important to view if the topic is the same for different certs. If it is, we can change the application ID associated to the device without any end user interaction or effect. If the topics are different, devices will need to re-enroll. From this query, you will have the ProductionCertificateID and ApplicationID.
SELECT a.ProductionCertificateID, lg.LocationGroupID, lg.Name, a.*
FROM deviceApplication.Application a (NOLOCK)
 JOIN dbo.LocationGroupFlat lgf (NOLOCK)
  ON a.RootLocationGroupID = lgf.ChildLocationGroupID
 JOIN dbo.LocationGroup lg (NOLOCK)
  ON lgf.ChildLocationGroupID = lg.LocationGroupID
WHERE lgf.ParentLocationGroupID = ###
 AND a.PackageID like '%com.apple.mgmt%'
  1. This statement will confirm if the certificates are alive by viewing the date. You simply need to input the ProductionCertificateID found using the queries previous.
SELECT *
FROM dbo.Certificate c (NOLOCK)
where c.CertificateID IN (###)
  1. This statement will tell the devices that are associated with the Application ID. You simply need to input the ApplicationID found using the queries previous. This will assist if the customer has multiple APNs certificates active in the console and you are trying to determine which to use. Obviously, we will want to choose the cert with the most devices for the least end user impact.
SELECT *
FROM deviceApplication.ApplicationDeviceCredential adc (NOLOCK)
WHERE adc.ApplicationId IN (###)

Condensed query:

SELECT adc.DeviceID, a.ApplicationID, a.PackageID, adc.NotificationID
FROM deviceApplication.ApplicationDeviceCredential adc
LEFT JOIN
deviceApplication.Application a
ON adc.ApplicationID = a.ApplicationIDWHERE DeviceID =
  1. This statement will show the serial number associated with the APNs certificate. This will assist if you need to relay this information to Apple to determine if they have a corrupt certificate where AirWatch is unable to continue troubleshooting. Simply input the name of the Location Group where the APNs Certificate is uploaded.
SELECT lg.Name, c.SerialNumber from dbo.Certificate c
inner join deviceApplication.Application a
on ProductionCertificateID = CertificateID
inner join dbo.LocationGroup lg
on lg.MDMAppleAPplicationID = a.ApplicationID
where lg.Name like '%__%'
  1. This statement will show you all the APNs certificates that have been uploaded to the console at or below the LG provided. You simply need to input the group ID found in the first query.
SELECT c.SerialNumber,c.SubjectName, c.NotBefore, c.NotAfter, ags.*
FROM dbo.ApnsGenerationStatus ags (NOLOCK)
JOIN dbo.Certificate c (NOLOCK)
ON c.CertificateID = ags.IssuedCertID
WHERE ags.LocationGroupID IN (###)
ORDER BY CreatedDate DESC

PushMagic Token

In Database:

SELECT *
FROM dbo.AppleDeviceEx
WHERE DeviceID = ###

APNs in logs

Console (CN) server logs:

  • MessengingServiceLog (C:\AirWatch → Logs → Services) Search for:
  • gateway.push.apple.com
  • APNs token
  • PushMagic token

Device Command Queues

Device operation sequence

A. AirWatch server sends out a PUSH notifications to device:

  • AirWatch prepares command, stores in command queue (SQL)
  • AirWatch sends check-in request to Messaging Service, one of the following:
    1. Google Cloud Messaging: formerly C2DM (Cloud 2 Device Messaging) deprecated 2012 →  Google Cloud Messaging (GCM) deprecated April 2019 → Firebase Cloud Messaging (FCM)
    2. Internal message bus for Android & Windows devices: AirWatch Cloud Messenger (AWCM)
    3. Message bus for iOS: Apple Push Notification Service (APNs)
  • Messaging Service relays check-in request to managed device
  • Managed device checks into MDM server
  • MDM Server delivers commands to device, one-by-one

B. Device performs the commands; C. Device contacts AirWatch server to report the result of the last commands and request new commands, if any.

Command Queue

Every pending command is in SQL database:

SELECT * FROM deviceCommandQueue.DeviceQueue;

Every processed command is in SQL database:

SELECT * FROM deviceCommandQueue.DeviceQueueAudit;

Commands description is in SQL database:

SELECT * FROM deviceCommandQueue.DeviceQueueCommand;

Command statuses in SQL:

SELECT * FROM deviceCommandQueue.DeviceQueueStatus;

Basic command types & statuses…

Device Management commands

  • Lock device
  • Query device
  • Send Message to device

Application commands

  • Install
  • Removal
  • Settings

Profile commands

  • Install
  • Removal

Normal Command Status Flow 1 = Queued 2 = Pending 3 = Processed 7 = Held (if batching is enabled)

Troubleshooting Command Logs

Device Services (DS) server (front-end communication with managed device) logs:

  • DeviceServicesLogs (C:\AirWatch → Logs → DeviceServices)
  • InterrogatorQueueService (C:\AirWatch → Logs → Services)

Console (CN) server (communication with external services: APNs, GCM/FCM, AWCM, ACC etc.) logs:

  • MessengingServiceLog (C:\AirWatch → Logs → Services)
  • BulkProcessingServiceLog (C:\AirWatch → Logs → Services)
  • SchedulerServiceLog (C:\AirWatch → Logs → Services)

Targeted logging: AirWatch console → Device Details → More → Targeted Logging

Device Status

Workspace ONE Registered – Unmanaged WS1 device (MAM Only) ManagedBy column in dbo.Device = 2 EnrollmentCategoryID column in dbo.DeviceExtendedProperties = 1 Query:

select d.ManagedBy, dep.EnrollmentCategoryID, d.DeviceID from dbo.Device d INNER JOIN
dbo.DeviceExtendedProperties dep
ON
d.DeviceID = dep.DeviceID
where d.ManagedBy = 2
AND dep.EnrollmentCategoryID = 1

Workspace ONE Managed – MDM on the device, but did not enroll through agent (Adaptive Mgmt) ManagedBy column in dbo.Device = 1 EnrollmentCategoryID column in dbo.DeviceExtendedProperties = 1 Query:

select d.ManagedBy, dep.EnrollmentCategoryID, d.DeviceID from dbo.Device d INNER JOIN
dbo.DeviceExtendedProperties dep
ON
d.DeviceID = dep.DeviceID
where d.ManagedBy = 1
AND dep.EnrollmentCategoryID = 1

Workspace ONE Managed – MDM on the device, but did not enroll through agent (Direct Enrollment) ManagedBy column in dbo.Device = 1 EnrollmentCategoryID column in dbo.DeviceExtendedProperties = 3 Query:

select d.ManagedBy, dep.EnrollmentCategoryID, d.DeviceID from dbo.Device d INNER JOIN
dbo.DeviceExtendedProperties dep
ON
d.DeviceID = dep.DeviceID
where d.ManagedBy = 1
AND dep.EnrollmentCategoryID = 3

Agent Enrollment with Workspace ONE – Workspace ONE as an app catalog, pushed as a managed app. ManagedBy column in dbo.Device = 1 EnrollmentCategoryID column in dbo.DeviceExtendedProperties = NULL or 0 Query:

select d.ManagedBy, dep.EnrollmentCategoryID, d.DeviceID from dbo.Device d INNER JOIN
dbo.DeviceExtendedProperties dep
ON
d.DeviceID = dep.DeviceID
where d.ManagedBy = 1
AND dep.EnrollmentCategoryID IN (NULL, 0)

EnrollmentCategoryID is mapped to following enum:

public enum EnrollmentCategory
{
    Unknown = 0,
    VmWorkspace = 1,
    Offline = 2,
    WorkSpaceOneDirectEnrollment = 3
}

ManagedBy is mapped to following enum:

public enum DeviceManagedBy
{
    /// <summary>
    /// Unknown value
    /// </summary>
 
    Unknown = 0,
 
    /// <summary>
    /// Managed By MDM
    /// </summary>
 
    MDM = 1,
 
    /// <summary>
    /// Managed by Workspace (newer terminology is 'Container')
    /// </summary>
 
    Workspace = 2,
 
    /// <summary>
    /// Managed by Application Catalog (Web MAM enrollment)
    /// </summary>
 
    AppCatalog = 3,
 
    /// <summary>
    /// Managed by Application
    /// </summary>
 
    AppLevel = 4,
 
    /// <summary>
    /// managed by VmWorkspace
    /// </summary>
 
    VmWorkspace = 5,
 
    /// <summary>
    /// managed by Offline (Enrolled Offline)
    /// </summary>
 
    Offline = 6
}

ENSv2 Database

ENS Database Schema

The ENS system leverages Amazon RDS(SQL Server) instance for its data persistence needs and makes use of 3 main tables.

EnsSubscriptionInfo

Keeps all user records with keys needed for decryption of url callbacks and device tokens to send messages to

APITokens

Keeps all valid API tokens for authenticating with the API

EWSUrlCache

Security

For security, apart from leveraging best practices defaults followed by Amazon on their RDS instances, the following has been enforced to protect customers data and integrity of the system:

  1. Database never contain user’s emails or passwords. All identification is done via the userId that gets matched to a public/private key pair
  2. Database access by ENS machines is limited via a single AWS IAM account with limited role, it has read/write records permission but no database/table level access.

API/Services Client Authentication

Authentication by clients is done via API keys created by us. The API token is sent as a header parameter (key: “ApiToken”) for all outgoing connections and authenticated by ENS server by checking against a table of authorized API keys. The api key is provided to the clients via a configuration pushed down by the console. Each VMware/Airwatch client is provided 1 API key for all their devices.
API Keys can be revoked and invalidated as explained in the API Key Revocal/Refresh section.

Boxer client activation via console

There are some keys that can be configured via console to activate the feature on Boxer iOS. These are described in Managed App Configuration

API/Services Provided by ENS Server to client devices.

Alive: Used to determine if ENS the service is up and running.

Task Endpoint Description Params
REQUEST/GET /alive Check if the system is currently up -

Get Public Key:

Gets a public key from the server.

Task Endpoint Description Params
REQUEST/POST /getpublickey Based on email passed, ENS server creates a record with a public/private key pair and a userId. At this point no devices are added and registered = false userid = sha256Hash( lowercase(email))
RESPONSE { pubKey = <public key value including header/footer>, userid = , responseCode = <response_code> }

Register Device V2:

Register devices so ENS starts monitoring its email Inbox Task: REQUEST/POST Endpoint: /registerdevicev2 Description: Based on credentials passed, ENS should now autodiscover what is the EWS endpoint it needs to subscribe to for push notifications. Upon receiving a success it will update the registered column in the DB to true and the endpoint as well. Any errors should be reported to the client. Note: We can force a registerdevicev2 call to ignore current user status (including Subscribed) and force a resubscription by putting a “force” flag equal to 1 in the payload Params:

userid = sha256Hash( lowercase(email))

email = email address of the user to aid in autodiscovery.

*creds = {
apnsToken: <device apns token>,
ewsUrl: <EWS endpoint url>,
**oauthAccessToken: <oauth access token>,
**userName: <username for account>,
**password: <password for accounts>,

**CBAData: {
certificate: <pkcs12 certificate>
password: <certificate password>
}
}

bundleid = string with application id
exchVersion = integer specifying which exchange version is to be used first on autodiscovery
force = parameter that forces the server to ignore the users current status (subscribed) and redoes the registration completely
defaultSound = string with soundname to be returned back on the notification payload.
ewsDeviceId = Eas device identifier which will be sent to SEG to evaluate the policies, depending on allowed status we will send the notification to device.

*parameter is encrypted with publickey and encoded in base 64. base64(RSAEncryptWithKey(creds))  
**optional values depending on authentication type

Response:

{  
responseCode = <response_code>
}

Unregister:

Tells the server that we want to remove a specific device from using push notifications. Task: REQUEST/POST Endpoint: /unregister Description: Based on userId and device token passed ENS will update corresponding record.

  1. Get record and list of devices
  2. Remove device, and Update Record
  3. If its last device on list unregister from Exchange push subscription, and delete record Params:
userid = sha256Hash( lowercase(email))  
token =  
base64(RSAEncryptWithKey(apnstoken, ensPublicKey))

Response:

{  
responseCode = <response_code>
}

Update Device:

Task: REQUEST/POST Endpoint: /updatedevice Description: Based on userId and device token passed ENS will update corresponding record.

  1. Get record and list of devices
  2. Replace old token for new one
  3. Update Record or delete record Params:
userid = sha256Hash( lowercase(email))  
token =  
base64(RSAEncryptWithKey(apnstoken, ensPublicKey))

Response:

{  
responseCode = <response_code>
}

### Get Subscription Status V1 and V2:
This service let's us "checkin" with ENS to keep our subscription active as well as help clients know if they need to resubscribe. This is also helpful for debugging purposes
Task: REQUEST/POST
Endpoint: /getsubscriptionstatus
Description: Based on userId passed, ENS checks what is the corresponding status for that account
Params:
```json
userid = sha256Hash( lowercase(email))  
lastMessageDate = string date "MM/dd/yyyy HH:mm:ss.SSS"

Endpoint: /getsubscriptionstatusv2 Description: Based on userId-devicetoken passed, ENS checks what is the corresponding status for that account. This was introduced so we could have more granularity and be able to better cleanup “stale” records for users who may not be using device anymore Params:

userid = emailhash string  
*subscriptionPayload = {  
pushToken: <apnsToken>  
lastMessageDate: <string date "MM/dd/yyyy HH:mm:ss.SSS"> ignoreLastMessageDate: <1 or 0>
}

*parameter is encrypted with publickey and encoded in base 64. base64(RSAEncryptWithKey(subscriptionPayload))  
= base64(RSAEncryptWithKey(apnstoken, lastMessageDate)) note: lastMessageDate format = "MM/dd/yyyy HH:mm:ss.SSS"

Update Sync Key:

Register devices so ENS starts monitoring its email Inbox Task: REQUEST/POST Endpoint: /updatesynckey Description: Based on information sent, we update the background sync key and expiration time of the system which is sent down to the device with each push notification. Params:

userid = sha256Hash( lowercase(email))

*info = {
apnsToken: <device apns token>,
backgroundSyncKey: <new background sync key>,
syncKeyExpirationTime: <expiration time of sync key in UTC "MM/dd/yyyy HH:mm:ss.SSS">
}  

*parameter is encrypted with publickey and encoded in base 64. base64(RSAEncryptWithKey(info))

Extra Header parameters:

On each request we can send some extra parameters which influences how ENS handles the subscriptions as well.

  1. badge - Whenever we send this on the header of a request (with value 1) we know that this device needs to be badged and hence ENS will include the badge on every notification that it sends down
  2. development - This header needs to be included (with value 1) to know that this request is coming from a development device/build. This helps in sending the push notification to the correct push notification environment among other things we do server side that are special to development devices.

Response Codes for applicable services:

• UserSubscriptionNotFound
• UserRecordPresentNotSubscribed • UserAlreadySubscribed
• UserUpdated
• UserSubscribed
• DeviceUnregistered • UpdateSuccess • UpdateFail

API Key Revocal/Refresh:

ENS server machines keep a cache of all API keys to speed up the process of validating every single call they receive. They refresh their keys on a schedule so that we can confidently remove api tokens and make them unusable

Encryption:

The ENS system uses asymmetric key encryption to keep all customer data transferred safe and invulnerable to various types of malicious attacks. Specifically, we use RSA Encryption to encrypt any crucial data passed around from client/server (see API/Services section for details).

A public key is created per user and is provided to the client upon authenticated request. Using this key, RSA encryption following PKCS1 standards needs to be performed on the client side for any services requiring it.

GPS History Table

  • GPS Poll Time Interval - X mins (configured in AirWatch Console, for example, in Android Agent settings)
  • Data Transmit Interval - Y mins

Privacy settings for “GPS Data” needs to be set to Collect and Display for required device Ownership type.

The samples which are transmitted by the Agent as per the Y interval are all stored in the database in the dbo.GPSLog table. To see if any samples have been reported for your specific device, use the following query:

SELECT * from dbo.LogSample (nolock) ls
inner join dbo.GPSLog (nolock) gl
on ls.logsampleID = gl.logsampleid
WHERE deviceid = ####

Under device details > Location tab on console, not all location data points will be reported:

  • Consecutive duplicate samples will not be reflected
    Note

    Consecutive samples which are not at least 100 ft. apart will not be listed corresponding to the particular time-stamp as can be seen in the DB.

Insert GPS History directly

Copy GPS coordinates from one device to another:

select * from device where Name like '%User01%' --deviceID = 7
select * from device where Name like '%User02%' --deviceID = 52
  
Insert dbo.LogSample ([DeviceUid] ,     [IpAddress] ,[SampleTime],[TransmitTime] ,[DeviceID] )
select      
--[LogSampleID] [int] IDENTITY(1,1) NOT NULL,
             [DeviceUid] ,[IpAddress] ,[SampleTime],[TransmitTime] ,
             52--[DeviceID] ,
       from dbo.LogSample  ls where deviceId = 7 and LogSampleID =8
  
  
select *  from dbo.LogSample  ls where deviceId = 52 --LogSampleID =9
  
Insert dbo.GPSLog ( [LogSampleID] ,[SampleTime],[Latitude] ,[Longitude] ,[Elevation] ,[Speed] ,
[Note] ,[LatitudeInternal] ,[LongitudeInternal] ,[ElevationInternal] ,       [SpeedInternal]
--,    [RowVersion]
                    )
       select 9 ,--[LogSampleID] ,      
             [SampleTime],[Latitude] ,[Longitude] ,  [Elevation] ,[Speed] ,[Note] ,[LatitudeInternal] ,[LongitudeInternal] ,     [ElevationInternal] ,     
[SpeedInternal] --, [RowVersion]
       from dbo.GPSLog  gl where LogSampleID in (select  LogSampleID from dbo.LogSample  ls where deviceId = 7and LogSampleID =8)

MEM Tables

What information is saved to the SQL database for the MEM module during enrollment and profile deployment.

The SQL flow will be different depending on whether you use SEG, Powershell, or Google Apps for Work.

SEG & Powershell SQL Flow Diagram

graph TD D -->|1| AW AW[WS1 UEM] -->|2| D[Device] D -->|3| AW AW --> DB[(DB: 1,3)]
  1. The device enrolls in the VMware Workspace ONE. We save some information to the SQL database;
  2. A profile is pushed down to the device;
  3. Sample data is sent from the device. Data is saved to the SQL database.
    Note

    Usually by the time you run a SQL query, all of these steps have already occurred as this happens very quickly.

Google Apps for Work SQL Flow Diagram

graph TD D -->|1| AW AW --> DB[(DB: 1,1)]
  1. The device enrolls in VMware Workspace ONE. We only save password 1 if the customer is
    using Google Apps direct integration. If the customer is using Google Apps with password retention, we also save password 2 to the database.

Useful SQL Queries

You can see what information is saved in the SQL database by running SQL queries. Our primary SQL table is mobileEmailGateway.MEMDevice but we also use other tables including MobileEmailGateway.MEMDeviceActivity, mobileEmailGateway.EasDeviceType, and mobileEmailGateway.MEMConfigproxy to name a few. Each entry in the MEMDevice table relates to one email client. You can think of this table like a giant Microsoft Excel spreadsheet where MEMDeviceID is the row number.

List Information on MEMDevice

For our first query, we want to see the details about all the email clients associated with a specific device. For any SQL troubleshooting issue, this is generally the first query you will want to run.

select * From MobileEmailGateway.MemDevice (nolock) Where DeviceID = 637368

For this query to work, you will need to know what your DeviceID value is. You can find that value by going to the VMware AirWatch Console.

For the second query, we want to see all the email client entries in the SQL database. You might want to see this data after a client enrolls to troubleshoot various issues.

select * from mobileEmailGateway.MEMDevice where LocationGroupID = 8998

The above query returns information on each entry in the MEMDevice table where the location group is equal to 8998. To run this query, you only need to know the location group ID where the device enrolled. To find the location group ID, you can go to your OG and right-click the OG name and choose “Inspect”. From there you can see the OG group ID on the right. See the image below: The result of this query shows you the following information for each MEMDeviceID.

  • MEMDeviceID: You will have a different MEMDeviceID for each email client a user has. You need to have 3 things to have a new MEMDeviceID. DeviceID, EasDeviceIdentifier, and EmailAddress.
  • EasDeviceIdentifier: This is the Exchange ActiveSync ID. Each mail client will have a unique ID for this that can be used to identify the device on the Exchange Server.
  • DeviceID: This is the DeviceID the device received when it enrolled. Each device will have a unique ID that our infrastructure will use to manage the device in the console.
  • IsManaged: This value indicates whether or not the email client is managed or not by the VMware UEM console. This is different for IsEnrolled as that refers to the device, IsManaged is referring to the email client.
  • LocationGroupID: This is the ID of the location group where the user enrolled.
  • AgentReportedPackageName: The name of the email client. For native iOS, this value will always be “NULL”.
  • IsEnrolled: Indicates whether or not the device is enrolled in the VMware UEM. This is different for IsManaged as that refers to the email client, IsEnrolled is referring to the device.
  • IsCompromised: Indicates whether or not the device is jailbroken or rooted.
  • IsDataProtected: Indicates whether the device is encrypted or not.
  • IsModelCompliant: If you are using the managed device policy for Model (under Email\Compliance Policies), this indicates whether or not this device is compliant based on that.
  • IsOSCompliant: If you are using the managed device policy for Operating System (under Email\Compliance Policies), this indicates whether or not this device is compliant based on that.
  • IsMDMCompliant: If you are using any device compliance policies (under Devices\Compliance Policies\List View), this indicates whether or not the device is compliant based on that.
  • AccessState: Based on the evaluation of all of the compliance policies that might affect the device or email client, this indicates if the device is allowed or blocked.
  • DiagnosticsEnabled: Unknown.
  • DiagnosticsEnabledOn: Unknown.
  • EmailAddress: The email address associated with the email client.
  • EasMailboxIdentity: The returned Exchange ActiveSync mailbox identify if available.
  • EasMailboxDisplayName: The returned Exchange ActiveSync mailbox display name if available.
  • EasDeviceGUID: Unknown.
  • EasDeviceTypeID: The Exchange ActiveSync device type returned from the table mobileEmailGateway.EasDeviceType.
  • EasDeviceOS: Displays the operating system version information when available. This appears to usually be “NULL”. If you don’t find any data here, you will usually find this information in EasDeviceFriendlyName.
  • EasDeviceUserAgent: Usually “NULL” but may contain additional details about the email client when available.
  • EasDeviceFriendlyName: Displays model and version information.
  • EasDeviceIdentity: Usually “NULL” or 0. Uknown.
  • EasDeviceImei: Displays the devices IMEI information.
  • CreatedOn: The date the SQL records was created.
  • ModifiedOn: The last date the SQL record was modified.
  • EasProfileInstall: If you are using the managed device policy for Require ActiveSync Profile (under Email\Compliance Policies), this indicates whether or not a valid profile has been pushed. If a profile has never been pushed, this will be “NULL”. You will see a value of “1” if a valid profile has been pushed and a value of “0” if a profile has been removed.
  • LastDeviceStateChangeType: Unknown.
  • ResoruceId: Usually “NULL”. Unknown.

List Information on MEMDeviceActivity

If you are using Powershell, you might want to see what the status of the last Powershell command was after enrollment. In the case of SEG, you can use this to see if the last single device policy update failed or not. To do that, use the following query.

select * From MobileEmailGateway.MEMDeviceActivity (nolock) Where MemDeviceId = 927453

The above query returns information on each entry in the MeMDeviceActivityID table where the memDeviceID is equal to 927453. To run this query, you only need to know the memDeviceID. You can get this value by running the query from the “List Information on MEMDevice”.

The result of this query shows you the following information for each MemDeviceActivityID.

• MemDeviceActivityID: You will have a different MemDeviceActivityID for each activity transaction in the SQL database.
• MemDeviceId: You will have a different MemDeviceId for each email client a user has. You need to have 3 things to have a new MEMDeviceID. DeviceID, EasDeviceIdentifier, and EmailAddress.
• MemConfigId: When you create a new MEM configuration by going to Email > Email Settings > Add, a MEM configuration ID is created in the database. This field indicates the ID associated with this email client. • UserName: Unknown.
• EmailUserIdentity: The email address associated with the account.
• IPAddress: If we were able to determine the IP address for the device, it will be listed here.
• MailClientName: If we were able to determine the name of the mail client, you will see it listed here.
• DeviceAccessStateReason: If we are able to see the reason why the device is allowed or blocked, you will see it here.
• AllowReason: If the device is allowed, you will see the reason it is allowed here if available.
• BlockedReason: If the device is blocked, you will see the reason it is blocked here if available.
• LastGatewayServer: This will list the SEG server that was used last by the mobile email client.
• LastCommand: This will display the status of the last command we issued to the SEG or the Powershell endpoint.
• LastRequestDateTime: This will show the date that the email client last made a request.
• LastUpdate: This will display the last update response from the SEG or Powershell endpoint.

List Information on MEMConfig

select * From MobileEmailGateway.MEMConfig (nolock) Where MemConfigId = 8

The above query returns information on the MEM configuration where the MEM configuration ID is equal to 8. The result of the query shows you the following information for individual MEM configurations.

  • MemConfigId: This is the unique ID that each MEM configuration is assigned. This is an auto-generated value and is unique within an environment.
  • LocationGroupId: This is the Location Group Id where the MEM configuration is created.
  • DeploymentId: This column can hold two values. 1 corresponds to a Proxy configuration while 2 corresponds to a Direct configuration.
  • ProxyTypeId: Unknown.
  • EmailTypeID: The value in this column corresponds to the type of email infrastructure. Exchange = 2, IBM Notes = 5, Google = 7 .
  • EmailVersionID: This corresponds to the email server version. Example: Exchange 2010 = 4, Exchange 2013 = 5, Exchange 2016 = 6, Exchange Online = 7.
  • DisableCompliance: This column can hold two values: 0 = Email compliance engine is enabled, 0 = Email compliance engine is disabled. ◦ NOTE: This is not specific to the MEM configuration but is a property of the location group where the MEM configuration is present.
  • FriendlyName: This is the name given to the MEM configuration.
  • IsEnabled: This column can holds two values: 1 = MEM configuration is active/enabled, 0 = MEM configuration is inactive/disabled.
  • TestModeEnabled: This column holds two values. 1 = Test mode is enabled, 0 = Test mode is disabled. ◦ When Test mode is enabled, you can test Email Compliance Policies without affecting email access for your devices. All devices are allowed for email and the Email Dashboard reflects the expected access state results of your Email Compliance Policies on these devices. This only applies to SEG.
  • UseRecommendedSettings: This column hold two values: 1 = The MEM configuration is using the Recommend Settings under the Advanced Section. 0 = The adminstrator has overriden the ‘Use Default Settings’ option under Advanced Settings.
  • MEMconfigGuid: This is the unique ID assigned to the MEM configuration. This is separate from the MEMConfigId.

Memcached Activation

Activate usage of Memcached

DECLARE @myid uniqueidentifier
DECLARE @GroupOverrideID int
DECLARE @userID int
DECLARE @CacheEndpoint varchar(255)
DECLARE @EnabledOverrideID int
DECLARE @SolutionOverrideID int
DECLARE @EndpointOverrideID int
SET @myid = NEWID()
SET @userID = 52
SET @CacheEndpoint = '[
  {
    "name" : "mc_node1",
    "address" : "192.168.1.1",
    "port" : "11211"
  },
  {
    "name" : "mc_node2",
    "address" : "192.168.1.2",
    "port" : "11211"
  }
]
'
EXEC SystemCodeGroupOverride_Save @GroupOverrideID, 132, 7, NULL, 1, 1, 1, @UserID
IF (@GroupOverrideID IS NULL)
BEGIN
    SELECT TOP (1) @GroupOverrideID =  scgo.SystemCodeGroupOverrideID
    FROM dbo.SystemCodeGroupOverride scgo
    WHERE [SystemCodeGroupID] = 132
    AND LocationGroupID=7
    AND LocationID IS NULL
END
EXEC SystemCodeOverride_Save @EnabledOverrideID, 877, @GroupOverrideID, 'True', @userID
EXEC SystemCodeOverride_Save @SolutionOverrideID, 4147, @GroupOverrideID, '0', @userID
EXEC SystemCodeOverride_Save @EndpointOverrideID, 829, @GroupOverrideID, @CacheEndpoint, @userID
UPDATE dbo.DatabaseVersion set buildkey = CONVERT(varchar(255), @myid)
DELETE from dbo.SystemCodeOverride where SystemCodeid in (829,877,4147)
DELETE from dbo.SystemCodeGroupOverride where SystemCodeGroupID = 132

Check Memcached in SQL

SELECT * from dbo.SystemCodeOverride where SystemCodeID IN (829 , 827, 877, 4147)

Query Internal Apps

Check app versions

Find out the list and version of applications installed on the device by running the following SQL query:

select * from interrogator.applicationlist al
inner join interrogator.application a
on a.applicationid = al.applicationid
where al.deviceid = #affecteddeviceid

How to verify if devices have installed the latest provisioning profile for an internal application?

Search for devices who are reporting the application as installed, but do not have the updated provisioning profile installed

SELECT * FROM interrogator.ApplicationList (NOLOCK)
WHERE DeviceID in (
    SELECT iaa1.DeviceID From deviceApplication.InternalAppAssignment (NOLOCK) iaa1
    WHERE InternalAppID = {INTERNAL_APP_ID}
    AND iaa1.DeviceId NOT IN (
        SELECT iaa.DeviceID FROM deviceApplication.InternalAppAssignment (NOLOCK) iaa
        JOIN deviceProfile.DeviceProfileDevicePool dpdp
        ON iaa.DeviceID = dpdp.DeviceID
        WHERE iaa.InternalAppID = {INTERNAL_APP_ID}
        AND dpdp.DeviceProfileID = {DEVICE_PROFILE_ID}
    )
)
AND ApplicationID = {APPLICATION_ID}
AND IsInstalled = 1

The logic behind this script is:

  • Check for devices reporting the provisioning profile as installed (deviceProfile.DeviceProfileDevicePool)
  • Check against devices that are supposed to be assigned this application (deviceApplication.InternalAppAssignment)
  • Check against devices that are reporting the application as installed (interrogator.ApplicationList)

To find the variables in this query:

  • INTERNAL_APP_ID - This is found in the deviceApplication.Application database table, or in the URL when viewing the summary page for the internal application.
  • DEVICE_PROFILE_ID - This is found in the deviceApplication.Application table as the ProvisioningDeviceProfileID.
  • APPLICATION_ID - This is found in the interrogator.Application table as the ApplicationID.  To find this value you can use the following query:
SELECT ia.* FROM interrogator.Application (NOLOCK) ia
JOIN DeviceApplication.Application daa
    ON ia.Identifier = daa.PackageID
    AND ia.VersionHash = daa.VersionHash
WHERE daa.ApplicationID = {INTERNAL_APP_ID}

Verify if commands are in the command queue to update the provisioning profile

The following query can be used to identify if, for devices that have not yet installed the updated profile, commands have been queued to install it on the next device check in:

SELECT * FROM DeviceCommandQueue.DeviceQueue (nolock) dq
JOIN deviceProfile.DeviceProfileVersion dpv
    ON dq.DeviceProfileVersionID = dpv.DeviceProfileVersionID
WHERE dq.CommandID = 13 --CommandID for InstallProvisioningProfile
AND dpv.DeviceProfileID = {DEVICE_PROFILE_ID}

Search for application upload events

The following query is generated by HUB -> Reports and Analytics -> Events -> Console Events in the console, but the console only searches for Last Month, while manual procedure allows to search further back

exec eventLog.EventLog_Search @LocationGroupID=571,@SeverityID=NULL,@eventGroupTypeID=2,@Module=N'Apps',@StartDateTime='2018-06-01 15:32:34.843',@EndDateTime='2018-07-27 15:33:34.843',
@EventDirectionID=NULL,@EventTypeID=0,@EventActionID=0,@StartRowIndex=50,@MaximumRows=1000,@SortExpression=NULL,@SortAscending=1,@SearchText=NULL

Smart Groups Queries

SQL queries for Smart Groups

List all the smart groups present in that particular environment:

select * from smartgroup.smartGroup

Current version of the smart group being used can be found in the table generated by the query below:

select * from smartgroup.SmartGroupVersion where sgdm.SmartGroupId = ##

Details of devices associated with a smart group:

select * from SmartGroup.SmartGroupDeviceMap sgdm Inner Join smartgroup.SmartGroupVersion sgv on sgdm.smartGroupId = sgv.smartGroupID
and sgdm.SmartGroupVersion = sgv.SmartGroupVersion where sgdm.SmartGroupId = ##

Relationship between smart group and entity can be found from the table generated by the query mentioned below:

select * from smartGroup.AWEntitySmartGroupAssignmentMap where SmartGroupId = ##

The device ID of device to which the entity is assigned can be obtained from the tables generated by the queries mentioned below:

  • Profile
  • InternalApp
  • ExternalApp
select * from deviceProfile.ProfileAssignment where ProfileID = ##
select * from [deviceApplication].[InternalAppAssignment] where InternalAppID = ##
select * from [deviceApplication].[ExternalAppAssignment] where ExternalAppID = ##

Devices associated with a smart group can be seen in the table generated by the query mentioned below:

select batchEvaluateLogic from smartGroup.smartGroupRuleLogic where smartGroupOverrideID = ##
Note

Apps and books are together considered as an app in SQL.

Troubleshooting Smart Groups

Warning

Do not name a Smart Group with the same name as an existing User Group - this leads to Smart Group being “locked” from deletion

Problem 1:

  • After assigning a smart group to the application and selecting Save & Publish , the app assignment in application grid shows 0/0/0 even though the View Device assignment shows all devices in smart group.

Go to Deployment tab and check the deployment time and if possible change it to 12.00 AM same day. If the time is already set at 12.00 AM, check the admin user’s time zone and compare it with OG time zone. Also check if app wrapping is enabled and whether it is still in progress or not completed.

Problem 2:

  • When a smart group is attempted to be deleted, the following error appears: " Deletion is unsuccessful. This smart group is currently being used in assignment. Please remove the smart group from the assignment, and try again."

Check if the smart group is currently assigned to any app, profiles, compliance etc. Check if there is any user group which is having the same name as the smart group which the smart group uses.

Problem 3:

  • Certain apps are not being shown in App Catalog for some users.

Create a new smart group which contains these users. After that edit the app and add this smart group in exclusion list.

SQL Recovery

General

VMware AirWatch Installation Guide (9.2), page 17 recommends Recovery Model = Full.

Also, in AirWatch SQL FAQ we see a comment: When trying to login to the AirWatch console or perform actions within the console you may receive the error message “The transaction log for database ‘AirWatch’ is full.”

This typically means that there is no available space on your SQL server. The common cause for this is backups not being deleted even though newer backups are being done, or lack of transaction log backups. To fix this issue you should first consult your local database admin and make them aware of the issue. The fix that we recommend is to perform a full database backup and then switch the database to simple recovery mode. Your database admin should be familiar with this and know how to perform these actions.

Shrinking the Log File When Disk Space is Full

The log file cannot be shrunk until it is truncated and free space is available. If a log file has grown to the point that it is full or consuming all available disk space, truncate it immediately in order to restore normal database operations. (If the log file has reached the maximum size database property and free disk space is still available, increase the value of maximum size property or set it to ‘unlimited’. A shrink operation should not be required in this case.)

To truncate a log file immediately, set the recovery model to Simple then shrink log file. The following commands are examples of how to do this. They may require editing to tailor them to the customer’s environment.

-- look up the name of the log file
 SELECT name as log_file_name FROM sys.database_files WHERE type_desc='LOG'
ALTER DATABASE [AirWatch] SET RECOVERY SIMPLE; 
DBCC SHRINKFILE ('AirWatch_log',0,TRUNCATEONLY);

-- replace 'log_file_name' with the actual name of the log file
 ALTER DATABASE [AirWatch] MODIFY FILE ( NAME = AirWatch_log, SIZE = 10GB , FILEGROWTH = 128MB );
ALTER DATABASE [AirWatch] SET RECOVERY FULL;

-- Full backup
 BACKUP DATABASE [AirWatch] TO DISK = 'E:\MSSQL\Backup\AirWatch_20160123_full.bak' WITH STATS; 

Setting the recovery model to Simple erases all of the transaction log records. For this reason, it is essential to create a full backup of the database immediately afterward the shrink procedure to ensure full data recoverability in the event of a database failure.

Preventative Measures

Make sure that the regular database and transaction log backups are scheduled for the AirWatch database. There are various methods: SQL scripts, third party tools, or SQL Server maintenance plans, which can be used.

Full Recovery Mode caveats

Generally the reason why DB log file is large is because the required DB maintenance has not been implemented. Typically, the size of the transaction log file stabilizes when it can hold the maximum number of transactions that can occur between transaction log truncations that are triggered by either checkpoints or transaction log backups.

What Scenarios can cause the Log to Keep Growing?
There are many reasons, but usually these reasons are of the following two patterns

Recovery process overview… In SQL Server, there are three recovery models - Full, Bulk-Logged and Simple. We’ll ignore Bulk-Logged hybrid model: most people who are in this model are there for a reason and understand recovery models. The two we care about are Simple and Full.

Before we talk about Recovery Models - Let’s talk about recovery in general.
The transaction log file is there for crash/restart recovery. For the rolling forward and rolling back of work that was either done (rolling forward/redo) before a crash or restart and the work that was started but not finished after a crash or restart (rolling back/undo). It is the job of the transaction log to see that a transaction started but never finished (rolled back or crash/restart happened before the transaction committed). In that situation It is the log’s job to say “hey.. this never really finished, let’s roll it back” during recovery. It is also the log’s job to see that you did finish something and that your client application was told it was finished (even if it hadn’t yet hardened to your data file) and say “Hey.. this really happened, let’s roll it forward, let’s make it like the applications think it was” after a restart. Now there is more but that is the main purpose.

The other purpose for a transaction log file is to be able to give us the ability to recover to a point in time due to an “oops” in a database or to guarantee a recovery point in the event of a hardware failure involving the data and/or log files of a database. If this transaction log contains the records of transactions that have been started and finished for recovery, SQL Server can and does then use this information to get a database to where it was before an issue happened. But that isn’t always an available option for us. For that to work we have to have our database in the right recovery model, and we have to take log backups.

Simple Recovery Model

In this model, you are telling SQL Server - I am fine with you using your transaction log file for crash and restart recovery (You really have no choice there.. Look up ACID properties and that should make sense quickly), but once you no longer need it for that crash/restart recovery purpose, go ahead and reuse the log file.

SQL Server listens to this request in Simple Recovery and it only keeps the information it needs to do crash/restart recovery. Once SQL Server is sure it can recover because data is hardened to the data file (more or less), the data that has been hardened is no longer necessary in the log and is marked for truncation - which means it gets re-used.

Full Recovery Model

With Full Recovery, you are telling SQL Server that you want to be able to recover to a specific point in time, as long as your log file is available, or to a specific point in time that is covered by a log backup. In this case when SQL Server reaches the point where it would be safe to truncate the log file in Simple Recovery Model, it will not do that. Instead It lets the log file continue to grow and will allow it to keep growing, until you take a log backup (or run out of space on your log file drive) under normal circumstances.

If you just switch into Full Recovery mode, but never take an initial Full Backup, SQL Server will not honor your request to be in Full Recovery model. Your transaction log will continue to operate as it has in simple until you switch to Full Recovery Model AND Take your first Full Backup.

So, that’s the most common reason for uncontrolled log growth: Being in Full Recovery mode without having any log backups.

Upgrade Rollback

The rollback process contains two phases.  First, a backup of the database (that was taken prior to the upgrade) is restored and configured.  Next, either backups of any application servers (Console, Device Services, etc) are applied, or the previous version of the AirWatch application is simply reinstalled on those servers after uninstalling the current version.

Restoring a database backup

The following steps can be used to restore a database backup to an existing database server and apply the necessary configuration changes.  This process assumes that a backup has been created prior to any upgrades, when the environment was fully functional.

  • Before working with the database, make sure that, for any AirWatch application servers (Console, Device Services, etc),  all AirWatch services are stopped.  Additionally, stop the World Wide Publishing Service.  Finally, make sure that IIS is stopped as well.
  • Open Microsoft SQL Server Management Studio, right-click on Databases and select Restore Database.
  • Under the General tab on the left and within the Source for restore section, select From device, then select ‘…’ button.
  • From the Specify Backup page, select Add. Locate the backup, select it, and click OK. The database backup will display on the Specify Backup page. Click OK.
  • Select the Restore check box.
  • Under Destination for restore, select the To database drop-down list and select the AirWatch database name. Note: This should be the last database in the list, in the event you have multiple backups all named AirWatch and are unsure of which one to select. Click OK to start the database restoration.

On the old database: Next you need to note the user permissions of the old database’s AirWatch SQL Service Account. To do this:

  • Open Microsoft SQL Server Management Studio
  • Navigate to Security > Logins > to locate your DB User in the Object Explorer, and then right-click and choose Properties.
  • Navigate to the Server Roles tab. Write down the roles listed
  • Select User Mapping. Write down the user mappings listed and the role membership permissions
Warning

Take note of all of the role memberships for AirWatch, master, model, msdb, and tempdb.

On the new database:

  • Delete the AirWatch SQL Service Account, which was created when you restored.
  • Create the new AirWatch SQL Service Account. To do this navigate to Security > Logins,  right-click, and select New Login.
Warning

This procedure is only for the main AirWatch SQL Service Account If you had any other custom-created SQL accounts you will need to perform this procedure for each of them.

  •     Enter the following:

A.   Select whether to use Windows or SQL Server authentication. For SQL Server authentication, enter your user credentials. Note: The username needs to exactly match the username of the old database.

B.   Uncheck Enforce password policy.

C.   Select the AirWatch database as the Default database.

D.   For Server Role, enter the roles you noted previously.

E.   For User Mapping, enter the user mappings and permissions you noted previously. IMPORTANT: This should include all of the permissions that you copied for AirWatch, master, model, msdb, and tempdb.

Next you need to migrate any AirWatch-related jobs.

Warning

The steps below are for the purge job, but any other AirWatch-related jobs need to be similarly migrated using the procedure below

On the old database:

  • Navigate to SQL Server Agent > Jobs, right-click <AirWatch_DB> - Purge Expired Sample Data, and select Script Job as > CREATE To > New Query Editor Window.
  • Save as the query.
  • Transfer the query to your new database.
  • On your new database: Execute the query.
Warning

To reiterate, any other AirWatch-related jobs need to be similarly migrated using the procedure above

On the new database: Perform a test query. For example, one for device count, to ensure proper functioning. To do this:

  • Right-click on the AirWatch database under Databases and select New Query.
  • Enter the query as shown below.

  • Select Execute.

Rename the old database, for example, to AirWatch_OLD. To kill all connections and rename the database, run the following script, replacing ‘AirWatch’ with the name of your old AirWatch DB and AirWatch_OLD with what you would like to rename the old database to.

Restoring a previous version of the application

With the database backup restored, the next step is to restore the proper version of the application.  Ideally, there will be a snapshot or backup of each application server from the same time as the database backup.  In this case, simply restore these backups in order to restore functionality.

However, if there are no available snapshots or backups of the servers, then the previous version of the AirWatch software must be reinstalled.

In these cases, identify which application servers have an updated version of the application (this will likely be all application servers, but may only be a subset of them).  For each of these, ensure that a copy of the AirWatch installer for the correct version has been installed on the server. Uninstall the current version of the software through Computer > Uninstall or change a program by selecting the AirWatch application.  With this uninstalled, open previous version of the AirWatch installer and proceed as normal.  When configuring the database connection, ensure the configurations used match those of the restored database.

Finally, when both the database and application has been restored properly, ensure that IIS and all AirWatch services (as well as the World Wide Web Publishing service) are properly started.

Changing the database connection string

In some cases, it may be necessary to change the database connection string of an application server without fully reinstalling the software.  For example, if a database migration has occurred, the name of the database may have simply changed.  Perform the following steps to update the database connection string on each AirWatch application server.  Note that this must be done on every application server so that they are pointing to the new database.

Note
  • For deployments with dedicated API and AWCM servers: Dedicated API and AWCM servers are considered application servers, similar to the AirWatch Console and Device Services. You should therefore perform the steps below regarding re-pointing app servers on these servers if you have dedicated servers for these components.
  • EIS, SEG, ACC/ESC are considered auxiliary components and you do not need to perform this step for these components.

Steps:

  • Navigate to AirWatch Root Folder on the application server.
  • Navigate to AirWatch X.X\Supplemental Software\Tools\UpdateSQLServerInfo.
  • Launch UpdateSQLServerInfo.exe.
  • Update the Server Hostname, Database Name, Username and Password. If Windows authentication is being used, the password field may be blank.
  • Make sure to restart IIS and all AirWatch services on each server after updating the SQL connection string.

Useful SQL Queries

SQL Database Health

Index Fragmentation Audit

This query provides you with the index fragmentation % for every table in the DB.

/**************************************************************
   You can use the following script to determine index 
   fragmentation by table. This can help to determine 
   that the database is causing performace problems.
**************************************************************/ 

SELECT OBJECT_NAME(object_id), index_id, index_type_desc, index_level, avg_fragmentation_in_percent, avg_Page_space_used_in_percent, page_count 
FROM sys.dm_db_index_physical_stats(DB_ID(N'AirWatch'), null, null, null, 'SAMPLED')
ORDER BY avg_fragmentation_in_percent DESC

Table Size Audit

This query will provide physical sizing information of all the tables in the database.

/**************************************************************
    This query can help you determine what table is taking
    up the most disk space and potentially what tables have
    too many rows.
**************************************************************/ 

--If the temp table exisits drop temp table
IF EXISTS ( SELECT * FROM sys.objects 
     WHERE object_id = OBJECT_ID(N'#Sizes') 
     AND type in (N'U') ) 
     BEGIN 
     DROP TABLE #Sizes; 
     END 

--Create temp table, #Sizes 
CREATE TABLE #Sizes 
( 
--Table name 
     table_name nvarchar(255), 
--Number of rows 
     table_rows char(11),
--Physical space table is using
     table_reserved varchar(18),
--Phyiscal space table data is using
     table_data varchar(18),
--Physical space idexes are using
     table_index_size varchar(18),
--Physical space reserved
     table_unused varchar(18)
)

EXEC sp_MSforeachtable @command1='INSERT #Sizes (table_name, table_rows, table_reserved,
table_data, table_index_size, table_unused) EXEC sp_spaceused ''?'''
--This query orders the results by actual phyisical table size
SELECT * FROM #Sizes ORDER BY CAST ( SUBSTRING (table_data, 0, LEN(table_data) - 2) AS InT) DESC
--This query orders the results by row count
--SELECT * FROM #Sizes ORDER BY CAST (table_rows AS int) DESC

Table Row Count Only

A more efficient script to get table row counts.

/**************************************************************
    This is a more efficient way to get Row Counts
    but will not include any Physical Sizing data
   
**************************************************************/ 

SELECT 
     sc.name +'.'+ ta.name TableName
     , SUM(pa.rows) RowCnt
FROM 
     sys.tables ta
     INNER JOIN sys.partitions pa
          INNER JOIN sys.schemas sc
          ON ta.schema_id = sc.schema_id
WHERE 
     ta.is_ms_shipped = 0 
      AND pa.index_id IN (1,0)
GROUP BY 
     sc.name
     , ta.name
ORDER BY 
     SUM(pa.rows) DESC;

Database IO Stalls

You can use this query to see which queries are causing IO stalls.

/**************************************************************
   The table valued dynamic management function, 
   sys.dm_io_virtual_file_stats provides a breakdown of SQL 
   Server reads, writes, and io_stalls for a particular
   database or transaction log file. IO_stalls is the total 
   cumulative time, in milliseconds, that users waited for
   I/O to be completed on the file since the last restart    of SQL Server. 
**************************************************************/ 

SELECT
     DB_NAME(fs.database_id) AS [DB Name]
     , fs.file_id
     , mf.physical_name
     , io_stall_read_ms
     , num_of_reads
     , CAST(io_stall_read_ms / ( 1.0 + num_of_reads ) AS NUMERIC(10 , 1)) AS 'avg_read_stall_ms'
     , io_stall_write_ms
     , num_of_writes
     , CAST(io_stall_write_ms / ( 1.0 + num_of_writes ) AS NUMERIC(10 , 1)) AS 'avg_write_stall_ms'
     , io_stall_read_ms + io_stall_write_ms AS io_stalls
     , num_of_reads + num_of_writes AS total_io
     , CAST(( io_stall_read_ms + io_stall_write_ms ) / ( 1.0 + num_of_reads
          + num_of_writes ) AS NUMERIC(10 ,
          1)) AS 'avg_io_stall_ms'
FROM
     sys.dm_io_virtual_file_stats(NULL , NULL) AS fs
INNER JOIN sys.master_files AS mf
          ON fs.database_id = mf.database_id
          AND fs.[file_id] = mf.[file_id]
ORDER BY
     avg_io_stall_ms DESC ;
GO

Identify Expensive Operations

/**************************************************************
   This query provides you with operations that are expensive
   from a database standpoint. This query is useful in
   determining what is causing performance problems on a server
**************************************************************/ 

SELECT TOP 25
     DB_NAME(qp.[dbid]) AS dbname
     , qp.[dbid]
     , qp.objectid
     , qp.number 
     --, qp.query_plan 
     --the query plan can be *very* useful; enable if desired 
     , qt.[text]
     , SUBSTRING(qt.[text] , 
          ( qs.statement_start_offset / 2 ) + 1 ,
          ( ( CASE statement_end_offset
          WHEN -1 THEN DATALENGTH(qt.text)
          ELSE qs.statement_end_offset
     END - qs.statement_start_offset ) / 2 ) + 1) 
         AS statement_text
     , qs.creation_time
     , qs.last_execution_time
     , qs.execution_count
     , qs.total_worker_time / qs.execution_count 
         AS avg_worker_time
     , qs.total_physical_reads / qs.execution_count 
         AS avg_physical_reads
     , qs.total_logical_reads / qs.execution_count 
         AS avg_logical_reads
     , qs.total_logical_writes / qs.execution_count 
         AS avg_logical_writes
     , qs.total_elapsed_time / qs.execution_count 
         AS avg_elapsed_time
     , qs.total_clr_time / qs.execution_count
     nbsp;   AS avg_clr_time
     --, qs.total_worker_time 
     --, qs.last_worker_time 
     --, qs.min_worker_time 
     --, qs.max_worker_time 
     , qs.total_physical_reads
     , qs.last_physical_reads
     , qs.min_physical_reads
     , qs.max_physical_reads 
     --, qs.total_logical_reads 
     --, qs.last_logical_reads 
     --, qs.min_logical_reads 
     --, qs.max_logical_reads 
     --, qs.total_logical_writes 
     --, qs.last_logical_writes 
     --, qs.min_logical_writes 
     --, qs.max_logical_writes 
     --, qs.total_elapsed_time 
     --, qs.last_elapsed_time 
     --, qs.min_elapsed_time 
     --, qs.max_elapsed_time
     --, qs.total_clr_time 
     --, qs.last_clr_time 
     --, qs.min_clr_time 
     --, qs.max_clr_time 
     --, qs.[sql_handle] 
     --, qs.statement_start_offset 
     --, qs.statement_end_offset 
     --, qs.plan_generation_num 
     --, qp.encrypted 
FROM
     sys.dm_exec_query_stats AS qs
CROSS APPLY sys.dm_exec_query_plan(qs.plan_handle) AS qp
CROSS APPLY sys.dm_exec_sql_text(qs.[sql_handle]) AS qt
     --sample WHERE 
WHERE
     last_execution_time > '20120912 12:15'
     AND creation_time > '20130101'
     AND execution_count > 10
     --AND SUBSTRING(qt.text, (qs.statement_start_offset/2) + 1,
     --((CASE statement_end_offset 
     --WHEN -1 THEN DATALENGTH(qt.text)
     --ELSE qs.statement_end_offset END 
     -- - qs.statement_start_offset)/2) 
     -- + 1)
     -- LIKE '%MyText%'

     --sample ORDER BY
     --ORDER BY qs.execution_count DESC --Frequency
     --ORDER BY qs.total_worker_time DESC --CPU
     --ORDER BY avg_worker_time DESC --CPU
     --ORDER BY qs.total_elapsed_time DESC --Durn
     --ORDER BY qs.total_logical_reads DESC --Reads 
     --ORDER BY qs.total_logical_writes DESC --Writes
     --ORDER BY qs.total_physical_reads DESC --PhysicalReads 
     --ORDER BY avg_worker_time DESC --AvgCPU
     --ORDER BY avg_elapsed_time DESC --AvgDurn 
     --ORDER BY avg_logical_reads DESC --AvgReads
     --ORDER BY avg_logical_writes DESC --AvgWrites
ORDER BY
     avg_physical_reads DESC --AvgPhysicalReads

General SQL Queries

Table search by column name

This query allows you to see what tables / procedures have a specified column name in them.

/**************************************************************
   Example(s) of common column names:
   DeviceId, LocationGroupId, ApplicationId, ProfileId, 
   CoreUserId
**************************************************************/ 

SELECT * FROM sysobjects WHERE id IN 
      (SELECT id FROM syscolumns WHERE name LIKE '%ColumnName%')

SP_WHO2 Advanced Query

This allows you to filter on SP_WHO2, which helps when backing up and restoring the AirWatch database.

/**************************************************************  
   This query may look complex but can really  
   narrow down if something is connected to   
   AirWatch or not. When you are upgrading  
   or restoring a database you need to ensure  
   that nothing is locking the databse. **************************************************************/   
CREATE TABLE #sp_who2(   
     SPID INT,   
     Status VARCHAR(1000) NULL,   
     Login SYSNAME NULL,   
     HostName SYSNAME NULL,   
     BlkBy SYSNAME NULL,   
     DBName SYSNAME NULL,   
     Command VARCHAR(1000) NULL,   
     CPUTime INT NULL,   
     DiskIO INT NULL,   
     LastBatch VARCHAR(1000) NULL,   
     ProgramName VARCHAR(1000) NULL,   
     SPID2 INT   
)   
  
INSERT INTO #sp_who2 EXEC sp_who2   
  
SELECT *   
FROM #sp_who2   
WHERE DBName LIKE '%AirWatch%'  
GO   
  
DROP TABLE #sp_who2   
GO

Get all devices from Location Group

This query can return all devices under one locationgroup tree. This is an especially helpful query when used with other queries.

/**************************************************************
   This script provides all devices residing under
   the parent location group. You can combine this
   query with other queries by using the WHERE column
   IN (SELECT query) filter (this script uses the
   same filter to filter the location groups)
**************************************************************/ 

SELECT *
FROM Device d (nolock)
INNER JOIN Location l (nolock)
ON l.LocationId = d.LocationId
INNER JOIN LocationGroup lg (nolock)
lg.DefaultLocationId = l.LocationId
WHERE lg.LocationGroupID IN 
(
   /*****************************************************
      The below sub query provides all children
      Location Groups under one Parent. This can 
      also be useful with other queries.
     ***************************************************/ 

   SELECT lgf.ChildLocationGroupID
   FROM LocationGroup lg (nolock)
   INNER JOIN LocationGroupFlat lgf (nolock)
   ON lgf.ParentLocationGroupID = lg.LocationGroupID
   WHERE lg.Name LIKE '%Customers LG%'
)

This query allows you to search the event log. This is imperitive since the event log often times out from the console.

/**************************************************************  
   This script provides you with a list of modules in the  
   event log. These can be used to filter the following  
   query.  
**************************************************************/   
SELECT DISTINCT el.Module   
FROM eventLog.EventLog el (nolock)   
  
/**************************************************************  
   This query will select information from the event log   
   and allow you to filter on Modules. Modules provide you  
   with context of the event that occured. The Name is   
   the actual name of the event that occured. This will   
   only show you the last 30 days. You can decrease this   
   number to improve performance.  
     
   You can also filter on the LocationGroupID if you know   
   the location group name. Keep in mind that some modules   
   only report at Global.  
  
   The Username is the Admin user who performed the task.   
   sysadmin is the system user for running stored procedures   
   however some modules (like the device module) show   
   sysadmin even if a user performed the action.  
**************************************************************/  
  
SELECT cu.UserName, el.Module, e.Name, el.*  
FROM eventLog.EventLog el (nolock)  
INNER JOIN eventLog.Event e (nolock)  
ON e.EventId = el.EventId  
INNER JOIN CoreUser cu (nolock)  
ON cu.CoreUserId = el.ActionBy  
WHERE el.CreatedOn > DATEADD(DAY, -30, GETUTCDATE())  
AND el.Module LIKE 'Dashboard'  
--AND LocationGroupID IN (SELECT LocationGroupID   
--FROM LocationGroup WHERE Name LIKE '%AirWatch%')

Role compare script

This script shows you two roles side by side for comparison.

/**************************************************************
   Update the values in @Role1 and @Role2
   to compare two roles. If you need to find
   why a custom role is missing use the next
   script.
**************************************************************/ 

DECLARE @Role1 AS nvarchar(50);
DECLARE @Role2 AS nvarchar(50);
SET @Role1 = 'AirWatch Administrator';
SET @Role2 = 'System Administrator';

SELECT re.ResourceID, c.Name AS 'Module', re.Name, m1.Allow AS 'Role1 Allowed', m2.Allow AS 'Role2 Allowed'
FROM Resource re 
INNER JOIN Category c (nolock)
ON c.CategoryID = re.CategoryID
INNER JOIN Mode m1 (nolock)
ON re.ResourceID = m1.ResourceID
INNER JOIN Role r1 (nolock)
ON r1.RoleId = m1.RoleId
INNER JOIN Mode m2 (nolock)
ON re.ResourceID = m2.ResourceID
INNER JOIN Role r2 (nolock)
ON r2.RoleID = m2.RoleID 
WHERE r1.Name = @Role1 AND r2.Name = @Role2
ORDER BY Module, ResourceID

Missing Custom Role script

This script shows you two roles side by side for comparison.

/**************************************************************
   Update the values in the variables to check
   if resources are missing from the role who is
   supposed to have higher priviledges.
**************************************************************/ 

DECLARE @HigherRole AS nvarchar(50);
DECLARE @LowerRole AS nvarchar(50);
SET @HigherRole = 'Role with higher privileges';
SET @LowerRole = 'Role with less privileges';

SELECT m.ModeID, r.Name, re.Name, m.Allow 
FROM Mode m (nolock)
INNER JOIN Role r (nolock)
ON r.RoleId = m.RoleId
INNER JOIN Resource re 
ON re.ResourceID = m.ResourceID
WHERE r.Name LIKE @LowerRole AND m.Allow = 1 AND
       m.ResourceID IN
       (SELECT ResourceID FROM Mode m 
       INNER JOIN Role r
       ON r.RoleId = m.RoleId
       WHERE r.Name LIKE @HigherRole AND
       m.Allow = 0
       )

### Status of Events stuck in ‘Processing’ or ‘Failed’ to ‘Ready for Processing’

UPDATE adp.AdpExportTracking
    SET [Status] = 2 -- 2 = Completed Event
WHERE [Status] = 1; -- 2 = Pending Event
 
 
-- This script updates the status of Error Exports in adp.ADPExportTracking table
 
SET NOCOUNT ON;
 
BEGIN TRY
    IF  OBJECT_ID(N'adp.ADPExportTracking') IS NOT NULL
    BEGIN
        UPDATE adp.ADPExportTracking
            SET [Status] = 2
        WHERE [Status] = -1 -- -1 = Failed Status
    END
     
END TRY
BEGIN CATCH
    DECLARE    @error_severity        INT,
            @error_state        INT,
            @error_message        NVARCHAR(2048);
 
    SELECT    @error_severity = ERROR_SEVERITY(),
            @error_state = ERROR_STATE(),
            @error_message = ERROR_MESSAGE();
 
    RAISERROR(@error_message, @error_severity, @error_state);
 
END CATCH
GO

VMware Tunnel in SQL

ws1tunnel.TunnelConfiguration

ColumnDescriptionType of Data
UuidUUID based primary Keyuniqueidentifier
NameName of the Tunnel Configurationnvarchar(255)
Enabled

If Tunnel Configuration

is enabled or not

bit
PacFileSupportEnabled

If PAC File Upload support

is enabled or not

bit
OrganizationGroupUuid

UUID based Organization

Group Id

uniqueidentifier
VpnVersionVpn Config versionnvarchar(255)
VpnVersionUtcDateTimeVpnConfigVersionUtcDateTimenvarchar(255)
CreatedByCore user who created this recorduniqueidentifier
CreatedOnCreated on Datedatetime
ModifiedByCore user who modified thisuniqueidentifier
ModifiedOnModified Datedatetime

ws1tunnel.TunnelClientConfiguration

ColumnDescriptionType of Data
UuidUUID based primary Keyuniqueidentifier
TunnelConfigUuidUuid of Tunnel Configurationuniqueidentifier
AuthenticationTypeAuthentication Typeint
CertificateSourceVPN Certificate Sourcenvarchar(255)
CertificateAuthorityUuidVPN  CA Id.uniqueidentifier
CertificateTemplateUuidAirwatch VPN CA Template Id.uniqueidentifier
ScepCertificateTemplateUuidSCEP Certificate Template ID
for Client Certificate for Tunne
uniqueidentifier
ProfileScepCertificateTemplateUuidthe Airwatch CA template id
for SCEP template with key 4096
uniqueidentifier
CreatedByCore user who created this recorduniqueidentifier
CreatedOnCreated on Datedatetime
ModifiedByCore user who modified thisuniqueidentifier
ModifiedOnModified Datedatetime

ws1tunnel.TunnelServerConfiguration

ColumnDescriptionType of Data
UuidUUID based primary Key for the tableuniqueidentifier
TunnelConfigUuidUuid of Tunnel Configurationuniqueidentifier
AwCertificateThumbprintVPN Certificate Thumbprintnvarchar(255)
AwIdentifierUnique string for AWCM to identify an installation of AirWatch for VPNnvarchar(1024)
VpnIdentifierUnique string for AWCM to identify an installation of VPN servernvarchar(1024)
TunnelConfigurationTypeWhether VPN is in a standard or endpoint-relay configuration.int
EndpointHostVPN server endpoint Host namenvarchar(255)
RelayEndpointPortVPN server portint
VpnServerHostNameVPN server Host namenvarchar(255)
VpnServerPortVPN server portint
PublicSslVPNWhether to use a public SSL certificate for vpn.bit
LogLevelVPN logging levelint
ApiViaProxyWhether to enable API and AWCM outbound calls via proxy.bit
AccessLogModeAccessLogMode (rsyslog or file)int
EnableAccessLogVpn EnableAccessLogbit
VpnServerConnectionTypeVPN server connection typeint
SyslogHostSyslog Hostnamenvarchar(255)
SyslogPortSyslog Portint
CreatedByCore user who created this recorduniqueidentifier
CreatedOnCreated on Datedatetime
ModifiedByCore user who modified thisuniqueidentifier
ModifiedOnModified Datedatetime

ws1tunnel.ConfigCertificateMapping

ColumnDescriptionType of Data
UuidUUID based primary Key for the tableuniqueidentifier
TunnelConfigUuidReference Identifier to TunnelConfiguration tableuniqueidentifier
CertificateUuidUUID for Certificateuniqueidentifier
CertificateTypeCertficate typeint
IsActiveActively used certificatebit
CreatedByUser ID that created this recorduniqueidentifier
CreatedOnDateTime when the record was created in the DBdatetime
ModifiedByUser ID that modified this recorduniqueidentifier
ModifiedOnDateTime when the record was last modified in the DBdatetime

ws1tunnel.TunnelTestConnectionStatus

ColumnDescriptionType of Data
UuidUUID based primary Key for the tableuniqueidentifier
TunnelConfigUuidReference Identifier to TunnelConfiguration tableuniqueidentifier
TestConnectionStateCurrent state of Test connectionnvarchar(20)
TestConnectionInitiatedTimeTime at which test connection was initiateddatetime
AWCMReachableAWCM is Reachable or notbit
AWCMConnectionFailureReasonReason for AWCMC not being reachablenvarchar(50)
TestConnectionResultTest connection was successful or notbit
CreatedByCreated by for the Test connection status recorduniqueidentifier
CreatedOnCreated on for the Test connection status recorddatetime
ModifiedByModified by for the Test connection status recorduniqueidentifier
ModifiedOnModified on for the Test connection status recorddatetime

WS1 UEM DB Important Tables

Each application uploaded to AirWatch would have its own row in the deviceApplication.Application table.  Each column contains specific attributes for that object.  For example, an application in the deviceApplication.Application table would have a column for Name, PackageID, LocationGroup, etc.

Device Management Tables

dbo.DeviceOperatingSystem: OS versions supported by AirWatch Console in current version

dbo.DeviceModelInfo: Device models (for Apple) or Android vendors supported by AirWatch Console in current version

User Management Tables

dbo.CoreUser: Admins – UserName, TimeZone, LastLogin, etc.

mobileManagement.EnrollmentUser: End users – UserName, LocationGroupID, DisplayName, UserPrincipleName, etc.

mobileManagement.CurrentDeviceEnrollmentUser: Ties enrolled devices to enrollment users

dbo.UserGroup: FriendlyName,DN,RootLocationGroupID, domain, etc

MAM Tables

deviceApplication.Application: Internal applications

deviceApplication.RecommendedExternalApplication: All things public application

deviceApplication.VPPLicensePool: SmartGroup, Allocated licenses, Redeemed licenses, etc.

deviceApplication.ApplicationGroup: For blacklists/whitelists/required lists of apps

smartGroup.SmartGroup: info about the Smart Group

smartGroup.AWEntitySmartGroupAssignmentMap: Maps a Smart Group to the App/Book that uses it for assignment

MEM Tables

mobileEmailGateway.MEMConfig Contains attributes related to the Email Configuration, including pre-7.1 sets of System Codes

mobileEmailGateway.MEMDevice Contains all the attributes for each device, whether managed or unmanaged

mobileEmailGateway.MEMDeviceActivity Contains the status and activity info for each device

mobileEmailGateway.MEMConfigProfile Contains associations between MEMConfig and EAS device profiles

mobileEmailGateway.MEMDeviceConfig Contains associations between MEMConfig and MEMDevice. When an enrolled device receives an EAS profile, MEMDevice will receive corresponding MEMConfig

mobileEmailGateway.MEMDeviceDiagnostic Contains diagnostics data for devices

mobileEmailGateway.{all the rest}  Email Policy configurations, such as attachment encryption or OS policies

MCM Tables

enterpriseContent.Content All settings pertaining to uploaded content – offlineViewing, allowEmail, Name, etc.

enterpriseContent.ContentRepository Details for repositories – Name, Link, AuthenticationUsername, User/Admin Repository

enterpriseContent.ContentMap Links DeviceID, EnrollmentUserID, and ContentVersionID

enterpriseContent.ContentVersion BlobID, Version, Size, Author, etc

MDM Tables

dbo.Device Device centric information – OS, Friendly Name, Serial Number, Phone Number, Enrollment Date, etc.

Profiles

DeviceProfile.DeviceProfile Settings for profiles – Name, Description, DevicePlatformID, ProfileAssignmentTypeID, etc.

deviceProfile.DeviceProfileVersion Shows historic versions of a profile and their modified dates

deviceProfile.DeviceProfileDevicePool Maps devices to profiles and profileVersions

deviceProfile.deviceProfileSettingValue Maps actual profile settings to profileVersions

Telecom

dbo.Plan Telecom plan details

dbo.PlanAssignmentRule Assignment rule details

dbo.CellCard Cellular details

MobileDataUsage, CellCardUsage Interrogator tables

Compliance

DeviceComplianceStatus Compliance status of devices

Policy List of policies

Device Samples

ManagedAppList,ProfileListSample, SecurityInformationSample Interrogator tables

Reports

dbo.Report Info on all available reports

dbo.ReportSubscription Created subscriptions

Miscellaneous tables

dbo.Certificate Thumbprint, SubjectName, Validity Dates, etc.

dbo.LocationGroup OG info – LocationGroupID, Name, CreatedOn, CreatedBy, MDMSettingsInheritable

deviceCommandQueue.DeviceQueue / deviceQueueAudit Current commands queued / already executed commands for devices

dbo.SystemCode All settings and default values configurable in the AirWatch Console - Groups & Settings

EventLog.EventLog Where console Event Logs pull from

Example of deleting “wipe device” commands from Message Queue

This is needed in case of accidental mass wipe created as result of incorrect compliance policy etc.

SELECT * FROM deviceCommandQueue.DeviceQueue WHERE commandid = 10;
DELETE * FROM deviceCommandQueue.DeviceQueue WHERE commandid = 10;

WS1 UEM DB Monitoring

Recommendations for monitoring on the WS1 UEM database

Monitor Descritpion
Data Files Monitor and alert for resizing when free space in data files drops below 10%.
Transaction Logs Monitor and resize if free space in log drops below 10%.
Waiting Tasks Waiting tasks in the SQL activity monitor must be under 10 on average. Ideally waiting tasks should be between 0 and 2 when compared to 20,000 batch requests per second.
Index Rebuild Monitor for fragmentation between 10% and 29%. Reorganize with an update of statistics. Indexes with fragmentation greater than 29% should be rebuilt.
SQL Server CPU Monitor sustained high CPU utilization (Over 90% for a 15 minute duration).
SQL Server Job History Monitor failed SQL Server Agent Jobs (in particular, AirWatch Jobs).
SQL Server Page Life Expectancy Monitor SQL Server Page Life Expectancy (dropping below 3000).
SQL Server Disk Space Monitor disk space usage on all Data and Log Drives for ‘AirWatch’ and ‘tempdb’ Databases.
SQL Server Disk Queuing Monitor Disk Queuing on all Data and Log Drives for ‘AirWatch’ and ‘tempdb’ Databases. Check Disk Queue Length via Task Manager > Performance > Resource Monitor > Dist Tab > Storage. It should average between 2 and 4. It could increase or decrease, but on average it should be between those values.
Page Life Expectancy Page Life Expectancy is an indication of whether the database server has memory pressure. The expected number is over 1,000 (seconds). If it is low, this is a first indicator of memory pressure. This may not be an issue if:
- The PLE is increasing over time. If it is increasing, but is still less than 1,000, then that is a sign of a memory pressure.
- After an index maintenance job, the PLE can be low. This needs to be monitored for a few hours to see if it goes up.
Index Fragmentation Level A high fragmentation level means data retrieval becomes less efficient and reduces database performance. Run the defragmentation job on a nightly basis. The script below shows the fragmentation level (in percent) against all the tables. The recommended fragmentation level is less than 30% when the page size is more than 1,000.
SELECT OBJECT_NAME(object_id), index_id, index_type_desc, index_level,
avg_fragmentation_in_percent, avg_Page_space_used_in_percent, page_count
 FROM sys.dm_db_index_physical_stats(DB_ID(N'AirWatch'), null, null,
null, 'SAMPLED') ORDER BY avg_fragmentation_in_percent DESC

If the database is highly fragmented, it is recommended that you perform an index reorganize or rebuild.

Health Checks

Synthetic transactions are the strongest indicator of a healthy AirWatch environment. They can mimic end user actions (for example, enrollment) and report if there are issues. Many different use cases could be considered, and high-use scenarios should be tested with synthetic transactions. An example synthetic transaction could be:

  1. Navigate to the AirWatch Console.
  2. Log in using credentials.
  3. Navigate to Hub > Reports & Analytics > Reports > List View.
  4. Run a report.
  5. Log out.

Typically, a tool like Keynote or AlertSite would be used to generate and monitor synthetic transactions.

WS1 UEM Hardening

Linked Articles

Tools to check SSL/TLS certificates and protocol

Subsections of WS1 UEM Hardening

IIS Hardening

Device Services IIS

Disable insecure TLS/SSL protocol support
- Yes, you can disable this and this will not have any impact on AirWatch Applications because we have made the necessary changes in our components as well. 
POODLE attack, SSLv3 etc have been taken care by our developers in console version 8.1 and above.

Remove the default page or stop/disable the IIS server
- Yes, you can remove the default page, but do not disable the IIS server. Recommended not to disable the IIS server.

Disable TLS/SSL support for RC4 ciphers

Disable SSLv2, SSLv3, and TLS 1.0

Regarding Ciphers suites

  •  Be it any kind of Ciphers(Static key cipher, 3DES cipher, Strong cipher) the best solution for this  is to enable TLS. Also, the MicrosoftKB article 245030 as mentioned in the ticket is the best solution for all the Cipher questions.
    https://support.microsoft.com/en-us/kb/245030

The RC4 cipher can be completely disabled on Windows platforms by setting the “Enabled” (REG_DWORD) entry to value 00000000 in the following registry locations:

• HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC4 128/128
• HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC4 40/128
• HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC4 56/128

Disabling SSL and enabling TLS is out of Airwatch’s scope as you have to make registry changes locally on the server:

Disabling SSL 3.0- https://www.digicert.com/ssl-support/iis-disabling-ssl-v3.htm
Enabling TLS - https://support.quovadisglobal.com/KB/a433/how-to-enable-tls-12-on-windows-server-2008-r2.aspx
https://www.hass.de/content/setup-your-iis-ssl-perfect-forward-secrecy-and-tls-12

Disabling RC4 on Java:
http://stackoverflow.com/questions/18589761/restict-cipher-suites-on-jre-level

Example:

jdk.certpath.disabledAlgorithms=MD2
jdk.tls.disabledAlgorithms=MD5, SHA1, RC4, RSA keySize < 1024

We can either do it at a JRE system wide level or at a JVM instance (such as AWCM) level adding RC4 as a disabled algorithm when a choice has to be made as part of SSL handshake.
In the latter case, It will be a config change on AWCM Service parameters (only change being the added restriction option in $AWCM_HOME/service/AWCMService.exe.parameters).

Tip

IIS Crypto Tool can be used to turn off weak ciphers in Windows Server 2008+

Warning

Usage of iiscrypto tool to disable Cipher Suites, as well as registry keys can break communication between AirWatch components.

Use with extreme caution, ONLY AFTER AirWatch was deployed and tested to be working. Disable Cipher Suites one by one and re-test AirWatch functionality after each change!

Security scanner sees IIS vulnerabilities:

  • SWEET32
  • POODLE
  • TLS_FALLBACK_SCSV

Hardening:

  • POODLE - need to disable SSL 3.0 protocol. Open registry: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\Schannel\Protocols and create keys SSL 3.0\Server (if not created previously), create a DWORD value Enabled = 0;

  • SWEET32 - need to disable weak ciphers. Open registry: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\Triple DES 168, create a DWORD value Enabled = 0. Create keys “RC4 56/128”, “RC4 40/128”, “RC4 128/128” create a DWORD value in all keys called Enabled = 0;

Warning

When turning off Triple DES the RDP protocol to server may stop working. Need to patch RDP to use modern ciphers to solve this problem.

  • TLS_FALLBACK_SCSV (only for Windows 2003-2008! see KB from Microsoft: https://support.microsoft.com/kb/980436/en-us) - Open registry: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL and create DWORD called UseScsvForTls with parameters:
    • UseScsvForTls = 0 # Client sends Renegotiation Info extension for TLS protocol
    • UseScsvForTls = 1 # Client sends SCSV for TLS protocol - use this to solve problem
Warning

AirWatch Self-Service portal uses TLS1.0/RC4-type cryptography, and gives Error and blank page after IIS hardening!

iOS 11 & Wi-Fi TLS 1.2 Requirements

With the release of iOS 11, TLS 1.2 will now be the default for EAP-TLS negotiation. This may cause an issue with older clients that still need to connect on TLS 1.0 or 1.1. Apple has allowed for a method to override this default setting with a configuration profile sent down to the device via MDM. In order to ensure your iOS devices maintain Wi-Fi connection when upgrading to iOS 11, please follow the steps below:

Note: If you already have a successfully deployed iOS Wi-Fi with EAP-TLS configured, skip to step 3. 

  1. Create a new profile with a Wi-Fi payload using EAP-TLS and General payload configured.
  2. Ensure that the profile successfully configures Wi-Fi on an iOS device.
  3. From your profile list view, select the Wi-Fi with EAP-TLS created profile and choose to view XML.
  4. Export or copy the XML of the profile.
  5. Edit the XML to remove everything prior to the first and after its corresponding .
  6. Edit the XML again to add the following bolded key/values (accepted values are 1.0, 1.1, and 1.2). These should be a part of the EAPClientConfiguration key
<key>EAPClientConfiguration</key>
<dict>
<key>AcceptEAPTypes</key>
<array>
<integer>13</integer>
</array>
<key>TLSMaximumVersion</key>
<string>1.1</string>
<key>TLSMinimumVersion</key>
<string>1.0</string>
</dict>
  • Edit the XML a final time to create a unique identifier for the payload. Locate the PayloadUUID key and edit the values that correspond to the ‘X’s to random values. Please ensure these values are as random as possible to avoid issues with duplicate identifiers (e.g. 123456, 111111, 101010).
<key>PayloadUUID</key>
      <string>352B3FD9-B875-45C5-AA0E-AAFEE3XXXXXX</string>
  1. Create another new profile and configure the General payload
  2. Paste your edited XML into the Custom Settings payload and publish to devices

IISCrypto config from AirWatch

iOS supports all latest ciphers and encryptions – however there are questions with Android, so Android 4.4 will be the baseline.

Protocols:  SSL 2.0, SSL 3.0, TLS 1.0, TLS 1.1 can be turned off as all of the platforms supports newer protocols.

Warning

Test results at client site (AirWatch ver. 9.2.3): disabling TLS 1.0 showed 4 services failing at AirWatch Device Services Server role - proceed with caution!

Cipher Suites, which can be turned off...

VMware Assist

External link: https://stackoverflow.com/questions/52212866/how-can-i-disable-tls-rsa-with-aes-128-cbc-sha-without-disabling-others-as-well

Hardening Assist is basically hardening of Java, disabling TLS_RSA_WITH_AES… in it. Delete RSA Keysize, and add specific protocols.

WS1 UEM Installation

Articles in section

Installation

❗️All Windows systems, which will be used to deploy AirWatch / Workspace One UEM, in localization settings, there should be language = US-EN.

❗️Special attention to Regional Settings on floating point identifier: it must be a dot, not a comma!

❗️All Windows systems, which will be used to deploy AirWatch / Workspace One UEM, should have all current patches applied. For example, early versions of Win2012R2 have broken ASP.NET and break AirWatch installation.

Legend

  • BE - (BackEnd server) WS1 UEM Admin Console
  • FE - (FrontEnd server) WS1 UEM Device Services
  • SQL - Microsoft SQL Database Server
  • UEM - AirWatch / Workspace One UEM

Database Deployment

❗️See first SQL Recommendations page before production or semi-production deployment.

  • Install SQL
  • Login to the SQL server, launch SQL Management Studio;
  • Create a new database. In database settings apply General → Autogrowth / Maximize → File Growth → In Megabytes = 128;
  • Choose collation: Options → Collation → SQL_Latin1_General_CP1_CI_AS;
  • SQL 2008 and MS SQL 2008R2 are not supported anymore. For MS SQL 2016 choose Options → Compatibility Level = 2014

❗️Issues currently detected with installing Workspace ONE UEM up to version 1909 in Microsoft SQL 2017. Services do not start after install, console does not launch. As recommended by Microsoft for SQL 2017, services should use the element to improve startup performance. Using this element can also help avoid delays that can cause a time-out and the cancellation of the service startup. See Microsoft KB article.

  • Create a user in Mixed-mode SQL (non-domain), with sysadmin permissions for server and db_owner for database. Gice the user permissions for msdb: SQLAgentUserRole, SQLAgentReaderRole, db_datareader roles. Do not forget to cancel password expiration for this user;
  • If there is not Internet on DB server - download Microsoft .NET Framework 4.6.2 from Microsoft website for English Windows on separate computer and copy to this server (during setup WorkspaceONE_UEM_DB_XX.YY.Z.K_Setup tries to download the framework itself, with no Internet it may hang the installation process for some time);
  • Copy WorkspaceONE_UEM_DB_XX.YY.Z.K_Setup files to the server, launch it. Copy to **C:\Distr**;

❗️Do NOT launch installer from C:\Users\Documents and Settings\Downloads etc folders - long path may cause unpacking error.

  • Enter “localhost” in install wizard, login and password of the SQL user, and choose the UEM database;
  • ❓️ If the database is created on AlwaysOn Cluster - turn on Using SQL AlwaysON Availability Groups option;
  • Wait for installer to end (10-15min). Install progress can be seen as log file growth in c:\AirWatch\AirWatch 1811\Database\AWDatabaseLog.txt* (the log will grow up to 2.3Mb when the installation will finish);
  • Check the installation, use SQL Management Studio to launch a script:

select * from dbo.DatabaseVersion;

The answer should be the UEM version number.

❓️ For AlwaysOn cluster - do not forget to clone the database Jobs on the other cluster nodes!

Device Services Front-End (FE) Server

  • Enter Windows Server Manager and check the following roles/features (double-check official doc for feature list):
    • Web Server (IIS)
    • Web Server (IIS) → Web Server → Common → Static Content, Default Document, Directory Browsing, HTTP Errors, HTTP Redirection
    • Web Server (IIS) → Web Server → Performance → Dynamic Content Compression
    • Web Server (IIS) → Web Server → ASP
    • Web Server (IIS) → Web Server → ASP.NET 4.5
    • Web Server (IIS) → Web Server → Security → IP & Domain Restrictions
    • Web Server (IIS) → Web Server → Health & Diagnostics → Request Monitor
    • Web Server (IIS) → Web Server → Application Development → Server Side Includes
    • .NET Framework 4.5 → WCF → HTTP Activation
    • Message Queuing
    • Telnet Client

❗️ DON NOT turn on Web Server (IIS) → Web Server → Common → WebDav Publishing - this will lead to multiple bugs in managing iOS devices

  • ❓️ If there is not Internet on FE server - download and install NET Framework 4.6.2 (Microsoft .NET Framework 4.6.2 (Offline Installer) for Windows 7 SP1, Windows 8.1, Windows Server 2008 R2 SP1, Windows Server 2012 and Windows Server 2012 R2). Reboot server;

  • ❓️ If there is not Internet on FE server - download and install URL Rewrite Module 2.0 (https://go.microsoft.com/?linkid=9722532) for IIS. Old version of Rewrite Module 2.0 provided on this page as attachment in case of need;

  • Upload an external certificate in PFX format with private key. Password used to protect the certificate MUST be 6+ characters long. Short password will lead to problems with AWCM Java keystore! Install the certificate into Local Machine account, leaving Automatic Detect option for certificate type. Also install any root and intermediate certificates of the certificate trust chain. Subject Alternative Name of the certificate MUST contain the external DNS name of the server!

  • Check correct start config of IIS - use browser to go to http://127.0.0.1/ (start page of IIS must be present)

  • Go to IIS admin console, bind the certificate: in sites tree choose Default Web Site → Bindings menu → Add.., choose https, in SSL Certificates list choose the certificate from previous step. Enter the external DNS name of the server, which is written in the certificate. 

❗️Port binding is needed ONLY for Device Service and Console Service.

  • Launch installer WorkspaceONE_UEM_Application_X.X.X.X_Full_Install. Choose Continue setup without importing/exporting config file;
  • In modules selection choose only Device Services, select This feature will not be available for Admin Console, continue installation;

❗️For AirWatch 9.2.2+: during installation, AirWatch installer deploys SQL Native Client, which may not have enough time to initialize during the work of the wizard. During SQL check, an error may be generated, that SQL is not found. Press Cancel and reboot the server, then re-launch the setup process.

  • Enter SQL data: in full database name, only enter the server name, do not enter SQL Instance name;
  • Specify the DNS name for reaching the server by HTTPS from outside and inside. Do not choose SSL Offload - it is much easier to make all connections as HTTPS and then edit configuration;

⭐️ Instead of choosing different DNS names and then have issues with AWCM, I recommend to enter the same external name for Device Services and Web Console (check Same as above? option). After this, make an alias on the local DNS server, or use the hosts file on Admin Console/BE server to alias the external name of Directory Services/FE to an internal IP address.

  • Choose Default Web Site as install target;
  • Leave AWCM listening IP as 0.0.0.0 since it is installed locally, and port 2001 for connection. Install the PFX certificate and enter its’ password; 

❗️The PFX certificate MUST be created with Export All Properties option! Or the Java keytool will not be able to import it into awcm.keystore, and it will not give errors in the log! But AWCM will not work!

  • Choose Implicit Clustering (do not cluster AWCM);
  • Wait for install completion. AirWatch Certificate Installation Wizard will open, click Next and choose SQL Authentication. If Internet is accessible, a code must be entered. For offline installation, click Get File and save the *.plist fiel on disk;
  • Go to my.workspaceone.com: My Workspace One menu → My Company → Certificate Signing Portal → Authorize Install → Generate a token (for Internet access); OR
  • My Workspace One menu → My Company → Certificate Signing Portal → Authorize Install → Upload Your File (for offline), and upload *.plist file.
  • Save the certs.plist answer file and upload it in the installation wizard, thus ending the installation.

❗️AirWatch (WOne UEM 1909) services may not start due to timeout error on Windows 2008-2012. Increase Timeout time in Windows registry: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control → ServicesPipeTimeout=180000 External link: https://kb.vmware.com/s/article/50105044?lang=en_US

Admin Console Back-End (BE) Server

  • Enter Windows Server Manager and check the following roles/features (double-check official doc for feature list):
    • Web Server (IIS)
    • Web Server (IIS) → Web Server → Common → Static Content, Default Document, Directory Browsing, HTTP Errors, HTTP Redirection
    • Web Server (IIS) → Web Server → Performance → Dynamic Content Compression
    • Web Server (IIS) → Web Server → ASP
    • Web Server (IIS) → Web Server → ASP.NET 4.5
    • Web Server (IIS) → Web Server → Security → IP & Domain Restrictions
    • Web Server (IIS) → Web Server → Health & Diagnostics → Request Monitor
    • Web Server (IIS) → Web Server → Application Development → Server Side Includes
    • .NET Framework 4.5 → WCF → HTTP Activation
    • Message Queuing
    • Telnet Client

❗️ DON NOT turn on Web Server (IIS) → Web Server → Common → WebDav Publishing - this will lead to multiple bugs in managing iOS devices

  • ❓️ If there is not Internet on FE server - download and install NET Framework 4.6.2 (Microsoft .NET Framework 4.6.2 (Offline Installer) for Windows 7 SP1, Windows 8.1, Windows Server 2008 R2 SP1, Windows Server 2012 and Windows Server 2012 R2). Reboot server;
  • ❓️ If there is not Internet on FE server - download and install URL Rewrite Module 2.0 (https://go.microsoft.com/?linkid=9722532) for IIS. Old version of Rewrite Module 2.0 provided on this page as attachment in case of need;
  • Check correct start config of IIS - use browser to go to http://127.0.0.1/ (start page of IIS must be present)
  • Configure the certificate on IIS - for Admin Console on BE a self-signed certificate may be used:
    • Enter IIS Admin Console, choose Server Certificates, and in the right column menu choose Create Self-Signed Certificate;
    • Enter a name for the certificate, type = Web Hosting, click ОК;
    • Go to IIS admin console, bind the certificate: in sites tree choose Default Web Site → Bindings menu → Add.., choose https, in SSL Certificates list choose the certificate from previous step.

❗️Port binding is needed ONLY for Device Service and Console Service.

  • Launch installer WorkspaceONE_UEM_Application_18.11.0.3_Full_Install. Choose Continue setup without importing/exporting config file;
  • In modules selection choose only the Admin Console, choose This feature will not be available for Device Services, continue the installation;

❗️For AirWatch 9.2.2+: during installation, AirWatch installer deploys SQL Native Client, which may not have enough time to initialize during the work of the wizard. During SQL check, an error may be generated, that SQL is not found. Press Cancel and reboot the server, then re-launch the setup process.

  • Enter SQL data: in full database name, only enter the server name, do not enter SQL Instance name;
  • Specify the FQDN name for HTTPS access on Admin Console from the inside. Do NOT use a short name of DNS alias. Choose an External DNS name for access via HTTPS on Device Services server. Check the absence of space characters before or after the names. An error in this form may be corrected only by re-installing UEM!
  • Choose Default Web Site as the install target;
  • In Company Profile choose the company name and installation type = Production;

❗️AirWatch (WS1 UEM 1909+) services may not start due to timeout error on Windows 2012R2+. Increase Timeout time in Windows registry: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control → New 32-Bit DWORD: ServicesPipeTimeout, Decimal=200000 (Decimal=60000 too small, put more!)

External link: https://support.microsoft.com/en-us/kb/922918

Manually running services: under Supplemental Software → QueueSetup locate and run the InstallQueues.Bat file.

  • Check the installation:
  • Enter the UEM console. Use Login: administrator, Password: airwatch. Choose a new password, choose a PIN-code and secret questions/answers pairs.

⭐️ Hardening of the IIS web-server for AirWatch/UEM Device Services is described in this article.

Subsections of WS1 UEM Installation

WS1 First Time Config

GroupID

  • Switch to tenant - Company
  • Switch to Groups & Settings → Groups → Organization Groups → Organization Groups Detail
  • Set the company name, set GroupID, country and time zone

iOS Agent

  • Switch to Groups & Settings → All Settings → Devices & Users → Apple → Apple iOS → Intelligent Hub Settings, click Override
  • Set Background App Refresh - for the AirWatch Agent works in the background and does not interfere with other apps
  • Turn on Collect Location Data - collection of data from GPS (Location Services)
  • Do not touch SDK profiles, leave settings as is currently

AWCM

  • Launch the Admin Console on the exact server, where AWCM is installed (FE) and go to Groups & Settings > All Settings > Device & Users > Android > Intelligent Hub Settings

  • Switch Use AWCM Instead of C2DM as Push Notification Service - if you have Android devices with no Google Apps/Services, no Google Account, or want to restrict PUSH notifications to MDM-direct

  • Switch AWCM Client Deployment Type to Always Running, click Save

  • Go to Groups & Settings > All Settings > System > Advanced > Secure Channel Certificate

  • Check/configure the Windows environment variable JAVA_HOME - it must point to the last version installed c:\Program Files\jre-<номер версии>

  • Click Download AWCM Secure Channel Certificate Installer and launch the cert installer

  • Check the cert install: open cmd as Administrator and enter command such as:

    c:\Program Files\Java\jre1.8.0_131\bin>keytool.exe -list -v -keystore “C:\AirWatch\AirWatch 1811\AWCM\config\awcm.truststore”

    Enter password as the password for the keystore, and check there are 2 certificates entered, including the secure channel certificate Switch to tenant = Global, go to Groups & Settings > All Settings > System > Advanced > Site URLs, click Enable AWCM Server button at the end of the page

  • Check AWCM settings: internal and external DNS names (they MUST be exactly those used in the corp certificate!) and port number (TCP2001).

Port TCP 2001 MUST be open FROM the outside to server with AWCM (Device Services - FE) in order for direct PUSH to work with Android, and Windows Phone/Desktop devices.

If the external DNS name is published on an external proxy or load balancer, and the inner servers do not know this, use hosts file on Admin Console (BE) server and AirWatch Cloud Connector (ACC) / Enterprise Systems Connector (ESC) to make an alias of external DNS name and internal IP of Device Services (FE) server.

AWCM Troubleshooting - see article.

APNs certificate for Apple and SSL for Apple profiles

  • Launch Admin Console using Firefox, Safari or Chrome (IE not supported!). Go to Groups & Settings → All Settings → Devices & Users → AppleAPNs for MDM
  • Download plist file.
  • Prepare an AppleID account, click Go to Apple
  • On Apple website click Create certificate, accept the terms, upload the plist file, download the corresponding PEM file on local disk
  • Return to the Admin Console - click Next, upload the PEM file and enter the corresponding AppleID click Save. Enter the PIN code of the console administrator
  • Go to Groups & Settings → All Settings → Devices & Users → AppleProfiles
  • Click Upload and choose the PFX file of the corp certificate enter the password of the PFX container

Google Play Registration and Android for Enterprise/Legacy Enrollment

  • Launch Admin Console using Firefox, Safari or Chrome (IE not supported!). Switch to Groups & Settings → All Settings → Devices & Users → Android → Android EMM Registration
  • Click Register with Google
  • Proceed with steps on Google website, entering the GMail Account (each GMail Account may only be used ONCE for 1 ЕММ system (any))

For old AirWatch Console 9.0.1 and earlier, if upgraded to latest version of UEM, the Enable Play Store button should be clicked.

  • Open Enrollment Restrictions tab: choose Define the enrollment method for this organization group. Default is Always use Android, which means to always use Android for Enterprise of type = Device Work Profile (duplication of software into BYOD/Corp containers). If devices with potential AfE support in current group are to be enrolled and managed using Android Legacy ELM/POEM drivers, then choose Always use Android (Legacy) in list, or choose hybrid mode by defining user groups for AfE: Define Assignment Groups that use Android.

After choosing the EMM registration method, DO NOT CHANGE IT with many devices enrolled. Consequences:

  • Profiles can still be installed on the device as they’re being installed directly from Workspace ONE;
  • Communication is maintained between the device and Workspace ONE UEM;
  • You will be unable to leverage any Play store services;
  • No new apps added to Workspace ONE will be visible on the device managed play store;
  • Previously added applications in Workspace ONE will no longer be deployable from the console.

Source - https://blog.eucse.com/things-not-to-do-workspace-one-changing-android-emm-registration/

AirWatch Cloud Connector (ACC) / Enterprise Systems Connector

  • Launch the Admin Console on the exact server, where ACC is to be installed and switch to a non-Global Tenant

You cannot download the ACC distrib on one server, then copy and launch on another!

  • Go to Groups & Settings > All Settings > System > Enterprise Integration > Cloud Connector, turn on Override, switch Enable AirWatch Cloud Connector and Enable Auto Update

It is strongly recommended to configure ACC on non-Global level

  • Switch to Advanced tab, click Generate Certificate button to create the connection certificate to AWCM
  • Choose Use Internal AWCM URL - if the connector is in LAN, and AWCM - in DMZ
  • Use buttons to switch ON services/components, which will talk to the connector (the usual minimum is LDAP, CA, SCEP, Syslog, )
  • Switch back to General tab, choose Download Enterprise Systems Connector Installer link, enter password
  • Install .NET Framework 4.6.2 on server
  • Launch the downloaded installer
  • Enter the password for certificate

Check the installation

  • Go to Groups & Settings > All Settings > System > Enterprise Integration > Cloud Connector
  • Click Test Connection and check that the connector is available

“Error : Reached AWCM but VMware Enterprise Systems Connector is not active” is resolved by server reboot and opening TCP2001 port from Cloud Connector to AWCM.

Active Directory

  • Use Company tenant, go to Groups & Settings > All Settings > System > Enterprise Integration > Directory Services
  • Choose Skip wizard and configure manually (alternative path Accounts > Administrators > Settings > Directory Services)
  • Enter domain data:
    • Directory Type = Active Directory
    • Server - domain controller name
    • Encryption Type = None
    • Bind Authentication Type - connection type = GSS-Negotiate is recommended, which means choose automatically Kerberos or NTLM depending on what is available
    • Bind Username - enter the service account user for reading the domain as <user>
    • Bind Password - enter the domain service account password
    • In Domain - Server fields enter the suffix of the domain and the name of the domain controller
  • Click Test Connection, check there is network access to the domain controller
  • Choose the User tab, DN field - choose the topmost level from the list
  • Choose the Group tab, DN field - choose the topmost level from the list
  • Click Save

Troubleshooting connection to Active Directory - see article.

Self-enrollment of Active Directory Users

  • In Company tenant go to **Groups & Settings > All Settings > Device & Users > General > Enrollment **
  • In Authentication Mode(s) choose Directory checkbox
  • Go to Restrictions, make sure that Restrict Enrollment To Known Users and Restrict Enrollment To Configured Groups are disabled.

Batch Import and Message Templates

To Batch Import users in an AirWatch group, this group needs a Group ID, which allows Enrollment into it.

During user import, a connection token can be distributed via EMail. The template language depends on the localization configuration of the specific Organization Group.

When defining localization on the topmost level, the sub-groups of the lower level may have a strange setting like “Select*”. It is recommended to specify the localization settings on each group and sub-group, so there is no obscurity in the settings.

Configure SDK default profiles

  • Switch to Groups & Settings → All Settings → Apps → Settings & policies → Security Policies, click Override
  • Leave Passcode turned on
  • Activate Single Sign-On
  • Activate Integrated Authentication to auto-enter apps and websites, put Enrollment Credentials and write star symbol ( * ) in mask = all websites

WS1 Installation Problems

Failed to extract custom package message

Console installer needs to be extracted to the *root disk C:*

AirWatch Services are not started before timeout

  1. Click Start, click Run, type regedit, and then click OK.
  2. Locate and then click the following registry subkey: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control
  3. In the right pane, locate the ServicesPipeTimeout entry.

Note: If the ServicesPipeTimeout entry does not exist, you must create it. To do this, follow these steps:

 - On the Edit menu, point to New, and then click DWORD Value.

 - Type ServicesPipeTimeout, and then press ENTER.

  1. Right-click ServicesPipeTimeout, and then click Modify.

  2. Click Decimal, type 60000 (default is 30000, use numbers from 60000 to 125000), and then click OK.

 - This value represents the time in milliseconds before a service times out.

  1. Restart the computer.

Database is not getting updated on upgrade of WS1 UEM

Error detected on upgrade to WS1 UEM 2108+

SQL update fails with error:

Only members of sysadmin role are allowed to update or delete jobs owned by a different login.

⭐️ Give account used for WS1 UEM db the MSSQL server role of sysadmin.

AirWatch Self-Service Portal gives error

“HTTP error 503: The service is unavailable”

The Self Service Portal has an associated App Pool in IIS. If the App Pool is not started we see this error.

Ensure that the SSP App Pool is started in IIS.

Log collection

See AirWatch Services and Devices log collection page

App Catalog not seen after successful enroll

Problem: App Catalog does not automatically appear after device enrollment. Solution:

  1. In AirWatch Admin Console go to Groups & Settings > All Settings > Apps > Catalog > General > Publishing > repeat SAVE procedure
  2. Alternative: create separate webclip profile for all devices with URL: https://{DS_URL}/Catalog/ViewCatalog/{SecureDeviceUdid}/{DevicePlatform} External link: https://kb.vmware.com/s/article/50100220?lang=en_US

WS1 Verify Installation

Verify Installation

  • Open AirWatch Console
  • Choose About Airwatch - check the version
  • Check the Site Links: open Groups & Settings > All Settings > System > Advanced > Site URLs and look through the links
  • Check the connection with Device Services server with a defined in the install phase external URL, signed with external certificate (type of link: https://<DS_URL>/DeviceManagement/Enrollment  )
  • Check the AWCM component, using link https://<DS_URL>:2001/awcm/status*
  • Check AirWatch services - launch services.msc in Windows Server and check that AirWatch services are Started
  • Check the GEM Inventory Service: go to the AirWatch Console server, in the folder C:\AirWatch\Logs\Services\ and delete the file AirWatchGemAgent.log; open services.msc and restart GEM Inventory Service. New log will either NOT show up, or show up without errors.  
Chapter 2

[VMW] WS1 Access

Статьи по установке, настройке и решению проблем с Workspace ONE Access (бывший Identity Manager).

Subsections of [VMW] WS1 Access

Access Installation

Connected Articles

Components

Identity Manager Appliance

  • The Service; User portal, Built-in AuthN and idP (TCP443)
  • Certificate Proxy Service (TCP5262)
  • Kerberos Key Distribution Center (KDC) (TCP/UDP 88)
  • User database (vPostgres OR external MS SQL)
  • OS = PhotonOS
  • Main WS1 Access service = horizon-workspace

Enterprise Connector

  • ❗️Old connector version 1811 needed for ThinApp publishing
  • Modern Connector is a Java Microservices app, with 4 microservices: User Auth, Kerberos Auth, Virtual App (1Gb RAM set for each service) and Directory Sync (4Gb RAM)

Increasing memory of services on connector:

  1. Log in to the Windows server in which the Workspace ONE Access enterprise service is installed.
  2. Navigate to the INSTALL_DIR\Workspace ONE Access\serviceName folder.
  3. Open the serviceName.xml file in a text editor.
  4. Change the Xmx1g entry to Xmxng where n is the maximum heap memory you want to allocate. Example: Xmx5g
  5. Save file, restart service.

Network ports for connector: Inbound & outbound TCP443 (many uses); Outbound TCP389, TCP636, TCP3268, TCP3269 (LDAP); Outbound TCP88, UDP88, TCP464, TCP135, TCP445 (Kerberos, Directory Sync); Outbound TCP53, UDP53 (DNS); Outbound TCP5555 (RSA SecurID); Outbound UDP514 (Syslog).

Intelligent Hub This is a client simultaneously for WS1 UEM and WS1 Access (Corp apps marketplace + Hub Services: people search, notifications)

Old components

Identity Manager on Windows Server (Old, uses only Cloud KDC)

  • The Service; User portal, Built-in AuthN and idP (TCP443)
  • The Connector; AuthN and User, ThinApp and Horizon Sync (TCP8443)
  • Certificate Proxy Service (TCP5262)

! For IDM on Windows do NOT use non-English localized Windows versions. Workaround: change the regional number setting for decimal to use a period “.” instead of a comma “,”. ! For IDM on Windows shutdown IIS to free up port TCP80. It is not used, but it is needed for IDM install.

Workspace One Client Mobile App App Bundle ID: com.air-watch.appcenter

Subsections of Access Installation

Managing Certificates

External links:

General Commands

Most common command is to build PFX-file from PEM files:


openssl pkcs12 -export -out certificate.pfx -inkey privkey.pem -in fullchain.pem -certfile cert.pem

WS1 Access always has to be signed with corporate or trusted certificates. If Access is clustered, sign the load-balanced name with external trusted certificate and the 3 nodes - with certs from corporate CA. 

  • Go to WS1 Access web console
  • On the top, click the Appliance Settings tab,
  • On the left, click the VA Configuration node.
  • On the right, click Manage Configuration. You will be redirected to a separate portal
  • Login as admin account
  • On the left, click Install TLS Certificates.
  • On the right, in the upper box, delete the certificate and key that are currently displayed.
  • Paste in the new PEM certificate and RSA private key. Paste every certificate in the chain: server + intermediate + root. Click Save.

❗️The order of certificates is important! First server, then intermediate, then root.

Certificate Requests


######### The cert request in idm01.domain.local.inf file

[Version]
Signature= "$Windows NT$" 
 
[NewRequest]
Subject = "CN=idm01.domain.local,OU=IT,O=Horn_n_hooves,L=Moscow,S=Moscow,C=RU"
KeySpec = 1
KeyLength = 2048
Exportable = TRUE
MachineKeySet = TRUE
UseExistingKeySet = FALSE
ProviderName = "Microsoft RSA SChannel Cryptographic Provider"
ProviderType = 12
RequestType = PKCS10
KeyUsage = 0xa0
FriendlyName = "vdm" ; needed for Horizon Connection Server only!
 
[EnhancedKeyUsageExtension]
OID=1.3.6.1.5.5.7.3.1
 
[RequestAttributes]
CertificateTemplate = WebServerExportable2008
 
[Extensions]
; If your client operating system is Windows Server 2008, Windows Server 2008 R2, Windows Vista, or Windows 7
; SANs can be included in the Extensions section by using the following text format.Note 2.5.29.17 is the OID for a SAN extension.
 
2.5.29.17 = "{text}"
_continue_ = "dns=idm01.domain.local&dns=idm01&dns=idm02.domain.local&dns=idm02&dns=idm03.domain.local&dns=idm03&dns=idm.domain.local"
##################################################################################

Bat script to submit certificate request:


set srvname=idm01.domain.local
cd C:\temp

Certreq -New -f %srvname%.inf %srvname%.req
Certreq -submit %srvname%.req %srvname%.crt
Certreq -accept %srvname%.crt
Certutil -exportpfx -p 12345 %srvname% "%srvname%.pfx"

In order to copy and paste the private key from PFX certificate for vIDM, you need a decrypted version of the key. Use OpenSSL to obtain this key:


openssl pkcs12 -in idm01.domain.local.pfx -nocerts -out idm01.domain.local_encrypted.key
openssl rsa -in idm01.domain.local_encrypted.key -out idm01.domain.local_decrypted.key

Open idm01.domain.local_decrypted.key with a text editor and copy the key from there. After inserting certificates, click OK to restart WS1 Access web service.

Password Reset

External Links:

❗️The admin user password must be at least 12 characters in length.

SSH User Password Change

Connect to WS1 Access Appliance by SSH, run command:


passwd sshuser

Web Console Admin Password Reset

  1. Log in to the URL as a root user: https://ApplianceFQDN:8443/cfg/changePassword

  2. Access WS1 Access using ssh (sshuser password needed), do su and use this command to reset password:

  • Admin console site:

/usr/sbin/hznAdminTool setOperatorPassword --pass newsecretpassword
  • Configurator page:

/usr/sbin/hznAdminTool setSystemAdminPassword --pass newsecretpassword

Root User Password Reset

WS1 Access works on VMware PhotonOS variation of Linux. It has a Single User Mode.

  1. Go to vCenter Server list of VMs;
  2. Right-click the affected WS1 Access OVA and click Open Console;
  3. Under the VM menu, click Power->Shut Down Guest;
  4. When the shutdown completes, Power On the WS1 Access Appliance;
  5. When the GNU GRUB menu displays, press p and enter the configured bootloader password = ⭐️ H0rizon! ⭐️ ; ❗️GRUB menu appears for a few seconds. If you miss it, reboot and try again;
  6. Use the up and down arrow keys to navigate to the first entry, and press e to edit the relevant boot parameters;
  7. Use the arrow keys to navigate to the line beginning with kernel and press e to edit;
  8. The cursor is at the end of the line, type a space and then append init=/bin/bash to the line;
  9. Press Enter to confirm the changes;
  10. Press b to execute the boot;
  11. After boot, you have ROOT access and you are able to set new passwords, for root: passwd and for sshuser: passwd sshuser; ❗️Follow password policies on new passwords!
  12. Shutdown by using command: shutdown -h -P now; ❗️VMware Tools do NOT work in Single User Mode, so shutdown from vCenter will NOT work;
  13. Start WS1 Access Appliance.

SQL Preparation

Attachments

Manual

Internal PostgreSQL

To login into the DB, get the PostgreSQL password: first login to console with SSH.


cat /usr/local/horizon/conf/db.pwd

Copy password, then login with it:


psql saas horizon

External MS SQL Preparation

❗️ Database schema name must be ‘saas’, cannot be changed. ❗️Collation must be ‘Latin1_General_CS_AS’, could be changed but change not recommended. ❗️The server role used to grant server-wide security privileges is set to public. The database role membership is db_owner.

Microsoft SQL Database Using Local SQL Server Authentication Mode for Workspace ONE Access (replace values in brackets < > ):


CREATE DATABASE saas
COLLATE Latin1_General_CS_AS;
ALTER DATABASE saas SET READ_COMMITTED_SNAPSHOT ON;
GO

BEGIN
CREATE LOGIN <loginusername> WITH PASSWORD = N'<password>';
END
GO

USE <saasdb>; 
IF EXISTS (SELECT * FROM sys.database_principals WHERE name=N'<loginusername>')
DROP USER [<loginusername>]
GO

CREATE USER [<loginusername>] FOR LOGIN [<loginusername>]
WITH DEFAULT_SCHEMA=saas;
GO

CREATE SCHEMA saas AUTHORIZATION <loginusername>
GRANT ALL ON DATABASE::saas TO <loginusername>;
GO

ALTER ROLE [db_owner] ADD MEMBER <loginusername>;
GO

JDBC URLs

SQL local user jdbc:sqlserver://<DB_VM_IP_ADDR>;DatabaseName=saas

jdbc:sqlserver://<DB_VM_IP_ADDR>\INSTANCE_NAME:PORT;DatabaseName=saas (you can remove the instance name if default)

AD domain user jdbc:jtds:sqlserver://<DB_VM_IP_ADDR>:1433/saas;integratedSecurity=true;domain=LAB.LOCAL;useNTLMv2=true

Multi-site, SQL Always On jdbc:sqlserver://;DatabaseName=saas;multiSubnetFailover=true

Troubleshooting Issues

Troubleshooting JDBC URL Wizard page

Back and continue buttons become greyed out and unclickable.

  • Use Web Admin tools of the Browser (Firefox, Chrome);
  • Right click and “Inspect Element” on the disabled button. Find the ID tag for “nextButton”;
  • Within this line of text there is a value = “is-disabled”. Remove “is-disabled” from the line by clicking and typing into the inspector;
  • Return to web page, button should start working. Click & proceed.

Database Locked Error

See resolution link

Check the result of the following DB query, If DB is locked, you should see an entry with the locked value set to 1 or TRUE.

	
SELECT * FROM DATABASECHANGELOGLOCK 

if locked=TRUE then run the below UPDATE statement to release/reset the DB lock.

USE [DBNAME]
GO 

UPDATE [saas].[DATABASECHANGELOGLOCK]
   SET [LOCKED] = 0 

 ,[LOCKGRANTED] =NULL 

 ,[LOCKEDBY] = NULL
 WHERE ID = 1 

GO

Restart horizon-workspace service on each node one after another. ❗️DO NOT restart all nodes of WS1 Access cluster simultaneously.

Important Queries

Update the Connector if there is more than 1 sync and Authentication


UPDATE Connector SET isDirectorySyncEnabled=false WHERE host=<auth_connector_hostname>
	

Check if IDM certificate is present in the console from DB side


SELECT * from dbo.coreuser WHERE isactivedirectoryuser=0  
SELECT * from dbo.certificate WHERE certificatethumbprint like '%%'  
SELECT * from dbo.UserLink WHERE coreuserid = < username >
SELECT * from dbo.role WHERE roleid =3
	

List of UUID with Super Admin access


SELECT strUsername, strFirstName, StrLastName,strExternalId, stremail, uuid FROM slesdb.saas.Users U (nolock) INNER JOIN  
(SELECT * FROM Slesdb.saas.ACS_RuleSetAssociation (nolock)  
WHERE ruleSetId LIKE  
(SELECT id FROM slesdb.saas.ACS_RuleSet (nolock)  
WHERE name LIKE 'Super Admin)) as A  
On U.uuid = A.SubjectUUID
ORDER BY strUsername
	

Update email address of the user or admin


UPDATE sles.saas.Users
SET strEmail = 'value'  
WHERE username LIKE 'value'
	

Update the value of the Identity Provider with correct connector name


UPDATE slesdb.saas.IdentityProviders  
SET strDescription = (  
SELECT host FROM slesdb.saas.Connector WHERE id = (

SELECT idConnector FROM slesdb.saas.IdpJoinConn WHERE idIdentityProvider = (  
SELECT id FROM slesdb.saas.IdentityProviders WHERE strFriendlyName LIKE '%Workspace%'))) 
	

Update the attribute column to make it mandatory or non-mandatory


 SELECT id, * FROM slesdb.saas.userattributedefinition WHERE idorganization IS NOT NULL AND ownerUuid IS NULL AND strName LIKE '<attribute to be updated>
	

Then run:


UPDATE slesdb.saas.userattributedefinition SET bIsRequired = 0 WHERE id = <id identified above>
	

Connector Sync Validation


SELECT id, uuid, tenantID, host, domainJoined, createdDate, oAuth2ClientId, isDirectorySyncEnabled FROM saas.Connector;

SELECT idSyncProfile, directoryConfigId, syncConnectors FROM saas.DirectorySyncProfile
	

⭐️ FROM saas.Connector isDirectorySyncEnabled = 1 means connector is the connector set as Sync. If not, you can update:


UPDATE saas.connector SET isDirectorySyncEnabled=0 WHERE id=1;
	

⭐️ FROM saas.DirectorySyncProfile syncConnector = saas.Connector uuid value. You can update if needed as well:


SET isDirectorySyncEnabled=0 WHERE id=1;  
update saas.DirectorySyncProfile SET syncConnectors='["12345678-abcd-1234-1234-0123a678b78"]' WHERE idSyncProfile=1;
	

AD Connection Removal

Supported removal

  • Directory cannot be removed, because an active Connector is attached to it;
  • Connector cannot be removed because an Identity Provider (IdP) is associated with it (BuiltIn IdP);
  • BuiltIn IdP cannot be dissociated, because it is locked to the Connector.
  • If you need to delete the Connector, you have to delete User Directory;
  • If you need to delete User Directory, then Built-in IDP should not be associated with it.

Steps:

  • Check the Built-in Identity Provider and take note of the access policies associated with the Built-in Identity Provider;
  • Un-associate all the policy rules. Edit Identity & Access Management → Policies tab (Setup mode) - remove “Password (Cloud Directory)” from policies, put “Password (Local Directory)” instead;
  • Delete the Built-in IdP:
    • Remove the connector associated with the Built-in IDP and save it. Uncheck the box for Password (Cloud deployment) in connector properties and save it;
    • Click on red cross to delete Connector in Built-In properties;
    • Click on Delete IDP. Click OK to delete IdP.
  • Now you can go to Directories and delete the User Directory;
  • Wait until the directory is deleted. Check by refreshing directory page;
  • Delete the Connector using the red delete button on the right of its’ name in the list;
  • Go via RDP to the server with Enterprise Connector, uninstall it.

Unsupported removal

Access saas database of IDM and delete association of IdP with Connector.

For PostgreSQL

If database of IDM is deployed as inherent PostgreSQL, then use SSH to connect to the primary appliance. ❗️Database on primary appliance is in read/write mode.

Connect to database:


su - postgres /opt/vmware/vpostgres/current/bin/psql vcac

Change schema to saas:


SET SCHEMA 'saas';

For Microsoft SQL Server

Use SQL Management Studio to connect to DB server, open saas database.

Find Connector table (example shown for MS SQL Server 2014) and ‘disconnect’ it from AD:


SELECT * FROM saas.saas.Connector;

-- Check the ID of your AD connector

UPDATE saas.saas.Connector
SET domainJoined = NULL
WHERE id = N; -- set N to the ID of your AD connector

❗️You cannot delete the Connector in SQL - it has dependencies from foreign keys. But all you need to do is find the ID of BuiltIn IdP and disconnect it:


SELECT * FROM saas.saas.IdentityProviders;
SELECT * FROM saas.saas.IdpJoinConn;

-- Check the ID of your IdP defore doing this!!

DELETE FROM saas.saas.IdpJoinConn
WHERE idIdentityProvider = N; -- set N to the ID of your IdP
  • After this, you can delete the Connector in Identity Manager Console. Then delete the AD Directory;
  • Reinstall Identity Manager Connector and activate it.

❗️After adding a new AD do not forget to follow the guide backwards: do not forget to add “Password (Cloud Directory)” back to Policies!

❗️The BuiltIn IdP must be associated back with the new Connector. This can be done using the console, or it can also be done in SQL:


SELECT * FROM saas.saas.IdpJoinConn;

-- Double-check the ID of your IdP and NEW Connector defore doing this!!

INSERT INTO saas.saas.IdpJoinConn
VALUES (X, Y) -- set X to the ID of your IdP, set Y to the ID of Connector

Elasticsearch

Background

Elasticsearch in VMware Workspace ONE Access is used for storing records on:

  • Audit Events and Dashboards (all events)
  • Sync Logs (LDAP)
  • Search data, Peoplesearch (users, groups, apps)
  • Entitlements (apps)

These records are collected on the local SAAS node and are being sent to RabbitMQ instance on the local node once per second. They are then picked up and sent to Analytics Service on local node, which then sends the records for storing in Elasticsearch. RabbitMQ acts as a buffer, so that if Elasticsearch is down, the records are not lost. It also decouples the generation rate from the storing rate, so if records are coming in faster than they can be stored, they can be queued up and left in RabbitMQ.

RabbitMQ previously WAS clustered, so records generated on one node could be picked up and processed by any other node in the cluster. This meant that if the local Elasticsearch instance is having issues, another node could grab the message and process them. Now RabbitMQ is NOT clustered (it had complex node stop/start order requirements to maintain a stable cluster, and maintaining a RabbitMQ cluster was more hassle than it was worth), so it is possible for the records generated on one node to start piling up in RabbitMQ. This is less robust, but makes it easier to spot when a specific node is having a problem.

Services connected to Elasticsearch:

  • elasticsearch
  • horizon-workspace
  • hzn-dots
  • hzn-network
  • hzn-sysconfig
  • rabbitmq-server
  • thinapprepo (optional, only if ThinApp is used)
  • vpostgres (optional, only for internal DB)

Elasticsearch Indices and Shards

Elasticsearch is a distributed, replicated data store, which means each node does not have full copy of the data. In a 3-node cluster, each node will have 2/3 of the data, so that all 3 nodes combined provides 2 full copies, and you can lose one node without losing any data. This distribution and replication is achieved by splitting each index into 5 shards and each shard has a primary copy and a replica (backup) copy. All ten shards for an index are then distributed across all three nodes so that no single node has both the primary and replica copy of any one shard, or all the shards for one index.

Audit data and sync logs are stored in a separate index for each day. The index is versioned based on the document structure so that you can seamlessly migrate from an old format to a new format. The audit indexes have the format "v<version>_<year>-<month>-<day>'. For example, version 4 of the document structure will be "v4_2022-03-04".

Search data (users, groups, applications) used by the admin UI for search and autocomplete, and the PeopleSearch feature, is stored in a separate index called "v<version>_searchentities", and currently uses version 2 of that document structure.

⭐️ The indices can be seen in the /db/elasticsearch/horizon/nodes/0/indices directory or listed with useful stats via: curl localhost:9200/_cat/indices?v

Elasticsearch Node Discovery

A cluster has a master node which has the additional job of coordinating the distribution of data and farms out the search and document retrieval requests to the other nodes based on which node has which shard. Any node can be the master node, the nodes choose the master themselves based on the makeup of the cluster. If the master node disappears, one of the remaining nodes is chosen as the new master.

Elasticsearch uses a list to determine how many nodes it should be able to see before a master can be elected and the cluster formed (and the other nodes its should be reaching out to). This is done to prevent a split-brain situation, where due to network issues not all the nodes can see each other, so form independent clusters, which require resetting the errant node and losing its data. So when WS1 Access tells Elasticsearch there should be 3 nodes, it knows it must see at least 2 nodes to form a cluster.

❗️If Elasticsearch is up before WS1 Access (horizon-workspace service), there will be timeout failures in the logs, which can result in the cluster getting stuck. If the other nodes in the cluster are not up, there will be error messages about trying to add them to the cluster and then removing them again because they did not respond.

Data Retention

❗️Due to rules and regulations in some countries and companies around data retention of audit data, by default historical data is not deleted.

Elasticsearch maintains a very complex series of lookup tables in memory that allows it to determine which documents match a particular query in just a couple of milliseconds, whether there is one document to query or a billion.

By default, to reduce memory footprint, the number of days (indices) allowed for Elasticsearch to keep in memory (or open) is limited to 90 days.
Older indices are closed to remove them from memory and are no longer searchable. Once a day, just after midnight (00:30am), the retention policy is executed, and any open index that is older than the 90 days is closed.

Also the open indices that are no longer being written to (any open index older than the current day) are optimized: As documents come in, Elasticsearch writes them to lots of small files on disk. This lets it store documents quickly, but slows down retrieval and consumes a huge amount of open file descriptors. When an index is no longer being written to (for example, because it is for yesterday and no dated records for yesterday should be coming in anymore to write to it), Elasticsearch takes all those little files and combines them into one big file. This reduces the number of file descriptors needed and speeds up document retrieval for older documents.

When the retention policy runs, there will be messages in the analytics-service log saying it has started, and for each open index, whether it was closed because it is now past the cut-off date, or was optimized (the optimization is blindly optimizing all open, older-than-today indices, because if it is already optimized, Elasticsearch does nothing).

com.vmware.idm.analytics.elasticsearch.ElasticSearchHttpStorageAdapter - Executing analytics retention policy, cutoff date is 2019-04-07
com.vmware.idm.analytics.elasticsearch.ElasticSearchHelper - Closed index: v4_2019-04-06
com.vmware.idm.analytics.elasticsearch.ElasticSearchHelper - Optimizing index v4_2019-05-02 
com.vmware.idm.analytics.elasticsearch.ElasticSearchHelper - Optimizing index v4_2019-06-28 
com.vmware.idm.analytics.elasticsearch.ElasticSearchHttpStorageAdapter - Analytics retention policy completed.

Elasticsearch and RabbitMQ Health Status

Time skew

Elasticsearch will be out of sync when the WS1 Access nodes are with different times. The easy way of checking is from the System Diagnostics Page -> Clocks - it should show all the clock times.

Check using CLI on all nodes: watch -n 1 date

Disk space

Due to default policy of never deleting old data, the /db filesystem usage will continue to grow until it runs out of space.

  • Elasticsearch will stop writing documents when it get 85% full (set via the cluster.routing.allocation.disk.watermark.low setting).
  • RabbitMQ will stop working when it gets down to just 250MB free (set via the disk-free-limit setting).

Check the free disk space for the /db filesystem with the command: df -h

RabbitMQ Queues

An indicator that something is wrong is if the size of pending documents in the analytics queue is growing. RabbitMQ is used for other types of messages, which go in their own queues, but the analytics queue is the only one that does not have a TTL, so it is the only one that can grow. This is done so that its’ messages never get deleted and the audit records never get lost. Typically the analytics queue should have just 1-2 messages waiting to be processed, unless something is wrong or there is a temporary spike in records being generated (during a large directory sync the queue might temporarily grow, if the system cannot keep up).

Check queue size on each node with CLI: rabbitmqctl list_queues | grep analytics

If the number reported is larger than 100, its time to investigate why.

⭐️ This is also reported as the AuditQueueSize value in WS1 Access health API: /SAAS/API/1.0/REST/system/health/

General Elasticsearch Checks

Overall cluster health:

curl http://localhost:9200/_cluster/health?pretty

Check that the node counts (number_of_nodes and number_of_data_nodes) equals the size of the cluster and the state is green (except for single nodes, whose state will always be yellow)

GREEN = good, there are enough nodes in the cluster to ensure at least 2 full copies of the data spread across the cluster;

YELLOW = ok, there are not enough nodes in the cluster to ensure HA (single-node cluster will always be in the yellow state);

RED = bad, unable to query existing data or store new data, typically due to not enough nodes in the cluster to function or out of disk space.

Check that nodes agree on which ones are actually in the cluster and who the master is (to ensure you don’t have errant nodes, or split-brain with two clusters instead of one).

Run this on each node and verify the output is the same for every node:

curl http://localhost:9200/_cluster/state/nodes,master_node?pretty

This information is also reported in the “ElasticsearchNodesCount”, “ElasticsearchHealth”, “ElasticsearchNodesList” values in the health API, but that will only be coming from one of the nodes as picked by the gateway: /SAAS/API/1.0/REST/system/health/

Other useful commands:

curl http://localhost:9200/_cluster/health?pretty=true
curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'

curl http://localhost:9200/_cluster/state/master_node,nodes?pretty
curl http://localhost:9200/_cluster/stats?pretty=true
curl http://localhost:9200/_nodes/stats?pretty=true
curl http://localhost:9200/_cluster/state?pretty=true

curl http://localhost:9200/_cat/shards | grep UNASSIGNED 
curl -XGET ‘http://localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason’ | grep UNASSIGNED 

Proactive support

Avoiding disk space issues

Determine the growth rate of data nodes can store. Use that to either schedule disk size increase, or determine how many days of data with their usage they can keep.

After a few weeks of running the system, look at the size of each index and determine the average day’s size using:

curl localhost:9200/_cat/indices?v

Extrapolate how many days nodes have space for. For safety, use 50-60% of space as the benchmark. You can change the data retention policy to automatically delete old data so that the filesystem usage is capped. This will not interrupt service and is done by a configuration change that must be done on each node, setting the maxQueryDays to the number of days to keep data. The new policy will take affect at 00:30 am:

  • Edit /usr/local/horizon/conf/runtime-config.properties on each node to add/modify the following lines:
analytics.deleteOldData=true   (default is false, close the older indices)
analytics.maxQueryDays=90      (default is 90)
  • Restart the workspace service after making the change. To avoid any interruption in service, wait for the service to be fully back up before moving on to the next node:
service horizon-workspace restart
  • When all nodes have been updated and restarted, pick one of the nodes and tell the cluster to re-open any old, closed indices so that the new policy will find and delete them:
curl -XPOST http://localhost:9200/_all/_open

Avoid full cluster restart issues

WS1 Access can take a long time to come up, because the first node has the default configuration of only expecting one node in the cluster and decides it is the master. Then it checks what indices it has and discovers a bunch a shards it cannot find. When the second node comes up, it is told by the first node that the first node is the master, but the first node is stuck because of the missing shards, so refuses to act like a master. The same thing happens when the third node comes up. This situation can be fixed by restarting Elasticsearch on the first node. This can be completely avoided by changing the configuration, so that on restart all the nodes know to expect more nodes in the cluster, so they wait and only elect a master when the second node comes up.

On each node, edit /opt/vmware/elasticsearch/config/elasticsearch.yml to add these lines, which tell the cluster to not form until at least 2 nodes are in communication, and to wait up to 10 minutes for a third node to come online before deciding to start copying shards between the two nodes that are there to ensure HA. It will also prevent accidentally starting Elasticsearch more than once on a node:

discovery.zen.minimum_master_nodes: 2
gateway.recover_after_nodes: 2
gateway.expected_nodes: 3
gateway.recover_after_time: 10m
node.max_local_storage_nodes: 1

❗️❗️ NOTES: ❗️❗️

  • Make sure there are no leading spaces in front of the entries, otherwise elasticsearch will not start (no “initializing” message in the log).
  • This should only be done for a 3 node cluster, this will break single nodes.
  • A restart after applying this configuration is NOT needed, the config is only applicable when a restart is happening, so it will take affect the next time the cluster restarts.

Elasticsearch Manual Reindex

. /usr/local/horizon/scripts/hzn-bin.inc && /usr/local/horizon/bin/curl -v -k -XPUT -H "Authorization:HZN <cookie>" -H "Content-Type: application/vnd.vmware.horizon.manager.systemconfigparameter+json" https://localhost/SAAS/jersey/manager/api/system/config/SearchCalculatorMode -d '{ "name": "SearchCalculatorMode", "values": { "values": ["REINDEX"] } }'

Troubleshooting Elasticsearch and RabbitMQ

Elasticsearch cluster health returns MasterNotDiscoveredException

When you try to check the elasticsearch cluster health you get:

{
  "error" : "MasterNotDiscoveredException[waited for [30s]]",
  "status" : 503
}

You can enable additional logging by editing /opt/vmware/elasticsearch/config/logging.yml and adding the following line after the “org.apache.http: INFO” one.
com.vmware: DEBUG OR com.vmware: TRACE

Causes:

  1. Stale/incorrect node entries stored in the DB being reported to Elasticsearch. Look at the contents of the “ServiceInstance” DB table. If there are any wrong entries manually delete them. Check the “status” value is 0. Any other value is indicates an issue with that node that needs to be looked at (1 = Inactive, 2 = error, 3 = invalid master key store, 4 = pending removal). When the issues with the ServiceInstance table have been addressed, within 30 seconds the elasticsearch plugin on all the nodes should now get just the correct records and sort itself out. If it hasn’t done that within a few minutes, restart elasticsearch on each node (service elasticsearch start).

  2. Mis-configured multi-DC setup, resulting in the service instance REST API reporting incorrect nodes to elasticsearch. On the affected nodes check the output of:

/usr/local/horizon/bin/curl -k https://localhost/SAAS/jersey/manager/api/system/clusterInstances

If it does not show the entries you expect, look in the “ServiceInstance” table in the DB and check the “datacenterId” column value is correct for the nodes (so they are identified as part of the correct per-DC cluster. Check the “status” value is 0. Any other value indicates an issue with that node that needs to be looked at (1 = inactive, 2 = error, 3 = invalid master key store, 4 = pending removal).

Elasticsearch unassigned shards

❗️ DO NOT DELETE UNASSIGNED SHARDS
You need to address the underlying cause, then the unassigned shards will be resolved by the cluster on its own. It is extremely rare for unassigned shards to be a real issue needing direct action (eg. filesystem corruption, accidental data deletion).

Elasticsearch cluster health yellow

One of the nodes is down, unreachable or broken. Find out which one is in trouble by getting the list of nodes and seeing which one is missing:

curl http://localhost:9200/_cluster/state/nodes,master_node?pretty

Do not worry about unassigned shards in the health API output, getting the node back in the cluster will allow the master to sort out who has what shard and within minutes the unassigned shards will be resolved and the cluster green.

Causes:

  1. If the node is down, bring it back up.

  2. Check the node is reachable from the other nodes in the cluster. If it isn’t, fix the networking/firewall issue so that it can rejoin the cluster.

  3. On the missing node, check the disk space is ok. If it’s full, either:

    a. increase the size of the filesystem, OR

    b. free up disk space to get the cluster back to green by manually deleting the older indices, based on their date. To do that, first stop Elasticsearch on all the nodes elasticsearch service stop. Then delete the directories for the oldest indices in the /db/elasticsearch/horizon/nodes/0/indices directory on the node with disk space issues. When there is at least 500MB-1GB free space again on that node, delete the exact same ones on the other two nodes. Finally, restart Elasticsearch on each node service elasticsearch start and the cluster should go back to green and then you can adjust the retention policy.

Elasticsearch cluster health red

At least two of the nodes are broken, the analytics queue in RabbitMQ will start to grow.

Causes:

  1. Check for the same causes as the yellow health.
  2. All nodes are up but the master node is misbehaving. You may see messages like this in the logs of the non-master nodes where they try to connect to the master, but are refused:
[discovery.zen ] [Node2] failed to send join request to master [Node1], 
reason [RemoteTransportException[[Node1]][internal:discovery/zen/join]]; nested: 
ElasticsearchIllegalStateException[Node [Node1]] not master for join request from [Node2]

Determine the master node from the log (eg Node2 in the example above) or with: curl http://localhost:9200/_cluster/state/nodes,master_node?pretty

Restart Elasticsearch on the master node service elasticsearch restart . One of the other two nodes should immediately become the master (re-run the curl on the other nodes again to verify) and the old master will re-join the cluster as a regular node when it comes up.

RabbitMQ analytics queue is large

The analytics queue size in RabbitMQ should usually only show 0-5 messages when the system is functioning correctly. If you see larger values, this means messages were not able to be delivered to elasticsearch.

Causes:

  1. The issue is with sending messages due to infrastructure issues. Look in the logs for error messages from “AnalyticsHttpChannel” to see if there is an issue sending (bad cert or hostname).
  2. The issue is with connecting to RabbitMQ. Look in the logs for messages that contain “analytics” and “RabbitMQMessageSubscriber”.
  3. The Elasticsearch status is yellow or red. Follow the steps above to remedy.
  4. It’s an old message. If the Elasticsearch status is green, look in the analytics-service log for details on why the message is unable to be processed. If you see a bulk store failure message containing “index-closed-exception”, then its because the message contains a record with a timestamp older than maxQueryDays. Allow the message to get processed by opening all the indices (the nightly data retention policy will then close any that need it again): curl -XPOST http://localhost:9200/_all/_open

You can use following command to check the indices status:

curl http://localhost:9200/_cat/indices

The messages should now be able to get processed and the queue size should drop.

  1. Its a bad message. If the Elasticsearch status is green, look in the analytics-service log for details on why the message is unable to be processed. If you see a bulk store failure message containing "parse_exception", then there is a malformed document.
  • You can clear the entire queue by executing rabbitmqctl purge_queue <queue>.
    Note: This will cause the loss of any other audit events, sync logs and search records that were also queued up, so the search index must be rebuilt to ensure the search/auto-complete functionality continues to work fully.

  • You can also use the management UI to remove just the bad message. Go to the queue in the Management UI and select the “Get messages” option with re-queue set to false to pull just the first message off the queue (which should be the bad message). Once the bad message is cleared, the others should get processed successfully.

  1. If the queue is for replicating to a secondary DC, but are old (DNS name change, or used IP addresses (other than 127.0.0.1) instead of FQDN), then the queue must be manually deleted from each node to prevent it from continuing to accumulate messages by doing the following on each node:

a. Enable firewall access to the RabbitMQ management UI: - Edit /usr/local/horizon/conf/iptables/elasticsearch to add “15672” to the ELASTICSEARCH_tcp_all entry, => ELASTICSEARCH_tcp_all="15672" - Re-apply the firewall rules by executing the /usr/local/horizon/scripts/updateiptables.hzn script.

b. You can only access the RabbitMQ Management UI using the default credentials of guest/guest on localhost, so you will have to create a new admin user in RabbitMQ. To create a user called “admin” with the password “s3cr3t”:

       rabbitmqctl add_user admin s3cr3t
       rabbitmqctl set_user_tags admin administrator
       rabbitmqctl set_permissions -p / admin ".*" ".*" ".*"

c. In a browser, access the RabbitMQ Management UI by going to http://host-or-ip-of-node:15672

  • log in with the newly created user
  • click on the “queues” tab
  • select the bad queue from the list
  • select delete from the “Delete / purge” option at the bottom of the page.

d. For security, re-disable remote access to the RabbitMQ Management UI by re-doing the a) step, but removing the 15672 port: ELASTICSEARCH_tcp_all="", then re-apply the firewall rules with the updateiptables.hzn script.

Diagnostics Page or Health API shows “Messaging Connection Ok: false”

This means RabbitMQ is in a bad state. Causes:

  • Out of disk space. Check the RabbitMQ status and compare the "disk_free_limit" and "disk_free" values to verify:
rabbitmqctl status

See cause #3 in the “Elasticsearch cluster health yellow” section for the steps to free up disk space.

  • Recovery file corruption, possibly due to an abrupt power off/failure on the VM See Article

RabbitMQ is not able to start, i.e. rabbitmqctl status gives a “nodedown” error and you see this error in /db/rabbitmq/log/startup_log:

      BOOT FAILED ... {error, {not_a_dets_file, 
      "/db/rabbitmq/data/rabbitmq@<YOUR_HOSTNAME>/recovery.dets"}}}

This will require deleting the recovery file and starting RabbitMQ again:

a. Stop all RabbitMQ processes by doing kill -9 on all processes found with ps -ef | grep -i rabbit, until none remain.

b. rm /db/rabbitmq/data/rabbitmq@<YOUR_HOSTNAME>/recovery.dets

c. rabbitmq-server -detached &

Once RabbitMQ is running again, workspace should be able to connect to it. Note, it may take up to 10 minutes before re-connection is tried. Look in horizon.log for messages from RabbitMQMessageSubscriber and RabbitMQMessagePublisher. If you see messages that indicate they were shutdown instead of connected or there are ProviderNotAvailableExceptions in the log, then workspace must be restarted service horizon-workspace restart to recreate them.

New/changed users or groups are not showing up in the search or assign to application boxes

  1. Ensure the cluster is healthy and all the nodes agree on the who is in the cluster and who the master node is by following the steps in “Elasticsearch Cluster Health” above;
  2. Ensure RabbitMQ queue size is not large, indicating messages containing the changes are undelivered to elasticsearch;
  3. If, after addressing any issues found with the previous checks, trigger a re-index of the search data.

Reset Everything (NUCLEAR BUTTON)

❗️❗️ DO NOT USE UNLESS NOTHING ELSE HELPS ❗️❗️

Option 1

Stop elasticsearch on all nodes, delete the data in elasticsearch and RabbitMQ, restart the elasticsearch process on each node, and trigger a re-index. This will result in the loss of the historic sync logs and audit data.

NOTE: If this is a Multi-Site setup:

  • Only steps 1-6 should be done on the DR site first (don’t do step 7, the reindex, it should ONLY to be done on the primary site, the results of that will then get replicated over to the DR site.).
  • When the DR site is OK, perform all 7 steps on the PRIMARY site.

Step 1. Stop elasticsearch on every node in the site: service elasticsearch stop

Step 2. When stopped on all nodes, clear the data, by doing the following on each node:

rm -rf /db/elasticsearch/horizon
rm -rf /opt/vmware/elasticsearch/logs 
//so we start with fresh logs to make it easier to see what is happening.

Get the analytics queue name(s) and purge them. On the primary site there should be two analytics RabbitMQ queues that need purging, one for 127.0.0.1 ("-.analytics.127.0.0.1") and one for the DR site (as configured via the analytics.replication.peers setting in the runtime-config.properties file, “-.analytics.dr.site.com”):

  rabbitmqctl list_queues | grep analytics
  rabbitmqctl purge_queue <analytics-queue-name>

Step 3. Verify the data retention policy configuration on each node is the same by looking at the /usr/local/horizon/conf/runtime-config.properties file and the “analytics.maxQueryDay” and “analytics.deleteOldData” values. If any node is different, then change them to match and restart WS1 Access on the node that did not match with: service horizon-workspace restart

Wait for WS1 Access to be back up before restarting another and before going on to step 4.

Step 4. Add the additional config to Elasticsearch from the above chapter “Avoid full cluster restart issues” preventative measure to help it recover from a full cluster restart, by adding these lines to /opt/vmware/elasticsearch/config/elasticsearch.yml on each node if missing:

   discovery.zen.minimum_master_nodes: 2
   gateway.recover_after_nodes: 2
   gateway.expected_nodes: 3
   gateway.recover_after_time: 10m

Step 5. Restart elasticsearch on each node (preferably quickly) service elasticsearch start

Step 6. Verify the cluster is functioning fully by running the following on each node and making sure the results match (ie, nodes are 3, same master and the node list is the same) and health is green:

  curl http://localhost:9200/_cluster/state/nodes,master_node?pretty
  curl http://localhost:9200/_cluster/health?pretty

If the output for those two commands do not match on all nodes stop, do not proceed to step 7. Gather the logs in /opt/vmware/elasticsearch/logs for each node.

STEP 7. Trigger a re-index of the search data.

To manually force a re-index, choose either the zero-down time option if you’re comfortable finding the HZN cookie value, or the down-time option by modifying the value directly in the DB.

The re-index should only take a few minutes. You can verify it started by looking in the workspace horizon.log for a message like:

com.vmware.horizon.search.SearchCalculatorLogic - 
Keep existing index. Search calculator mode is: REINDEX

Zero down-time

  • Log in to VIDM as the operator/first admin.
  • Use the developer tab in your browser to find the cookies (eg for Firefox, select Web Developer→Network. Then load a page and select the first request. Then select the “Cookies” tab and scroll down to the HZN cookie).
  • Copy the value of the HZN cookie.
    
  • ssh into a node make the following REST API call, replacing <cookie_value> with the HZN cookie value obtained from the browser.
    
. /usr/local/horizon/scripts/hzn-bin.inc && /usr/local/horizon/bin/curl -k -XPUT -H "Authorization:HZN <cookie_value>" -H "Content-Type: application/vnd.vmware.horizon.manager.systemconfigparameter+json" https://localhost/SAAS/jersey/manager/api/system/config/SearchCalculatorMode -d '{ "name": "SearchCalculatorMode", "values": { "values": ["REINDEX"] } }'

After initiated “REINDEX”, you can change the log to confirm it is started

cat /opt/vmware/horizon/workspace/logs/horizon.log | grep REINDEX

With down-time

Stop horizon-workspace on all nodes: service horizon-workspace stop

Edit the DB:

UPDATE "GlobalConfigParameters" SET "strData"='REINDEX' WHERE "id"='SearchCalculatorMode';

Restart horizon-workspace on all nodes: service horizon-workspace start

Verify progress

The reindex should start within a couple of seconds. Verify the reset was successful by looking for the message “Forcing a REINDEX of the search index” in the logs.

Reindexing can take hours, depending on the size of the data. You can monitor its progress by looking in the logs for “SearchCalculatorLogic” statistic messages, which show the counts of the objects its currently processing (typically it will be 5000 at a time when doing a re-index). When the all counts reach zero, it means it has finished. eg:

2020-02-28T05:54:08,208 WARN (pool-37-thread-2) [ACME;-;-;-] 
com.vmware.horizon.common.datastore.BaseCalculator - ++SearchCalculatorLogic$
$EnhancerBySpringCGLIB$$3e4c8f8c [ 0] 0.000000/Object ([ 0] 0.000000/Group, [ 0] 0.000000/Users, 
[ 0] 0.000000/ResourceDb)

Option 2

Re-Doing the Elastic Search Index.

Symptom: in case the user/group search is not working properly.

Resolution: Stop workspace service (on all nodes):

service horizon-workspace stop

  1. For User Entitlement:

update saas.GlobalConfigParameters set "strData"=-1 where "id"= 'LatestUserEntitlementVersion’;

For Search:

update saas.GlobalConfigParameters set "strData"=-1 where "id"= 'LatestSearchVersion’; 

For User Group association:

update saas.GlobalConfigParameters set "strData"=-1 where "id"=   'LatestUserGroupVersion’;
  1. Start service (on all nodes):
service horizon-workspace start

Mobile SSO - iOS

External Links:

❗️❗️ Do NOT use UAG in front of WS1 Access with MobileSSO for iOS scenario!

Schema

KDC Schema is here

Configuation Logic

ADMINISTRATOR CONFIGURATION

  • Following the red lines, the Administrator is going to configure REALM DNS entries (for on-premises only).
  • The admin will configure the Certificate Authority as well as the Cert templates and get the CA Cert for uploading into AirWatch and IDM.
  • The admin will configure AirWatch with the Cert Authority configurations, including setting up the certificate template and device profiles for pushing certificates.
  • Finally the admin will configure Identity Manager via the Admin Console to setup the Built-In Kerberos adapter and KDC with Cert Authority CA Root Certs from both AirWatch admin and Certificate Authority as necessary. Additionally, the admin will configure the Built-In IDP and authentication policies within Identity Manager to properly use the CA and Device certificates delivered through AirWatch to the end device.

USER ENROLLMENT

  • Next the user will enroll the device with AirWatch.
  • The enrollment request to AirWatch will request a device certificate from the Certificate authority.
  • The certificate bundle will be delivered back to the device during the completion of the enrollment.
  • Additionally, any native AirWatch applications such as Workspace ONE and other apps will be pushed to the iOS device.

USER APP LAUNCH

  • User launches application such as SalesForce. This requests gets passed down via TCP 443 through the load balancer, and hits the Built-In IDP and Built-In Kerberos Adapter. The Authentication Policy responds with the type of authentication it is expecting to see (i.e. Built-In Kerberos). This is passed back to the iOS device within a token which then tells the device to authenticate to a specific REALM.
  • The iOS device then does a REALM lookup against DNS and gets the specific KDC server it should authenticate against.
  • The iOS device requests authentication via TCP 88 to the KDC server, sending the Device certificate to the KDC, which then responds with an authentication approval.
  • Once IDM receives the authentication approval, the app launches on the iOS device and the user goes about their day.

iOS Login Flow

iOS Login Flow Schema is here

Login Flow Steps:

  • Managed iOS device is deployed with Kerberos SSO profile, a user certificate and a vanilla SalesForce app.
  • SalesForce.com has been configured to use Access/vIDM for authentication using SAML.
  • On launch, the salesforce app connects to salesforce.com and is redirected to VMware Access/vIDM.
  • Access/vIDM evaluates the authentication policy and determines that built-in KDC authentication should be used for the request. The built-in KDC adapter sends a SPNEGO 401 response to the device.
  • iOS on the device intercepts the response and performs Kerberos PKINIT authentication using the certificate. An OCSP check can optionally be configured. This results in the device receiving a Kerberos service ticket which is submitted to the IDM built-in KDC adapter in an Authorization header.
  • The built-in KDC adapter validates the ticket and completes the authentication on IDM. The UUID is extracted from the ticket and optionally verified with WOne UEM / AirWatch. 
  • Upon successful completion of all checks, a SAML assertion is generated and sent back to the salesforce.com app. 
  • The salesforce app can then present the SAML assertion to SalesForce.com and get logged in.

DNS

Given L4 Load Balancer at 1.2.3.4 and a DNS domain of example.com and a realm of EXAMPLE.COM:

kdc.example.com.     1800 IN  AAAA           ::ffff:1.2.3.4 kdc.example.com.     1800 IN  A                   1.2.3.4 _kerberos._tcp.example.com.   IN  SRV  10  0   88 kdc.example.com. _kerberos._udp.example.com.   IN  SRV  10  0   88 kdc.example.com.

❗️ There MUST be an IPv6 AAAA entry - iPhone requires it to work correctly.

AAAA entry may have to be converted to IPv6 format: :::::ffff:102:304 ⭐️ Use web-tool: https://www.ultratools.com/tools/ipv4toipv6

Troubleshooting DNS records

To test the DNS settings, you can use the dig command (built-in to Mac and Linux) or NSLOOKUP on Windows.

Here is what the dig command looks like for DNS records:


dig SRV _kerberos._tcp.example.com
dig SRV _kerberos._udp.example.com

# You may wish to define the name server to use with dig by using the following command:
dig @ns1.no-ip.com SRV _kerberos._tcp.example.com

Checking DNS entries with tools

Use Google Toolbox to check the SRV entries:

https://toolbox.googleapps.com/apps/dig/#SRV/_kerberos._tcp.example.com

The result must contain string of type:


;ANSWER
_kerberos._tcp.example.com. 3599 IN SRV 10 0 88 krb.example.com.

Use NC to check UDP/Kerberos services:


nc -u -z kdc.example.com 88

# Answer:
#> Connection to kdc.example.com port 88 [udp/kerberos] succeeded!

Use nslookup to check the SRV entries:


nslookup -q=srv _kerberos._tcp.vmwareidentity.eu
Server:		10.26.28.233
Address:	10.26.28.233#53

Non-authoritative answer:
_kerberos._tcp.vmwareidentity.eu	service = 10 0 88 kdc.vmwareidentity.eu.

Certificates

Cert Trust Schema is here

CA Requirements (including CloudKDC)

Mobile Device Profile

The mobile device profile for the Cloud KDC feature must include the following:

  • Add KDC server root certificate from previous step to credentials
  • Add credential for Certificate Authority issued certificate
  • Configure the SSO profile
  • A user principal name that does not contain an “@” symbol, i.e., it is is typically set to “{EnrollmentUser}". 
  • The realm name that identifies the site where the tenant is deployed. This is the domain name for the site in upper case, e.g., VMWAREIDENTITY.COM. 
  • Specify the Kerberos principal name: {EnrollmentUser}
  • Specify the certificate credential
  • Specify the URL List: https://tenant.example.com/
  • The URLPrefixMatches value in the Kerberos dictionary in the com.apple.sso payload type must include https://tenant-vidm-url/ where “tenant-vidm-url” is the FQDN of the tenant’s WS1 Access address. Note that this FQDN may be different than the one that is used for the realm, e.g., workspaceair.com vs. vmwareidentity.com

Specify a list of app bundle ids:

  • com.apple.mobilesafari (enables access from the Safari browser)
  • com.apple.SafariViewController (enables apps that use Safari View Controller)
  • com.air-watch.appcenter (enables the Workspace ONE native app)

OCSP Requirements

The CA OCSP responder must be accessible on the Internet so that it can be accessed by the KDC. The URL for the OCSP responder is provided in the client certificate in the usual manner.

WS1 UEM has confirmed that the AW CA will meet this requirement.

KDC in vIDM Schema (old)

Port 88 must be open TCP+UDP. Use packet sniffer to check packets from device.

AirWatch-KDC Schema here

Enable KDC

For a single Workspace ONE environment or primary domain:


/etc/init.d/vmware-kdc init --realm MY-IDM.EXAMPLE.COM
# Restart horizon-workspace service:
service horizon-workspace start
# Start KDC service:
service vmware-kdc start

Full instructions link

Reinitialize the KDC

Login to the console (or SSH and SU to root) and do the following: a. Stop the vmware-kdc service b. Force reinitialization of the KDC service (note the use of the ‘–force’ switch at the end) c. Restart the vmware-kdc service d. Restart the horizon-workspace service

⭐️ Last 2 steps can be replaced with rebooting the appliance.


service vmware-kdc stop
/etc/init.d/vmware-kdc init --realm EXAMPLE.COM --subdomain idm.example.com --force
service vmware-kdc start
service horizon-workspace restart
#Or just type reboot to restart the appliance.

Enable KDC using subdomain

This is useful for multiple Workspace ONE environments (Prod, UAT, DEV). Remember that realm must be all CAPs and subdomain is all lowercase.

  • /etc/init.d/vmware-kdc init --force --realm SUBDOMAIN.IDM.DOMAIN.COM --subdomain subdomain.idm.customer.com
  • Restart horizon-workspace service: service horizon-workspace start
  • Start KDC service: service vmware-kdc start

Flags: –force (force init erasing previous initialized KDC) –realm (by convention, this is all caps, this is the Kerberos realm and is usually the DNS domain of the customer. It can also be a DNS sub domain if the customer needs to have multiple Kerberos realms for multiple systems –subdomain (this is not related to subdomain in the conventional sense. This should be your WS1 Access FQDN that the end users use)

Change KDC using subdomain

Please be aware that this procedure is unnecessary if the vmware-kdc init command is run with the correct values in the first place. If a reinitialization is necessary, then there are two better options.                                                                      

If the KDC CA cert doesn’t have to be maintained, then just run vmware-kdc init --force with the right arguments, or

If the KDC CA cert does need to be maintained (so that the WS1 UEM profile doesn’t have to be updated), then copy the kdc-cacert.pem, kdc-cert.pem, and kdc-certkey.pem files from /opt/vmware/kdc/conf to a temporary area, make them readable by the horizon user, and then run vmware-kdc init --force but use the arguments that allow the certs to be passed in (–kdc-cacert, –kdc-cert, and –kdc-certkey).

How to update kdc subdomain without regenerating certificates

Navigate to the kdc-init.input file and change the subdomain there to the new value

  1. service vmware-kdc update
  2. service vmware-kdc restart
  3. Enter command /etc/init.d/vmware-kdc kadmin
  4. Next command: addprinc -clearpolicy -randkey - requires preauth HTTP/(insertnewsubdomain)
  5. Next command: ktadd -k kdc-adapter.keytab HTTP/(insertnewsubdomain)
  6. Next command: listprincs
  7. You should see the HTTP/(newsubdomain) after the listprincs command. This confirms a successful update of the subdomain.

Old Windows-based IDM

Windows VIDM uses the cloud KDC which only requires the appliance to reach outbound on 443 and 88. The clients also need to be able to reach outbound on 443 and 88.

REALM used in WS1 UEM and in Windows VIDM is: OP.VMWAREIDENTITY.COM

Troubleshooting MobileSSO for iOS

Reasons for failure

  1. The device isn’t enrolled
  2. The device is enrolled but cannot access port 88 on the VIDM service
  3. Response from port 88 on the VIDM service cannot reach device
  4. DNS entries not configured correctly
  5. The device is enrolled but the profile has the wrong realm
  6. The device is enrolled but doesn’t have a certificate in the profile
  7. The device is enrolled but has the wrong certificate in the profile
  8. The device is enrolled but the Kerberos principal is wrong
  9. The certificate SAN values cannot be matched with the Kerberos principal
  10. The device is enrolled but the KdcKerberosAuthAdapter has not been configured with the issuer certificate
  11. The device is enrolled but the user’s certificate has been revoked (via OCSP)
  12. The device is enrolled but doesn’t send the certificate during login
  13. The device is enrolled but the app bundle id for the app isn’t in the profile
  14. Time is not synced between device and KDC (this will cause the TGT to be issued and a TGS_REQ to be made, but the device will reject the service ticket and asking for an “http/fqdn” ticket)
  15. Time is not synced between KDC nodes (this will cause save of adapter settings to fail)
  16. KDC subdomain value is set incorrectly (this causes the TGS_REQ to be rejected)

General checks

  • What is not working?

    1. Unable to save Mobile SSO for iOS adapter settings
      1. Is the error message “Checksum failed”?
        1. Is this a cluster?
          1. How was the data copied from one node to another?  / Was “vmware-kdc init” run on multiple nodes?
            1. Use dump/load to replicate the data to all nodes rather than running init on all nodes.
    2. Device cannot login via Safari
    3. Device cannot login via Workspace ONE app
    4. Device cannot login via some other native app
      1. Is the app bundle id included in the profile
      2. Does the app use Safari View Controller?
        1. Is com.apple.SafariViewController included in the list of app bundle ids in the profile?
  • If the device cannot login, why is the login failing? 

    1. Device makes no request at all to the KDC
      1. vIDM policy not setup correctly
      2. Device cannot reach the KDC 
        1. DNS lookup issue
        2. Networking issue with reaching the KDC
    2. AS_REQ received but certificate isn’t validated
      1. Does the certificate have acceptable SAN values?
      2. Is OCSP used? 
        1. Is the OCSP server reachable?
        2. Is the OCSP server validating the certificate?
    3. TGT issued but TGS_REQ not issued
      1. Is the subdomain set correctly
    4. Is the service ticket ignored by the device
      1. Is the time on the KDC service sync’d with the device?
  • Is it failing for all devices or just a subset or one device?

  • What version of iOS is being used?

    1. iOS 9.3.5 is known not to work for Safari
    2. iOS 10.3.3 has a known issue with native apps that use SafariViewController
  • Can iOS device reach the KDC?  (use sniffer to check)

    1. There may be a networking issue. Sometime firewalls or load balancers do not pass through UDP and TCP traffic on port 88
  • Are the DNS SRV entries set up correctly?

  • Is the subdomain value for the on-premises KDC set correctly?

  • If using Hybrid KDC, does registration fails to save the adapter configuration?

    1. Is the vIDM service node able to resolve the _mtkadmin._tcp. SRV entry?
    2. Is the vIDM service node able to reach the https://hybrid-kdc-admin./mtkadmin/rest/health URL?
    3. Is the vIDM service node able to reach UDP port 88 for the KDC?
    4. Are responses from the KDC with source UDP port 88 able to reach the vIDM service node?
  • Is the AirWatch tunnel being used? 

    1. Is the tunnel bypassing port 88 traffic to allow it to reach the KDC?
  • Is REALM domain in KDC and DNS all CAPITAL LETTERS, and subdomain all lower letters?

  • If you are using {Enrollment User} as the Kerberos principal in the device profile and {Enrollment User} is mapped to the sAMAccountName, and the sAMAccountname does not match the first portion of the user’s UPN – which is everything before the @ sign – then mobile SSO will fail.  

  • For certificate authentication to work (Android SSO included), UserPrincipalName value in SAN needs to be set to user’s UPN or Email, in user’s certificate or SAN must have user’s email if UPN is missing.

  • When creating the certificate template in ADCS - use the KerberosAuthentication template with the following modifications: change the template name, change Subject Name to accept it from the request, add a new “Kerberos Client Authentication” auth policy (oid=1.3.6.1.5.2.3.4), add the AirWatch service account to the list of users that can use the certificate.

    ❗️The authentication policy oid must be exactly as specified

WS1 Logs

Built-in Kerberos adapter log:  /opt/vmware/horizon/workspace/logs/horizon.log KDC logs: /var/log/messages log and grep for ‘pkinit’ to see the errors returned.


grep -i 'pkinit' /var/log/messages

This may return errors like vmw_service_matches_map: “HTTP/idm.test.ws@TEST.WS” is not a match for the service_regex and Server not found in Kerberos database. Basically, either DNS is not correct or built-in KDC Server is not properly initialized.

How to Enable Kerberos iOS Device Logs to troubleshoot Mobile SSO

When having issues troubleshooting Mobile SSO for iOS, install the GSS Debug Profile on the device to retrieve more verbose Kerberos logs.

  • Copy this config file to your desktop.
  • Email the config file to the device.
  • From the device, tap on the attachment, tap on Install.  Enter the device passcode when prompted.
  • Tap on Install in the upper right on the Apple Consent notification.

Link: https://developer.apple.com/services-account/download?path=/iOS/iOS_Logs/Enterprise_SSO_and_Kerberos_Logging_Instructions.pdf ❗️Action requires Apple Dev Account!

Packet Sniffing

There is no tcpdump on WS1 Access, and there is no possibility to install it due to dependency problems (written & tested on SUSE Ent Linux version of WS1 Access).

Use Python2 to write your own sniffer of packets for Access-vIDM (fresh build on GitHub):


# Filename = sniffer.py
# Packet sniffer script 0.3
# Made by Alexei Rybalko for vIDM-Access Server
# Based on SUSE Ent. Linux 11 with python2

# Usage:
# python sniffer.py 192.168.1.1
# Will sniff any packets going from or coming into IP=192.168.1.1, includes ping-ICMP/TCP/UDP

import socket, sys
from struct import *
 
if __name__ == "__main__":
    s = socket.socket(socket.AF_PACKET, socket.SOCK_RAW, socket.ntohs(0x0003))

    if (not sys.argv[1]):
        print("Enter IP Address to filter packets from!")
        sys.exit(0)
 
    while True:
        packet = s.recvfrom(65565)[0]
        eth_header = packet[:14]
        eth = unpack('!6s6sH', eth_header)
        eth_protocol = socket.ntohs(eth[2])
 
        if eth_protocol == 8: # IP
            ip_header = packet[14:34]
            iph = unpack('!BBHHHBBH4s4s', ip_header)
            ttl = iph[5]
            protocol = iph[6]
            s_addr = socket.inet_ntoa(iph[8])
            d_addr = socket.inet_ntoa(iph[9])
            #print "Source IP: " + s_addr
            #print "Destination IP: " + d_addr
 
            if (s_addr == sys.argv[1]) or (d_addr == sys.argv[1]): # IP Address only the one provided as argument to script
                if protocol == 6: # TCP
                    tcp_header = packet[20:40]
                    tcph = unpack('!HHLLBBHHH', tcp_header)
                    source_port = tcph[0]
                    dest_port = tcph[1]
                    print("--TCP--")
                    print "Source port: " + str(source_port)
                    print "Destination port: " + str(dest_port)
 
                elif protocol == 1: # ICMP
                    icmp_header = packet[20:24]
                    icmph = unpack('!BBH', icmp_header)
                    icmp_type = icmph[0]
                    code = icmph[1]
                    checksum = icmph[2]
                    print("--ICMP--")
                    print "Type: " + str(icmp_type)
                    print "Code: " + str(code)
 
                elif protocol == 17: # UDP
                    udp_header = packet[20:28]
                    udph = unpack('!HHHH', udp_header)
                    source_port = udph[0]
                    dest_port = udph[1]
                    print("--UDP--")
                    print "Source port: " + str(source_port)
                    print "Destination port: " + str(dest_port)
 
                else:
                    print('Unknown Protocol!')
Chapter 3

[VMW] Gateway (UAG)

Статьи по установке, настройке и решению проблем с Unified Access Gateway (бывший Access Point Server).

Subsections of [VMW] Gateway (UAG)

Content Gateway Cert Error

WS1 UEM Content installation Certificate error

In some rare cases (for UAG release 2106 or maybe later) installation Content Gateway on UAG can be failed with messager in logs like this:

13:10:03.127 [main]  INFO c.v.enterprise.content.Application - Starting Application v21.06.0 on <UAG-hostname> with PID 18264 (/opt/vmware/content-gateway/content-gateway.jar started by gateway in /opt/vmware/content-gateway)  
13:10:03.132 [main]  INFO c.v.enterprise.content.Application - No active profile set, falling back to default profiles: default. 
13:10:05.159 [main]  INFO  c.v.enterprise.content.Application - PostConstruct: Application instance created.
13:10:05.850 [main]  ERROR c.v.e.c.security.AuthenticatorImpl - **Exception while generating public key from modulus and exponent**
13:10:05.851 [main]  WARN  o.s.c.a.AnnotationConfigApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'server': Unsatisfied dependency expressed through field 'connectivityAdapter'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'connectivity': Unsatisfied dependency expressed through field 'authenticator'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authenticatorImpl': Invocation of init method failed; nested exception is java.lang.IllegalStateException: java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException:  
**RSA keys must be at least 512 bits long**
13:10:05.877 [main]  ERROR o.s.boot.SpringApplication - Application startup failed

❗️This artice is only workaround for error and don’t contain deep explanation root cause problem analyze

  1. If you already upload public certificate to Groups & Settings > All Settings > System > Enterprise Integration > Content Gateway, delete it and save settings
  2. Option step: upload/update your default UAG certificate for UAG console in TLS Server Certificate Settings pointed it to Internet Interface (more detailed here: Update SSL Server Signed Certificates)
  3. Enable Content service in Edge Service Settings. It should be started after a couple of minutes with green light. At this stage you may notice URL your content gateway is reacheble, but get TLS/SSL error
  4. Repopulate your public certificete deleted on Stage 1.
  5. Disable and Enable Content service in Edge Service Settings to reread settings from API server. After that your Content Gateway URL should work as expected with your public certificate.

SEG on UAG

UAG Architecture

SEG Architecture

  • SEG process running inside the container also behaves as just any other process on UAG, from networking perspective.
  • Handling ports - there is no port mapping between UAG host and the container, as container runs with host network stack itself. There are fixed set of ports reserved on UAG for running SEG. These are listed below:

UAG Admin UI and Logs

  • Admin UI: https://UAG IP:9443/admin
  • Show Edge service settings.
  • Log archive: UAG + All edge services
  • Log level settings: UAG logs level only.

Important logs:

  • Appliance-agent: Configuration related.
  • SEG: SEG service-related.
  • Docker: Docker related.

Logs file path: SEG logs: /var/log/vmware/docker/seg UAG logs: /opt/vmware/gateway/logs and /var/log/vmware/appliance-agent/ (Appliance agent logs)

SEG Configuration

  • General mem configuration.
  • Outbound proxy configuration.
  • Trusted certificates.
  • Host file entries.
  • Add SSL certificate.
  • Save (Install and configure)

Troubleshooting SEG

SEG configuration issues

  • Check Appliance-agent logs.
  • Check v2-seg-installer and app logs (Available inside SEG folder).
  • If no SEG folder then check docker logs.
  • If docker fails to start, then check Journalctl -xe

SEG application issues

Enable SSH on UAG

  • Connect to UAG using vCentre web console
  • Navigate cd /opt/vmware/gateway/bin/ and execute ./enable_ssh.sh

Check Docker Status

systemctl status docker.service
docker ps

To check all required service is up and running: netstat –ntlp

Firewall and connection issues - https://docs.vmware.com/en/Unified-Access-Gateway/3.10/com.vmware.uag-310-deploy-config.doc/GUID-390D3A2A-0CB7-4A82-9B0F-D525B74CF55B.html

SSL Offload

##SSL offloading on UAG

SSL offload to external LoadBalancer usually a good practice to improve performance DS’s and UAG servers, avoiding it to extra work with encrypt traffic Also you should to change public Certificate in one place instead of replace it in each server

Note: all above are valid if you have alot devices registred in WS1 (more than 2000). For smaler deployments usually you may stay at much easy deployment without load balancers.

In case of SSL offloadin you should use one of the scenarios:

  1. Traffic isn’t encrypted after termination on LoadBalancer
    SSL termination without reencryption

This can be used for traffic to Devcie Services endpoint only. Even if you setup SSL offload check box during DS installation AWCM endpoint will use encriprion with self-generated certificate for traffic on port 2001. In this case you need to enshure tant LoadBalancer trust for this cert (or ignores SSL errors)

❗️All you internal connection to AWCM must go though LoadBalancer enpoint to proper trust

  1. Traffic is reencrypted after termination on LoadBalancer with local cert
    SSL termination without reencryption

Despite the availability SSL offload check box in SEG and Content setup in WS1 Console, UAG server can work only with encripted http-traffic.

SEG
The UAG does not support any non-encrypted protocols. Therefore, SEG only supports SSL re-encryption (SSL bridging) or SSL pass through

Content:
HTTP traffic is not allowed for Content Gateway on port 80 on Unified Access Gateway because TCP port 80 is used by the edge Service Manager.

Chapter 4

[KL] Kaspersky Labs

Статьи по установке, настройке и решению проблем с Kaspersky Secure Mobile Management.

KSMM Architecture (KSC + Gateways)

Subsections of [KL] Kaspersky Labs

KSMM Certificates

Articles

General Schema of certificates storage and flows

The following schema is used in the KSMM architecture with KSC server in LAN, Network Agents in DMZ with Gateway and iOS Server roles, plus a corporate apps (Self-Service) portal is used. Client devices include Android, iOS/iPadOS and Windows desktops. KSMM/KSC Certificate general schema

Important Notes

  • By default, certificate-based authentication of KES devices is disabled. Certificates are NOT checked by KSC, until registry key is created (see link on enabling certificate-based auth above). Use default behaviour for test and proof of concept projects only;

  • PKI SSL protocol is used by KSC to transfer only Mail and VPN. After configuring PKI integration and choosing the CA template, check Mail or VPN certificate settings. Next, manually create a certificate in the Certificates tab. Choose creation of certificates on KSC - this will in reality activate creation of the certificate on CA via the new integration. Certificate is first created manually, then auto-renewed by KSC;

  • SCEP / NDES protocol schema not only requires CA to be exposed for device access. The certificate request (CSR) on stage 1 of protocol exchange is also done using plain text HTTP. This poses risk of man-in-the-middle attacks, use only in closed environments with extreme caution;

  • Current version of KSC Web console (NWC) does NOT allow usage of manual certificates. Only self-signed certificates from KSC may be created. This is due to absence of an internal encryption mechanism for certificates and passwords in the web console code. Use MMC console or OpenAPI methods instead;

  • Manual change of certificates in folders of KSC for Win server and iOS Server is prohibited! Certificates have references to them in the database, so change of file will lead to breaking in functionality of components. Use provided utilities (see schema and links above);

  • Corp App Portal certificate is a file in a folder, but changing it manually is not recommended: potential side effects are not explored. Use Windows Uninstaller to Change certificate:

Subsections of KSMM Certificates

PKI Integration

Integration Schema

PKI Integration is done using native Microsoft Active Directory domain services.

Warning

MS CA server must be in Enterprise Mode of operation. To install CA in Enterprise mode, Enterprise Admin account is needed. CA in Enterprise Mode is installed on forest level! This means Domain Controllers will have links to this CA, even if it is installed in a subdomain.

graph TD KSC[KSC for Win] -->|1. CSR| WIN[Win Server in AD Domain] WIN -->|2. Enrolment Agent| CA[Microsoft CA in AD Domain] KSC -->|4. User Cert| A[Android] CA -->|3. User Cert| KSC KSC -->|4. User Cert| I[iOS Profile] CA --> AD[/Active Directory\]

Create Agent Account

On AD Domain Controller:

  • Create a service account for Agent (example: “agentca”);

Add a Service Account on the CA

Open the Certificate Authority (CA):

  • Launch the Certification Authority Console from the Administrative Tools in Windows.
  • In the left pane, select (+) to expand the CA directory;
  • Right-click the name of the CA and select Properties. The CA Properties dialog box displays.
  • Click the Security tab;
  • Click Add. The Select Users, Computers, Service Accounts, or Groups dialog box displays;
  • Click within the Enter the object names to select field and type the name of the service account (e.g., Ima Service);
  • Click OK. The CA Properties dialog box displays;
  • Select the service account you added in the previous step (e.g., Ima Service) from the Group or user names list;
  • Select the Read, the Issue and Manage Certificates, and the Request Certificates checkboxes to assign permissions to the service account. Click OK.

Create the Restricted Enrolment Agent Certificate Template

Open the Certificate Authority (CA):

  • Expand the CA Name, Right click Certificate Templates, and select Manage;
  • Right click the Enrollment Agent (Computer) template and select Duplicate Template (Do not choose Enrollment Agent user cert!). Name it per your preference;
  • Select Windows Server 2008+ Enterprise;
  • On the Request Handling tab, select Allow Private Key to be Exported;
  • In the Subject Name tab, make sure Build from this Active Directory Information is activated and Subject Name format is set to Fully distinguished name. Click OK;
  • Navigate back to the CA, right click Certificate Templates, select New, and select Certificate Template to Issue;
  • Select the duplicate copy of the template created in the previous step. Click OK.

Create the User Certificate Template

Open the Certificate Authority (CA):

  • Open the CA (certsrv) window.
  • In the left pane, select (+) to expand the CA directory.
  • Right-click the Certificate Template folder and select Manage. The Certificate Templates Console window displays.
  • Select the desired template (User) under Template Display Name, and right-click Duplicate Template. The Duplicate Template dialog box displays.
    Tip

    For Wi-Fi, VPN, or Exchange Active Sync (EAS) client authentication select User template.

  • Select the Windows Server that represents the oldest enterprise version being used within the domain to ensure backward compatibility of the certificate that was issued.
  • Click OK. The Properties of New Template dialog box displays.
Edit User Template Properties
  • Click the General tab;
  • Type the name of the template displayed to users in the Template display name field. The Template name field auto-fills with the template display name without spaces;
  • Select the desired length of time for the certificate to be active from the Validity period entry field/drop-down menu. Click Apply;
  • Click the Request Handling tab;
  • Select the appropriate client authentication method from the Purpose: drop-down menu. This selection might be based on the application of the certificate being issued, although for general purpose client authentication, select Signature and Encryption;
  • Select the Allow private key to be exported checkbox;
    Warning

    For a certificate to be installed on an iOS device, this checkbox MUST be selected.

  • Click Apply;
  • Select the Subject Name tab;
  • Select Supply in the request or Build from this Active Directory.

Create Enrolment Agent Certificate

On KSC server:

  • Login as Administrator user. Go to Local Server Policies, find “Run as service” and add the “agentca” account to this role (SeServiceLogonRight permission);
  • Give Local Administrator permission to “agentca” (needed for saving certs in MMC);
    Warning

    Service user account, which is used by KSMM to create a certificate request, must be a domain account and have enough permissions to access Windows certificate store. Standard user will not work. Local admin rights on the computer is a simple solution.

  • Login as “agentca” to the Windows server with KSC;
  • Launch Certificate snap-in (certmgr.mmc);
  • Expand Certificates (Local Computer), double click Personal, right click Certificates, select All Tasks, and select Request New Certificate. Click Next;
  • In the Wizard, Active Directory Registration Policy list, choose Enrolment Agent template. Choose its’ Options: Private key tab, choose Make private key exportable;
  • Move the created certificate in Local Computer certificate storage;
  • Give “agentca” permissions to read the certificate private key: right-click on the certificate, choose All Tasks -> Manage Private Keys…, click Add and choose the “agentca” account. Choose Full Control. Click Apply.

Create PKI Integration

On KSC MMC Console:

  • Go to Mobile Device Management -> Certificates section;
  • Click Integrate with public key infrastructure button;
  • In the window choose Integration with PKI section, enter the “agentca” service account login and password. Choose the User Template created earlier from CA in the drop-down list to use as default;
  • Go to Issuance of mail certificates and Issuance of VPN certificates sections. In the Certificate Source drop-down list choose PKI. Do not change the default template, choose auto-renewal time period in days. Click Apply;
    Warning

    Do NOT change the Mobile Certificate Issuance - this may severe the KSC - Agent communication protocol.

Issue a certificate to a mobile device

On KSC MMC Console:

  • Go to Mobile Device Management -> Certificates section;
  • Choose the Add certificate button;
  • In the wizard chose Mail or VPN certificate, Next, choose iOS or Android mobile devices, Next;
  • Choose user account, who is a mobile device (controlled by KSMM) owner, for whom to issue a certificate;
  • Choose Issue certificate using Kaspersky Security Center - this will launch the request procedure using the configured PKI integration.

Troubleshooting

Error

0x80094806 описана у MS https://learn.microsoft.com/en-us/windows/win32/com/com-error-codes-4 - CERTSRV_E_BAD_RENEWAL_SUBJECT 0x80094806  The request was made on behalf of a subject other than the caller. The certificate template must be configured to require at least one signature to authorize the request.

Solution

Add signatures in User Certificate Template:

KSMM Network Ports

Network Troubleshooting Articles

Devices to KSMM/KSN

Sender Recipient Ports Description Port may be changed
(WAN/DMZ) Mobile Device with KES Agent (DMZ) NAgent +Gateway mode TCP13292, TCP13293 Mobile device connects to KSC via Gateway Yes - on KSC console
(WAN/DMZ) Mobile Device with KES Agent (DMZ/LAN) KSC TCP13292, TCP13293 Mobile device connects to KSC directly Yes - on KSC console
(DMZ/LAN) Device with KES Agent (DMZ/LAN) Activation Proxy-server on KSC klactprx TCP17000, TCP17100 License activation proxy for desktop devices (TCP17000) OR mobile devices (TCP17100) IF devices do not have internet connection Yes - on KSC console
(WAN/DMZ) iOS Device with Control Profile (DMZ) iOS MDM Server, kliosmdmservicesrv.exe TCP443 Sending data to iOS devices Yes - during iOS MDM Server installation
(WAN/DMZ) Browser on Device with KES Agent (DMZ) Corp Catalog, Apache Web Server TCP8071 User device downloads apps from Corp App Catalog component Yes - during installation
(WAN/DMZ/LAN) Desktop (Win/macOS/NIX) Device with KES Agent (WAN) KSN Cloud Proxy TCP13111, UDP15111 Access of controlled devices to Kaspersky Security Network (Cloud, optional) Yes - on KSC console
(WAN/DMZ) Desktop (Win/macOS/NIX) Device with KES Agent (DMZ) NAgent +Gateway mode TCP13000 Connection of NAgent from client desktop to NAgent Gateway proxy for data exchange with KSC Yes - on KSC console
(DMZ/LAN) Desktop (Win/macOS/NIX) Device with KES Agent (LAN) KSC TCP13000, UDP13000, TCP14000 Connection of NAgent from client desktop directly to KSC. UDP13000 needed for NAgent status update Yes - on KSC console
(WAN/DMZ) Desktop (Win/macOS/NIX) Device with KES Agent (DMZ) NAgent +Gateway mode TCP13295 PUSH-notifications to desktop machines Yes - on KSC console
(DMZ/LAN) Device with KES Agent (DMZ/LAN) KSC TCP8060(HTTP), TCP8061(HTTPS) Installation packets request and download directly from KSC (if Corp App Component is not available) Yes - on KSC console
(DMZ/LAN) OS Windows Device with KES Agent (DMZ/LAN) KSC Web Console - klsctunnel TCP19170 Remote desktop assist connection of tech support to user device Yes - on KSC console

KSMM to PUSH Cloud Services

Sender Recipient Ports Description Port may be changed
(DMZ/LAN) KSC Google Firebase Cloud Messaging (FCM) ASN15169 IP block: android.googleapis.com, play.google.com, android.clients.google.com, accounts.google.com, fonts.googleapis.com TCP5228, TCP5229, TCP5230 PUSH notifications for Android devices No
(DMZ) iOS MDM Server Apple Cloud APNs IP Block 17.0.0.0/8: gateway.push.apple.com, feedback.push.apple.com, mdmenrollment.apple.com, *.itunes.apple.com, *.mzstatic.com, *phobos.apple.com, *phobos.apple.com.edgesuite.net TCP2195, TCP2197 PUSH notifications for iOS No

KSMM to KSMM Components

Sender Recipient Ports Description Port may be changed?
(LAN) KSC (DMZ) NAgent +Gateway Mode TCP13000 Request to NAgent to establish tunnel session for connecting devices from WAN/DMZ Yes - in KSC Console
(DMZ) Corp App Catalog klakaut.exe (DMZ) NAgent +Gateway Mode TCP13000 Corp App Catalog connects via Gateway to KSC for data exchange with KSC Yes - during Corp App Catalog install
(DMZ) Subordinate KSC (LAN) Main KSC OR NAgent +Gateway Mode TCP13000 In cascade KSC Architecture, two KSC servers communicate between each other directly OR via NAgent Gateway Yes - in KSC Console
(LAN) MMC Admin Console (LAN) KSC TCP13291 Connection of remote MMC admin console to KSC Yes - in KSC Console
(DMZ/LAN) NAgent, NAgent +Gateway, KSC (DMZ/LAN) NAgent, NAgent +Gateway, KSC UDP15000 Installation packets, status update messages, NAgent discovery of other NAgents in broadcast domain Yes - in KSC Console
(LAN) Web Admin Console (New Web Console = NWC) (LAN) KSC TCP8080 Connection from browser to Web admin console https://ksc.local:8080 Yes - in KSC Web Console settings
(LAN) Third-party scripts, Web Admin Console (NWC) (LAN) KSC OpenAPI Interface TCP13299 KSC klserver component REST API port for OpenAPI commands Yes - in KSC Console

Subsections of KSMM Network Ports

Troubleshooting Ports

Network Agent + Connection Gateway

  • Network schema:
graph LR K[KSC] --> |TCP13000| A[NAgent +Distr Point & Gateway]; A <--> |TCP13292| M[Mobile Device]

Listening service: klnagent.exe, opens TCP13000

  • Check open port:
netstat -napo tcp | find "13000" # port must be LISTENING
  • Check gateway function: Use klnagchk.exe tool (see link above), you should see This device is a connection gateway in output text.

  • Check gateway port:

netstat -napo tcp | find "13292" # port must be LISTENING
  • Check gateway certificate:
openssl s_client -connect ksmm.lab.local:13292
You should see certificate data with correct Subject CN.

iOS MDM Server

  • Network schema:
graph LR I[iOS for MDM] <--> |TCP443| D[iOS Device]; I --> |TCP2197| A(((APNs Cloud, 17.0.0.0/8))) A --> D

Listening service: kliosmdmservicesrv2.exe, opens TCP443

  • Check open port:
netstat -napo tcp | find "443" # port must be LISTENING
  • Check iOS MDM certificate:
openssl s_client -connect ksmm.lab.local:443

Corporate Apps Portal

  • Network schema:
graph LR C[Corp Portal] --> |klakaut.exe TCP13291| K[KSC]; A[Admin] --> |TCP8070| C U[User] --> |TCP8071| C
  • Check open port:
tasklist | find "klakaut" # => PID
netstat -napo tcp | find "PID" # port must be ESTABLISHED

KSMM SQL

Recommendations

  • MS SQL Express 2019 is embedded in the current version of KSC/KSMM. This is a limited version of SQL. It is not available in case of 10000+ device install and/or Program Control component enabled;

  • MySQL v8.0.20+ may be used for installations of <50000 devices (Program Control component NOT enabled);

  • MariaDB v10.3+ may be used for installations of <20000 devices (Program Control component NOT enabled);

MS SQL Express 2019 w/o CU12 update error

Symptom: very slow response from MMC console of KSC. Very high resources utilisation of KSC server.

USE KAV  
GO  
ALTER DATABASE SCOPED CONFIGURATION SET TSQL_SCALAR_UDF_INLINING = OFF 
GO

“KAV” - database default name. Change if using another name for the database.

  • After completing the SQL query above, restart the SQL Server (SQLEXPRESS) service using Sql Server Configuration Manager.

KSMM Windows Desktops

Windows Desktop Management

Desktop management is done using scripts. Scripts are packed into “Installation packs”: Entry point for an installation pack is a “.BAT”, “.EXE” or “.MSI/.MSP” file. A BAT-file may be used to launch Powershell, Python or any other type of scripts.

Scripts are launched using Scheduled Tasks of Remote App Install type:

Python script launch

Example of entry-point BAT-file runpy.bat running Python script and then logging a success in a text file:

c:\python3\python.exe testpy.py
@echo SCRIPT RUN SUCCESS>%SYSTEMDRIVE%\LOG.txt

Python testpy.py payload script to create a file needs to be placed in the same installation pack:

if __name__ == "__main__":
    text = "File write"
    file1 = open("c:\\testFile.txt", "w")
    file1.write(text)
    file1.close() 

Powershell script launch

Example of entry-point BAT-file runps.bat running Powershell script and then logging a success in a text file:

Powershell.exe -executionpolicy remotesigned -File certinstall.ps1
@echo CERT INSTALL SUCCESS>%SYSTEMDRIVE%\LOG.txt

Powershell payload script certinstall.ps1 to decrypt and install a PFX certificate test2.pfx provided in the installation pack in the Windows certificate store (Local Machine store):

$Pass = ConvertTo-SecureString -String '12345' -Force -AsPlainText
$User = "lab\administrator"
$Cred = New-Object -TypeName "System.Management.Automation.PSCredential" -ArgumentList $User, $Pass
Import-PfxCertificate -FilePath test2.pfx -CertStoreLocation Cert:\LocalMachine\My -Password $Cred.Password

Parameters transfer

Parameters can be transferred using the UI, parameter section in the installation pack: Parameters are sent to the entry-point file (BAT-file) and should be handled there. Example of the certinstall.bat BAT-file running Powershell and transferring a “seconds” parameter in %1:

Powershell.exe -executionpolicy remotesigned -File timeout.ps1 -sec %1 > c:\PSLOG.txt

Powershell script payload timeout.ps1 to insert a value of seconds in the Windows registry, for the system to timeout (default of 120sec in case parameter is not provided) and lock screen in case of inactivity (restart needed for registry parameter to be applied):

param([Int32]$sec=120)

Write-Host 'Current timeout value: '

Get-ItemPropertyValue 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System' 'InactivityTimeoutSecs' 

Set-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System' -Name  InactivityTimeoutSecs -Value $sec

Write-Host 'New timeout value: '

Get-ItemPropertyValue 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System' 'InactivityTimeoutSecs'

Restart-Computer -Force

OpenAPI

Warning

KlAkOAPI Python Package is a wrapper over the regular REST API calls. I do NOT recommend its’ usage, because it does not cover all classes, parameters and attributes available.

Articles

Config / Auth

Access with OpenAPI requires 2 auth elements:

  • Local technical user account login/password (Basic Auth method)
  • KSC Server certificate verification (optional)

For KSC for Windows, server certificate is located at path: %ProgramData%\KasperskyLab\adminkit\1093\cert\klserver.cer Certificate needs to be copied to the machine, from which a script is run.

Login example

REST API Classic login:

import requests
import base64
import json
import urllib3
from pprint import pprint

urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

ksc_server = "https://ksmm.demo.local:13299"
url = ksc_server + "/api/v1.0/login"
user = "user_api"
password = "P@ssw0rd"
data = {}

user = base64.b64encode(user.encode('utf-8')).decode("utf-8")
password = base64.b64encode(password.encode('utf-8')).decode("utf-8")
session = requests.Session()

auth_headers = {
    'Authorization': 'KSCBasic user="' + user + '", pass="' + password + '", internal="1"',
    'Content-Type': 'application/json',
}

response = session.post(url=url, headers=auth_headers, data=data, verify=False)
print(f'Server response = {response.status_code}')

KlAkOAPI Python Package Login:

import socket
import uuid
import datetime
from sys import platform
from KlAkOAPI.Params import KlAkArray, paramBinary, strToBin, dateTimeToStr
from KlAkOAPI.AdmServer import KlAkAdmServer

def GetServer():
    server_address = 'ksmm.demo.local'
    server_port = 13299
    server_url = 'https://' + server_address + ':' + str(server_port)
    
    username = 'user_api'
    password = 'P@ssw0rd'
    SSLVerifyCert = 'C:\\Lab\\klserver.cer'

    server = KlAkAdmServer.Create(server_url, username, password, verify = SSLVerifyCert)
    return server

# Call server login function:
server = GetServer()

Subsections of OpenAPI

OpenAPI - Certificates

Send a certificate created on KSC

This is a programmatic way to do Create Certificate -> Mail or VPN certificate (placed in User Certificate Store) -> Self-Signed Certificate from KSC. The certificate is generated on the KSC and send to a user and his device - to the device, where the specific user is the owner. User is chosen by his unique ID, ul_binId parameter. See List Users page for details on how to get unique user IDs.

userID = 'YbTpoXJ4XkSxzy5hcXm75w=='

url = ksc_server + "/api/v1.0/MdmCertCtrlApi.SetCertificateAsync2"
# "NSDomain" - Domain Auth 
# "CPKES" - certificate for OS Android
# "CTMail" - Mail certificate type
# "ul_binId" - paramBinary complex data with base64-encoded unique user ID

data = {'pAuthType':{'NSDomain': True},'pCertificate':{'CPKES':True, 'CTMail':True}, "pRecipient":{"ul_binId":{"type":"binary","value":userID}}}

response = session.post(url=url, headers=common_headers, data=json.dumps(data), verify=False)
wstrIteratorId = json.loads(response.text)

paramBinary

This is a complex data type with base64 encoded string data. Example:

import base64
paramBinary = {"type": "binary", "value": "c29tZXRleHQ="}
print(base64.b64decode("c29tZXRleHQ=")) # b'sometext'

This paramBinary type is used for transferring certificate PEM data, certificate password data etc. Note: although PFX container is supported in documentation, using it raises “cannot be JSON serialised” errors.

OpenAPI - List Data

List Data

SrvView Class, ResetIterator function is used to query and list inventory elements from KSC:

  • Choose a “view” type = which data to query (SrvView Views list link above). “GlobalUsersListSrvViewName” was chosen in the code below to query users and their unique IDs;
  • Choose which fields should be shown for the chosen View (for “GlobalUsersListSrvViewName”, available fields are listed in Users and groups list section). Display names (ul_wstrDisplayName) and unique IDs (ul_binId) are chosen fotr the example;
  • Get the record count - how many records are there to show. Use SrvView Class, GetRecordCount function for this;
  • Get the records themselves: Use SrvView Class, GetRecordRange function to provide the final records counting from the first record to the last, which was provided in the previous step.
data = {}

# CHOOSE VIEW TYPE, DATA FILEDS
url = ksc_server + "/api/v1.0/SrvView.ResetIterator"
data = {"wstrViewName": "GlobalUsersListSrvViewName", "vecFieldsToReturn": ["ul_wstrDisplayName","ul_binId"], "lifetimeSec": 3600}
response = session.post(url=url, headers=common_headers, data=json.dumps(data), verify=False)
wstrIteratorId = json.loads(response.text)['wstrIteratorId']

# FIGURE OUT THE NUMBER OF RECORDS AVAILABLE IN THE DATABASE
url = ksc_server + "/api/v1.0/SrvView.GetRecordCount"
data = {"wstrIteratorId": wstrIteratorId}
response = session.post(url=url, headers=common_headers, data=json.dumps(data), verify=False)
count = json.loads(response.text)
NUMBER_OF_RECORDS = int(count['PxgRetVal'])

# GET THE DATA ITSELF WITH ALL THE RESTRICTIONS PROVIDED EARLIER
url = ksc_server + "/api/v1.0/SrvView.GetRecordRange"
data = {"wstrIteratorId": wstrIteratorId, "nStart": 0, "nEnd": NUMBER_OF_RECORDS}
response = session.post(url=url, headers=common_headers, data=json.dumps(data), verify=False)
pRecords = json.loads(response.text)['pRecords']['KLCSP_ITERATOR_ARRAY']

# LIST THE DATA, FILTERING OUT EXTRA SYNTAX
for record in pRecords:
    print(f"{record['value']['ul_wstrDisplayName']} : {record['value']['ul_binId']['value']}" ) 

TLS Ciphers

KSC for Win 14.x

SrvUseStrictSslSettings

Switch the supported versions of TLS and cipher suites is done using SrvUseStrictSslSettings flag:

klscflag -fset -pv ".core/.independent" -s Transport -n SrvUseStrictSslSettings -v <value> -t d

Flag SrvUseStrictSslSettings has values from 0 to 5:

0 - allow all TLS versions and cipher suites; 1 - SSL v2 turned OFF; 2 - SSL v2 + SSL v3 turned OFF; 3 - ONLY TLS v1.2; 4 - ONLY TLS v1.2, and ONLY a group of cipher suites with TLS_RSA_WITH_AES_256_GCM_SHA384 - used by default; 5 - ONLY TLS v1.2 and ONLY a selected group of cipher suites;

Options 0 - 3 are NOT recommended, used for compatibility purposes only.

SrvUseStrictSslSettings = 0

| Hexcode | Cipher Suite Name (OpenSSL) | KeyExch Encryption Bits | Cipher Suite Name |(IANA/RFC) |

TLS 1 xc014 ECDHE-RSA-AES256-SHA ECDH 521 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA x35 AES256-SHA RSA AES 256 TLS_RSA_WITH_AES_256_CBC_SHA x84 CAMELLIA256-SHA RSA Camellia 256 TLS_RSA_WITH_CAMELLIA_256_CBC_SHA xc013 ECDHE-RSA-AES128-SHA ECDH 521 AES 128 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA x2f AES128-SHA RSA AES 128 TLS_RSA_WITH_AES_128_CBC_SHA x96 SEED-SHA RSA SEED 128 TLS_RSA_WITH_SEED_CBC_SHA x41 CAMELLIA128-SHA RSA Camellia 128 TLS_RSA_WITH_CAMELLIA_128_CBC_SHA x07 IDEA-CBC-SHA RSA IDEA 128 TLS_RSA_WITH_IDEA_CBC_SHA

TLS 1.1 xc014 ECDHE-RSA-AES256-SHA ECDH 521 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA x35 AES256-SHA RSA AES 256 TLS_RSA_WITH_AES_256_CBC_SHA x84 CAMELLIA256-SHA RSA Camellia 256 TLS_RSA_WITH_CAMELLIA_256_CBC_SHA xc013 ECDHE-RSA-AES128-SHA ECDH 521 AES 128 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA x2f AES128-SHA RSA AES 128 TLS_RSA_WITH_AES_128_CBC_SHA x96 SEED-SHA RSA SEED 128 TLS_RSA_WITH_SEED_CBC_SHA x41 CAMELLIA128-SHA RSA Camellia 128 TLS_RSA_WITH_CAMELLIA_128_CBC_SHA x07 IDEA-CBC-SHA RSA IDEA 128 TLS_RSA_WITH_IDEA_CBC_SHA

TLS 1.2 xc030 ECDHE-RSA-AES256-GCM-SHA384 ECDH 521 AESGCM 256 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 xc028 ECDHE-RSA-AES256-SHA384 ECDH 521 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 xc014 ECDHE-RSA-AES256-SHA ECDH 521 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA xcca8 ECDHE-RSA-CHACHA20-POLY1305 ECDH 521 ChaCha20 256 TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 xc077 ECDHE-RSA-CAMELLIA256-SHA384 ECDH 521 Camellia 256 TLS_ECDHE_RSA_WITH_CAMELLIA_256_CBC_SHA384 x9d AES256-GCM-SHA384 RSA AESGCM 256 TLS_RSA_WITH_AES_256_GCM_SHA384 xc0a1 AES256-CCM8 RSA AESCCM8 256 TLS_RSA_WITH_AES_256_CCM_8 xc09d AES256-CCM RSA AESCCM 256 TLS_RSA_WITH_AES_256_CCM x3d AES256-SHA256 RSA AES 256 TLS_RSA_WITH_AES_256_CBC_SHA256 x35 AES256-SHA RSA AES 256 TLS_RSA_WITH_AES_256_CBC_SHA xc0 CAMELLIA256-SHA256 RSA Camellia 256 TLS_RSA_WITH_CAMELLIA_256_CBC_SHA256 x84 CAMELLIA256-SHA RSA Camellia 256 TLS_RSA_WITH_CAMELLIA_256_CBC_SHA xc051 ARIA256-GCM-SHA384 RSA ARIAGCM 256 TLS_RSA_WITH_ARIA_256_GCM_SHA384 xc061 ECDHE-ARIA256-GCM-SHA384 ECDH 521 ARIAGCM 256 TLS_ECDHE_RSA_WITH_ARIA_256_GCM_SHA384 xc02f ECDHE-RSA-AES128-GCM-SHA256 ECDH 521 AESGCM 128 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 xc027 ECDHE-RSA-AES128-SHA256 ECDH 521 AES 128 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 xc013 ECDHE-RSA-AES128-SHA ECDH 521 AES 128 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA xc0a0 AES128-CCM8 RSA AESCCM8 128 TLS_RSA_WITH_AES_128_CCM_8 xc09c AES128-CCM RSA AESCCM 128 TLS_RSA_WITH_AES_128_CCM xc076 ECDHE-RSA-CAMELLIA128-SHA256 ECDH 521 Camellia 128 TLS_ECDHE_RSA_WITH_CAMELLIA_128_CBC_SHA256 x9c AES128-GCM-SHA256 RSA AESGCM 128 TLS_RSA_WITH_AES_128_GCM_SHA256 x3c AES128-SHA256 RSA AES 128 TLS_RSA_WITH_AES_128_CBC_SHA256 x2f AES128-SHA RSA AES 128 TLS_RSA_WITH_AES_128_CBC_SHA xba CAMELLIA128-SHA256 RSA Camellia 128 TLS_RSA_WITH_CAMELLIA_128_CBC_SHA256 x96 SEED-SHA RSA SEED 128 TLS_RSA_WITH_SEED_CBC_SHA x41 CAMELLIA128-SHA RSA Camellia 128 TLS_RSA_WITH_CAMELLIA_128_CBC_SHA xc050 ARIA128-GCM-SHA256 RSA ARIAGCM 128 TLS_RSA_WITH_ARIA_128_GCM_SHA256 xc060 ECDHE-ARIA128-GCM-SHA256 ECDH 521 ARIAGCM 128 TLS_ECDHE_RSA_WITH_ARIA_128_GCM_SHA256

SrvUseStrictSslSettings = 1

| Hexcode | Cipher Suite Name (OpenSSL) | KeyExch Encryption Bits | Cipher Suite Name |(IANA/RFC) |

TLS 1 xc014 ECDHE-RSA-AES256-SHA ECDH 521 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA x35 AES256-SHA RSA AES 256 TLS_RSA_WITH_AES_256_CBC_SHA xc013 ECDHE-RSA-AES128-SHA ECDH 521 AES 128 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA x2f AES128-SHA RSA AES 128 TLS_RSA_WITH_AES_128_CBC_SHA

TLS 1.1 xc014 ECDHE-RSA-AES256-SHA ECDH 521 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA x35 AES256-SHA RSA AES 256 TLS_RSA_WITH_AES_256_CBC_SHA xc013 ECDHE-RSA-AES128-SHA ECDH 521 AES 128 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA x2f AES128-SHA RSA AES 128 TLS_RSA_WITH_AES_128_CBC_SHA

TLS 1.2 xc030 ECDHE-RSA-AES256-GCM-SHA384 ECDH 521 AESGCM 256 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 xc028 ECDHE-RSA-AES256-SHA384 ECDH 521 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 xc014 ECDHE-RSA-AES256-SHA ECDH 521 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA xcca8 ECDHE-RSA-CHACHA20-POLY1305 ECDH 521 ChaCha20 256 TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 x9d AES256-GCM-SHA384 RSA AESGCM 256 TLS_RSA_WITH_AES_256_GCM_SHA384 x3d AES256-SHA256 RSA AES 256 TLS_RSA_WITH_AES_256_CBC_SHA256 x35 AES256-SHA RSA AES 256 TLS_RSA_WITH_AES_256_CBC_SHA xc02f ECDHE-RSA-AES128-GCM-SHA256 ECDH 521 AESGCM 128 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 xc027 ECDHE-RSA-AES128-SHA256 ECDH 521 AES 128 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 xc013 ECDHE-RSA-AES128-SHA ECDH 521 AES 128 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA x9c AES128-GCM-SHA256 RSA AESGCM 128 TLS_RSA_WITH_AES_128_GCM_SHA256 x3c AES128-SHA256 RSA AES 128 TLS_RSA_WITH_AES_128_CBC_SHA256 x2f AES128-SHA RSA AES 128 TLS_RSA_WITH_AES_128_CBC_SHA

SrvUseStrictSslSettings = 2

| Hexcode | Cipher Suite Name (OpenSSL) | KeyExch Encryption Bits | Cipher Suite Name |(IANA/RFC) |

TLS 1 xc014 ECDHE-RSA-AES256-SHA ECDH 256 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA x35 AES256-SHA RSA AES 256 TLS_RSA_WITH_AES_256_CBC_SHA x84 CAMELLIA256-SHA RSA Camellia 256 TLS_RSA_WITH_CAMELLIA_256_CBC_SHA xc013 ECDHE-RSA-AES128-SHA ECDH 256 AES 128 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA x2f AES128-SHA RSA AES 128 TLS_RSA_WITH_AES_128_CBC_SHA x41 CAMELLIA128-SHA RSA Camellia 128 TLS_RSA_WITH_CAMELLIA_128_CBC_SHA

TLS 1.1 xc014 ECDHE-RSA-AES256-SHA ECDH 256 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA x35 AES256-SHA RSA AES 256 TLS_RSA_WITH_AES_256_CBC_SHA x84 CAMELLIA256-SHA RSA Camellia 256 TLS_RSA_WITH_CAMELLIA_256_CBC_SHA xc013 ECDHE-RSA-AES128-SHA ECDH 256 AES 128 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA x2f AES128-SHA RSA AES 128 TLS_RSA_WITH_AES_128_CBC_SHA x41 CAMELLIA128-SHA RSA Camellia 128 TLS_RSA_WITH_CAMELLIA_128_CBC_SHA

TLS 1.2 xc030 ECDHE-RSA-AES256-GCM-SHA384 ECDH 256 AESGCM 256 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 xc028 ECDHE-RSA-AES256-SHA384 ECDH 256 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 xc014 ECDHE-RSA-AES256-SHA ECDH 256 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA xcca8 ECDHE-RSA-CHACHA20-POLY1305 ECDH 253 ChaCha20 256 TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 xc077 ECDHE-RSA-CAMELLIA256-SHA384 ECDH 256 Camellia 256 TLS_ECDHE_RSA_WITH_CAMELLIA_256_CBC_SHA384 x9d AES256-GCM-SHA384 RSA AESGCM 256 TLS_RSA_WITH_AES_256_GCM_SHA384 xc0a1 AES256-CCM8 RSA AESCCM8 256 TLS_RSA_WITH_AES_256_CCM_8 xc09d AES256-CCM RSA AESCCM 256 TLS_RSA_WITH_AES_256_CCM x3d AES256-SHA256 RSA AES 256 TLS_RSA_WITH_AES_256_CBC_SHA256 x35 AES256-SHA RSA AES 256 TLS_RSA_WITH_AES_256_CBC_SHA xc0 CAMELLIA256-SHA256 RSA Camellia 256 TLS_RSA_WITH_CAMELLIA_256_CBC_SHA256 x84 CAMELLIA256-SHA RSA Camellia 256 TLS_RSA_WITH_CAMELLIA_256_CBC_SHA xc051 ARIA256-GCM-SHA384 RSA ARIAGCM 256 TLS_RSA_WITH_ARIA_256_GCM_SHA384 xc061 ECDHE-ARIA256-GCM-SHA384 ECDH 253 ARIAGCM 256 TLS_ECDHE_RSA_WITH_ARIA_256_GCM_SHA384 xc02f ECDHE-RSA-AES128-GCM-SHA256 ECDH 256 AESGCM 128 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 xc027 ECDHE-RSA-AES128-SHA256 ECDH 256 AES 128 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 xc013 ECDHE-RSA-AES128-SHA ECDH 256 AES 128 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA xc0a0 AES128-CCM8 RSA AESCCM8 128 TLS_RSA_WITH_AES_128_CCM_8 xc09c AES128-CCM RSA AESCCM 128 TLS_RSA_WITH_AES_128_CCM xc076 ECDHE-RSA-CAMELLIA128-SHA256 ECDH 256 Camellia 128 TLS_ECDHE_RSA_WITH_CAMELLIA_128_CBC_SHA256 x9c AES128-GCM-SHA256 RSA AESGCM 128 TLS_RSA_WITH_AES_128_GCM_SHA256 x3c AES128-SHA256 RSA AES 128 TLS_RSA_WITH_AES_128_CBC_SHA256 x2f AES128-SHA RSA AES 128 TLS_RSA_WITH_AES_128_CBC_SHA xba CAMELLIA128-SHA256 RSA Camellia 128 TLS_RSA_WITH_CAMELLIA_128_CBC_SHA256 x41 CAMELLIA128-SHA RSA Camellia 128 TLS_RSA_WITH_CAMELLIA_128_CBC_SHA xc050 ARIA128-GCM-SHA256 RSA ARIAGCM 128 TLS_RSA_WITH_ARIA_128_GCM_SHA256 xc060 ECDHE-ARIA128-GCM-SHA256 ECDH 253 ARIAGCM 128 TLS_ECDHE_RSA_WITH_ARIA_128_GCM_SHA256

SrvUseStrictSslSettings = 3

| Hexcode | Cipher Suite Name (OpenSSL) | KeyExch Encryption Bits | Cipher Suite Name |(IANA/RFC) |

TLS 1.2 xc030 ECDHE-RSA-AES256-GCM-SHA384 ECDH 256 AESGCM 256 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 xc028 ECDHE-RSA-AES256-SHA384 ECDH 256 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 xc014 ECDHE-RSA-AES256-SHA ECDH 256 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA xcca8 ECDHE-RSA-CHACHA20-POLY1305 ECDH 253 ChaCha20 256 TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 xc077 ECDHE-RSA-CAMELLIA256-SHA384 ECDH 256 Camellia 256 TLS_ECDHE_RSA_WITH_CAMELLIA_256_CBC_SHA384 x9d AES256-GCM-SHA384 RSA AESGCM 256 TLS_RSA_WITH_AES_256_GCM_SHA384 xc0a1 AES256-CCM8 RSA AESCCM8 256 TLS_RSA_WITH_AES_256_CCM_8 xc09d AES256-CCM RSA AESCCM 256 TLS_RSA_WITH_AES_256_CCM x3d AES256-SHA256 RSA AES 256 TLS_RSA_WITH_AES_256_CBC_SHA256 x35 AES256-SHA RSA AES 256 TLS_RSA_WITH_AES_256_CBC_SHA xc0 CAMELLIA256-SHA256 RSA Camellia 256 TLS_RSA_WITH_CAMELLIA_256_CBC_SHA256 x84 CAMELLIA256-SHA RSA Camellia 256 TLS_RSA_WITH_CAMELLIA_256_CBC_SHA xc051 ARIA256-GCM-SHA384 RSA ARIAGCM 256 TLS_RSA_WITH_ARIA_256_GCM_SHA384 xc061 ECDHE-ARIA256-GCM-SHA384 ECDH 253 ARIAGCM 256 TLS_ECDHE_RSA_WITH_ARIA_256_GCM_SHA384 xc02f ECDHE-RSA-AES128-GCM-SHA256 ECDH 256 AESGCM 128 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 xc027 ECDHE-RSA-AES128-SHA256 ECDH 256 AES 128 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 xc013 ECDHE-RSA-AES128-SHA ECDH 256 AES 128 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA xc0a0 AES128-CCM8 RSA AESCCM8 128 TLS_RSA_WITH_AES_128_CCM_8 xc09c AES128-CCM RSA AESCCM 128 TLS_RSA_WITH_AES_128_CCM xc076 ECDHE-RSA-CAMELLIA128-SHA256 ECDH 256 Camellia 128 TLS_ECDHE_RSA_WITH_CAMELLIA_128_CBC_SHA256 x9c AES128-GCM-SHA256 RSA AESGCM 128 TLS_RSA_WITH_AES_128_GCM_SHA256 x3c AES128-SHA256 RSA AES 128 TLS_RSA_WITH_AES_128_CBC_SHA256 x2f AES128-SHA RSA AES 128 TLS_RSA_WITH_AES_128_CBC_SHA xba CAMELLIA128-SHA256 RSA Camellia 128 TLS_RSA_WITH_CAMELLIA_128_CBC_SHA256 x41 CAMELLIA128-SHA RSA Camellia 128 TLS_RSA_WITH_CAMELLIA_128_CBC_SHA xc050 ARIA128-GCM-SHA256 RSA ARIAGCM 128 TLS_RSA_WITH_ARIA_128_GCM_SHA256 xc060 ECDHE-ARIA128-GCM-SHA256 ECDH 253 ARIAGCM 128 TLS_ECDHE_RSA_WITH_ARIA_128_GCM_SHA256

SrvUseStrictSslSettings = 4 (default)

| Hexcode | Cipher Suite Name (OpenSSL) | KeyExch Encryption Bits | Cipher Suite Name |(IANA/RFC) |

TLS 1.2 xc030 ECDHE-RSA-AES256-GCM-SHA384 ECDH 256 AESGCM 256 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 xc028 ECDHE-RSA-AES256-SHA384 ECDH 256 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 xcca8 ECDHE-RSA-CHACHA20-POLY1305 ECDH 256 ChaCha20 256 TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 x9d AES256-GCM-SHA384 RSA AESGCM 256 TLS_RSA_WITH_AES_256_GCM_SHA384 xc02f ECDHE-RSA-AES128-GCM-SHA256 ECDH 256 AESGCM 128 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 xc027 ECDHE-RSA-AES128-SHA256 ECDH 256 AES 128 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256

SrvUseStrictSslSettings = 5

| Hexcode | Cipher Suite Name (OpenSSL) | KeyExch Encryption Bits | Cipher Suite Name |(IANA/RFC) |

TLS 1.2 xc030 ECDHE-RSA-AES256-GCM-SHA384 ECDH 256 AESGCM 256 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 xc028 ECDHE-RSA-AES256-SHA384 ECDH 256 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 xcca8 ECDHE-RSA-CHACHA20-POLY1305 ECDH 256 ChaCha20 256 TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 xc02f ECDHE-RSA-AES128-GCM-SHA256 ECDH 256 AESGCM 128 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 xc027 ECDHE-RSA-AES128-SHA256 ECDH 256 AES 128 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256