AWCM and ECC Troubleshooting

General Rules

Do not install Enterprise Service Connector/AirWatch Cloud Connector (ESC/ACC), until you are absolutely sure AWCM is working
Do not install AWCM or ESC (ACC) in Global tenant
Check there is a Device Root Certificate in the Organization Group, in which work is done. It is located in \system configuration\system\advanced\device root certificate. If there is nothing, click Generate.

AWCM Installation

When installing AWCM, DO NOT use the self signed SSL certificate, check the box for “custom SSL” which really means the public SSL Cert you put in IIS for Device Services. Notice that in the installation dialog box the two fields for password are NOT the same. One is for the SSL Cert you are importing and one is for the password to the Java Keystore.
Make sure that REST API is enabled in the OG where you are enabling AWCM.
Make sure that AWCM is enabled in the Site URL’s page. Also, put the correct information in the two fields. The External URL should NOT contain http:// or https://. The Internal Service URL should contain https:// instead of http:// and should have the port number after the URL and “/awcm” at the end. It should look like https://{url}:2001/awcm.
Download and run the AWCM Secure Channel Certificate program from the Secure Channel Certificate page ON THE SERVER RUNNING AWCM.

DO NOT download the program onto another computer and copy it to the AWCM server!

Download and run this program “As Administrator”.

There is a possibility that you will receive an error message that the application can’t find the Java Folder, this can be a result of not running the program “As Administrator”.

Browse to the AWCM Status page by going to https://{url}:2001/awcm/status. If this page doesn’t come up or if there is an SSL error stop and fix it before you go on. Check the SSL Certificate common name, it should match the name of the DS URL. If it says “Air Watch “ then you need to uninstall and reinstall AWCM, this time installing the correct SSL Certificate (see #1).

ESC/ACC WILL NOT WORK if you use the self signed certificate!

AWCM Status page MUST BE TRUSTED by AirWatch Console AND by ESC. Test by opening https://{url}:2001/awcm/status status page in browser - there MUST BE NO CERTIFICATE WARNING!

Confirm that the awcm.truststore and the awcm.keystores are not corrupt and contain the correct certificates. Run the keytool application (see next section) and list the contents of both stores.
- In awcm.keystore there should be 1 certificate and it should contain the SSL certificate for the site.
- In the awcm.truststore there should be 2 or 3 certificates: one of them is Secure Channel Certificate.
If the certificates do not exist in the stores then you may need to re-install AWCM. If the password is not accepted then the store may be corrupt and you will need to reinstall AWCM.

Java KeyStore

AWCM is a Java web application and stores its certificates in the Java Keystore as opposed to the Microsoft Certificate store. The Java Keystore and Java Truststore are located in the \airwatch\airwatch {version}\awcm\config folder.

There is a utility in Windows called “keytool”. With this utility you can view, add, and delete certificates from the Java Keystores.

⭐️ Password to awcm.truststore = “password”

Password to awcm.keystore = password to the PFX certificate uploaded on installation of AWCM. DO NOT use password less than 6 characters! Or you will not be able to change certificate in awcm.keystore.

Example of keytool commands:

# List the certificates in the store:
keytool -list -v -keystore awcm.truststore

# Import a certificate into a store:
keytool -import -trustcacerts -file {cert file} -alias {common name} -keystore $JAVA_HOME/jre/lib/security/cacerts

Replace database of AWCM

Run the following command to replace SSL cert on AWCM servers:

keytool -importkeystore -srckeystore <new-pfx-cert-name>.pfx -srcstoretype pkcs12 -destkeystore awcm.keystore.new -deststoretype JKS

Once this has completed successfully, you will now see a new file named awcm.keystore.new in the config directory.
Stop the AWCM service.
Rename the awcm.keystore to awcm.keystore.old.
Rename the awcm.keystore.new to awcm.keystore.
Start the AWCM service.

Reinstall of Secure Channel Certificate

If ESC is installed, then uninstall it and delete all its’ folders before reinstall of Secure Channel Certificate.

Before reinstall of Secure Channel Certificate, you must delete the old certificate from the AWCM Java database. Delete a certificate in the store:

keytool -delete -alias "aw secure channel certificate - {url}" -keystore awcm.truststore

AWCM Logs

See General Article on AirWatch Logs.

Verify correct work of AWCM

Verify AWCM is listening on the configured port through netstat:

netstat -ano | findstr LISTENING | findstr {port number}

Perform the following to make sure that AWCM is functioning accurately:

Confirm that there is a device root certificate in the relevant OG by navigating to Settings / System / Advanced / Device Root Certificate.
Make sure that REST API is enabled.
Make sure that AWCM is enabled in the URL page of the site.
- The external URL must not contain http:// or https://.
- Exact format of the URL: https://{url}:2001/awcm.
Browse to the AWCM Status page by selecting https://{url}:2001/awcm/status (You should see “OK”) and https://{url}:2001/awcm/statistics

❗️For clustering infrastructure on SaaS, browse over port 443 instead of 2001 (https://awcm118.awmdm.com/awcm/statistics) while testing the status page.

❗️For lor load balanced deployments:

Ensure that clients who are required to connect to AWCM are pointed to and are able to reach the endpoint on the load balancer. This means that if installation of AWCM is on the DS servers, then ensure that the requests for AWCM from the DS services are still accessing the load balancer so that they are subject to the set rules;
As per the Installation Guide, the preferred deployment for a customer using ESC/ACC with AWCM is to deploy multiple AWCM nodes in an active-passive configuration. This makes everything easier since persistence of connections doesn’t matter. There are no specific advantages with having two active nodes as the network load is not much while using only ESC/ACC.

Load Balancing AWCM - Persistence Rules with F5 LTM

See KB for details: https://kb.vmware.com/s/article/2960904

To deploy AWCM with multiple nodes behind a load balancer without clustering, you must account for persisting the connections to the AWCM servers. In the HTTP request that is sent to AWCM (from a device, the Device Services server, the Console Server, ACC, and so on), there is a cookie value called awcmsessionid, which is used to establish request level affinity to an AWCM node from a pool of nodes. You must configure your load balancer or proxy to parse the HTTP request for this value and use it for persistence. The persistence settings are only necessary for AWCM servers that are load-balanced in an active-active manner. The persistence settings will ensure that established connections are not dropped when the F5 switches from one AWCM server to the other to balance the load.

The iRule might vary based on a client’s existing configuration or best-practices, but the basics are straight forward:

Parse the HTTP request for the awcmsessionid cookie’s value
Set persistence with this value via the “persist carp” command.

For more background on this methodology, see the following F5 solution page for Overview of the CARP hash algorithm.

In Local Traffic → iRules : iRule List create an iRule to inspect the HTTP request for the value of the “awcmsessionid” cookie:

    when HTTP_REQUEST {

        if { [HTTP::cookie exists "awcmsessionid"] }{

            set awcm_session_id [string tolower [HTTP::cookie "awcmsessionid"]]

            persist carp $awcm_session_id

        }

    }

In Local Traffic → Profiles: Persistence create a persistence profile based on the default “hash” profile. All items should be default except the following:
- Algorithm: “CARP”
- iRule: created in Step 1.
Configure the Virtual server with the following settings:
- Select an HTTP profile from the “HTTP Profile” drop-down list;
- Select OneConnect profile = oneconnect (for HTTP request balancing, not just connection);
- Apply the persistence profile under the Resources tab

Load Balancing AWCM - Persistence Rules with Citrix NetScaler

See details in an archived article - http://web.archive.org/web/20190506104002/https://www.citrix.com/blogs/2010/06/01/configuring-verifying-and-monitoring-persistence-on-netscaler/

Check the Persistence Rule

Check for persistence by hitting the URL [https://awcm url:port/awcm/statistics?awcmsessionid=abc123](https://awcm urlport) on ACC server and on machine from outside network. If the servers are different then customer needs to change the persistence rule on load balancer on their end.

Common Errors

AWCM Status Error - DNS name

AWCM not working - page https://<DS_URL>:2001/awcm/status unavailable. Error log seen:

2017-08-15 12:02:44,229 ERROR (nioEventLoopGroup-3-3) [com.airwatch.awcm.event.AWCMChannelConnectedEventHandler] - java.nio.channels.ClosedChannelException
java.util.concurrent.ExecutionException: java.nio.channels.ClosedChannelException
<...>
at io.netty.handler.ssl.SslHandler.channelInactive(...)(Unknown Source) [netty-all-4.0.43.Final.jar:4.0.43.Final]

Solution: DNS name of Device Services is registered on external proxy and not known to servers. Go to C:\Windows\System32\drivers\etc\hosts file on AirWatch Admin Console, and also on server with Enterprise Connector Service and add the EXTERNAL public DNS name (listed in public certificate) of AWCM binded to its’ internal IP.

AWCM Status Error - Cryptography

AWCM not working - page https://<DS_URL>:2001/awcm/status unavailable. Error log seen:

2017-08-15 14:49:41,044 ERROR (nioEventLoopGroup-3-7) [com.airwatch.awcm.event.AWCMChannelConnectedEventHandler] - javax.net.ssl.SSLHandshakeException: no cipher suites in common
java.util.concurrent.ExecutionException: javax.net.ssl.SSLHandshakeException: no cipher suites in common

Solution: AWCM was installed AFTER crypto algorithms were disabled in IIS hardening, and it cannot launch normally. Reinstall of AWCM needed.

AWCM SSL Certificate Error

Certificate error while browsing the AWCM status page

Login to the AWCM server.
Open a command prompt, navigate to the following directory (E:\airwatch\airwatch\AWCM\config) and run the following: keytool -list -v -keystore awcm.keystore
Enter the password when prompted
Export a new SSL certificate from a machine.

❗️Make sure that the full signing chain is exported (settings that you select when exporting the certificate) and that the password used to export is same as the one used for the current awcm.keystore.

If the passwords are not same, the import happens but an error message appears when AWCM starts and the status page does not load (as the pre-configured password will be incorrect and the AWCM app will not be able to open the keystore).

AWCM_BRIDGE_FILE_TRANSFER_TIMEOUT_IN_MINUTES : 15 2013-09-23 10:46:22,036
Error (main) [com.airwatch.awcm.ssl.AWCMSSLContext]:

java.security.UnrecoverableKeyException: Cannot recover key java.security.UnrecoverableKeyException: Cannot recover key
<...>
2013-09-23 10:46:22,036 ERROR (main) [com.airwatch.awcm.server.AWCMServer] - Error initializing server environment, exiting

When the certificate is on the AWCM server (copy into the C:\airwatch\airwatch\AWCM\config directory), run the following command to replace SSL certificate:

keytool -importkeystore -srckeystore <new-pfx-cert-name>.pfx -srcstoretype pkcs12 -destkeystore awcm.keystore.new -deststoretype JKS

Once this has completed successfully, you will now see a new file named awcm.keystore.new in the config directory. Stop the AWCM service.
Rename the awcm.keystore to awcm.keystore.old.
Rename the awcm.keystore.new to awcm.keystore.
Start the AWCM service.
Using a valid AWCM URL, try to access the page (https://{url}:2001/awcm/status) and if the status page loads, then check the certificate details. It should display the values for the newly uploaded certificate.

If the status page does not load, check the log files.
If rollback is required, rename the awcm.keystore to awcm.keystore.new.
Then rename awcm.keystore.old to awcm.keystore. Restart AWCM to restore the old settings.

AWCM and Admin Console trust error

ESC/ACC starts and generates no errors in log, also no errors in AWCM. But error in console while performing Test connection for ESC/ACC: Undefined Error; Please check server logs.

Reason: there is no trust between AWCM and AirWatch Admin Console

**Remedy:
**Import Intermediate and Root certificates for public PFX certificate in AWCM server and AirWatch Admin Console Server

ACC Errors

ACC must be able to reach AWCM:

Protocol HTTPS
Telnet from ACC to AWCM Server on the relevant port (usually 2001 for On-Premise installations and 443 for SaaS environments)
Also, verify by opening a browser on the ACC server, entering https://:2001/awcm/status and /awcm/statistics to ensure there is no certificate trust error.
For On-Premise installations: if using ACC with AWCM and there are multiple AWCM servers and they required to be load-balanced them, persistence needs to be configured - see the above section.

ACC must be able to reach the Console Server:

Protocol: HTTP or HTTPS
Telnet from ACC to the Console URL on the relevant port (usually 80 or 443)
Also, verify by opening a browser on the ACC server, entering http(s)://
If auto-update is enabled, ACC must be able to query AirWatch Console for updates

ACC must be able to reach the API

Protocol: HTTPS, port TCP443
Verify by navigating to the URL of your API server on the ACC server: https:///API/help
When the credentials screen appears, enter the credentials of a console admin and the API Developer page should appear.
ACC to API access is required for the proper functioning of the AirWatch Diagnostics service.

Connectivity Errors

If you see errors in the ACC logs indicating connections being closed/aborted/terminated, check if there is any network device in between the ACC and AWCM that would close or terminate idle connections. The outbound connection required for use by ACC must remain open at all times. Check the TCP session timeout on this network device in between and see if this can be increased to a value >2 minutes;
ACC sends what is known as an IDLE message, by default every 2 minutes. This IDLE message by ACC helps ACC register itself as a listener on AWCM so that AWCM knows that this ACC is ready to take requests;
If there are any network devices between ACC and AWCM that closes the connection between these components deeming the connection as an idle connection, it could cause issues with this ACC/AWCM connectivity.

401 Errors on ACC

Check time on ACC and AWCM servers to make sure they are correct and synced to NTP preferably to avoid time synchronization issues.
Make sure secure channel certificate is installed on the AWCM server
For any certificate related issues, enable and check CAPI2 logs on ACC and make changes accordingly in certificate stores (adding root/intermediate CA certificates to establish trust)
CAPI2 Logging (https://social.technet.microsoft.com/wiki/contents/articles/242.windows-pki-troubleshooting-capi2-diagnostics.aspx)

ACC service does not start:

Error log contains error:

System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.
<...>

Reason: ESC/ACC service does not start because there is no trust between ESC/ACC and AWCM.

If this error is present after trying to hit Update/Check URL on the console, check the SSL certificate on the console and do the following: keytool -list -v -keystore "{AWCM install path}/awcm.truststore" > c:\test.txt

In the .txt file, search for the secure channel and it should match with the secure channel certificate in the console.

Remedy:

Generate new certificates for ESC/ACC and download the installer. Then, uninstall ACC and install the new ACC with the renewed certificates. Restart the AWCM service, if required.
Reinstall AWCM and download the installer from the console.