AWCM and ECC Troubleshooting
General Rules
- Do not install Enterprise Service Connector/AirWatch Cloud Connector (ESC/ACC), until you are absolutely sure AWCM is working
- Do not install AWCM or ESC (ACC) in Global tenant
- Check there is a Device Root Certificate in the Organization Group, in which work is done. It is located in \system configuration\system\advanced\device root certificate. If there is nothing, click Generate.
AWCM Installation
-
When installing AWCM, DO NOT use the self signed SSL certificate, check the box for “custom SSL” which really means the public SSL Cert you put in IIS for Device Services. Notice that in the installation dialog box the two fields for password are NOT the same. One is for the SSL Cert you are importing and one is for the password to the Java Keystore.
-
Make sure that REST API is enabled in the OG where you are enabling AWCM.
-
Make sure that AWCM is enabled in the Site URL’s page. Also, put the correct information in the two fields. The External URL should NOT contain http:// or https://. The Internal Service URL should contain https:// instead of http:// and should have the port number after the URL and “/awcm” at the end. It should look like https://{url}:2001/awcm.
-
Download and run the AWCM Secure Channel Certificate program from the Secure Channel Certificate page ON THE SERVER RUNNING AWCM.
DO NOT download the program onto another computer and copy it to the AWCM server!
Download and run this program “As Administrator”.
There is a possibility that you will receive an error message that the application can’t find the Java Folder, this can be a result of not running the program “As Administrator”.
- Browse to the AWCM Status page by going to https://{url}:2001/awcm/status. If this page doesn’t come up or if there is an SSL error stop and fix it before you go on. Check the SSL Certificate common name, it should match the name of the DS URL. If it says “Air Watch “ then you need to uninstall and reinstall AWCM, this time installing the correct SSL Certificate (see #1).
ESC/ACC WILL NOT WORK if you use the self signed certificate!
AWCM Status page MUST BE TRUSTED by AirWatch Console AND by ESC. Test by opening https://{url}:2001/awcm/status status page in browser - there MUST BE NO CERTIFICATE WARNING!
-
Confirm that the awcm.truststore and the awcm.keystores are not corrupt and contain the correct certificates. Run the keytool application (see next section) and list the contents of both stores.
- In awcm.keystore there should be 1 certificate and it should contain the SSL certificate for the site.
- In the awcm.truststore there should be 2 or 3 certificates: one of them is Secure Channel Certificate.
If the certificates do not exist in the stores then you may need to re-install AWCM. If the password is not accepted then the store may be corrupt and you will need to reinstall AWCM.
Java KeyStore
AWCM is a Java web application and stores its certificates in the Java Keystore as opposed to the Microsoft Certificate store. The Java Keystore and Java Truststore are located in the \airwatch\airwatch {version}\awcm\config folder.
There is a utility in Windows called “keytool”. With this utility you can view, add, and delete certificates from the Java Keystores.
⭐️ Password to awcm.truststore = “password”
Password to awcm.keystore = password to the PFX certificate uploaded on installation of AWCM. DO NOT use password less than 6 characters! Or you will not be able to change certificate in awcm.keystore.
Example of keytool commands:
# List the certificates in the store:
keytool -list -v -keystore awcm.truststore
# Import a certificate into a store:
keytool -import -trustcacerts -file {cert file} -alias {common name} -keystore $JAVA_HOME/jre/lib/security/cacerts
Replace database of AWCM
- Run the following command to replace SSL cert on AWCM servers:
keytool -importkeystore -srckeystore <new-pfx-cert-name>.pfx -srcstoretype pkcs12 -destkeystore awcm.keystore.new -deststoretype JKS
- Once this has completed successfully, you will now see a new file named awcm.keystore.new in the config directory.
- Stop the AWCM service.
- Rename the awcm.keystore to awcm.keystore.old.
- Rename the awcm.keystore.new to awcm.keystore.
- Start the AWCM service.
Reinstall of Secure Channel Certificate
If ESC is installed, then uninstall it and delete all its’ folders before reinstall of Secure Channel Certificate.
Before reinstall of Secure Channel Certificate, you must delete the old certificate from the AWCM Java database. Delete a certificate in the store:
keytool -delete -alias "aw secure channel certificate - {url}" -keystore awcm.truststore
AWCM Logs
See General Article on AirWatch Logs.
Verify correct work of AWCM
Verify AWCM is listening on the configured port through netstat:
netstat -ano | findstr LISTENING | findstr {port number}
Perform the following to make sure that AWCM is functioning accurately:
- Confirm that there is a device root certificate in the relevant OG by navigating to Settings / System / Advanced / Device Root Certificate.
- Make sure that REST API is enabled.
- Make sure that AWCM is enabled in the URL page of the site.
- The external URL must not contain http:// or https://.
- Exact format of the URL: https://{url}:2001/awcm.
- Browse to the AWCM Status page by selecting https://{url}:2001/awcm/status (You should see “OK”) and https://{url}:2001/awcm/statistics
❗️For clustering infrastructure on SaaS, browse over port 443 instead of 2001 (https://awcm118.awmdm.com/awcm/statistics) while testing the status page.
❗️For lor load balanced deployments:
- Ensure that clients who are required to connect to AWCM are pointed to and are able to reach the endpoint on the load balancer. This means that if installation of AWCM is on the DS servers, then ensure that the requests for AWCM from the DS services are still accessing the load balancer so that they are subject to the set rules;
- As per the Installation Guide, the preferred deployment for a customer using ESC/ACC with AWCM is to deploy multiple AWCM nodes in an active-passive configuration. This makes everything easier since persistence of connections doesn’t matter. There are no specific advantages with having two active nodes as the network load is not much while using only ESC/ACC.
Load Balancing AWCM - Persistence Rules with F5 LTM
See KB for details: https://kb.vmware.com/s/article/2960904
To deploy AWCM with multiple nodes behind a load balancer without clustering, you must account for persisting the connections to the AWCM servers. In the HTTP request that is sent to AWCM (from a device, the Device Services server, the Console Server, ACC, and so on), there is a cookie value called awcmsessionid, which is used to establish request level affinity to an AWCM node from a pool of nodes. You must configure your load balancer or proxy to parse the HTTP request for this value and use it for persistence. The persistence settings are only necessary for AWCM servers that are load-balanced in an active-active manner. The persistence settings will ensure that established connections are not dropped when the F5 switches from one AWCM server to the other to balance the load.
The iRule might vary based on a client’s existing configuration or best-practices, but the basics are straight forward:
- Parse the HTTP request for the awcmsessionid cookie’s value
- Set persistence with this value via the “persist carp” command.
For more background on this methodology, see the following F5 solution page for Overview of the CARP hash algorithm.
- In Local Traffic → iRules : iRule List create an iRule to inspect the HTTP request for the value of the “awcmsessionid” cookie:
when HTTP_REQUEST {
if { [HTTP::cookie exists "awcmsessionid"] }{
set awcm_session_id [string tolower [HTTP::cookie "awcmsessionid"]]
persist carp $awcm_session_id
}
}
-
In Local Traffic → Profiles: Persistence create a persistence profile based on the default “hash” profile. All items should be default except the following:
- Algorithm: “CARP”
- iRule: created in Step 1.
-
Configure the Virtual server with the following settings:
- Select an HTTP profile from the “HTTP Profile” drop-down list;
- Select OneConnect profile = oneconnect (for HTTP request balancing, not just connection);
- Apply the persistence profile under the Resources tab
Load Balancing AWCM - Persistence Rules with Citrix NetScaler
See details in an archived article - http://web.archive.org/web/20190506104002/https://www.citrix.com/blogs/2010/06/01/configuring-verifying-and-monitoring-persistence-on-netscaler/
Check the Persistence Rule
Check for persistence by hitting the URL [https://awcm url:port/awcm/statistics?awcmsessionid=abc123](https://awcm urlport) on ACC server and on machine from outside network. If the servers are different then customer needs to change the persistence rule on load balancer on their end.
Common Errors
AWCM Status Error - DNS name
AWCM not working - page https://<DS_URL>:2001/awcm/status unavailable.
Error log seen:
2017-08-15 12:02:44,229 ERROR (nioEventLoopGroup-3-3) [com.airwatch.awcm.event.AWCMChannelConnectedEventHandler] - java.nio.channels.ClosedChannelException
java.util.concurrent.ExecutionException: java.nio.channels.ClosedChannelException
<...>
at io.netty.handler.ssl.SslHandler.channelInactive(...)(Unknown Source) [netty-all-4.0.43.Final.jar:4.0.43.Final]
Solution: DNS name of Device Services is registered on external proxy and not known to servers. Go to C:\Windows\System32\drivers\etc\hosts file on AirWatch Admin Console, and also on server with Enterprise Connector Service and add the EXTERNAL public DNS name (listed in public certificate) of AWCM binded to its’ internal IP.
AWCM Status Error - Cryptography
AWCM not working - page https://<DS_URL>:2001/awcm/status unavailable.
Error log seen:
2017-08-15 14:49:41,044 ERROR (nioEventLoopGroup-3-7) [com.airwatch.awcm.event.AWCMChannelConnectedEventHandler] - javax.net.ssl.SSLHandshakeException: no cipher suites in common
java.util.concurrent.ExecutionException: javax.net.ssl.SSLHandshakeException: no cipher suites in common
Solution: AWCM was installed AFTER crypto algorithms were disabled in IIS hardening, and it cannot launch normally. Reinstall of AWCM needed.
AWCM SSL Certificate Error
Certificate error while browsing the AWCM status page
-
Login to the AWCM server.
-
Open a command prompt, navigate to the following directory (E:\airwatch\airwatch\AWCM\config) and run the following: keytool -list -v -keystore awcm.keystore
-
Enter the password when prompted
-
Export a new SSL certificate from a machine.
❗️Make sure that the full signing chain is exported (settings that you select when exporting the certificate) and that the password used to export is same as the one used for the current awcm.keystore.
If the passwords are not same, the import happens but an error message appears when AWCM starts and the status page does not load (as the pre-configured password will be incorrect and the AWCM app will not be able to open the keystore).
- AWCM_BRIDGE_FILE_TRANSFER_TIMEOUT_IN_MINUTES : 15 2013-09-23 10:46:22,036
- Error (main) [com.airwatch.awcm.ssl.AWCMSSLContext]:
java.security.UnrecoverableKeyException: Cannot recover key java.security.UnrecoverableKeyException: Cannot recover key
<...>
2013-09-23 10:46:22,036 ERROR (main) [com.airwatch.awcm.server.AWCMServer] - Error initializing server environment, exiting
- When the certificate is on the AWCM server (copy into the C:\airwatch\airwatch\AWCM\config directory), run the following command to replace SSL certificate:
keytool -importkeystore -srckeystore <new-pfx-cert-name>.pfx -srcstoretype pkcs12 -destkeystore awcm.keystore.new -deststoretype JKS
-
Once this has completed successfully, you will now see a new file named awcm.keystore.new in the config directory. Stop the AWCM service.
-
Rename the awcm.keystore to awcm.keystore.old.
-
Rename the awcm.keystore.new to awcm.keystore.
-
Start the AWCM service.
-
Using a valid AWCM URL, try to access the page (https://{url}:2001/awcm/status) and if the status page loads, then check the certificate details. It should display the values for the newly uploaded certificate.
- If the status page does not load, check the log files.
- If rollback is required, rename the awcm.keystore to awcm.keystore.new.
- Then rename awcm.keystore.old to awcm.keystore. Restart AWCM to restore the old settings.
AWCM and Admin Console trust error
ESC/ACC starts and generates no errors in log, also no errors in AWCM. But error in console while performing Test connection for ESC/ACC: Undefined Error; Please check server logs.
Reason: there is no trust between AWCM and AirWatch Admin Console
**Remedy:
**Import Intermediate and Root certificates for public PFX certificate in AWCM server and AirWatch Admin Console Server
ACC Errors
ACC must be able to reach AWCM:
- Protocol HTTPS
- Telnet from ACC to AWCM Server on the relevant port (usually 2001 for On-Premise installations and 443 for SaaS environments)
- Also, verify by opening a browser on the ACC server, entering https://:2001/awcm/status and /awcm/statistics to ensure there is no certificate trust error.
- For On-Premise installations: if using ACC with AWCM and there are multiple AWCM servers and they required to be load-balanced them, persistence needs to be configured - see the above section.
ACC must be able to reach the Console Server:
- Protocol: HTTP or HTTPS
- Telnet from ACC to the Console URL on the relevant port (usually 80 or 443)
- Also, verify by opening a browser on the ACC server, entering http(s)://
- If auto-update is enabled, ACC must be able to query AirWatch Console for updates
ACC must be able to reach the API
- Protocol: HTTPS, port TCP443
- Verify by navigating to the URL of your API server on the ACC server: https:///API/help
- When the credentials screen appears, enter the credentials of a console admin and the API Developer page should appear.
- ACC to API access is required for the proper functioning of the AirWatch Diagnostics service.
Connectivity Errors
- If you see errors in the ACC logs indicating connections being closed/aborted/terminated, check if there is any network device in between the ACC and AWCM that would close or terminate idle connections. The outbound connection required for use by ACC must remain open at all times. Check the TCP session timeout on this network device in between and see if this can be increased to a value >2 minutes;
- ACC sends what is known as an IDLE message, by default every 2 minutes. This IDLE message by ACC helps ACC register itself as a listener on AWCM so that AWCM knows that this ACC is ready to take requests;
- If there are any network devices between ACC and AWCM that closes the connection between these components deeming the connection as an idle connection, it could cause issues with this ACC/AWCM connectivity.
401 Errors on ACC
ACC service does not start:
Error log contains error:
System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.
<...>
Reason: ESC/ACC service does not start because there is no trust between ESC/ACC and AWCM.
If this error is present after trying to hit Update/Check URL on the console, check the SSL certificate on the console and do the following:
keytool -list -v -keystore "{AWCM install path}/awcm.truststore" > c:\test.txt
In the .txt file, search for the secure channel and it should match with the secure channel certificate in the console.
Remedy:
- Generate new certificates for ESC/ACC and download the installer. Then, uninstall ACC and install the new ACC with the renewed certificates. Restart the AWCM service, if required.
- Reinstall AWCM and download the installer from the console.
Subsections of Integration - EMail
Boxer and SCL Restrictions
What needs to be checked to restrict transfer of mail attachments and files transmission between Boxer and SCL for iOS and Android:
1. To restrict transfer of documents in specific controlled apps, on Organization Group level:
a. Allow DLP – SCL-step2 functionality
b. Search for and add Boxer for iOS/Android – SCL-step3
- Next step for devices in Organization Group:
a. Create a new profile for all devices of a group – SLC-step4.
b. Turn off “Allow documents from managed sources in unmanaged destinations” – SCL-step5
1. To restrict transfer of files from SCL into Boxer the file share, connected to SCL, has to be configured in the Security tab - SCL-step6:
a. Allow Open in Email = OFF
b. Allow Opent in Third Party Apps = ON
2. Check that in the security profile properties, Enable Composing Email = No (SCL-step7).
Boxer and SSL for PoC
Boxer has 2 types of accounts:
- main – is being added on install, is usually configured with policies. Cannot be deleted.
- additional – is added by users if this is allowed by policies. Can be configured manually.
Warning
(15.08.2017) For policies which are being distributed centrally for the main account there is no possibility to configure “Ignore SSL errors”. If in a PoC of Boxer you have to connect to a private/test mail server, this may pose a problem.
Variant 1 (recommended): server is signed by a cert, which was given by a inner CA (issued by =/= issued to)
Solution: on all devices in the test, you have to manually deploy the certificate chain in the trusted section:
iOS: http://longwhiteclouds.com/2013/01/03/installing-corporate-ca-certificates-on-iphone-or-ipad-for-use-with-vmware-view/
Android: https://support.google.com/nexus/answer/2844832?hl=en
Variant 2: server is signed by a self-signed cert (issued by == issued to)
Solution: use the first account to distribute policies from any public EMail. For example, Office365 from our VMTestDrive
Note
Account can be only one for all devices - this account will not be used
- After Boxer installation connect to the account provided by policies (VMTestDrive)
- Choose Add Account (“Добавить аккаунт”) in Boxer left menu (IMG_1455.jpg)
- Insert an address (IMG_1456.jpg) and later choose Manual configuration (“Ручная настройка”). Type -> Exchange Server.
- See config in IGM_1457.jpg ang IGM_1458.jpg as an example of such a connection. It is important to choose SSL (Accept any certificates / “принимать любые сертификаты”).
Warning
After adding the account first time, Inbox may sync for a long time (>30min)
- Choose the default account (IMG_1459.jpg)
Boxer Copy-Paste
Boxer Copy&Paste restrictions in two ways
Unlike all other WS1 SDK-enabled Apps, Boxer has two different approaches to restrict Copy&Paste:
At the Assignment stage in App Policies, there is a Copy Paste setting. As a result, copy&paste functions will be denied in ANY directions
❗️Unrestricted personal mail accounts in Boxer still can be troublmakers in this case. Recommend to disable it
If you need to do more granular restriction you need to implement this on SDK profile
- Recommended to set Native Boxer DLP capability to Unrestricted. It can be Restricted for potential more secure way, but SDK settings must be enabled after this settings
- The Boxer App must be published with SDK-profile enable. We use the Default profile, but it should work with Custom SDK-profile as well
- Before actual install Apps on devices, you need setup SDK-profile: Authentication Type =/= Disable; SSO must be enabled
- In DLP section (Security Policies) you may enable Copy&Paste Into to get user possibility to copy from unmanaged messengers/notes/etc into Boxer emails
As a result, copy&paste functions will be denied only in the desired way: into or out from managed Apps
Boxer for Android Fingerprint
- VMware Boxer v 4.5+ for Android
- AirWatch Console 9.0.5+
Following are the steps for fingerprint authentication on the Android Boxer app:
- In the Apps SDK settings (Groups & Settings > All Settings > Apps > Settings and Polices > Security Polices), enable the Biometric Mode.
- While deploying the Boxer app, enable the Application to use AirWatch SDK and Select the Global SDK for Android.
- In the Email settings, enable the Application Configuration and enter “AppForceActivateSSO” (without the quotes) under Configuration Key and Value Type as Boolean and Configuration Type as True.
- Make sure passcode is set as None
- Push the boxer app to the device and download it from the Play Store
ENS Basic Troubleshooting
Warning
Database for ENSv2 must be named “ENS”.
Network
ENSLINKAddress On-Premise, should point to the externally accessible hostname pointed to ENS service. A support ticket has to be made with VMware Airwatch to request API token (internally the support reaches out to Boxer product manager and requests API key).
MS Exchange Server
User agent is configured for ENSv2 on MS Exchange Server CAS role. User agent must have access to receive data from MS Exchange, or ENS will not be able to receive PUSH notifications.
Troubleshooting
ENSLINKAddress for On-Premise installation should point it correctly to the customer’s externally accessible hostname pointed to ENS service.
Autodiscovery errors showing on logs.
- Make sure that the EWSUrl key is configured in the console with a correct value for the EWS url for their exchange environments.
Test it out by opening it on the browser and making sure you are prompted for credentials.
Tip
Alternatively they can just turn ON autodiscovery on their environment.
Authentication errors (401s)
- Check what type of authentication is enabled in EWS? Make sure it has parity with whatever they are using for ActiveSync (Basic, OAuth, CBA) as Boxer will be the one to pass whatever type of credentials it has to ENS to use against EWS.
- Verify the EWSUrl is correct and resolves to Exchange environment and test the credentials used by navigating to the EWS URL and testing them there.
- Run “Outlook Connectivity Tests” on https://testconnectivity.microsoft.com
Tip
Try the Exchange Server or Office 365 tabs accordingly
ENSv2 not sending notifications but no logs found on help portal.
-
Verify the ENSAPIToken key is correct in the console configuration.
-
Verify the ENSLinkAddress key is correct in the console configuration
Tip
Try appending the “alive” endpoint for the environment and make sure it responds.
-
Verify the EWSUrl is correctly configured with a valid EWS value.
-
Verify that ENSv2 servers have inbound access to their EWS environment (firewall may be blocking access, they need to open the corresponding IPs)
-
Verify that EWS can send outbound traffic to the corresponding ENSv2 domain (https://.getboxer.com/api/ens)
General FAQ
- **How are credentials or authentications tokens handled?
**
- Although the client does share the credentials/tokens with the ENSv2 environment upon registration, they are not kept (saved anywhere) by AirWatch servers. Rather, the Exchange server passes them back to AirWatch, encrypted, as part of a notification it sends whenever a new email is available. From that notification (Exchange -> ENSv2), ENS decrypts the credentials and uses them to make any requests necessary to the Exchange server. After performing any necessary requests, the credentials are once again discarded.
Warning
Bug in MS Exchange detected: Exchange returns the email information even though the user is not the owner. This results in the notification payload being created for the wrong user and ultimately another user seeing the notification.
With ENSv2 1.2+, a new service object is created for each EWS request. This will prevent the application from making a request to the EWS endpoint with different credentials.
Patch for Exchange needed from Microsoft on this, since this is unexpected Exchange behavior.
- If credentials are not persisted, is there any data persistent at all by ENS? How is it secured?
- There is a secure database that keeps a list of devices and a list of public private key pairs used to unencrypt the credentials when they come from Exchange;
- Logs are also kept to aide in debugging issues and monitoring the system. These don’t contain any customer’s private information and access to them is also tightly secured via account permissions.
- What data is transmitted through the ENS server without being persisted? How is it secured?
- User credentials (encrypted with RSA encryption)
- Email subject and sender (sent via HTTPS)
- All communication is done via HTTPS
- What additional cloud services does ENS depend on?
- AWS Simple Notification Service (SNS) for push notification handling.
- Apple Push Notification Service (APNS) as it is the only way to pass notifications to Apple devices.
- AWS Relational database service (RDS) for data persistence.
- What is the user agent used by ENSv2 when sending requests to Exchange?
- MailNotificationService/v2 (ExchangeServicesClient/15.00.0913.015+ (will change as new libraries from Microsoft are released)
- What email folders does ENSv2 monitor for incoming messages and actions?
- Currently, ENSv2 only monitors each user’s Inbox folder.
Load Balancing ENSv2
For HA, it is recommended to load balance several ENS web servers as needed following the Hardware Requirements. All web servers should point to the same database server as this will be their shared source of state for each of the clients.
Since the ENS web application itself is stateless there are no requirements to configure any session handling (stickiness) in the loadbalancer so a straightforward configuration should suffice.
Integration - Microsoft IRM-RMS
RMS features in Boxer
According to Boxer User Guide for iOS 4.5.1, Boxer User Guide for Android 4.5.0
Main features
- Edit
- Reply
- Reply All
- Forward
- Copy-Paste
- Modify recipients
- Extract
- Print
- Export
- Content Expiry Date
Other Features
- Press and hold an email message to copy and paste it into the application.
- You cannot copy data from the Boxer application and paste anywhere outside the application. However, you can copy data from outside the application and paste into the Boxer application.
- If your email message has contact number details, tap hold on the number to immediately dial it.
- If restricted by your administrator, attachments may open through the VMware Content Locker and other AirWatch approved apps. Hyperlinks may open only through the VMware Browser.
- If configured by your administrator, you can preview emails and their attachments within Boxer (See Boxer supported files’ types).
- On the attachment preview screen, the Share icon will be unavailable. When tapped on Share icon, you are presented with a toast message “Disabled by your admin”.
- After performing an action on an email while viewing it, you can have Boxer either advance to the next message, the previous message, or return to the conversation list. This setting can be configured from Mail settings (navigate to Settings > Mail > More mail settings > Auto Advance).
RMS Attachments
Boxer does not open RMS-restricted attachments - it transmits them to Content Locker. To use Content Locker on iOS device, the following has to be done:
-
Root certificate must be placed on device
-
In new iOS version the root Trust has to be ACTIVATED in a special menu option
-
In order for Content Locker to access RMS attachments, it must be registered on the ADFS server with this command:
Add-AdfsClient -Name "<App name>" -ClientId "<ID name>" -RedirectUri "<RedirectUri>"
Example:
Client ID for VMware Content Locker for iOS is e9fcfce0-a20b-4d34-b580-909332723090
Tip
Client ID of application can be found in ADFS logs: every error Content Locker gives while trying to read a RMS-secured attachment is followed by its’ current Client ID.
SEG on Windows Proxy
External Link
SEG Guide - VMware AirWatch SEG Guide
Prevent usage of Native EMail on enrolled devices
Problem: Users can gain access to Exchange ActiveSync from uncontrolled devices and mail clients on them. Usage of SEG solves the problem of uncontrolled devices access.
You can enforce using Boxer/Inbox by creating an email compliance policy from the AirWatch console:
Email> Compliance Polices > General Email Policies > Mail Client
SEG as MS Exchange OWA Proxy
Warning
Article is for OLD separate SEG. NOT about SEG on UAG.
You can restrict mobile traffic to seg.company.com by installing IP and Domain restrictions on the IIS on the Exchange server, and then enable IP filtering to deny everyone but the SEG on the ActiveSync endpoint on IIS. This will ensure all enrolled mobile devices will access email through SEG. You can also implement email policies to ensure that unmanaged devices do not access the SEG.
AirWatch cannot block access to OWA for unenrolled mobile devices since SEG does not manage OWA. The only way to do so would be checking through the AD for unenrolled users and preventing them from webmail access from there.
Note
OWA traffic can be routed through the SEG however it will act as a simple pass through.
Warning
The OWA through SEG & proxying Webmail through SEG is not a supported setup as it could lead to a single point of failure for email access.
SEG Java Keystore
Warning
Article is for OLD separate SEG. NOT about SEG on UAG.
Default password for SEG Java Key store = changeit
SEG on Windows Java Memory
Warning
Article is for OLD separate SEG. NOT about SEG on UAG.
Zulu is the new Java Corporate middleware.
Resolution
- Upgrade to latest version of SEG (2.18+);
- Set the max heap size to 5Gb;
- Use Shenandoah as the garbage collection method.
Follow these steps to apply the settings:
- Stop SEG service;
- Go to SEG install directory and edit file SecureEmailGateway-2.18/service/conf/segServiceWrapper.conf
- Update max heap to 5Gb, look for “Xmx” and update the property to:
wrapper.java.additional.3=-Xmx5120m
- Use Shenandoah GC, look for “#wrapper.java.additional.38” and in the next line add:
wrapper.java.additional.39=-XX:+UseShenandoahGC
wrapper.java.additional.40=-Xlog:gc=debug:file=tmp/gc-%p-%t.log:time,level,tags:filecount=10,filesize=50m*
- Save file and start the SEG service.
- Observe the system resources once this change is placed + enable GC logs in the above settings.
If no issue is seen, then remove the GC logging setting by following these steps:
- Stop SEG service.
- Go to SEG install directory and edit file SecureEmailGateway-2.18/service/conf/segServiceWrapper.conf
- Remove line:
wrapper.java.additional.40=-Xlog:gc=debug:file=tmp/gc-%p-%t.log:time,level,tags:filecount=10,filesize=50m*
- Save and start SEG service.
SEG Clustering
If multiple SEG servers are load balanced, single policy broadcast messages apply to only one SEG. This includes the messages sent from the AirWatch Console to SEG upon enrollment, compliance violation, or correction. Use Delta Sync with a refresh interval of ten minutes to facilitate newly enrolled or compliant devices. These devices experience a waiting period of maximum ten minutes before email begins to sync. Benefits of this approach include:
- Updated policies from the same API source for all SEG servers.
- Smaller performance impact on API server.
- Reduced implementation or maintenance complexity compared to the SEG clustering model.
- Fewer failure points as each SEG is responsible for its own policy sets.
- Improved user experience.
SEG Clustering is also available to facilitate the sharing of single policy updates to all nodes of a SEG cluster.
SEG TLSv1
Please go through the following instructions in order to enable TLSv1.0 on SEG V2:
- Go to SEG installation directory -> {SEG_DIRECTORTY}/service/conf
- Edit file conf
- Look for following properties:
Property 1
wrapper.java.additional.9=-Djdk.tls.disabledAlgorithms=MD5\, RC4\, TLSv1\, SSLv2Hello\, SSLv3\, DSA\, DESede\, DES\, 3DES\, DES40_CBC\, RC4_40\, MD5withRSA\, DH\, 3DES_EDE_CBC\, DHE\, DH keySize < 1024\, EC keySize < 224
set this property as: (Removing TLSv1 from the disabled list):
wrapper.java.additional.9=-Djdk.tls.disabledAlgorithms=MD5\, RC4\, SSLv2Hello\, SSLv3\, DSA\, DESede\, DES\, 3DES\, DES40_CBC\, RC4_40\, MD5withRSA\, DH\, 3DES_EDE_CBC\, DHE\, DH keySize < 1024\, EC keySize < 224
Property 2
wrapper.java.additional.12=-Dhttps.protocols=TLSv1.1\,TLSv1.2
set this property as:
wrapper.java.additional.12=-Dhttps.protocols= TLSv1\,TLSv1.1\,TLSv1.2
Subsections of Internal Apps
Apps Deployment
AirWatch has 2,5 mechanisms to deploy apps:
- Apps & Books → Internal. You can only deploy 3 versions of an application called “Alpha”, “Beta”, “Production”;
- Devices → Staging & Provisioning →****Product List View → Add Product. Add Manifest → Install Application. Difference between this method and the one below is not clear, except slightly easier config;
- Devices → Staging & Provisioning → Components → Files/Actions. Add Files/Actions → Add Manifest → Install Unmanaged Application.
I recommend the 3rd method as the most descriptive and stable if several versions of an application are needed in one Organization Group.
Things to check in application deployment
Application ID
Check if there is a modification of the Application ID while uploading the application in the Console via Staging & Provisioning → Components → Applications or check while uploading the Application, is it fetching the Application ID and populating the same name in the Application ID field.
Once the app gets uploaded, AirWatch SQL writes down its’ metadata to SQL. Parsing the data:
SELECT a.name, a.versionhash, * FROM INTERROGATOR.APPLICATIONLIST AL
JOIN interrogator.Application A ON AL.ApplicationID=A.ApplicationID
WHERE DEVICEID in (1473) and isinstalled=1
See more on SQL querying apps
Warning
On AirWatch 9.2.3 version and lower, metadata provided by some build systems like Gradle 3 is incorrectly processed. Specifically a VersionHash is needed in the correct place for the app to see it.
Application Deployment Logs
- Device receives command to install application
- Device is directed to app destination for download via Manifest.plist file - so search for “manifest” in the logs:
- Device is redirected to Manifest.plist URL
- Device locates Blobhandler.pblob URL:
- Device is redirected Blobhandler.pblob URL
- Application download begins
Tip
You can copy the blob link (see picture) manually in the browser to check that download starts.
macOS Apps Deployment
External link:
You are able to deploy apps on Mac OSX devices either from Apps & Books or from Product & Provisioning method. If you have a pkg or dmg file, then you could simply use Apps & Books. However, if you have multiple files to be executed or scripts to run then you could use Product & Provisioning. In AirWatch 9.3+, all macOS application file types (.dmg, .pkg, .mpkg, .app) can be managed through the Internal Applications section. This new framework leverages the popular Mac Admin community tool, Munki.
Provisioning with Munki
Enabling Software Management
- Navigate to Settings > Devices & Users > Apple > Apple macOS > Software Management
- Enable Software Management , there is a check in place to verify if File Storage is enabled at this page. If there is no File Storage, the admin will be requested to enable File Storage.
- On-Prem deployments will need to enable CDN.
Before uploading macOS File to AirWatch
All primary macOS software file types will now be uploaded through Internal Applications (.pkg, .dmg, .mpkg). A .pkg file can be a Bootstrap Package or can be managed through full lifecycle management (this feature). In order to configure Advanced Management options for macOS software and for effective desired state management, which is achieved through the integrated Open-Source Munki library in the AirWatch Agent, a metadata file must be generated for the file separately before uploading it to the AirWatch Console. Munki’s deployment logic is built on the concept of pkginfo files, which are xml/plist files that contain metadata, configuration information, and more for a given application file. AirWatch requires this pkginfo file along with the application file to manage the deployment in the Console.
The pkginfo file is genetrated with VMware AirWatch Admin Assistant (see tools page for download). This is a GUI wrapper for a Munki command-line utility and is used to generate the pkginfo metadata file for a given application file.
-
Download and install the Admin Assistant tool to a macOS device or VM.
-
Open the Assistant. The Assistant dialog will ask for the application installer files to parse.
-
Upload an application installer file by dragging & dropping a .pkg, .dmg, .mpkg, .app file into the labeled field, or browse the local files on the machine in order to find an installer file.
Note
If the file is .app, it will be converted into a .dmg for deployment
-
Once the file is selected and uploaded, the Assistant will automatically start parsing process the process. Additional files can be added during this time if needed.
-
Once the parsing is complete, the tool will prompt to reveal the parsed metadata files in Finder. Store the metadata files in a local folder where they can be easily retrieved during the Software Distribution procedure.
There are multiple ways to obtain the metadata/pkginfo file aside from using the Admin Assistant. One of them is to use Autopkg and its’ GUI tool Autopkgr.
Uploading a macOS Application to the AW Console:
- Navigate to Apps & Books > Applications > Native > Internal
- Click Add Application
- Upload app/software file (.pkg, .mpkg, .dmg)
- Continue to the next screen and Upload the pkginfo .plist file
- Continue to customize the app deployment. The proceeding screens will display any available configurations present in the pkginfo file (for example, a pkginfo from AutoPkg may contain an install_check script). Additional deployment configurations can be defined, and will be merged with the existing configurations
- If the pkginfo file has one or more key and configuration that are not exposed in the UI, they will not be affected. This feature is important as it allows administrators to upload pkginfo files with keys that would be supported by the Munki client but would not be configurable in the UI. Any changes in the UI will be merged with the existing keys.
Pre/Post Install Scripts
A common reason to repackage software is to perform additional configuration tasks or to install additional items. A technique to avoid repackaging is to add pre-install scripts and/or post-install scripts to the configuration of an item. These scripts can take care of some of the tasks which previously may have required repackaging to implement.
An Exit Code of 0 will define the script as successfully executed in order to allow Munki to continue.
Note
Failure of the pre-install script will abort the installation attempt. Failure of the post-install script will log errors, but the installation will be considered complete.
Note
SCRIPT LOG
echo
statements will be logged to /Library/Application Support/AirWatch/Data/Munki/Managed Installs/Logs/ManagedSoftwareUpdate.log
Uninstall Methods
There are multiple options available for uninstallation of software. The appropriate option will be selected by default by the VMware Admin Assistant tool based on the software’s file type. If needed options to override the default values are available in the AirWatch Console.
Remove Copied Items
Used primarily for software installed from a .dmg
- Pulls from items_to_copy array[dicts] in the pkginfo file
- All file paths in this array will be deleted
- Future Console release will show the paths in the items_to_copy array in the UI
Remove App
- Pulls from installs array[dicts] in the pkginfo file
- All file paths in this array will be deleted
- Future Console release will show the paths in the installs array in the UI
Remove Packages
Used primarily for software installed from a .pkg
- Uses receipts and analyzes the packages to remove
- Tries to determine what files were installed via the Bom file
- Deletes receipt
- Will only remove if package is not associated with any other files or programs
- Future Console releases will show the receipts that Munki check for in the UI
Uninstall Script
Can be used for any installer type
- Written in shell script
- Used to perform custom uninstall operations if needed
- If the Admin has a customized deployment of an app, they will need to also write a corresponding uninstall script to remove their custom configurations
Install or Uninstall Verification
With some software, the Admin needs to configure what exactly defines a Successful Install or Uninstall. Munki allows this through setting an Install or Uninstall Check Script
Install Check Script - If present, this script is executed to determine if an item needs to be installed. A return code of 0 means install is needed; any other return code causes install to be skipped.
Uninstall Check Script - If present, this script is executed to determine if an item needs to be uninstalled. A return code of 0 means uninstall is needed; any other return code causes uninstall to be skipped.
Conditions
Conditions can defined on a per-application level so that they are evaluated prior to the download and installation of a software.
Warning
Custom conditions will be created in a later version of AirWatch.
Built-in Conditions
Currently available built-in attributes for conditional comparison:
Attribute | Type | Description | Example Comparison |
hostname | string | Hostname | hostname == "Lobby iMac" |
arch | string | Processor architecture. e.g. 'powerpc', 'i386', 'x86_64'. | arch == "x86_64" |
os_vers | string | Full OS Version e.g. "10.7.2" | os_vers BEGINSWITH "10.7" |
os_vers_major | integer | Major OS Version e.g. '10' | os_vers_major == 10 |
os_vers_minor | integer | Minor OS Version e.g. '7' | os_vers_minor == 7 |
os_vers_patch | integer | Point release version e.g. '2' | os_vers_patch >= 2 |
machine_model | string | 'Macmini1,1', 'iMac4,1', 'MacBookPro8,2' | machine_model == "iMac4,1" |
machine_type | string | 'laptop' or 'desktop' | machine_type == "laptop" |
ipv4_address | array of strings | This contains current IPv4 addresses for all interfaces. | ANY ipv4_address CONTAINS '192.168.161.' |
munki_version | string | Full version of the installed munkitools | munki_version LIKE '* 0.8.3* ' |
serial_number | string | Machine serial number | serial_number == "W9999999U2P" |
date | UTC date string | Date and time. Note the special syntax required to cast a string into an NSDate object. | date > CAST("2013-01-02T00:00:00Z", "NSDate") |
Example:
machine_type == “laptop” AND os_vers BEGINSWITH “10.7”
date > CAST(“2016-03-02T00:00:00Z”, “NSDate”)
Dates in conditions:
The date string must be written in UTC format, this format is interpreted as a local date/time. The condition date > CAST("2013-01-02T00:00:00Z", "NSDate")
is True
if the local time is after midnight local time on 02 Jan 2013.
Literal types in comparisons
-
Strings are delimited by either single or double-quotes: os_vers BEGINSWITH "10.7"
-
Integers have no quotes: os_vers_major == 10
-
Booleans are indicated as TRUE or FALSE (and have no quotes, or they’d be strings!): some_custom_condition == TRUE
-
Dates are possible, but they must be cast from ISO 8601 strings: date > CAST("2013-01-02T00:00:00Z", "NSDate")
Updates
Updates can be managed similarly to the other platforms in the AirWatch Console. If a new version of the file needs to be added, perform the following:
- Navigate to Apps & Books > Native
- Click on the App that requires an update. Clicking the app will navigate to the Details View page.
- In the top right side, click “Add Version”
- Upload the new installer for the new app version
- Upload the new pkginfo file for the new version
- Make any additional changes and then save the configuration
Troubleshooting
The AirWatch Console will show report macOS app installation data from a device in several locations:
- Apps & Books > Applications > Native > Internal. Click onto the Application to drill into Application Details > Devices Tab. The grid in this tab will display installation statuses for each device.
- Devices & Users > Devices > List View. Click on a device to drill into Device Details > Troubleshooting Tab. The grid on this tab will show activity on the device and provides filtering options to show information relating to Software Distribution.
Munki Logs can also be directly accessed on the device:
/Library/Application\ Support/AirWatch/Data/Munki/Managed\ Installs/Logs/ManagedSoftwareUpdate.log
tail -n 20 -F /Library/Application\ Support/AirWatch/Data/Munki/Managed\ Installs/Logs/ManagedSoftwareUpdate.log
Munki Known Issues
- Once software has been published and installed on a few devices, if any of the configuration options such as “install script”, “uninstall script” etc.. are changed, devices which already have the application installed successfully will not receive the updated options as these can only be updated when an “InstallApplication” command. This is the only command that can update the locally cached PKGInfo files on the device. In order to update the scripts, administrators will have to manually select all of the devices which have the software installed and repush the command. Repushing the command will update the respective cached PKGinfo file with the latest information.
Note: Repushing the command will NOT reinstall the application unless there is a change in the software version. Munki is has mechanisms in place to not reinstall software when the application is already installed.
- If a package has a dependency on another package or software then an administrator will have to manually provide a required key in the pkginfo of the package with a dependency. Providing the key in the pkginfo is the only way for Munki to recognize if there are any dependencies, and determine the sequence of installation steps for such dependent packages. Unless otherwise specified, packages are installed in random order. Such package installation will fail if a package dependency is not met during the installation.
Example: Netbeans requires that Java be present and installed on the machine. Netbeans will not implicitly install Java while installing the Netbeans software package. Even if administrators attempt to install Netbeans without utilizing M****unki or using standalone Munki, the installation will fail and will prompt the administrator to first install Java before installing Netbeans.
- For packages which have the same receipts array with same version even on an upgraded package, Munki will not be able to detect that a new package is an upgraded package. The reason is that the receipt array has the same version as the previous package. The impact of duplicated receipts arrays is that the packages will not be installed on User machines as Munki cannot determine that a new version needs to be installed.
Note: The best solution for packages with duplicate versions in the Receipts Array is to use an Autopkg Recipe, which adds an installs array in the pkginfo. This method allows Munki to detect upgraded packages; alternatively an administrator can manually add an installs array into the pkginfo.
Example: Wireshark is an example of a package which has identical receipts which contain the same version, thus Munki cannot detect the when upgraded packages are deployed.
Legacy methods (AirWatch 9.2 & older)
Apps & Books
- Go to Groups & Settings > All Settings > Devices and Users > Apple > MAC OS > Software Distribution.
- Enable Software distribution.
- Now go to Apps and Books > Native > Internal > Add Application.
- Upload the .pkg or .dmg file.
- Once uploaded, click on Continue. You will get an option to download VMware AirWatch Admin Assistance tool.
- Download the tool.
- Upload the .pkg or .dmg file there.
- You will see that it will open the package into .dmg file, .plist file and .png file.
- Go back to the Console and upload the .plist file for metadata.
- Click on upload and you will see the application description.
- You will get an option to upload the image. Upload .png file there.
- Now assign the application to the required Smart Group.
- Click on Save and Publish.
Product provisioning
It allows you to create, through AirWatch, products containing profiles, applications, and files/actions (depending on the platform you use). These products follow a set of rules, schedules, and dependencies as guidelines for ensuring your devices remain up to date with the content they need.
- Navigate to Devices > Staging & Provisioning > Components > Files/Actions
- Click on Add > General
- Navigate to Files > Add Files and enter the specific path
- Navigate to Manifest. Under Install Manifest, click Add Action. Choose the action type Install and provide the specific path.
- Under Uninstall Manifest, click Add Action and set the action type as Install, again providing the specific path.
- Under Uninstall Manifest, click on Add Action and choose the action type Uninstall, and provide the exact application name with the extension as ‘.app’.
- Select Save
- Navigate to Staging & Provisioning > Product List View and select Add Product.
- Under General, verify the details and assignment group are filled in.
- Navigate to the Manifest tab and click on Add. Choose the created component after selecting Install/Action.
With this configuration, the application will first be downloaded to the Downloads folder on the Mac. You can then install the application, which will be available in the Applications folder. If you want only to uninstall the application which was previously installed, you may choose the Delete Files option while creating files and actions.
Recipes
Script recipies
TextWrangler Post Install Script
Implement a post-install script that copies the command-line tools from the TextWrangler bundle to their intended locations.
After uploading the appropriate files to AirWatch, pre/post install scripts can be configured for the app in the Scripts tab of the Application Details. Simply paste the script into the appropriate field and AirWatch will format it to be used by Munki.
App recipes
Adobe Reader DC...
Adobe Reader DC
- Files:
- Upload the Acrobat DC Installer.pkg file and set to download to /tmp/Acrobat DC Installer.pkg
- Install Manifest:
- Action Type - Install: /tmp/Acrobat DC Installer.pkg
- Uninstall Manifest:
- Action Type - Run: rm -rf /Applications/Adobe\ Acrobat\ Reader\ DC.app/
- Action Type - Run: rm -rf /Library/Application\ Support/Adobe/Reader/
- Action Type - Run: rm -rf /Library/Application\ Support/Internet\ Plug-Ins/AdobePDFViewer.plugin/
- Action Type - Run: rm -rf /Library/Application\ Support/Internet\ Plug-Ins/AdobePDFViewerNPAPI.plugin/
Apple Enterprise Connect...
Apple Enterprise Connect
- Files:
- Upload the Enterprise Connect 1.6.3.pkg file and set to download to /tmp/Enterprise Connect 1.6.3.pkg
- Install Manifest:
- Action Type - Install: /tmp/Enterprise Connect 1.6.3.pkg
- Uninstall Manifest:
- Action Type - Uninstall: /Applications/Enterprise Connect.app
McAfee Endpoint Protection...
McAfee Endpoint Protection
- Files:
- Upload the install.sh and set it to download to /tmp/install.sh
- Install Manifest:
- Run Command: chmod +x /tmp/install.sh
- Run Command: sudo /tmp/install.sh -i > /dev/null 2>&1
- Uninstall Manifest:
- Action Type - Run: sudo /Library/McAfee/cma/scripts/uninstall.sh
Microsoft Skype For Business...
Microsoft Skype For Business
- Files:
- Upload the SkypeForBusinessInstaller-16.1.0.456.pkg file and set to download to /tmp/SkypeForBusinessInstaller-16.1.0.456.pkg
- Install Manifest:
- Action Type - Install: /tmp/SkypeForBusinessInstaller-16.1.0.456.pkg
Palo Alto GlobalProtect...
Palo Alto GlobalProtect
- Files:
- Upload the GlobalProtect-3.1.3.pkg and set it to download to /tmp/GlobalProtect-3.1.3.pkg
- Install Manifest:
- Install: /tmp/GlobalProtect-3.1.3.pkg
- Uninstall Manifest:
- Action Type - Run: sudo /bin/bash /Applications/GlobalProtect.app/Contents/Resources/uninstall_gp.sh
RSA SecureID...
RSA SecureID
- Files:
- Upload the RSASecurIDTokenAutoMac412x64.pkg and set it to download to /tmp/RSASecurIDTokenAutoMac412x64.pkg
- Install Manifest:
- Install: /tmp/RSASecurIDTokenAutoMac412x64.pkg
- Uninstall Manifest:
- Action Type - Run: sudo /usr/bin/python /Library/Application\ Support/SecurID/uninstall-rsasecurid.py &>/dev/null
Sophos Antivirus...
Sophos Antivirus
- Files:
- Upload the SophosInstall-Mac.zip and set it to download to /tmp/SophosInstall-Mac.zip
- Install Manifest:
- Install: /tmp/SophosInstall-Mac.zip
- Run Command: chmod a+x /tmp/TEMPDIR-SophosInstall-Mac\Sophos\ Installer.app/Contents/MacOS/Sophos\ Installer
- Run Command: chmod a+x /tmp/TEMPDIR-SophosInstall-Mac\Sophos\ Installer.app/Contents/MacOS/tools/com.sophos.bootstrap.helper
- Run Command: /tmp/TEMPDIR-SophosInstall-Mac\Sophos\ Installer.app/Contents/MacOS/Sophos\ Installer –install > /dev/null 2>&1
Symantec Removal Tool...
- Unpack the SymantecRemovalTool.zip.
- Upload the SymantecRemovalTool.command and set it to download to /tmp/SymantecRemovalTool.command
- Action Type - Run: chmod a+x /tmp/SymantecRemovalTool.command
- Action Type - Run: sudo /tmp/SymantecRemovalTool.command > /dev/null 2>&1
Manifest recipies
Quick fix for root user login bug (https://techcrunch.com/2017/11/28/astonishing-os-x-bug-lets-anyone-log-into-a-high-sierra-machine/ )
Enable root user with random password
Below are steps to fix the issue via Product Provisioning -
A. Create a File
- Go to Devices > Staging & Provisioning > Components > Files /Actions.
2. Add Files / Actions > macOS
- Manifest > Install Manifest > Run > Command Line and Arguments to run
/usr/bin/dscl . -passwd /Users/root $(/usr/bin/openssl rand -base64 20)
- Click Save
B. Create a Product to include above File
- Go to Devices > Staging & Provisioning > Products List View > Add Product > macOS
2. Click on Manifest > Add Manifest > Actions to Perform > Install Files / Actions > Files/Actions (Enter the saved file).
- Save & Activate the product
Unsigned application launch bypassing Security & Privacy
Use above recipe with command Manifest > Install Manifest > Run > Command Line and Arguments to run:
sudo xattr -rd com.apple.quarantine /Applications/@cursor # where '/Applications/@cursor' is the target app
Logs
Linked Articles
Collecting AirWatch Services Logs (On-Premise)
AirWatch Enterprise Systems Connector (ESC) / Cloud Connector (ACC)
To verbose the ACC log, perform the following:
- Open Windows Explorer on the ACC server, and browse to the C:\AirWatch\CloudConnector\folder
- Note the presence of two folders: Bank1 and Bank2. Every time the Cloud Connector software is updated, the update is applied to the inactive bank folder. The updated bank folder then becomes the active bank folder.
- Open each Bank folder and sort the file list by date modified. Compare the most recent date modified in each file. The current bank file has the most recent date modified.
- Within the current bank folder (C:\AirWatch\CloudConnector\Bank#), open the CloudConnector.exe.config file and change the level value in the from error to verbose and save the file.
- After reproducing the error, open Windows Explorer on the ACC server and browser to the C:\AirWatch\Logs directory. Copy the appropriate log to a new location for use in support/troubleshooting.
- Be sure to change the loggingConfiguration level value from verbose to error and save the file to prevent unnecessary impact to the ACC server.
AirWatch API Services (API)
To verbose the API Service Log, perform the following:
- On the server running API services, open Windows Explorer and browse to C:\AirWatch\AirWatch #.#\Websites\AirWatchApi. Note: You can determine the API server by browsing to Groups & Settings > All Settings > System > Advanced > Site URL’s.
- Open the web.config file, and look for the loggingConfiguration key.
- Change the value for level from error to verbose and save the web.config file.
- Restart IIS services.
- Reproduce your issue and then copy the log from C:\AirWatch\Logs\AirWatchAPI\webserviceapi.log.
- Change the value for level from verbose back to error and save the web.config file.
- Restart IIS Services.
AirWatch Cloud Messaging (AWCM)
To verbose the AWCM logs, please perform the following steps:
- Open the logback.xml file. The path to access the file:\AirWatch\AirWatch x.x\AWCM\config\logback.xml.
- Search for the following:
- Change the state from error to debug.
- Save the file and restart the AWCM services.
Note
Once the issue is reproduced, return logging level back to info and restart the AWCM services. Or the AWCM disk may overflow with logs.
Folder = AWCM
Log name = Awcm.log
Contains information on AWCM such as status, history, properties, and additional sub-services.
Log name = AWCMservice.log
Contains log information on AWCM Java service wrapper.
ACC Logs
Use these steps below to verbose ACC logs:
-
On the ACC server navigate to *\AirWatch\AirWatch #.#\CloudConnector\Bank#*
-
#.# will be the AirWatch version you are using, if there are multiple choose the most recent.
-
ACC utilizes two distinct banks: one active and one is used for installation of automatic updates. If you are unsure of which bank is active, make changes to the CloudConnector.exe.config file in each bank. If one bank is empty or does not have the file, it is not the active bank.
-
Edit the CloudConnector.exe.config lines:
-
Locate the log by looking at the filePath attribute from the line above. The path is included below as well \AirWatch\Logs\CloudConnector\CloudConnector.log
-
ACC doesn’t need to be restarted to pick up the logging level configuration change.
Console Services (CS)
To enable verbose logging for console and scheduler services, please perform the following steps:
- Log in to the AirWatch console in question.
- With the Global organization group selected, browse to Groups & Settings > All Settings > Admin > Diagnostics > Logging.
- Change the logging level for the services in question to verbose and click Save.
- Admin Console
- Self-Service Portal
- API
- Scheduled Services (such as Inventory, Workflow, and Monitor services)
- Reproduce your error, then open Windows Explorer and browser to C:\AirWatch\Logs_Service Folder_ and look for the latest log.
- Change the Device Services logging level back to Error. This prevents logging from impacting system performance.
Device Services – Targeted
Depending on the version of AirWatch, it is possible to collect verbose logs for an individual device without having to verbose the logs for all devices. This is particularly helpful when troubleshooting a single device in a large production deployment. To do this, perform the following steps:
- From within the AirWatch console device list view, click on your device to take you to the device details page.
- Click More > Targeted Logging. If necessary, click the Continue Targeted Logging File Path to ensure the logging path is configured.
- From the targeted logging page, click Create New Log and select the timeframe you want the logs to collect. Click Start. At any point, you can click Stop Logging to stop log collection for the device (such as after you have reproduced the issue).
- Once the tests are completed, go to the appropriate server and look for the TargetedLogging folder.
- Inside the folder is a zip file with the current date and time. Unzip the file to view the files.
Device Services – General
When you wish to verbose device services logging for all devices, perform the following:
- Log in to the AirWatch console in question.
- With the Global organization group selected, browser to Groups & Settings > All Settings > Admin > Diagnostics > Logging.
- Change the Device Services logging level to verbose and click Save.
- Reproduce your error, then open Windows Explorer and browser to C:\AirWatch\Logs\DeviceServices\ and look for the latest log.
- Change the Devices Services logging level back to Error. This prevents logging from impacting system performance.
SEG Console, Setup, and Integration Logs
To verbose the SEG logs for console/setup/integration, please perform the following steps:
- Open Windows Explorer on the SEG server and browser to C:\AirWatch\Logs
- Note the following Folders and change the appropriate config log level from error to verbose:
- Services – Contains the AW.EAS.IntegrationService.log file which details communications between the AirWatch API server and SEG server. Note: This log is verbosed by changing the level value in the key of the AW.ES.IntegrationService.Exe.config file in the C:\AirWatch\AirWatch #.#\AW.Eas.IntegrationService folder.
- SEG Setup – Contains the AW.EAS.Setup.log file which details activity related to the http://localhost/SEGSetup website. Note: This log is verbosed by changing the level value in the key of the web.config file in the C:\AirWatch\AirWatch #.#\AW.Eas.Setup folder.
- SEG Console – Contains the AW.EAS.Web.log file which details activity related to the http://localhost/SEGConsole website. Note: This log is verbosed by changing the level value in the key of the web.config file in the C:\AirWatch\AirWatch #.#\AW.Eas.web folder.
- Before reproducing your issue, making the necessary change to the LoggingConfiguration key for the service in question.
- After reproducing the error, open Windows Explorer on the SEG server and browse to the appropriate subfolder in the C:\AirWatch\Logs\ directory. Copy the appropriate log to a new location for use in support/troubleshooting.
- Be sure to change the loggingConfiguration level value (currently verbose) in the appropriate configuration file back to error to prevent unnecessary impact to the SEG server.
SEG Exchange ActiveSync (EAS) Listener Logs
To verbose the SEG EAS Listener logs, please perform the following steps:
- On the SEG server, open an Internet browser and navigate to http://localhost/SEGSetup
- In the “Log Level” drop down, select verbose and click Save. Note: This changes the level value in the key of the web.config file in the C:\AirWatch\AirWatch #.#\AW.Eas.Web.Listener folder.
- Copy the AW.Eas.Web.Listener.log to a new location for use in support/troubleshooting.
- In your Internet browser, change the “Log Level” drop down back to error and click Save.
FTP Relay Server (Rugged Management)
Logging for the relay server is saved in the following location:
- Browse to the C:\AirWatch\Logs\Service folder.
- The logging for the Relay Server is saved in the ContentDeliveryService.log file.
Collecting logs from the Admin Console
If you do not have immediate access to the on-premise servers to access the logs, you can retrieve SEG/ACC logs directly from the console from the following page:
- Navigate to System > Admin > Diagnostics > System Health. Click on the installed service you wish to pull the logs from.
- In the pop-up box now displayed, click on the “Acquire Logs,” for the required service, from the four-button menu on the right.
- Now the “Download” button is activated and you can click on it to download and view logs remotely.
Note: The System Health dashboard will be populated only if you have any of the services (ACC/SEG) already installed and running.
EMail Notification Service (ENS)
ENSv2 is a Windows Service ‘AWSubscription’. Like other AirWatch services, relevant logs can be found in the path {Installation Path}/Logs, and the logging level can be configured by editing parameters to traceEnabled=“true” Level=“Verbose” in the app config ({Installation Path}\Config\WebSites\Web.config) file located in the installation folder
ENSv2 Errors are in ENS.log and ReSubscriptionMechanism.log
Note
The service must be restarted for logging changes to take effect.
When set to verbose, you will be able to identify log messages pertaining to both new subscriptions being created, as well as any device compliance state-changes being identified. For example, if a device becomes compromised and is then marked as non-compliant. In the logs, a message indicating that a device’s access state is True indicates that the device is allowed, whereas False means the device is blocked.
Subsections of Logs
Boxer for Android Logs
Boxer for Android Architecture
Boxer communicates with a few different systems to provide these services.
The flow of the Android Boxer log file looks like this:
- Boxer reaches out to the console to get the profile information;
- Boxer reaches out to the console to get any S/MIME certificates and client authentication certificates available from the console;
- Boxer reaches out to the Exchange ActiveSync(EAS) email endpoint to make the options, provision, foldersync, and ping request;
- Boxer reaches out to the Email Notification S ervice (ENS) and provides the Exchange Web Services (EWS) credential information that was used for ActiveSync so it can subscrib e to email notifications.
graph TD
CS[Console Server] --> Boxer;
Boxer --> CS;
Exchange --> Boxer;
Boxer --> Exchange;
Boxer --> ENS;
ENS --> Boxer;
- Boxer logs are in GMT. That means the entries there are 5 hours ahead of EST time. In addition, the logs are in 24 hour time so you will need to subtract 12 if you want am/pm.
- The very first line in the Boxer logs has the last date of the logs in the file. For example
2019-01-08T21:28:04.539Z - [-] - ###--HEADER--###
- Boxer logs can persist in the app for up to 3 days, so they can be pulled for review even if the issue hasnt occurred recently, but at least within the 3 day period.
This is useful for intermittent types of issues, or where there is a delay in an affected user reporting the issue to their techsupport.
Boxer Communication Logs
Here is an example of what you might see in the logs followed by notes (in bold) that detail what the log entry tracks:
Boxer log sample...
2019-02-05T13:51:52.636Z - [-] - ###--HEADER--### | This is the time of the last log entry in this log file for easy reference.
-------------------------------------------------------------- DEVICE INFO
--------------------------------------------------------------
DEVICE ID = boxerc59d6cd3d34aba549 | the EAS device identifier
MANUFACTURER = samsung
MODEL = SM-G930F
SDK VERSION = 26
COMPROMISED = false | shows whether or not the device is jail broken
APP VERSION CODE = 758
APP VERSION NAME = 5.3.0.3
STANDALONE = false
ANCHOR APP = com.airwatch.androidagent | shows how the device is enrolled through the AirWatch agent, Boxer, or other method CONSOLE VERSION = 19.3.0.0
PASSCODE ENABLED = false | shows whether or not Boxer will prompt the user for a password when opening Boxer
AUTO-REFRESH FOLDERS = true | shows the sync frequency specified in the profile. "true" will be returned if it's set to "automatic" -- Stats --
EnsRegister_1 => {"total_events_today":1,"average_events_per_hour":1.0}
AttachmentServiceWakeLock => {"total_events_today":4,"average_events_per_hour":4.0} Eas.SSLException => {"total_events_today":30,"average_events_per_hour":3.0}
IrmSync => {"total_events_today":1,"average_events_per_hour":1.0} AccountProtocolCache.Cache_Used => {"total_events_today":103,"average_events_per_hour":103.0} EnsUpdateEnsConfig_-1 => {"total_events_today":1,"average_events_per_hour":1.0}
1/Socket Timeout => {"total_events_today":4,"average_events_per_hour":0.22222222} FetchManagedConfig => {"total_events_today":3,"average_events_per_hour":3.0} Eas.SyncAdapter => {"total_events_today":5,"average_events_per_hour":5.0} EnsGetSubscriptionStatus_1 => {"total_events_today":1,"average_events_per_hour":1.0} -- End Stats --
-- Power Info --
Power saving mode enabled = false | shows whether or no the application is excluded from power saving mode which will cause application and networking issues Ignoring battery optimizations = true | shows whether or not the application is excluded from battery optimizations which will cause application and networking issues Device idle enabled = false
-- End Power Info --
-- Permissions Info -- | shows whether or not the proper permissions are enabled for the application Calendar read: true
Calendar write: true
Contacts read: true
Contacts write: true Call: false
Phone state: false Storage: true Camera: false
-- End Permissions Info --
-------------------------------------------------------------- ACCOUNT SETTINGS
-------------------------------------------------------------- DESCRIPTION = nas98.airwlab.com:8002 DISPLAY_NAME = Legend Wilcox
EMAIL = lwilcox@labmail.airwlab.com
USERNAME = lwilcox@milkyway.local SERVER_ADDRESS = nas98.airwlab.com:8002 SSL_REQUIRED = null
PORT = null
USER_DOMAIN = milkyway
SIGNATURE =
SYNC_EMAIL = true
SYNC_CALENDAR = true
SYNC_CONTACTS = true
SYNC_EMAIL_LOOKBACK = 5
MAX_SYNC_EMAIL_LOOKBACK = 5
SYNC_CALENDAR_LOOKBACK = 7
MAX_SYNC_CAL_LOOKBACK = 7
LOGIN_CERTIFICATE = null
LOGIN_CERTIFICATE_NAME = null
TRUST_ALL_SSL_CERTS = null
SMIME_SIGNING_CERT_ISSUER = 09e219f7-7864-437c-8c26-cd6613375e09
SMIME_ENCRYPTION_CERT_ISSUER = 09e219f7-7864-437c-8c26-cd6613375e09
SMIME_SIGNING_CERT_ISSUER_TOKEN = 3D9B54CEC239582BFD471D2BB1BE4E55580DF302 | this is the SMIME signing certificate thumbprint ID SMIME_ENCRYPTION_CERT_ISSUER_TOKEN = 3D9B54CEC239582BFD471D2BB1BE4E55580DF302 | this is the SMIME encryption cetificate thumbprint ID ACCOUNT_AUTHENTICATION_TYPE = 1 | 0 for basic, 1 for certificate, 2 for both (dual factor), 3 Oauth, 4 Oauth with CBA
ALLOW_MOBILE_FLOWS = false MOBILE_FLOWS_HOST_URL = null MOBILE_FLOWS_VIDM_URL = null SMIME_POLICY_TRUST_STORE = 0 SMIME_POLICY_REVOCATION_STRICTNESS = 0 SMIME_POLICY_STRICT_EKU_CHECK = false TLS_POLICY_TRUST_STORE = 0 DEFAULT_SIGNING_ALGORITHM = null DEFAULT_ENCRYPTION_ALGORITHM = null SMIME_ENCRYPTION_CONFORMING_ALGORITHMS = null SMIME_SIGNING_CONFORMING_ALGORITHMS = null STRICT_TLS_TRUST_CHECK = false
-------------------------------------------------------------- ACCOUNT POLICY
--------------------------------------------------------------
ALLOW_CALLER_ID = true | caller ID restricted = false, caller ID unrestricted = true CALLER_ID_DEFAULT = true
APP_MUST_USE_AIRWATCH_BROWSER = false
APP_ALLOW_CRASH_REPORTING = true
ATTACHMENT_VIEWER_PACKAGE_NAMES = []
APP_PASSCODE_COMPLEXITY = 0 | passcode mode 0 for numeric, 1 for alphanumeric MIN_COMPLEX_CHARS_IN_APP_PASSCODE = 0
MIN_APP_PASSCODE_LENGTH = 4 | how many characters required for the Boxer application password MAX_APP_PASSCODE_FAIL_ATTEMPTS = 10
MAX_AGE_FOR_APP_PASSCODE = 90
MAX_LOOKBACK_FOR_REPEAT_PASSCODE_CHECK = 5
APP_PASSCODE_TIMEOUT = 2
ALLOW_OTHER_ACCOUNTS = true
ALLOW_COPY_PASTE = true
ALLOW_SCREEN_CAPTURE = true
ALLOW_ATTACHMENTS = true
MAX_ATTACHMENT_SIZE = 2147483647
DOCUMENT_SHARING_RESTRICTION = 0
ALLOW_CALENDAR_WIDGET = false
SMIME_POLICY_ALLOWED_KEY = 1
ALLOW_EMAIL_WIDGET = false
ALLOW_METRICS = true
APP_FORCE_ACTIVATE_SSO = false
ALLOW_ATTACHMENTS_FROM_DOC_PROVIDERS = true
POLICY_ALLOW_OPEN_IN = 1
APP_DEFAULT_BROWSER_EXCEPTIONS = null
EMAIL_NOTIFICATION_TEXT_POLICY = -1
SUPPORT_API_KEY having value = false
ALLOW_ACTION_ARCHIVE = true
ALLOW_ACTION_SPAM = true
POLICY_APP_SWIPES_LEFT_SHORT_DEFAULT = -1
POLICY_APP_SWIPES_LEFT_LONG_DEFAULT = -1
POLICY_APP_SWIPES_RIGHT_SHORT_DEFAULT = -1
POLICY_APP_SWIPES_RIGHT_LONG_DEFAULT = -1
APP_AVATAR_ENABLED = true
ALLOW_LOCAL_CALENDAR = true
ALLOW_LOCAL_CONTACTS = true
APP_DOMAINS_INTERNAL = null
APP_DOMAINS_WARNING = false
APP_RE_FETCH_EMPTY_LINKS_USING_MIME = false
SHOULD_USE_PLAIN_TEXT_FOR_SYNC = false
APP_LOG_AGGREGATION_URL = null
APP_LOG_AGGREGATION_MODE = 0
ALLOWED_MAIL_SERVER_CIPHERS = null
-------------------------------------------------------------- BOXER SETTINGS
-------------------------------------------------------------- DEVICE_ID = 1D62C4D972BB4A4799474C5C0E5BA437 NEW_SYNC_ENGINE = false ALLOW_ENTERPRISE_CONTENT = false VIP_NOTIFICATIONS = true
-------------------------------------------------------------- BOXER POLICY
-------------------------------------------------------------- ALLOW_LOGGING = null
ALLOW_METRICS = null
ALLOW_PRINT = null
ALLOW_ACTION_EVERNOTE = null IS_KEY_ESCROW_DISABLED = false
-------------------------------------------------------------- EMAIL CLASSIFICATION SETTINGS --------------------------------------------------------------
ENABLED = false DEFAULT CLASS = null X-HEADER KEY = null VERSION = null
SORT ORDER = 1 CLASSIFICATIONS = []
-------------------------------------------------------------- THROTTLE SETTINGS
-------------------------------------------------------------- POLICY_CMD_FREQUENCY = null POLICY_SYNC_CMDS = null POLICY_RECENT_CMDS = null
-------------------------------------------------------------- ENS SETTINGS
--------------------------------------------------------------
ENS_LINK_ADDRESS = https://europa.airwlab.com/MailNotificationService/api/ens ENS_API_TOKEN = 091d6a8a897b41528ea452088e7fc599 POLICY_ACCOUNT_NOTIFY_PUSH = true
EWS_URL = https://nas98.airwlab.com:8002/ews/exchange.asmx
ENS_STATE = (1 -> Subscribed) | the state of ENS when the logs were captured
PendingAccountEmail = null PendingAccountName = null UserDisabled = false
-------------------------------------------------------------- THREAD INFO
--------------------------------------------------------------
com.boxer.common.e.a@811edcf[pool size = 2, active threads = 1, queued tasks = 0, completed tasks = 59, largest pool size = 7]
[com.boxer.unified.utils.at:a x 2, com.boxer.email.activity.MailActivityEmail:a, com.boxer.app.d:a x 2, com.boxer.email.provider.AttachmentProvider:onCreate, com.boxer.common.e.f:a x 18, com.boxer.common.e.f:b x 7, com.boxer.unified.widget.BaseWidgetProvider:b, com.boxer.emailcommon.provider.i:a, com.boxer.email.a.b:a x 2, com.boxer.settings.fragments.CombinedSettingsFragment:s, com.boxer.sdk.AirWatchAccountSetupService:e, com.boxer.unified.ui.ConversationViewFragment:a x 2, com.boxer.unified.providers.MailAppProvider:g, com.boxer.unified.ui.ConversationListFragment:U, com.boxer.email.service.ImapPopWorkerService:b, com.boxer.common.passcode.n:t, com.boxer.app.ag:f, com.boxer.common.activity.n:e x 2, com.boxer.common.app.v26support.k:o x 2, com.boxer.common.activity.n:c x 6, com.boxer.calendar.provider.CalendarProvider2:e, com.boxer.common.activity.ManagedActivity:P x 2, com.boxer.unified.ui.MailActionBarView:a, com.boxer.exchange.service.EmailSyncAdapterService:a]
-------------------------------------------------------------- HEALTH STATS
--------------------------------------------------------------
[sync] com.boxer.contacts/com.boxer.exchange/eM_ADDR count=0 [sync] com.boxer.contacts/com.boxer.exchange/eM_ADDR time_ms=0 [sync] com.boxer.calendar/com.boxer.exchange/eM_ADDR count=0
[sync] com.boxer.calendar/com.boxer.exchange/eM_ADDR time_ms=0
[sync] com.boxer.email.provider/com.boxer.exchange/eM_ADDR count=0
[sync] com.boxer.email.provider/com.boxer.exchange/eM_ADDR time_ms=0
[jobs] com.boxer.email/com.boxer.calendar.CalendarProviderChangedJobService count=0 [jobs] com.boxer.email/com.boxer.calendar.CalendarProviderChangedJobService time_ms=0 [jobs] com.boxer.email/.AirWatchSDKIntentService count=0
[jobs] com.boxer.email/.AirWatchSDKIntentService time_ms=0
[jobs] com.boxer.email.provider/com.boxer.exchange/eM_ADDR:android count=0
[jobs] com.boxer.email.provider/com.boxer.exchange/eM_ADDR:android time_ms=0
[jobs] com.boxer.email/com.airwatch.bizlib.gcm.GCMReceiverService count=0
[jobs] com.boxer.email/com.airwatch.bizlib.gcm.GCMReceiverService time_ms=0
[jobs] com.boxer.email/com.boxer.ens.EnsStateManager$RequestHandlerService count=0 [jobs] com.boxer.email/com.boxer.ens.EnsStateManager$RequestHandlerService time_ms=0 [jobs] com.boxer.email/com.boxer.service.EasOptionJobSchedulerService count=0
[jobs] com.boxer.email/com.boxer.service.EasOptionJobSchedulerService time_ms=0
[jobs] com.android.contacts/com.boxer.exchange/eM_ADDR:android count=0
[jobs] com.android.contacts/com.boxer.exchange/eM_ADDR:android time_ms=0
[jobs] com.boxer.email/com.boxer.sdk.SDKConfigurationUpdateHandler count=0
[jobs] com.boxer.email/com.boxer.sdk.SDKConfigurationUpdateHandler time_ms=0
[jobs] com.android.calendar/com.boxer.exchange/eM_ADDR:android count=0
[jobs] com.android.calendar/com.boxer.exchange/eM_ADDR:android time_ms=0
[jobs] com.boxer.email/.service.AttachmentsDeleteIntentService count=0
[jobs] com.boxer.email/.service.AttachmentsDeleteIntentService time_ms=0
[jobs] com.boxer.email/com.boxer.contacts.services.BoxerCallerIdIntentService count=0 [jobs] com.boxer.email/com.boxer.contacts.services.BoxerCallerIdIntentService time_ms=0 [jobs] com.boxer.email/com.boxer.irm.IRMTemplatesSyncService count=0
[jobs] com.boxer.email/com.boxer.irm.IRMTemplatesSyncService time_ms=0
[jobs] com.boxer.email/com.airwatch.storage.SessionDataStorageService count=0
[jobs] com.boxer.email/com.airwatch.storage.SessionDataStorageService time_ms=0
[jobs] com.boxer.email/com.boxer.exchange.provider.GALDirectoryUpdateService count=0 [jobs] com.boxer.email/com.boxer.exchange.provider.GALDirectoryUpdateService time_ms=0 [jobs] com.boxer.contacts/com.boxer.exchange/eM_ADDR:android count=0
[jobs] com.boxer.contacts/com.boxer.exchange/eM_ADDR:android time_ms=0
[jobs] com.boxer.calendar/com.boxer.exchange/eM_ADDR:android count=0
[jobs] com.boxer.calendar/com.boxer.exchange/eM_ADDR:android time_ms=0
[foreground activity] count=0
[foreground activity] time_ms=0
[foreground_service] count=0
[foreground_service] time_ms=0
[top] count=0
[top] time_ms=0
[top_sleeping] count=0
[top_sleeping] time_ms=0
[background process state] count=0
[background process state] time_ms=0
[cached process state] count=0
[cached process state] time_ms=0
[cpu] user_cpu_time_ms=0
[cpu] system_cpu_time_ms=0
[user activity] button_user_activity_count=0
[user activity] touch_user_activity_count=0
[process] sessionTokenStorage.data anr_count=0
[process] sessionTokenStorage.data crash_count=0
[process] sessionTokenStorage.data foreground=0
[process] sessionTokenStorage.data starts=10
[process] sessionTokenStorage.data system_time=60
[process] sessionTokenStorage.data user_time=130
[process] com.boxer.email [process] com.boxer.email [process] com.boxer.email [process] com.boxer.email [process] com.boxer.email [process] com.boxer.email
[wifi] wifi_running_ms=0
[wifi] wifi_full_lock_ms=0
[wifi] wifi_rx_bytes=0
[wifi] wifi_tx_bytes=0
[mobile] mobile_rx_bytes=0 [mobile] mobile_tx_bytes=0 [mobile_radio_active] count=0 [mobile_radio_active] time_ms=0
anr_count=0 crash_count=0 foreground=289340 starts=10 system_time=58290 user_time=153290
-------------------------------------------------------------- ACCOUNTS INFO
--------------------------------------------------------------
NAME: lwilcox@labmail.airwlab.com, TYPE: 5, IS_MANAGED: true, FETCHES_MIME: false
Common Ways to Search Through the Logs
On the console where you configure your Boxer profile, you will have places where you can enter profile information like Account Name, Domain, User, etc… However, in the Boxer logs, when you are looking for the corresponding data there, the wording might be slightly different. You can use the mapping information below to see what this would say in the Boxer logs. The value on the left is what you will see in the console. The value after “=” is what you will see in the Boxer logs.
Account Name = DESCRIPTION
Exchange ActiveSync Host = SERVER_ADDRESS
Domain = USER_DOMAIN
User = USERNAME
Email Address = EMAIL
Email Signature = SIGNATURE
Authentication Type = ACCOUNT_AUTHENTICATION_TYPE | 0 basic, 1 certificate, 2 both Copy Paste = ALLOW_COPY_PASTE
Screenshots = ALLOW_SCREEN_CAPTURE
Allow email widget = ALLOW_EMAIL_WIDGET
Allow calendar widget = ALLOW_CALENDAR_WIDGET
Hyperlinks = POLICY_ALLOW_OPEN_IN
Sharing = DOCUMENT_SHARING_RESTRICTION
Caller ID = ALLOW_CALLER_ID
Personal Accounts = ALLOW_OTHER_ACCOUNTS
Personal Contacts = ALLOW_LOCAL_CONTACTS
Boxer reaches out to the console to get the profile information
2019-02-07T15:21:55.345Z I [18459-BoxerWorker-5] - App initialization step complete: 1
2019-02-07T15:21:55.578Z E [18459-main] - FLF.setSelectedAccount(null) called! Destroying existing loader.
2019-02-07T15:21:55.589Z I [18459-BoxerWorker-3] - Waiting for app restrictions
This entry is the best example we have of when Boxer reaches out to the console. You can use Fiddler to further determine when Boxer is reaching out to the console. In the Boxer logs, you can’t directly see this. The other place you can see this is in the ADB device logs. The line here that says “waiting for app restrictions” just indicates that Boxer is waiting to read from the database. Boxer reaches out to the console to get any S/MIME certificates and client authentication certificates available from the console.
2019-02-07T15:21:56.256Z I [18459-IntentService[AirWatchAccountSetupService]] - Fetching S/MIME signing certificate
2019-02-07T15:21:58.021Z I [18459-AsyncTask #2] - Certificate being fetched for : AccountAuthenticationCertificateId
2019-02-07T15:21:59.013Z I [18459-AsyncTask #2] - Certificate fetch successful
2019-02-07T15:21:59.755Z I [18459-Thread-7] - TrackingKeyManager: requesting a client cert alias for 66.170.96.7
2019-02-07T15:22:05.776Z I [18459-Thread-9] - Registering socket factory for certificate alias [AW-be6fad91fe2b4291b1d392c9eabf90be]
2019-02-07T15:22:05.846Z D [18459-Thread-9] - Found cert chain: [ [0] Version: 3
SerialNumber: 468315651040541492877707779630191635557515624
IssuerDN: DC=local,DC=milkyway,CN=milkyway-SUN-CA Start Date: Thu Feb 07 10:11:47 EST 2019
Final Date: Sat Mar 09 10:11:47 EST 2019
SubjectDN: CN=lwilcox
Public Key: RSA Public Key
modulus: ccbf28975eafd49dedfcfc43f9d66a309d36c4c636fbece2bf9faa87fe6544c8e025c3da260eb4bc28096f665a231202c5cf53dbb5f1d08ceac4d9af90f0605ee2951e6239576749d39d17e8d411b39c03e6a68e155083866a69de610853ab83d149c0228b6f0b0d67d4e9e1093ce8a3316a563c48d91fcc000bae01c150644113a527d611d83f7303825a58a4382c1818d8fd56b2064ef495c709e4ae4366e7793e29cd3b376e9660028269c8233ba8cd9a42e57dc2f07087d25fc6ce536d3bcbc8fabe2f6c408ddb2fefd8b80fdc029ba16237b48d26072d0f88f9e982ab37720883ff7bfb35bc409637fb314c00cbc1cb262e7406732c45c400ca45d9357b
public exponent: 10001
Boxer reaches out to the Exchange ActiveSync (EAS) email endpoint to make the options, provision, foldersync, and ping request.
Making the Options Request
Boxer Options Request...
2019-02-07T15:22:05.859Z D [18459-Thread-9] - EasServerConnection about to make request OPTIONS https+clientCert+aw-be6fad91fe2b4291b1d392c9eabf90be://nas98.airwlab.com:8002/Microsoft-Server-ActiveSync HTTP/1.1 X-VMware-Boxer-RequestId:5bde2467-38bc-4a8e-9599-256217cb8c6d
2019-02-07T15:22:05.859Z V [18459-Thread-9] - query: uri=content://com.boxer.email.provider/account/1, match is 1
2019-02-07T15:22:06.080Z I [18459-Thread-9] - Requesting a client cert alias for [RSA, EC]
2019-02-07T15:22:06.081Z I [18459-Thread-9] - Requesting a client private key for alias [AW-be6fad91fe2b4291b1d392c9eabf90be]
2019-02-07T15:22:06.082Z I [18459-Thread-9] - Requesting a client certificate chain for alias [AW-be6fad91fe2b4291b1d392c9eabf90be]
2019-02-07T15:22:06.178Z V [18459-Thread-9] - SSL socket using protocol: TLSv1.2
2019-02-07T15:22:11.485Z V [18459-Thread-9] - Response headers:
Header [Strict-Transport-Security], value [max-age=31536000;includeSubDomains] Header [X-Frame-Options], value [sameorigin]
Header [X-XSS-Protection], value [1;mode=block]
Header [X-Content-Type-Options], value [nosniff]
Header [Content-Security-Policy], value [default-src 'self'; font-src 'self' data:; script-src 'unsafe-eval' 'self'; style-src 'unsafe-inline' 'self'; object-src 'none';] Header [Cache-Control], value [private]
Header [Allow], value [OPTIONS,POST]
Header [Content-Type], value [application/vnd.ms-sync.wbxml]
Header [Server], value [Microsoft-IIS/8.5]
Header [request-id], value [73ab0f67-308f-45d2-a76b-04eb72fde4e8]
Header [X-CalculatedBETarget], value [earth.milkyway.local]
Header [MS-Server-ActiveSync], value [15.0]
Header [MS-ASProtocolVersions], value [2.0,2.1,2.5,12.0,12.1,14.0,14.1]
Header [MS-ASProtocolCommands], value [Sync,SendMail,SmartForward,SmartReply,GetAttachment,GetHierarchy,CreateCollection,DeleteCollection,MoveCollection,FolderSync,FolderCreate,FolderDelete,FolderUpdate,MoveItems,GetItemEstimate,MeetingResponse,Search,Settings,Ping,ItemOperations,Provision,ResolveRecipients,ValidateCert] Header [Public], value [OPTIONS,POST]
Header [X-MS-BackOffDuration], value [L/-470]
Header [X-DiagInfo], value [EARTH]
Header [X-BEServer], value [EARTH]
Header [X-AspNet-Version], value [4.0.30319]
Header [Persistent-Auth], value [false]
Header [X-Powered-By], value [ASP.NET]
Header [X-FEServer], value [MERCURY]
Header [WWW-Authenticate], value [Negotiate YIGVBgkqhkiG9xIBAgICAG+BhTCBgqADAgEFoQMCAQ+idjB0oAMCAReibQRr1j3vRSNgnWRAjRmh9+c4H68rnNGVHQxp2bq87jvJsxPfGt+Wi1ciWxUKztM5Rjq2HiDSiMttELVzigj9rrUB00pGrB/LemrzZeB1NU3oyWSlYfGMd1uvo96mRGykkn8vA0kW3mlXDKacEz0=]
Header [Date], value [Thu, 07 Feb 2019 15:22:00 GMT]
Header [X-AW-SEG-SERVER-INFO], value [***.***.***.43|2.9.0.1|JSEG-SEG368-JOB1-2|Tue 08 Jan 2019 03:32:23 PM EST -0500]
Header [X-AW-SEG-TRANSACTION-ID], value [c42418f6-55a1-4ffc-b4fd-6e94c2d70fe3]
Header [X-Correlation-ID], value [c42418f6-55a1-4ffc-b4fd-6e94c2d70fe3]
Header [Content-Length], value [0]
Header [Set-Cookie], value [ClientId="NGLCEUYUWLVTAAWHW";$Path="/";$Domain="nas98.airwlab.com:8002";HttpOnly]
Header [Set-Cookie], value [X-BackEndCookie="S-1-5-21-174156188-4291662551-709137977-1610=u56Lnp2ejJqByprLzcrHyMzSx8aantLLz8ic0p7Kmc7Sz52bnZqZy8nIy8mZgYHNz87G0s/M0s/Gq87Kxc3Nxc/P";$Path="/Microsoft-Server-ActiveSync";Secure;$Domain="nas98.airwlab.com:8002";HttpOnly]
2019-02-07T15:22:11.535Z D [18459-Thread-9] - Server supports versions: 2.0,2.1,2.5,12.0,12.1,14.0,14.1
2019-02-07T15:22:11.625Z V [18459-Thread-9] - <SyncKey>
2019-02-07T15:22:11.625Z V [18459-Thread-9]-0
2019-02-07T15:22:11.625Z V [18459-Thread-9] - </SyncKey>
2019-02-07T15:22:11.625Z V [18459-Thread-9] - </FolderSync>
Perform a FolderSync
Boxer FolderSync...
2019-02-07T15:22:11.634Z D [18459-Thread-9 ] - EasServerConnection about to make request POST https+clientCert+aw-be6fad91fe2b4291b1d392c9eabf90be://nas98.airwlab.com:8002/Microsoft-Server-ActiveSync?Cmd=FolderSync&User=milkyway%5Clwilcox%40milkyway.local&DeviceId=1D62C4D972BB4A4799474C5C0E5BA437&DeviceType=BoxerManagedAndroid HTTP/1.1 X-VMware-Boxer-RequestId:572f2ab2-cdf1-4960-b30a-6b078365b3aa
2019-02-07T15:22:11.805Z V [18459-Thread-9 ] - SSL socket using protocol: TLSv1.2
2019-02-07T15:22:15.574Z V [18459-Thread-9 ] - Response headers:
Header [Strict-Transport-Security], value [max-age=31536000;includeSubDomains] Header [X-Frame-Options], value [sameorigin]
Header [X-XSS-Protection], value [1;mode=block]
Header [X-Content-Type-Options], value [nosniff]
Header [Content-Security-Policy], value [default-src 'self'; font-src 'self' data:; script-src 'unsafe-eval' 'self'; style-src 'unsafe-inline' 'self'; object-src 'none';] Header [Cache-Control], value [private]
Header [Content-Type], value [application/vnd.ms-sync.wbxml]
Header [Vary], value [Accept-Encoding]
Header [Server], value [Microsoft-IIS/8.5]
Header [request-id], value [3cd7bb11-e3c4-4292-be4a-87cb5d388473]
Header [X-CalculatedBETarget], value [earth.milkyway.local]
Header [MS-Server-ActiveSync], value [15.0]
Header [X-MS-BackOffDuration], value [L/-469]
Header [X-DiagInfo], value [EARTH]
Header [X-BEServer], value [EARTH]
Header [X-AspNet-Version], value [4.0.30319]
Header [Persistent-Auth], value [false]
Header [X-Powered-By], value [ASP.NET]
Header [X-FEServer], value [MERCURY]
Header [WWW-Authenticate], value [Negotiate YIGVBgkqhkiG9xIBAgICAG+BhTCBgqADAgEFoQMCAQ+idjB0oAMCAReibQRrSwxIlKGW3nSAU46FtN7BJ3AO5tTFRrU/dbnLqIXVdl6/ySBXRFmuQ/AsFOP95xNJf0ze6ZG5Vrk/KeCaxlmsaSqlShiRmjPAxAyuIzJ8vKnjzQFf6MPFepwruykQpNDqjdGtRs7RV1/uy8M=] Header [Date], value [Thu, 07 Feb 2019 15:22:04 GMT]
Header [X-AW-SEG-SERVER-INFO], value [***.***.***.43|2.9.0.1|JSEG-SEG368-JOB1-2|Tue 08 Jan 2019 03:32:23 PM EST -0500]
Header [X-AW-SEG-TRANSACTION-ID], value [51e38c4f-5114-4465-a956-6f66f86c967d]
Header [X-Correlation-ID], value [51e38c4f-5114-4465-a956-6f66f86c967d]
Header [Set-Cookie], value [X-BackEndCookie="S-1-5-21-174156188-4291662551-709137977-1610=u56Lnp2ejJqByprLzcrHyMzSx8aantLLz8ic0p7Kmc7Sz52bnZqZy8nIy8mZgYHNz87G0s/M0s/Gq87Kxc3Nxc/L";$Path="/Microsoft-Server-ActiveSync";Secure;$Domain="nas98.airwlab.com:8002";HttpOnly] Header [content-encoding], value [gzip]
Header [transfer-encoding], value [chunked]
2019-02-07T15:22:15.667Z V [18459-Thread-9] - <Status>
2019-02-07T15:22:15.668Z V [18459-Thread-9] - Status: 144
2019-02-07T15:22:15.671Z V [18459-Thread-9] - </Status>
Perform a provision request
Boxer provision request...
2019-02-07T15:22:15.681Z I [18459-Thread-9 ] - Received needs provision response in EasOperation.performOperation
2019-02-07T15:22:15.723Z D [18459-Thread-9 ] - EasServerConnection about to make request POST https+clientCert+aw-be6fad91fe2b4291b1d392c9eabf90be://nas98.airwlab.com:8002/Microsoft-Server-ActiveSync?Cmd=Provision&User=milkyway%5Clwilcox%40milkyway.local&DeviceId=1D62C4D972BB4A4799474C5C0E5BA437&DeviceType=BoxerManagedAndroid HTTP/1.1 X-VMware-Boxer-RequestId:08e64270-501d-4732-9675-5309dd6a3675
2019-02-07T15:22:15.834Z V [18459-Thread-9 ] - SSL socket using protocol: TLSv1.2
2019-02-07T15:22:19.828Z V [18459-Thread-9 ] - Response headers:
Header [Strict-Transport-Security], value [max-age=31536000;includeSubDomains] Header [X-Frame-Options], value [sameorigin]
Header [X-XSS-Protection], value [1;mode=block]
Header [X-Content-Type-Options], value [nosniff]
Header [Content-Security-Policy], value [default-src 'self'; font-src 'self' data:; script-src 'unsafe-eval' 'self'; style-src 'unsafe-inline' 'self'; object-src 'none';] Header [Cache-Control], value [private]
Header [Content-Type], value [application/vnd.ms-sync.wbxml]
Header [Vary], value [Accept-Encoding]
Header [Server], value [Microsoft-IIS/8.5]
Header [request-id], value [dcda9893-be8c-4b17-8fe5-067f71dd57e0]
Header [X-CalculatedBETarget], value [earth.milkyway.local]
Header [MS-Server-ActiveSync], value [15.0]
Header [X-MS-BackOffDuration], value [L/-469]
Header [X-DiagInfo], value [EARTH]
Header [X-BEServer], value [EARTH]
Header [X-AspNet-Version], value [4.0.30319]
Header [Persistent-Auth], value [false]
Header [X-Powered-By], value [ASP.NET]
Header [X-FEServer], value [MERCURY]
Header [WWW-Authenticate], value [Negotiate YIGVBgkqhkiG9xIBAgICAG+BhTCBgqADAgEFoQMCAQ+idjB0oAMCAReibQRr9huiSZNPIN39dB0GIaIF4gSACDj8yrV879OKH5dUeSEQzFnRcPGIShPvLFbDCrDS+ii/N9SAvk/B20xq+MXtxXRBsHyJXFJo5vIRjkWq+3Qis5c/4+gP5Vi0bsvlVSKIW4yqRyGe5ZfGoNc=] Header [Date], value [Thu, 07 Feb 2019 15:22:08 GMT]
Header [X-AW-SEG-SERVER-INFO], value [***.***.***.43|2.9.0.1|JSEG-SEG368-JOB1-2|Tue 08 Jan 2019 03:32:23 PM EST -0500]
Header [X-AW-SEG-TRANSACTION-ID], value [13206ae4-e2c0-4579-a992-a3d9664a5763]
Header [X-Correlation-ID], value [13206ae4-e2c0-4579-a992-a3d9664a5763]
Header [Set-Cookie], value [X-BackEndCookie="S-1-5-21-174156188-4291662551-709137977-1610=u56Lnp2ejJqByprLzcrHyMzSx8aantLLz8ic0p7Kmc7Sz52bnZqZy8nIy8mZgYHNz87G0s/M0s/Gq87Kxc3Nxc/H";$Path="/Microsoft-Server-ActiveSync";Secure;$Domain="nas98.airwlab.com:8002";HttpOnly] Header [content-encoding], value [gzip]
Header [transfer-encoding], value [chunked]
2019-02-07T15:22:19.882Z D [18459-Thread-9 ] - Provision status: 1
Perform another FolderSync, this time it works. Start email sync.
Boxer another FolderSync...
2019-02-07T15:22:24.586Z I [18459-AccountsUpdateListener] - Accounts changed - requesting FolderSync for unsynced accounts
2019-02-07T15:22:24.975Z I [18459-SyncAdapterThread-1] - Calendar sync for account lwilcox@labmail.airwlab.com, with extras Bundle[{}]
2019-02-07T15:22:25.236Z I [18459-SyncAdapterThread-1] - Email sync for account lwilcox@labmail.airwlab.com, with extras Bundle[{}]
2019-02-07T15:22:29.810Z I [18459-Thread-10] - Initial sync requested for account: [nas98.airwlab.com:8002:lwilcox@labmail.airwlab.com:Legend Wilcox]
2019-02-07T15:22:30.430Z I [18459-BoxerWorker-9] - requestSync EmailProvider startSync Account {name=lwilcox@labmail.airwlab.com, type=com.boxer.exchange}, Bundle[{do_not_retry=false, __userRequest__=true, callback_method=sync_status, force=true, expedited=true, callback_uri=content://com.boxer.email.provider}]
2019-02-07T15:22:30.447Z I [18459-SyncAdapterThread-2] - Email sync for account lwilcox@labmail.airwlab.com, with extras Bundle[{ignore_settings=true, do_not_retry=false, __userRequest__=true, callback_method=sync_status, force=true, expedited=true, ignore_backoff=true, callback_uri=content://com.boxer.email.provider}]
2019-02-07T15:22:34.950Z I [18459-pool-21-thread-4] - Starting sync command
2019-02-07T15:22:34.993Z I [18459-pool-21-thread-2] - Syncing account 1 mailbox "Sent Items" (class Email) with syncKey 0, operation name = HtmlMailSync
2019-02-07T15:22:39.191Z I [18459-pool-21-thread-2] - Committing new sync key (684740726) for mailbox (Sent Items)
2019-02-07T15:22:39.216Z I [18459-pool-21-thread-2] - Sync command finished with result 1
2019-02-07T15:23:00.737Z I [18459-SyncAdapterThread-2] - Initial sync completed
Ping example
2019-02-07T15:40:12.911Z I [18459-PingTask-lwilcox@labmail.airwlab.com] - Ping task starting for 1
2019-02-07T15:40:12.912Z I [18459-PingTask-lwilcox@labmail.airwlab.com] - Exchange ping starting
2019-02-07T15:43:00.125Z I [18459-PingTask-lwilcox@labmail.airwlab.com] - Changes found in: 8
2019-02-07T15:43:00.126Z I [18459-PingTask-lwilcox@labmail.airwlab.com] - Ping found changed folders for account 1
2019-02-07T15:43:00.137Z I [18459-PingTask-lwilcox@labmail.airwlab.com] - requestSync EasOperation requestSyncForMailboxes Account {name=lwilcox@labmail.airwlab.com, type=com.boxer.exchange}, Bundle[{__mailboxCount__=1, force=true, expedited=true, __mailboxId0__=5, PING_ERROR_COUNT=3}] 2019-02-07T15:43:00.139Z I [18459-PingTask-lwilcox@labmail.airwlab.com] - Exchange ping finished with result 2
2019-02-07T15:43:00.200Z I [18459-SyncAdapterThread-8 ] - Email sync for account lwilcox@labmail.airwlab.com, with extras Bundle[{ignore_settings=true, __mailboxCount__=1, force=true, expedited=true, ignore_backoff=true, __mailboxId0__=5, PING_ERROR_COUNT=3}] 2019-02-07T15:43:00.316Z I [18459-SyncAdapterThread-8 ] - Starting sync command
Boxer reaches out to the Email Notification Service (ENS) and provides the Exchange Web Services (EWS) credential information that was used for ActiveSync so it can subscribe to email notifications.
2019-02-07T15:23:08.315Z I [18459-AsyncTask #4 ] - Ens registration for account (id=1) is successful!
2019-02-07T15:43:30.041Z I [18459-main ] - Sync triggered from distance
You can use Notepad++ to search the logs using the following search terms.
• “error” - This is a very general way to look for errors. You will have a lot of false/positives when using this search criteria.
• “exception “ - You can use this to search the file for exceptions that are generated when the application runs into an error.
• “Current Network” - This will show you what network connection the mobile device was connected to 4g, wi-fi, etc…
• “distance” - This shows you each time the ENS server has reached out to the mobile device to wake it up, send a notification, and trigger a sync • “transitioning” - Use this search term to tell when the application is transitioning from background to foreground and vice versa.
• “requestPing” - This will show you when the device reaches out to the OS to use the Android SyncAdaptor which it uses to request the ping. The response to this request will be “Email sync for account”.
• “Email sync for account” - This will be a response from the OS for a command “requestPing” or “requestSync “. Take note of how long between the request and the reply from the OS. If it’s a long time, it’s likely being throttled by the OS due to battery optimization or a third party product.
• “Changes found in: “ - is the response to a ping request. This tells us if the ping found any changes in the inbox. If it did, we will now do a “requestSync " to bring those emails or changes into Boxer.
• “requestSync” - This will show you when the device reaches out to the OS to use the Android SyncAdaptor which it uses to request email sync. The response to this request will be “Email sync for account”.
• “Q-AppInitialization “ - This shows you when the application is starting up. You will see this when you force close and reopen the application or when you start the application after a crash.
• “PingTask “ - shows you all activities related to pings
• “SyncAdapterThread “ - shows you all activities related to sync operations.
• “Email sync for account “ - The response from the OS after a “requestSync” or “RequestPing” operation.
• “ens” - Search the log files for ens errors and settings.
Collecting Boxer Logs When You Can’t Get Into Android Boxer
Depending on what model phone you have, you can follow one of the processes below to collect Boxer logs in the event that you can’t get into Boxer due to an error. You may have a different model device than the ones listed, but the process will be very similar.
Samsung S9+
• Go to “Settings” on the phone followed by “Apps” • Select “Boxer”
• Select “Mobile Data”
• Select “View App settings”
• Select “Send logs”
• Use one of the other mechanisms to send the logs (either by copying the logs out or another email client).
On Motorola X Pure
• Go to “Settings” on the phone followed by “Apps”
• Select “Boxer”
• Select “Data Usage”
• Select “App settings”
• Select “Send logs”
• Use one of the other mechanisms to send the logs (either by copying the logs out or another email client).
Device-side logs collection
Collecting Device-Side Logs
macOS Devices
With version 2.x of the macOS agent, you can now collect logs via the agent by performing the following steps:
- Open the agent by right-clicking the agent icon from the menu bar, and then clicking Preferences
- From the Status screen, click on Diagnostics
- Click the button to Send Logs to Administrator
- The agent will gather logs and zip them into an email for you automatically
To manually gather logs on older versions of the AirWatch agent, please perform the following:
- Enable Debug Logging in the Mac OS X Terminal:
- $ sudo defaults write /Library/Preferences/com.apple.MCXDebug debugOutput -2
- $ sudo defaults write /Library/Preferences/com.apple.MCXDebug collateLogs 1
- $ sudo touch /var/db/MDM_EnableDebug
- Open the Console application from the Launchpad. Select All Messages and then click Clear Display to clear out old logs.
- Reproduce the issue on Mac.
- Click File > Save a Copy As. and save a copy of the logs to be sent to AirWatch.
- Disable the Debug logging in the Mac OS X Terminal:
- $ sudo rm –rf /var/db/MDM_EnableDebug
- $ sudo defaults delete /Library/Preferences/com.apple.MCXDebug debugOutput
- $ sudo defaults delete /Library/Preferences/com.apple.MCXDebug collateLogs
iOS Devices (iPhone, iPad, Apple TV)
To gather logs from iOS devices (iOS 7.x and below), please perform the following steps:
- Install iPhone Configuration Utility (iPCU) on your workstation.
- With iPCU started, connect the iOS device to your computer via the USB cable.
- On the left-hand side of iPCU, select the iOS device where you wish to collect logs.
- Click on the Console tab at the top right-hand corner.
- Reproduce the issue with your iOS devices.
- Click the Save Console As button to save the text in the console to a file.
To gather logs from iOS devices (iOS 8.0+), please perform the following steps:
- Install xCode 6 on your OS X Device. Note: You cannot gather iOS 8 logs from a Windows-based computer. You must use a Mac OS X computer.
- With xCode started, connect the iOS device to your computer via the USB cable.
- From within xCode, click on the Window menu and click Devices.
- Select your iOS device from the left hand side, then select the up arrow at the bottom corner of the right hand side.
- Reproduce the issue with your iOS devices.
- Save the contents of the activity log to a file.
Android
To gather logs using Console, please see the following documentation.
To gather logs using ADB, please perform the following steps (see the following page for details):
- Download and set up the Android SDK per the SDK documentation.
- Open Windows Explorer and browse to the \platform-tools folder. Ensure you see the adb.exe file.
- Open a CMD window and navigate to the platform-tools folder. Or, in Windows 7, navigate back to the SDK folder, then Shift + Right-Click on the platform-tools folder, and select Open Command Window Here.
- Ensure USB Debugging is enabled on your Android device (from the Developer Settings menu). For more information on how to enable the Developer Settings menu, browse to http://www.androidcentral.com/how-enable-developer-settings-android-42
- In your Notification Center, you may need to make sure the device is not connected in USB Media Device mode.
- In the CMD window, type adb logcat –v long > androidlog.txt
- On the Android device, recreate whatever error you are trying to log.
- When complete, from within the CMD window use CTRL + C to end the logging.
- Go back to the platform-tools folder and find the log file (androidlog.txt) that you just created.
Windows 7/8/8.1/10/11
To gather logs, please perform the following steps:
- Click on Start > Run, type eventtvwr.msc and click OK. On Windows 8/8.1, from the start menu you can simply start typing Event and select the View Event Logs item returned from universal search.
- Expand Event Viewer (Local) > Windows Logs and select the Application log.
- You can filter logs by Event ID or Source if desired.
- To export for support, click on either Save All Events As or Save Selected Events to export the log entries as an *.evtx file which can be sent to support.
- You can also find logs in the following location: \AgentUI\Logs
- AwclClient.log - AWCM-related Issues
- AWProcessCommands.log - Issues with sending commands to the device
- NativeEnrollment.log - Issues with Enrollment
- TaskScheduler.log - Issues with samples sent to console
Windows Phone 8.1 (deprecated)
To gather logs, please perform the following steps:
- Ensure you have Visual Studios 2013 Update 3 installed. If not, perform the following:
- From your Windows Laptop of VM, browse to Windows Phone SDK Archives
- Download the Visual Studio Express 2013 for Windows and Install it.
- From within Visual Studios, click on Tools > Windows Phone 8.1 > Developer Unlock. Follow the prompts to unlock your Windows Phone 8.1 device.
- From within Visual Studios, click on Tools > Windows Phone 8.1 > Developer Power tools.
- Select Device from the Select Device dropdown, then click Connect. If prompted, click Install to install the Phone Tools Update Pack.
- Select the Performance Recorder tab, then check the Enterprise Management option under the Extras profile category.
- Click the Start button in the Developer Power Tools window to start a log.
- Run your scenarios and re-create the issue you’re experiencing.
- Click the Stop button in the Developer Power Tools window to stop logging and save the ETW to a local location.
- You will need to download the Windows Performance Analyzer to view the logs. This can be found in the Windows Performance Toolkit included in the Windows Assessment & Deployment Toolkit (ADK) and Windows Software Development Kit (SDK).
Windows Performance Toolkit
- Open the Windows Performance Analyzer and Open the ETL file.
- In the Graph Explorer window, expand System Activity and view the Generic Events window.
- Double-click the graphic bars in the Generic Events window to display an Analysis window.
- In the Analysis window, click Open View Editor to show a Generic Events View Editor window.
- In the Generic Events View Editor window, ensure the Message box is checked and click Apply:
- The Message field in the analysis window provides the MDM specific log message under various providers.
- Microsoft-WindowsPhone-Enrollment-API-Provider – ETW logs for MDM Enrollment and MDM Client Cert Renew Process.
- Microsoft-WindowsPhone-SCEP-Provider – SCEP Cert enrollment logging
- Microsoft-WindowsPhone-CmCspVpnPlus – VPN Configuration logging
Windows Mobile Devices with Agent 5.x
All log settings are configured in the log_config.cfg file in the \Program Files\AirWatch directory on the device. The file will resemble the following:
[*]
trace_level=5
max_file_size_kb=256
files_to_keep=2
log_file_path=\Program Files\AirWatch\Logs
use_local_time=false
[aw_setup]
trace_level=5
max_file_size_kb=256
files_to_keep=2
log_file_path=\
use_local_time=false
[awregisterdevice.exe]
trace_level=3
max_file_size_kb=256
files_to_keep=2
log_file_path=\Program Files\AirWatch\Logs
use_local_time=false
[awapplyprofile.exe]
trace_level=5
max_file_size_kb=256
files_to_keep=2
log_file_path=\Program Files\AirWatch\Logs
use_local_time=false
[awremotecontrol.exe]
trace_level=1
max_file_size_kb=256
files_to_keep=2
log_file_path=\Program Files\AirWatch\Logs
use_local_time=false
In general, the following notes apply to Windows Mobile device logging:
- The logging level can be modified as a whole, or on an individual basis:
- The asterisk configuration is the default config for all logs. Trace levels vary from 1 (basic) to 5 (verbose/debug).
- Each individual section, which can be used to increase logging to override the default setting from the asterisk section.
- The log files which are available can vary (based on configuration and OEM), but the following are the most common:
- aw_setup - Provides logging information relating to the AWMasterSetup utility, which is responsible for initiating the agent install and uninstall process on a device. This is the only log file that is not located in the “\Program Files\AirWatch” directory and is instead located in the root of the file system.
- awacmclient - Provides logging information relating to the AWCM client on the device
- awapplicationmanager - Provides logging information relating to product provisioning
- awprocesscommands - Provides logging information relating to the execution of MDM commands and installation of profiles
- AWService - Provides information about the AWService.exe component, which is responsible for managing beacon and interrogator samples
- awapplyprofile - Relates to installation of the agent settings xml file which occurs during the enrollment process
- awregisterdevice - Provides information about the registering of the device that occurs during the enrollment process
- awapplauncher - Provides information about the Application Launcher executable. This log will only be present if the App Launcher utility is assigned to and being used by a device.
- fusionwlansetup - Provides information about configuring and setting up the Fusion WiFi driver on Motorola devices.
The general process for configuring log files is as follows:
- Transfer the log file to your machine. This can be done through the file manager utility in device details or through remote management if a client has that configured.
- Open the log file via a basic text editor such as notepad.
- Edit the desired trace level to the needed value.
- Save the log file.
- Transfer the log file back down to the “Program Files\AirWatch” directory on the devices. This can be accomplished via file manager, remote manager, or product provisioning. To be safe, you may elect to first delete the old log_config.cfg file.
- Restart AWService on the device once it has the updated log_config.cfg file. This can be accomplished by directly restarting the AWService through the “Restart AirWatch Agent” or the “Warm Boot” MDM commands that are available in the AirWatch Console.
- Once the AWService has been restarted, the new logging configuration will take effect. Reproduce your issue and then repeat the steps to turn the logging back down on the device.
Collecting Service/Functionality Specific Logs
Product Provisioning
Review the AirWatch Agent Logs and look for the following items to help you troubleshoot what is occurring:
- If the device is newly enrolled, you’ll see the following in the logs: A message from [AWProductHandler sendProductResponses] stating “Products: No products with results to be sent!”
- A message from [AWEnhancedProductsHandler handleCommand:] stating “Got Products New Manifest”
- Note: In the manifest will be a line entry called ProductID". You’ll want to save this for later on.
- Depending on the number of products being installed, you may see an entry for each product that is required.
- Messages from [AWAppDataManager readJobProduct:] looking to see if the product is downloaded to the local cache
- Messages from [AWOSXUtils deleteFile:] where it attempts to delete any pre-existing plist file for the products.
- Messages from [AWJob printJob] which show the sequence number assigned to the Product which will be installed.
- From this point forward you can search the log by the sequence number assigned to the product install job:
- Messages about the job being queued
- Messages about the job being started.
- The line will look like this: airwatchd[PID] : - [AWJobQueue doJob] [Line 98] THREAD: Current Job: where PID is the AirWatch Agent Process ID and the JobID is the Sequence Number assigned to the product.
- You can get additional information about the product actions occurring by searching from that point forward for entries from the process ID!
- Messages about any files being downloaded to the product cache
- Messages about Job Status Change. You’ll want to search for a line ending in Job Status changed ========> :AWJobStatusFailed!" From that point, search up in the log for messages relating to the JobID and/or the ProductID (as found in the manifest). All these messages should be coming from the Process ID of the Airwatch agent that initially started the install.
Screen Recording
Screen Recording
Windows 7/8/8.1/10/11
To document click-through steps on Windows machines, perform the following:
- Click on Start > Run and type psr.exe to bring up the Problem Steps Recorder (or PSR, a built-in Windows utility).
- Click on Start Record to begin capturing steps. Note: PSR captures screenshots of ALL monitors; no scoping.
- Each Mouse-Click you make captures a screenshot. At any time during the session, click on Add Comment to provide more details about the screen, error, etc.
- When finished, click Stop Record.
- Choose where to Save the PSR file – it outputs a zip file containing a pre-compiled HTML (*.mhtml) file with all your screenshots and comments.
macOS
To document click-through steps on macOS machines, perform the following:
- Launch QuickTime Player. You’ll find it in the Other folder within Launchpad.
- From the QuickTime menu bar, click File > New Screen Recording. Click the red record button.
- Optionally you may wish to select View > Float on Top before you start recording.
- Optionally, you can select the upside-down triangle in the record screen to include audio recording during the screen capture for annotation.
- Click the screen (or Click-Drag to select part of the screen) for recording.
- When complete, click the Stop button that appears in the menu bar of the screen where you’re recording.
- Click File > Save (or simply quit QuickTime) to be prompted with a location to save the screen capture. Note: Keep it moving when you record these; they create full-blown movies and the file gets large quickly.
REST API
### URI
Old API URI example: https://host/api/v1/mam/apps/public/{applicationid}/addsmartgroup/{smartgroupid}
New API URI example: https://host/api/mam/apps/public/{applicationid}/smartgroup/{smartgroupid}
External links
Attached Documents
Articles in section
Preparation
You need a user account to run scripts in AirWatch REST API. Create a local account, take its’ credentials: domain\login:password, and go encode it at https://www.base64encode.org/
For example, 'lab\restguy:P@ssw0rd' ==> 'bGFiXHJlc3RndXk6UEBzc3cwcmQ='._
Turn API access ON for this account in the AirWatch console System -> Advanced -> API -> REST, we also should get an API Key for it.
Warning
-
AirWatch requires all requests to be made over SSL.
-
AirWatch limits the number of API requests that can be made during certain time periods. Rate limiting is done per Organization Group (OG) based on the API key used in the request.
**Server Throttling** - The limit imposed on an Organization Group for a 1 minute interval.
**Daily Quota** - The limit imposed on an Organization Groupvfor a 24-hour interval.
Local REST API Help
REST API Help is built into the AirWatch API Server host, go to link: https:///api/help/
Examples of using REST API
Let’s take some device from those enrolled in AirWatch and check out the apps list on it.
import requests
consoleURL = 'https://airwatch.example.com'
b64EncodedAuth = 'bGFiXHJlc3RndXk6UEBzc3cwcmQ='
API_Key = 'S390lvIVeFRBSO4InP73MFG3YSFDNKL+BXKW7Wv5Wwo='
basic_headers={"Authorization": "Basic " + b64EncodedAuth, "aw-tenant-code": APItenantCode,"Accept": "application/json"}
appsTest = requests.get(consoleURL + "/api/mam/apps/internal/46", headers=basic_headers)
print(appsTest.json())
46 is the device ID inside AirWatch. How do you find the ID of a device? - simplest way to do this is open AirWatch console, device list page, and hover your mouse over any device name: you will see the ID in the browser URL status bar. Now, let’s see where the device has been for the last 2 days:
gps2days = requests.get(consoleURL + "/api/mdm/devices/46/gps?dayrange=2", headers=basic_headers)
print(gps2days.json())
If there are any GPS points recorded for the device, we will get a list of them here. How can we use this? For example, customers say they hate Bing Maps embedded in AirWatch console. So we can build a little portal for searching devices, embed Google Maps into it and use the coordinates list to draw points.
One step deeper: suppose we need to show a customer how GPS coordinates are being collected, but the device enrolled is fresh, did not catch any sattelites yet or has problems with GPS module. Since GPS data is actually stored and taken from the SQL database, we can insert some coordinates there manually and see them right away on the Bing map in the console. See article on inserting false GPS history for a device in SQL section
Last example in this article I got from a cool client. They give out corporate iPhones to their employees, and all of those devices are supposed to be supervized. Only supervized devices are to be enrolled in AirWatch and get corp data, no personal gadgets coming through! AirWatch does have a tag for Apple device status (is it Supervized/DEP/Education?), but uses it only for reporting - there is no Compliance rule or filtering around this currently (a ticket in Jira on the topic is promised to be closed in AirWatch 9.6+). So the client gets a list of all enrolled devices and filters them manually:
import json
devicelist = requests.get(consoleURL + '/api/mdm/devices/search', headers=basic_headers)
for i in devicelist.json()["Devices"]:
if i["Platform"] == "Apple" and i["IsSupervised"] == False and i["EnrollmentStatus"] == "Enrolled":
# Tag the toxic devices, or enterprise wipe them
By running this script every half-hour, all unsupervised devices are Enterprise Wiped shortly after they are enrolled.
WS1 Launcher and Restrictions
Custom XML Profiles for Android
External link
Custom XML for Workspace ONE Launcher - https://docs.vmware.com/en/VMware-Workspace-ONE-UEM/2209/Launcher_Publication/GUID-CustomXML.html?hWord=N4IghgNiBcIMYFcDOAXA9gWwAQA8MSwAcAnNAMwEsIBTLCMBAOzgAtriQBfIA
When using Ws1 Launcher for Android, it will block Phone app and incoming phone call notifications. In order to allow phone dialer app (example: com.samsung.android.dialer) a custom profile has to be added:
<characteristic type="com.airwatch.android.androidwork.launcher" uuid="568bc89d-1df8-4ce9-a041-e5a24acdb7ec">
<parm name="SkipCosuSetup" value="True"/>
Enable WiFi in Android Kiosk
By default, WiFi is blocked by kiosk mode. To unlock possibility for user to choose WiFi network do this:
- Check the APP ID of “Settings” application on Android OS;
- Add Settings app APP ID to AirWatch Launcher as Hidden Apps.
Custom XML Profiles for iOS
Custom Settings profile payload allows admins to enter their own XML into a profile and apply the profile to devices.
(For iOS 11.3+ and macOS 10.13.4+)
- XML should contain the complete block of code as listed below, from to .
- Administrators should configure each setting from to as desired.
- If certificates are required, then configure a Certificate payload within the profile and reference the Payload UUID in the Custom Settings payload.
iOS 11.3
Restrictions
<dict>
<key>allowUSBRestrictedMode</key>
<true/>
<key>forceClassroomRequestPermissionToLeaveClasses</key>
<true/>
<key>forceDelayedSoftwareUpdates</key>
<true/>
<key>enforcedSoftwareUpdateDelay</key>
<integer>30</integer>
<key>PayloadDisplayName</key>
<string>Restrictions</string>
<key>PayloadDescription</key>
<string>RestrictionSettings</string>
<key>PayloadIdentifier</key>
<string>7480b205-2e1c-40fe-bd59-b53db434652d.Restrictions</string>
<key>PayloadOrganization</key>
<string></string>
<key>PayloadType</key>
<string>com.apple.applicationaccess</string>
<key>PayloadUUID</key>
<string>99b5b40b-5683-4315-9ec2-f9e014a6XXXX</string>
<key>PayloadVersion</key>
<integer>1</integer>
</dict>
Home Screen Layout
Note
Web clips should be configured and deployed using the current web clip payload UI
<dict>
<key>Dock</key>
<array>
<dict>
<key>Type</key>
<string>WebClip</string>
<key>URL</key>
<string>https://google.com</string>
</dict>
</array>
<key>Pages</key>
<array>
<array>
<dict>
<key>Type</key>
<string>WebClip</string>
<key>URL</key>
<string>https://yahoo.com</string>
</dict>
<dict>
<key>Type</key>
<string>Folder</string>
<key>DisplayName</key>
<string>My Web Clip</string>
<key>Pages</key>
<array>
<array>
<dict>
<key>Type</key>
<string>WebClip</string>
<key>URL</key>
<string>https://www.vmware.com</string>
</dict>
</array>
</array>
</dict>
</array>
</array>
<key>PayloadDisplayName</key>
<string>Home Screen Layout</string>
<key>PayloadDescription</key>
<string>HomeScreenLayout</string>
<key>PayloadIdentifier</key>
<string>97213d06-b750-466b-8a89-782d8a406f86.Home Screen Layout</string>
<key>PayloadOrganization</key>
<string></string>
<key>PayloadType</key>
<string>com.apple.homescreenlayout</string>
<key>PayloadUUID</key>
<string>2fa8fe03-30fa-4189-aa00-ba752eabXXXX</string>
<key>PayloadVersion</key>
<integer>1</integer>
</dict
Enable Bluetooth Command
Unlike custom profiles, the payload content and UUID are not required for custom commands. This command will not take place if the Allow Bluetooth Settings Modification restriction is enforced.
<dict>
<key>RequestType</key>
<string>Settings</string>
<key>Settings</key>
<array>
<dict>
<key>Item</key>
<string>Bluetooth</string>
<key>Enabled</key>
<true/>
</dict>
</array>
</dict>
macOS 10.13.4 Custom XML
Restrictions
<dict>
<key>forceDelayedSoftwareUpdates</key>
<true/>
<key>enforcedSoftwareUpdateDelay</key>
<integer>30</integer>
<key>PayloadDisplayName</key>
<string>Restrictions</string>
<key>PayloadDescription</key>
<string>RestrictionSettings</string>
<key>PayloadIdentifier</key>
<string>7480b205-2e1c-40fe-bd59-b53db434652d.Restrictions</string>
<key>PayloadOrganization</key>
<string></string>
<key>PayloadType</key>
<string>com.apple.applicationaccess</string>
<key>PayloadUUID</key>
<string>99b5b40b-5683-4315-9ec2-f9e014a6XXXX</string>
<key>PayloadVersion</key>
<integer>1</integer>
</dict>
Autonomous Single App (ASA) Mode
<dict>
<key>AllowedApplications</key>
<array>
<dict>
<key>BundleIdentifier</key>
<string>com.sample.app1</string>
<key>TeamIdentifier</key>
<string>ABCDEFG1HI</string>
</dict>
<dict>
<key>BundleIdentifier</key>
<string>com.sample.app2</string>
<key>TeamIdentifier</key>
<string>ABCDEFG1HI</string>
</dict>
</array>
<key>PayloadDisplayName</key>
<string>Autonomous Single App Mode</string>
<key>PayloadDescription</key>
<string>AutonomousSingleAppMode</string>
<key>PayloadIdentifier</key>
<string>7480b205-2e1c-40fe-bd59-b53db434652d.AutonomousSingleAppMode</string>
<key>PayloadOrganization</key>
<string></string>
<key>PayloadType</key>
<string>com.apple.asam</string>
<key>PayloadUUID</key>
<string>91b5e40b-5683-4376-9ec2-f9e214a6XXXX</string>
<key>PayloadVersion</key>
<integer>1</integer>
</dict
Warning
- Can only be installed on User Approved MDM Enrolled devices. Must be installed as Device profile. Only one payload allowed per machine.
- To be granted access, applications must be signed with the specified Bundle Identifier and Team Identifier using an Apple-issued production developer certificate. Applications must specify the com.apple.developer.assessment entitlement with a value of true.
- The application’s bundle identifier. BundleIdentifier must be unique. If two dictionaries contain the same BundleIdentifier but different TeamIdentifiers, this will be considered a hard error and the payload will not be installed.
To check if the .app has the correct entitlement noted above:
codesign –d –entitlements - /Applications/Example.app
This will print out XML with the entitlements. It needs to have the com.apple.developer.assessment entitlement with a value of true.
To get the Bundle & Team Identifier for an .app:
codesign –dvvvv /Applications/Example.app
The Bundle Identifier will be in the ‘Identifier’ field. The Team Identifier will be a 10 character string in the ‘TeamIdentifier’ field.
Content Caching
<dict>
<key>AllowPersonalCaching</key>
<true/>
<key>AllowSharedCaching</key>
<true/>
<key>AutoActivation</key>
<true/>
<key>CacheLimit</key>
<integer>100000000</integer> <!--100 MB example-->
<key>DataPath</key>
<string>/Library/Application Support/Apple/AssetCache/Data</string>
<key>DenyTetheredCaching</key>
<false/>
<key>ListenRanges</key>
<array>
<dict>
<key>type</key>
<string>IPV4</string>
<key>first</key>
<string>0.0.0.0</string>
<key>last</key>
<string>255.255.255.255</string>
</dict>
</array>
<key>ListenRangesOnly</key>
<false/>
<key>ListenWithPeersAndParents</key>
<true/>
<key>LocalSubnetsOnly</key>
<true/>
<key>LogClientIdentity</key>
<false/>
<key>Parents</key>
<array>
<string>1.1.1.1</string>
<string>2.2.2.2</string>
</array>
<key>ParentSelectionPolicy</key>
<string>round-robin</string> <!-- Possible values are
round-robin, first-available, url-path-hash, random, and
sticky-available-->
<key>PeerFilterRanges</key>
<array>
<dict>
<key>type</key>
<string>IPV4</string>
<key>first</key>
<string>0.0.0.0</string>
<key>last</key>
<string>255.255.255.255</string>
</dict>
</array>
<key>PeerListenRanges</key>
<array>
<dict>
<key>type</key>
<string>IPV4</string>
<key>first</key>
<string>0.0.0.0</string>
<key>last</key>
<string>255.255.255.255</string>
</dict>
</array>
<key>PeerLocalSubnetsOnly</key>
<true/>
<key>Port</key>
<integer>0</integer>
<key>PublicRanges</key>
<array>
<dict>
<key>type</key>
<string>IPV4</string>
<key>first</key>
<string>0.0.0.0</string>
<key>last</key>
<string>255.255.255.255</string>
</dict>
</array>
<key>PayloadDisplayName</key>
<string>Content Caching</string>
<key>PayloadDescription</key>
<string>ContentCaching</string>
<key>PayloadIdentifier</key>
<string>7480b205-2e1c-40fe-bd59-b53db434652d.ContentCaching</string>
<key>PayloadOrganization</key>
<string></string>
<key>PayloadType</key>
<string>com.apple.AssetCache.managed</string>
<key>PayloadUUID</key>
<string>98f5b40b-5683-2415-9ec2-f9e014a6XXXX</string>
<key>PayloadVersion</key>
<integer>1</integer>
</dict>
Subsections of WS1 SQL Database
Admin Password Reset
Changing an administrator’s password
Changing an administrator’s password requires first getting the CoreUserID for that user, and then updating the password and password salt for that user in the database.
Getting the CoreUserID
Run the following SQL query to get the CoreUserID for the user. Make sure to replace with the administrator’s username. If more than one result are returned, make sure you identify the right entry.
Select * from CoreUser where UserName = '<USERNAME>';
Updating the password and password salt
The following query will update the administrator’s password. Make sure to replace with the proper CoreUserID identified in the first section. Have updating the password, log in with the account and reset the password to something else.
-- This Updates the password to 'a123b456c789'
UPDATE CoreUser SET Password = 'c2otxl1SURGxVibCX1K9IJvyizYl4ylnIIfXhwNtfe1iCuuVM8LNPK1oWWSwE3C3BB3AYxspGqrfXaVnryxjzw==' ,
PasswordSalt = '6eklCHP5ixaMT9RREfQbmi4Z2jc=' WHERE CoreUserID = <COREUSERID>
-- This Updates the password to 'Password123!'
UPDATE dbo.CoreUser SET Password = 'awhash4:IGdeYQ7eaHqyh3g6RelU0zkbHjoRW111/dg2xyqjiK0=:100000:K91XOTYtcw/20ZxDOUm7pe3DVLFA3XrzuUAYtWdl21M=' WHERE UserName = 'Administrator'
Account lock-out
Check IsLockedOut flag - it must be set to 0 for active/unlocked account.
UPDATE dbo.CoreUser SET IsLockedOut = 0 WHERE UserName = 'Administrator'
UPDATE dbo.CoreUser SET LastLoginAttempts = 0 WHERE UserName = 'Administrator'
APNs Troubleshooting
External links
APNs incorrect renewal
- First you need to get the GroupID of the organization group that has APNs configured. In Chrome, you can do this by right-clicking the organization group drop down as shown below and choosing Inspect Element. In the HTML, search for data-current-lg and note the GroupID.
- Perform the following SQL query in the database to retrieve information about the APNs certificate:
SELECT DISTINCT DC.CertificateID, DAGS.appleid, DC.SubjectName,
DC.serialnumber, DC.certificate thumprint,
DC.notBefore, DC.NotAfter
FROM
dbo.Certificate (NOLOCK) AS DC
JOIN
dbo.ApnsGenerationStatus (NOLOCK) as DAGS
ON DC.CertificateID = DAGS.IssuedCertID
WHERE
DAGS.LocationGroupID = 10289
ORDER BY NotBefore ASC
- Validate the results of this query. In a correct renewal, both the appleid and SubjectName columns of the results should match between the latest certificate and the previous certificate. You can tell which certificate is which based on the notBefore and notAfter columns, which match the validity dates of the certificate.
CertificateID |
appleid |
SubjectName |
serialnumber |
thumprint |
notBefore |
NotAfter |
378672 |
userA@yourcompany.com |
C=US,CN=APSP:06efab8c-1d88-4fed-8c4d-a92f88ea9281,OID.0.9.2342.19200300.100.1.1=com.apple.mgmt.External.06efab8c-1d88-4fed-8c4d-a92f88ea9281 |
52DEAB4B44A18861 |
0x30820C9F0… |
9/5/2014 |
9/5/2015 |
30874966 |
userA@yourcompany.com |
C=US, CN=APSP:06efab8c-1d88-4fed-8c4d-a92f88ea9281, OID.0.9.2342.19200300.100.1.1=com.apple.mgmt.External.06efab8c-1d88-4fed-8c4d-a92f88ea9281 |
0EA275019DF7D704 |
0x30820C975… |
8/28/2015 |
8/27/2016 |
- If the appleid field is different, then the administrator used an incorrect Apple ID when renewing the APNs certificate, and so the SubjectName will not match. The administrator must use the same Apple ID when renewing APNs, or all devices will lose the ability to communicate. If appleid matches but SubjectName is different, then the administrator used the correct Apple ID but chose the wrong certificate it Apple’s portal to renew. They must go back to Apple’s portal and renew the correct certificate.
- Note: All devices enrolled after the notBefore date of an incorrect APNs certificate must re-enroll when the certificate is corrected. These devices will lose communication once the certificate is correct.
- Note: If the administrator previously cleared out the APNs certificate (by selecting the Clear button in the APNs settings page and saving), they will lose communication with all devices. In this event, a database backup is required to restore the tokens, and this process generally cannot be completed on SaaS environments.
APNs Database Queries
Below are a list of helpful SQL queries specifically when troubleshooting APNs. All of the queries below are SELECT statements, which allows us to view the information but not update it.
- This statement will display an overall view of the location group. This allows us to look at certain fields as well as the Location Group ID for the location where the APNs certificate is being uploaded. By inputting the name, you can receive the Group ID.
SELECT * FROM dbo.locationgroup lg (NOLOCK) WHERE lg.Name like '%%'
- This statement displays the APNs settings at and above the LG provided. We are looking for these first four fields and which have values that are not NULL. It should only be configured at one level and inherited below. You simply need to input the group ID found in the first query.
SELECT lgf.LGlvl, lg.MDMAppleApplicationID, lg.MDMAppleDeviceProfileID, lg.AppleMDMEnabled, lg.MDMSettingsInheritable,lg.*
FROM dbo.LocationGroup lg (NOLOCK)
JOIN dbo.LocationGroupFlat lgf (NOLOCK)
ON lgf.ParentLocationGroupID = lg.LocationGroupID
WHERE lgf.ChildLocationGroupID = ###
ORDER BY lgf.LGlvl DESC
- This statement shows the APNs settings at or below the LG provided. You simply need to input the group ID found in the first query. The first four columns after the level should be NULL at every level below the level where the cert was uploaded at. You can also view the levels of the tree in the first column. If you do see multiple levels with values not being NULL, we can look at the different APNs certs and the number of devices associated. This will require some update statements which a DB or T3 Member can assist with.
SELECT lgf.LGlvl, lg.MDMAppleApplicationID, lg.MDMAppleDeviceProfileID, lg.AppleMDMEnabled, lg.MDMSettingsInheritable,lg.*
FROM dbo.LocationGroup lg (NOLOCK)
JOIN dbo.LocationGroupFlat lgf (NOLOCK)
ON lgf.ChildLocationGroupID = lg.LocationGroupID
WHERE lgf.ParentLocationGroupID = ###
ORDER BY lgf.LGlvl ASC
- This statement will display all the APNs certificates in the environment. You simply need to input the group ID found in the first query. It is important to match this with the previous query so we know which cert is at each level. It is also important to view if the topic is the same for different certs. If it is, we can change the application ID associated to the device without any end user interaction or effect. If the topics are different, devices will need to re-enroll. From this query, you will have the ProductionCertificateID and ApplicationID.
SELECT a.ProductionCertificateID, lg.LocationGroupID, lg.Name, a.*
FROM deviceApplication.Application a (NOLOCK)
JOIN dbo.LocationGroupFlat lgf (NOLOCK)
ON a.RootLocationGroupID = lgf.ChildLocationGroupID
JOIN dbo.LocationGroup lg (NOLOCK)
ON lgf.ChildLocationGroupID = lg.LocationGroupID
WHERE lgf.ParentLocationGroupID = ###
AND a.PackageID like '%com.apple.mgmt%'
- This statement will confirm if the certificates are alive by viewing the date. You simply need to input the ProductionCertificateID found using the queries previous.
SELECT *
FROM dbo.Certificate c (NOLOCK)
where c.CertificateID IN (###)
- This statement will tell the devices that are associated with the Application ID. You simply need to input the ApplicationID found using the queries previous. This will assist if the customer has multiple APNs certificates active in the console and you are trying to determine which to use. Obviously, we will want to choose the cert with the most devices for the least end user impact.
SELECT *
FROM deviceApplication.ApplicationDeviceCredential adc (NOLOCK)
WHERE adc.ApplicationId IN (###)
Condensed query:
SELECT adc.DeviceID, a.ApplicationID, a.PackageID, adc.NotificationID
FROM deviceApplication.ApplicationDeviceCredential adc
LEFT JOIN
deviceApplication.Application a
ON adc.ApplicationID = a.ApplicationIDWHERE DeviceID =
- This statement will show the serial number associated with the APNs certificate. This will assist if you need to relay this information to Apple to determine if they have a corrupt certificate where AirWatch is unable to continue troubleshooting. Simply input the name of the Location Group where the APNs Certificate is uploaded.
SELECT lg.Name, c.SerialNumber from dbo.Certificate c
inner join deviceApplication.Application a
on ProductionCertificateID = CertificateID
inner join dbo.LocationGroup lg
on lg.MDMAppleAPplicationID = a.ApplicationID
where lg.Name like '%__%'
- This statement will show you all the APNs certificates that have been uploaded to the console at or below the LG provided. You simply need to input the group ID found in the first query.
SELECT c.SerialNumber,c.SubjectName, c.NotBefore, c.NotAfter, ags.*
FROM dbo.ApnsGenerationStatus ags (NOLOCK)
JOIN dbo.Certificate c (NOLOCK)
ON c.CertificateID = ags.IssuedCertID
WHERE ags.LocationGroupID IN (###)
ORDER BY CreatedDate DESC
PushMagic Token
In Database:
SELECT *
FROM dbo.AppleDeviceEx
WHERE DeviceID = ###
APNs in logs
Console (CN) server logs:
- MessengingServiceLog (C:\AirWatch → Logs → Services)
Search for:
- gateway.push.apple.com
- APNs token
- PushMagic token
Device Command Queues
Device operation sequence
A. AirWatch server sends out a PUSH notifications to device:
- AirWatch prepares command, stores in command queue (SQL)
- AirWatch sends check-in request to Messaging Service, one of the following:
- Google Cloud Messaging: formerly C2DM (Cloud 2 Device Messaging) deprecated 2012 → Google Cloud Messaging (GCM) deprecated April 2019 → Firebase Cloud Messaging (FCM)
- Internal message bus for Android & Windows devices: AirWatch Cloud Messenger (AWCM)
- Message bus for iOS: Apple Push Notification Service (APNs)
- Messaging Service relays check-in request to managed device
- Managed device checks into MDM server
- MDM Server delivers commands to device, one-by-one
B. Device performs the commands;
C. Device contacts AirWatch server to report the result of the last commands and request new commands, if any.
Command Queue
Every pending command is in SQL database:
SELECT * FROM deviceCommandQueue.DeviceQueue;
Every processed command is in SQL database:
SELECT * FROM deviceCommandQueue.DeviceQueueAudit;
Commands description is in SQL database:
SELECT * FROM deviceCommandQueue.DeviceQueueCommand;
Command statuses in SQL:
SELECT * FROM deviceCommandQueue.DeviceQueueStatus;
Basic command types & statuses…
Device Management commands
- Lock device
- Query device
- Send Message to device
Application commands
Profile commands
Normal Command Status Flow
1 = Queued
2 = Pending
3 = Processed
7 = Held (if batching is enabled)
Troubleshooting Command Logs
Device Services (DS) server (front-end communication with managed device) logs:
- DeviceServicesLogs (
C:\AirWatch → Logs → DeviceServices
)
- InterrogatorQueueService (
C:\AirWatch → Logs → Services
)
Console (CN) server (communication with external services: APNs, GCM/FCM, AWCM, ACC etc.) logs:
- MessengingServiceLog (
C:\AirWatch → Logs → Services
)
- BulkProcessingServiceLog (
C:\AirWatch → Logs → Services
)
- SchedulerServiceLog (
C:\AirWatch → Logs → Services
)
Targeted logging: AirWatch console → Device Details → More → Targeted Logging
Device Status
Workspace ONE Registered – Unmanaged WS1 device (MAM Only)
ManagedBy column in dbo.Device = 2
EnrollmentCategoryID column in dbo.DeviceExtendedProperties = 1
Query:
select d.ManagedBy, dep.EnrollmentCategoryID, d.DeviceID from dbo.Device d INNER JOIN
dbo.DeviceExtendedProperties dep
ON
d.DeviceID = dep.DeviceID
where d.ManagedBy = 2
AND dep.EnrollmentCategoryID = 1
Workspace ONE Managed – MDM on the device, but did not enroll through agent (Adaptive Mgmt)
ManagedBy column in dbo.Device = 1
EnrollmentCategoryID column in dbo.DeviceExtendedProperties = 1
Query:
select d.ManagedBy, dep.EnrollmentCategoryID, d.DeviceID from dbo.Device d INNER JOIN
dbo.DeviceExtendedProperties dep
ON
d.DeviceID = dep.DeviceID
where d.ManagedBy = 1
AND dep.EnrollmentCategoryID = 1
Workspace ONE Managed – MDM on the device, but did not enroll through agent (Direct Enrollment)
ManagedBy column in dbo.Device = 1
EnrollmentCategoryID column in dbo.DeviceExtendedProperties = 3
Query:
select d.ManagedBy, dep.EnrollmentCategoryID, d.DeviceID from dbo.Device d INNER JOIN
dbo.DeviceExtendedProperties dep
ON
d.DeviceID = dep.DeviceID
where d.ManagedBy = 1
AND dep.EnrollmentCategoryID = 3
Agent Enrollment with Workspace ONE – Workspace ONE as an app catalog, pushed as a managed app.
ManagedBy column in dbo.Device = 1
EnrollmentCategoryID column in dbo.DeviceExtendedProperties = NULL or 0
Query:
select d.ManagedBy, dep.EnrollmentCategoryID, d.DeviceID from dbo.Device d INNER JOIN
dbo.DeviceExtendedProperties dep
ON
d.DeviceID = dep.DeviceID
where d.ManagedBy = 1
AND dep.EnrollmentCategoryID IN (NULL, 0)
EnrollmentCategoryID is mapped to following enum:
public enum EnrollmentCategory
{
Unknown = 0,
VmWorkspace = 1,
Offline = 2,
WorkSpaceOneDirectEnrollment = 3
}
ManagedBy is mapped to following enum:
public enum DeviceManagedBy
{
/// <summary>
/// Unknown value
/// </summary>
Unknown = 0,
/// <summary>
/// Managed By MDM
/// </summary>
MDM = 1,
/// <summary>
/// Managed by Workspace (newer terminology is 'Container')
/// </summary>
Workspace = 2,
/// <summary>
/// Managed by Application Catalog (Web MAM enrollment)
/// </summary>
AppCatalog = 3,
/// <summary>
/// Managed by Application
/// </summary>
AppLevel = 4,
/// <summary>
/// managed by VmWorkspace
/// </summary>
VmWorkspace = 5,
/// <summary>
/// managed by Offline (Enrolled Offline)
/// </summary>
Offline = 6
}
GPS History Table
- GPS Poll Time Interval - X mins (configured in AirWatch Console, for example, in Android Agent settings)
- Data Transmit Interval - Y mins
Privacy settings for “GPS Data” needs to be set to Collect and Display for required device Ownership type.
The samples which are transmitted by the Agent as per the Y interval are all stored in the database in the dbo.GPSLog table. To see if any samples have been reported for your specific device, use the following query:
SELECT * from dbo.LogSample (nolock) ls
inner join dbo.GPSLog (nolock) gl
on ls.logsampleID = gl.logsampleid
WHERE deviceid = ####
Under device details > Location tab on console, not all location data points will be reported:
- Consecutive duplicate samples will not be reflected
Note
Consecutive samples which are not at least 100 ft. apart will not be listed corresponding to the particular time-stamp as can be seen in the DB.
Insert GPS History directly
Copy GPS coordinates from one device to another:
select * from device where Name like '%User01%' --deviceID = 7
select * from device where Name like '%User02%' --deviceID = 52
Insert dbo.LogSample ([DeviceUid] , [IpAddress] ,[SampleTime],[TransmitTime] ,[DeviceID] )
select
--[LogSampleID] [int] IDENTITY(1,1) NOT NULL,
[DeviceUid] ,[IpAddress] ,[SampleTime],[TransmitTime] ,
52--[DeviceID] ,
from dbo.LogSample ls where deviceId = 7 and LogSampleID =8
select * from dbo.LogSample ls where deviceId = 52 --LogSampleID =9
Insert dbo.GPSLog ( [LogSampleID] ,[SampleTime],[Latitude] ,[Longitude] ,[Elevation] ,[Speed] ,
[Note] ,[LatitudeInternal] ,[LongitudeInternal] ,[ElevationInternal] , [SpeedInternal]
--, [RowVersion]
)
select 9 ,--[LogSampleID] ,
[SampleTime],[Latitude] ,[Longitude] , [Elevation] ,[Speed] ,[Note] ,[LatitudeInternal] ,[LongitudeInternal] , [ElevationInternal] ,
[SpeedInternal] --, [RowVersion]
from dbo.GPSLog gl where LogSampleID in (select LogSampleID from dbo.LogSample ls where deviceId = 7and LogSampleID =8)
Memcached Activation
Activate usage of Memcached
DECLARE @myid uniqueidentifier
DECLARE @GroupOverrideID int
DECLARE @userID int
DECLARE @CacheEndpoint varchar(255)
DECLARE @EnabledOverrideID int
DECLARE @SolutionOverrideID int
DECLARE @EndpointOverrideID int
SET @myid = NEWID()
SET @userID = 52
SET @CacheEndpoint = '[
{
"name" : "mc_node1",
"address" : "192.168.1.1",
"port" : "11211"
},
{
"name" : "mc_node2",
"address" : "192.168.1.2",
"port" : "11211"
}
]
'
EXEC SystemCodeGroupOverride_Save @GroupOverrideID, 132, 7, NULL, 1, 1, 1, @UserID
IF (@GroupOverrideID IS NULL)
BEGIN
SELECT TOP (1) @GroupOverrideID = scgo.SystemCodeGroupOverrideID
FROM dbo.SystemCodeGroupOverride scgo
WHERE [SystemCodeGroupID] = 132
AND LocationGroupID=7
AND LocationID IS NULL
END
EXEC SystemCodeOverride_Save @EnabledOverrideID, 877, @GroupOverrideID, 'True', @userID
EXEC SystemCodeOverride_Save @SolutionOverrideID, 4147, @GroupOverrideID, '0', @userID
EXEC SystemCodeOverride_Save @EndpointOverrideID, 829, @GroupOverrideID, @CacheEndpoint, @userID
UPDATE dbo.DatabaseVersion set buildkey = CONVERT(varchar(255), @myid)
DELETE from dbo.SystemCodeOverride where SystemCodeid in (829,877,4147)
DELETE from dbo.SystemCodeGroupOverride where SystemCodeGroupID = 132
Check Memcached in SQL
SELECT * from dbo.SystemCodeOverride where SystemCodeID IN (829 , 827, 877, 4147)
Query Internal Apps
Check app versions
Find out the list and version of applications installed on the device by running the following SQL query:
select * from interrogator.applicationlist al
inner join interrogator.application a
on a.applicationid = al.applicationid
where al.deviceid = #affecteddeviceid
How to verify if devices have installed the latest provisioning profile for an internal application?
Search for devices who are reporting the application as installed, but do not have the updated provisioning profile installed
SELECT * FROM interrogator.ApplicationList (NOLOCK)
WHERE DeviceID in (
SELECT iaa1.DeviceID From deviceApplication.InternalAppAssignment (NOLOCK) iaa1
WHERE InternalAppID = {INTERNAL_APP_ID}
AND iaa1.DeviceId NOT IN (
SELECT iaa.DeviceID FROM deviceApplication.InternalAppAssignment (NOLOCK) iaa
JOIN deviceProfile.DeviceProfileDevicePool dpdp
ON iaa.DeviceID = dpdp.DeviceID
WHERE iaa.InternalAppID = {INTERNAL_APP_ID}
AND dpdp.DeviceProfileID = {DEVICE_PROFILE_ID}
)
)
AND ApplicationID = {APPLICATION_ID}
AND IsInstalled = 1
The logic behind this script is:
- Check for devices reporting the provisioning profile as installed (deviceProfile.DeviceProfileDevicePool)
- Check against devices that are supposed to be assigned this application (deviceApplication.InternalAppAssignment)
- Check against devices that are reporting the application as installed (interrogator.ApplicationList)
To find the variables in this query:
- INTERNAL_APP_ID - This is found in the deviceApplication.Application database table, or in the URL when viewing the summary page for the internal application.
- DEVICE_PROFILE_ID - This is found in the deviceApplication.Application table as the ProvisioningDeviceProfileID.
- APPLICATION_ID - This is found in the interrogator.Application table as the ApplicationID. To find this value you can use the following query:
SELECT ia.* FROM interrogator.Application (NOLOCK) ia
JOIN DeviceApplication.Application daa
ON ia.Identifier = daa.PackageID
AND ia.VersionHash = daa.VersionHash
WHERE daa.ApplicationID = {INTERNAL_APP_ID}
Verify if commands are in the command queue to update the provisioning profile
The following query can be used to identify if, for devices that have not yet installed the updated profile, commands have been queued to install it on the next device check in:
SELECT * FROM DeviceCommandQueue.DeviceQueue (nolock) dq
JOIN deviceProfile.DeviceProfileVersion dpv
ON dq.DeviceProfileVersionID = dpv.DeviceProfileVersionID
WHERE dq.CommandID = 13 --CommandID for InstallProvisioningProfile
AND dpv.DeviceProfileID = {DEVICE_PROFILE_ID}
Search for application upload events
The following query is generated by HUB -> Reports and Analytics -> Events -> Console Events in the console, but the console only searches for Last Month, while manual procedure allows to search further back
exec eventLog.EventLog_Search @LocationGroupID=571,@SeverityID=NULL,@eventGroupTypeID=2,@Module=N'Apps',@StartDateTime='2018-06-01 15:32:34.843',@EndDateTime='2018-07-27 15:33:34.843',
@EventDirectionID=NULL,@EventTypeID=0,@EventActionID=0,@StartRowIndex=50,@MaximumRows=1000,@SortExpression=NULL,@SortAscending=1,@SearchText=NULL
Smart Groups Queries
SQL queries for Smart Groups
List all the smart groups present in that particular environment:
select * from smartgroup.smartGroup
Current version of the smart group being used can be found in the table generated by the query below:
select * from smartgroup.SmartGroupVersion where sgdm.SmartGroupId = ##
Details of devices associated with a smart group:
select * from SmartGroup.SmartGroupDeviceMap sgdm Inner Join smartgroup.SmartGroupVersion sgv on sgdm.smartGroupId = sgv.smartGroupID
and sgdm.SmartGroupVersion = sgv.SmartGroupVersion where sgdm.SmartGroupId = ##
Relationship between smart group and entity can be found from the table generated by the query mentioned below:
select * from smartGroup.AWEntitySmartGroupAssignmentMap where SmartGroupId = ##
The device ID of device to which the entity is assigned can be obtained from the tables generated by the queries mentioned below:
- Profile
- InternalApp
- ExternalApp
select * from deviceProfile.ProfileAssignment where ProfileID = ##
select * from [deviceApplication].[InternalAppAssignment] where InternalAppID = ##
select * from [deviceApplication].[ExternalAppAssignment] where ExternalAppID = ##
Devices associated with a smart group can be seen in the table generated by the query mentioned below:
select batchEvaluateLogic from smartGroup.smartGroupRuleLogic where smartGroupOverrideID = ##
Note
Apps and books are together considered as an app in SQL.
Troubleshooting Smart Groups
Warning
Do not name a Smart Group with the same name as an existing User Group - this leads to Smart Group being “locked” from deletion
Problem 1:
- After assigning a smart group to the application and selecting Save & Publish , the app assignment in application grid shows 0/0/0 even though the View Device assignment shows all devices in smart group.
Go to Deployment tab and check the deployment time and if possible change it to 12.00 AM same day. If the time is already set at 12.00 AM, check the admin user’s time zone and compare it with OG time zone. Also check if app wrapping is enabled and whether it is still in progress or not completed.
Problem 2:
- When a smart group is attempted to be deleted, the following error appears: " Deletion is unsuccessful. This smart group is currently being used in assignment. Please remove the smart group from the assignment, and try again."
Check if the smart group is currently assigned to any app, profiles, compliance etc. Check if there is any user group which is having the same name as the smart group which the smart group uses.
Problem 3:
- Certain apps are not being shown in App Catalog for some users.
Create a new smart group which contains these users. After that edit the app and add this smart group in exclusion list.
SQL Recovery
General
VMware AirWatch Installation Guide (9.2), page 17 recommends Recovery Model = Full.
Also, in AirWatch SQL FAQ we see a comment:
When trying to login to the AirWatch console or perform actions within the console you may receive the error message “The transaction log for database ‘AirWatch’ is full.”
This typically means that there is no available space on your SQL server. The common cause for this is backups not being deleted even though newer backups are being done, or lack of transaction log backups. To fix this issue you should first consult your local database admin and make them aware of the issue. The fix that we recommend is to perform a full database backup and then switch the database to simple recovery mode. Your database admin should be familiar with this and know how to perform these actions.
Shrinking the Log File When Disk Space is Full
The log file cannot be shrunk until it is truncated and free space is available. If a log file has grown to the point that it is full or consuming all available disk space, truncate it immediately in order to restore normal database operations. (If the log file has reached the maximum size database property and free disk space is still available, increase the value of maximum size property or set it to ‘unlimited’. A shrink operation should not be required in this case.)
To truncate a log file immediately, set the recovery model to Simple then shrink log file. The following commands are examples of how to do this. They may require editing to tailor them to the customer’s environment.
-- look up the name of the log file
SELECT name as log_file_name FROM sys.database_files WHERE type_desc='LOG'
ALTER DATABASE [AirWatch] SET RECOVERY SIMPLE;
DBCC SHRINKFILE ('AirWatch_log',0,TRUNCATEONLY);
-- replace 'log_file_name' with the actual name of the log file
ALTER DATABASE [AirWatch] MODIFY FILE ( NAME = AirWatch_log, SIZE = 10GB , FILEGROWTH = 128MB );
ALTER DATABASE [AirWatch] SET RECOVERY FULL;
-- Full backup
BACKUP DATABASE [AirWatch] TO DISK = 'E:\MSSQL\Backup\AirWatch_20160123_full.bak' WITH STATS;
Setting the recovery model to Simple erases all of the transaction log records. For this reason, it is essential to create a full backup of the database immediately afterward the shrink procedure to ensure full data recoverability in the event of a database failure.
Preventative Measures
Make sure that the regular database and transaction log backups are scheduled for the AirWatch database. There are various methods: SQL scripts, third party tools, or SQL Server maintenance plans, which can be used.
Full Recovery Mode caveats
Generally the reason why DB log file is large is because the required DB maintenance has not been implemented. Typically, the size of the transaction log file stabilizes when it can hold the maximum number of transactions that can occur between transaction log truncations that are triggered by either checkpoints or transaction log backups.
What Scenarios can cause the Log to Keep Growing?
There are many reasons, but usually these reasons are of the following two patterns
Recovery process overview…
In SQL Server, there are three recovery models - Full, Bulk-Logged and Simple. We’ll ignore Bulk-Logged hybrid model: most people who are in this model are there for a reason and understand recovery models. The two we care about are Simple and Full.
Before we talk about Recovery Models - Let’s talk about recovery in general.
The transaction log file is there for crash/restart recovery. For the rolling forward and rolling back of work that was either done (rolling forward/redo) before a crash or restart and the work that was started but not finished after a crash or restart (rolling back/undo). It is the job of the transaction log to see that a transaction started but never finished (rolled back or crash/restart happened before the transaction committed). In that situation It is the log’s job to say “hey.. this never really finished, let’s roll it back” during recovery. It is also the log’s job to see that you did finish something and that your client application was told it was finished (even if it hadn’t yet hardened to your data file) and say “Hey.. this really happened, let’s roll it forward, let’s make it like the applications think it was” after a restart. Now there is more but that is the main purpose.
The other purpose for a transaction log file is to be able to give us the ability to recover to a point in time due to an “oops” in a database or to guarantee a recovery point in the event of a hardware failure involving the data and/or log files of a database. If this transaction log contains the records of transactions that have been started and finished for recovery, SQL Server can and does then use this information to get a database to where it was before an issue happened. But that isn’t always an available option for us. For that to work we have to have our database in the right recovery model, and we have to take log backups.
Simple Recovery Model
In this model, you are telling SQL Server - I am fine with you using your transaction log file for crash and restart recovery (You really have no choice there.. Look up ACID properties and that should make sense quickly), but once you no longer need it for that crash/restart recovery purpose, go ahead and reuse the log file.
SQL Server listens to this request in Simple Recovery and it only keeps the information it needs to do crash/restart recovery. Once SQL Server is sure it can recover because data is hardened to the data file (more or less), the data that has been hardened is no longer necessary in the log and is marked for truncation - which means it gets re-used.
Full Recovery Model
With Full Recovery, you are telling SQL Server that you want to be able to recover to a specific point in time, as long as your log file is available, or to a specific point in time that is covered by a log backup. In this case when SQL Server reaches the point where it would be safe to truncate the log file in Simple Recovery Model, it will not do that. Instead It lets the log file continue to grow and will allow it to keep growing, until you take a log backup (or run out of space on your log file drive) under normal circumstances.
If you just switch into Full Recovery mode, but never take an initial Full Backup, SQL Server will not honor your request to be in Full Recovery model. Your transaction log will continue to operate as it has in simple until you switch to Full Recovery Model AND Take your first Full Backup.
So, that’s the most common reason for uncontrolled log growth: Being in Full Recovery mode without having any log backups.
Upgrade Rollback
The rollback process contains two phases. First, a backup of the database (that was taken prior to the upgrade) is restored and configured. Next, either backups of any application servers (Console, Device Services, etc) are applied, or the previous version of the AirWatch application is simply reinstalled on those servers after uninstalling the current version.
Restoring a database backup
The following steps can be used to restore a database backup to an existing database server and apply the necessary configuration changes. This process assumes that a backup has been created prior to any upgrades, when the environment was fully functional.
- Before working with the database, make sure that, for any AirWatch application servers (Console, Device Services, etc), all AirWatch services are stopped. Additionally, stop the World Wide Publishing Service. Finally, make sure that IIS is stopped as well.
- Open Microsoft SQL Server Management Studio, right-click on Databases and select Restore Database.
- Under the General tab on the left and within the Source for restore section, select From device, then select ‘…’ button.
- From the Specify Backup page, select Add. Locate the backup, select it, and click OK. The database backup will display on the Specify Backup page. Click OK.
- Select the Restore check box.
- Under Destination for restore, select the To database drop-down list and select the AirWatch database name. Note: This should be the last database in the list, in the event you have multiple backups all named AirWatch and are unsure of which one to select. Click OK to start the database restoration.
On the old database: Next you need to note the user permissions of the old database’s AirWatch SQL Service Account. To do this:
- Open Microsoft SQL Server Management Studio
- Navigate to Security > Logins > to locate your DB User in the Object Explorer, and then right-click and choose Properties.
- Navigate to the Server Roles tab. Write down the roles listed
- Select User Mapping. Write down the user mappings listed and the role membership permissions
Warning
Take note of all of the role memberships for AirWatch, master, model, msdb, and tempdb.
On the new database:
- Delete the AirWatch SQL Service Account, which was created when you restored.
- Create the new AirWatch SQL Service Account. To do this navigate to Security > Logins, right-click, and select New Login.
Warning
This procedure is only for the main AirWatch SQL Service Account If you had any other custom-created SQL accounts you will need to perform this procedure for each of them.
A. Select whether to use Windows or SQL Server authentication. For SQL Server authentication, enter your user credentials. Note: The username needs to exactly match the username of the old database.
B. Uncheck Enforce password policy.
C. Select the AirWatch database as the Default database.
D. For Server Role, enter the roles you noted previously.
E. For User Mapping, enter the user mappings and permissions you noted previously. IMPORTANT: This should include all of the permissions that you copied for AirWatch, master, model, msdb, and tempdb.
Next you need to migrate any AirWatch-related jobs.
Warning
The steps below are for the purge job, but any other AirWatch-related jobs need to be similarly migrated using the procedure below
On the old database:
- Navigate to SQL Server Agent > Jobs, right-click <AirWatch_DB> - Purge Expired Sample Data, and select Script Job as > CREATE To > New Query Editor Window.
- Save as the query.
- Transfer the query to your new database.
- On your new database: Execute the query.
Warning
To reiterate, any other AirWatch-related jobs need to be similarly migrated using the procedure above
On the new database: Perform a test query. For example, one for device count, to ensure proper functioning. To do this:
- Right-click on the AirWatch database under Databases and select New Query.
- Enter the query as shown below.
Rename the old database, for example, to AirWatch_OLD. To kill all connections and rename the database, run the following script, replacing ‘AirWatch’ with the name of your old AirWatch DB and AirWatch_OLD with what you would like to rename the old database to.
Restoring a previous version of the application
With the database backup restored, the next step is to restore the proper version of the application. Ideally, there will be a snapshot or backup of each application server from the same time as the database backup. In this case, simply restore these backups in order to restore functionality.
However, if there are no available snapshots or backups of the servers, then the previous version of the AirWatch software must be reinstalled.
In these cases, identify which application servers have an updated version of the application (this will likely be all application servers, but may only be a subset of them). For each of these, ensure that a copy of the AirWatch installer for the correct version has been installed on the server. Uninstall the current version of the software through Computer > Uninstall or change a program by selecting the AirWatch application. With this uninstalled, open previous version of the AirWatch installer and proceed as normal. When configuring the database connection, ensure the configurations used match those of the restored database.
Finally, when both the database and application has been restored properly, ensure that IIS and all AirWatch services (as well as the World Wide Web Publishing service) are properly started.
Changing the database connection string
In some cases, it may be necessary to change the database connection string of an application server without fully reinstalling the software. For example, if a database migration has occurred, the name of the database may have simply changed. Perform the following steps to update the database connection string on each AirWatch application server. Note that this must be done on every application server so that they are pointing to the new database.
Note
- For deployments with dedicated API and AWCM servers: Dedicated API and AWCM servers are considered application servers, similar to the AirWatch Console and Device Services. You should therefore perform the steps below regarding re-pointing app servers on these servers if you have dedicated servers for these components.
- EIS, SEG, ACC/ESC are considered auxiliary components and you do not need to perform this step for these components.
Steps:
- Navigate to AirWatch Root Folder on the application server.
- Navigate to AirWatch X.X\Supplemental Software\Tools\UpdateSQLServerInfo.
- Launch UpdateSQLServerInfo.exe.
- Update the Server Hostname, Database Name, Username and Password. If Windows authentication is being used, the password field may be blank.
- Make sure to restart IIS and all AirWatch services on each server after updating the SQL connection string.
Useful SQL Queries
SQL Database Health
Index Fragmentation Audit
This query provides you with the index fragmentation % for every table in the DB.
/**************************************************************
You can use the following script to determine index
fragmentation by table. This can help to determine
that the database is causing performace problems.
**************************************************************/
SELECT OBJECT_NAME(object_id), index_id, index_type_desc, index_level, avg_fragmentation_in_percent, avg_Page_space_used_in_percent, page_count
FROM sys.dm_db_index_physical_stats(DB_ID(N'AirWatch'), null, null, null, 'SAMPLED')
ORDER BY avg_fragmentation_in_percent DESC
Table Size Audit
This query will provide physical sizing information of all the tables in the database.
/**************************************************************
This query can help you determine what table is taking
up the most disk space and potentially what tables have
too many rows.
**************************************************************/
--If the temp table exisits drop temp table
IF EXISTS ( SELECT * FROM sys.objects
WHERE object_id = OBJECT_ID(N'#Sizes')
AND type in (N'U') )
BEGIN
DROP TABLE #Sizes;
END
--Create temp table, #Sizes
CREATE TABLE #Sizes
(
--Table name
table_name nvarchar(255),
--Number of rows
table_rows char(11),
--Physical space table is using
table_reserved varchar(18),
--Phyiscal space table data is using
table_data varchar(18),
--Physical space idexes are using
table_index_size varchar(18),
--Physical space reserved
table_unused varchar(18)
)
EXEC sp_MSforeachtable @command1='INSERT #Sizes (table_name, table_rows, table_reserved,
table_data, table_index_size, table_unused) EXEC sp_spaceused ''?'''
--This query orders the results by actual phyisical table size
SELECT * FROM #Sizes ORDER BY CAST ( SUBSTRING (table_data, 0, LEN(table_data) - 2) AS InT) DESC
--This query orders the results by row count
--SELECT * FROM #Sizes ORDER BY CAST (table_rows AS int) DESC
Table Row Count Only
A more efficient script to get table row counts.
/**************************************************************
This is a more efficient way to get Row Counts
but will not include any Physical Sizing data
**************************************************************/
SELECT
sc.name +'.'+ ta.name TableName
, SUM(pa.rows) RowCnt
FROM
sys.tables ta
INNER JOIN sys.partitions pa
INNER JOIN sys.schemas sc
ON ta.schema_id = sc.schema_id
WHERE
ta.is_ms_shipped = 0
AND pa.index_id IN (1,0)
GROUP BY
sc.name
, ta.name
ORDER BY
SUM(pa.rows) DESC;
Database IO Stalls
You can use this query to see which queries are causing IO stalls.
/**************************************************************
The table valued dynamic management function,
sys.dm_io_virtual_file_stats provides a breakdown of SQL
Server reads, writes, and io_stalls for a particular
database or transaction log file. IO_stalls is the total
cumulative time, in milliseconds, that users waited for
I/O to be completed on the file since the last restart of SQL Server.
**************************************************************/
SELECT
DB_NAME(fs.database_id) AS [DB Name]
, fs.file_id
, mf.physical_name
, io_stall_read_ms
, num_of_reads
, CAST(io_stall_read_ms / ( 1.0 + num_of_reads ) AS NUMERIC(10 , 1)) AS 'avg_read_stall_ms'
, io_stall_write_ms
, num_of_writes
, CAST(io_stall_write_ms / ( 1.0 + num_of_writes ) AS NUMERIC(10 , 1)) AS 'avg_write_stall_ms'
, io_stall_read_ms + io_stall_write_ms AS io_stalls
, num_of_reads + num_of_writes AS total_io
, CAST(( io_stall_read_ms + io_stall_write_ms ) / ( 1.0 + num_of_reads
+ num_of_writes ) AS NUMERIC(10 ,
1)) AS 'avg_io_stall_ms'
FROM
sys.dm_io_virtual_file_stats(NULL , NULL) AS fs
INNER JOIN sys.master_files AS mf
ON fs.database_id = mf.database_id
AND fs.[file_id] = mf.[file_id]
ORDER BY
avg_io_stall_ms DESC ;
GO
Identify Expensive Operations
/**************************************************************
This query provides you with operations that are expensive
from a database standpoint. This query is useful in
determining what is causing performance problems on a server
**************************************************************/
SELECT TOP 25
DB_NAME(qp.[dbid]) AS dbname
, qp.[dbid]
, qp.objectid
, qp.number
--, qp.query_plan
--the query plan can be *very* useful; enable if desired
, qt.[text]
, SUBSTRING(qt.[text] ,
( qs.statement_start_offset / 2 ) + 1 ,
( ( CASE statement_end_offset
WHEN -1 THEN DATALENGTH(qt.text)
ELSE qs.statement_end_offset
END - qs.statement_start_offset ) / 2 ) + 1)
AS statement_text
, qs.creation_time
, qs.last_execution_time
, qs.execution_count
, qs.total_worker_time / qs.execution_count
AS avg_worker_time
, qs.total_physical_reads / qs.execution_count
AS avg_physical_reads
, qs.total_logical_reads / qs.execution_count
AS avg_logical_reads
, qs.total_logical_writes / qs.execution_count
AS avg_logical_writes
, qs.total_elapsed_time / qs.execution_count
AS avg_elapsed_time
, qs.total_clr_time / qs.execution_count
nbsp; AS avg_clr_time
--, qs.total_worker_time
--, qs.last_worker_time
--, qs.min_worker_time
--, qs.max_worker_time
, qs.total_physical_reads
, qs.last_physical_reads
, qs.min_physical_reads
, qs.max_physical_reads
--, qs.total_logical_reads
--, qs.last_logical_reads
--, qs.min_logical_reads
--, qs.max_logical_reads
--, qs.total_logical_writes
--, qs.last_logical_writes
--, qs.min_logical_writes
--, qs.max_logical_writes
--, qs.total_elapsed_time
--, qs.last_elapsed_time
--, qs.min_elapsed_time
--, qs.max_elapsed_time
--, qs.total_clr_time
--, qs.last_clr_time
--, qs.min_clr_time
--, qs.max_clr_time
--, qs.[sql_handle]
--, qs.statement_start_offset
--, qs.statement_end_offset
--, qs.plan_generation_num
--, qp.encrypted
FROM
sys.dm_exec_query_stats AS qs
CROSS APPLY sys.dm_exec_query_plan(qs.plan_handle) AS qp
CROSS APPLY sys.dm_exec_sql_text(qs.[sql_handle]) AS qt
--sample WHERE
WHERE
last_execution_time > '20120912 12:15'
AND creation_time > '20130101'
AND execution_count > 10
--AND SUBSTRING(qt.text, (qs.statement_start_offset/2) + 1,
--((CASE statement_end_offset
--WHEN -1 THEN DATALENGTH(qt.text)
--ELSE qs.statement_end_offset END
-- - qs.statement_start_offset)/2)
-- + 1)
-- LIKE '%MyText%'
--sample ORDER BY
--ORDER BY qs.execution_count DESC --Frequency
--ORDER BY qs.total_worker_time DESC --CPU
--ORDER BY avg_worker_time DESC --CPU
--ORDER BY qs.total_elapsed_time DESC --Durn
--ORDER BY qs.total_logical_reads DESC --Reads
--ORDER BY qs.total_logical_writes DESC --Writes
--ORDER BY qs.total_physical_reads DESC --PhysicalReads
--ORDER BY avg_worker_time DESC --AvgCPU
--ORDER BY avg_elapsed_time DESC --AvgDurn
--ORDER BY avg_logical_reads DESC --AvgReads
--ORDER BY avg_logical_writes DESC --AvgWrites
ORDER BY
avg_physical_reads DESC --AvgPhysicalReads
General SQL Queries
Table search by column name
This query allows you to see what tables / procedures have a specified column name in them.
/**************************************************************
Example(s) of common column names:
DeviceId, LocationGroupId, ApplicationId, ProfileId,
CoreUserId
**************************************************************/
SELECT * FROM sysobjects WHERE id IN
(SELECT id FROM syscolumns WHERE name LIKE '%ColumnName%')
SP_WHO2 Advanced Query
This allows you to filter on SP_WHO2, which helps when backing up and restoring the AirWatch database.
/**************************************************************
This query may look complex but can really
narrow down if something is connected to
AirWatch or not. When you are upgrading
or restoring a database you need to ensure
that nothing is locking the databse. **************************************************************/
CREATE TABLE #sp_who2(
SPID INT,
Status VARCHAR(1000) NULL,
Login SYSNAME NULL,
HostName SYSNAME NULL,
BlkBy SYSNAME NULL,
DBName SYSNAME NULL,
Command VARCHAR(1000) NULL,
CPUTime INT NULL,
DiskIO INT NULL,
LastBatch VARCHAR(1000) NULL,
ProgramName VARCHAR(1000) NULL,
SPID2 INT
)
INSERT INTO #sp_who2 EXEC sp_who2
SELECT *
FROM #sp_who2
WHERE DBName LIKE '%AirWatch%'
GO
DROP TABLE #sp_who2
GO
Get all devices from Location Group
This query can return all devices under one locationgroup tree. This is an especially helpful query when used with other queries.
/**************************************************************
This script provides all devices residing under
the parent location group. You can combine this
query with other queries by using the WHERE column
IN (SELECT query) filter (this script uses the
same filter to filter the location groups)
**************************************************************/
SELECT *
FROM Device d (nolock)
INNER JOIN Location l (nolock)
ON l.LocationId = d.LocationId
INNER JOIN LocationGroup lg (nolock)
lg.DefaultLocationId = l.LocationId
WHERE lg.LocationGroupID IN
(
/*****************************************************
The below sub query provides all children
Location Groups under one Parent. This can
also be useful with other queries.
***************************************************/
SELECT lgf.ChildLocationGroupID
FROM LocationGroup lg (nolock)
INNER JOIN LocationGroupFlat lgf (nolock)
ON lgf.ParentLocationGroupID = lg.LocationGroupID
WHERE lg.Name LIKE '%Customers LG%'
)
Event Log Search
This query allows you to search the event log. This is imperitive since the event log often times out from the console.
/**************************************************************
This script provides you with a list of modules in the
event log. These can be used to filter the following
query.
**************************************************************/
SELECT DISTINCT el.Module
FROM eventLog.EventLog el (nolock)
/**************************************************************
This query will select information from the event log
and allow you to filter on Modules. Modules provide you
with context of the event that occured. The Name is
the actual name of the event that occured. This will
only show you the last 30 days. You can decrease this
number to improve performance.
You can also filter on the LocationGroupID if you know
the location group name. Keep in mind that some modules
only report at Global.
The Username is the Admin user who performed the task.
sysadmin is the system user for running stored procedures
however some modules (like the device module) show
sysadmin even if a user performed the action.
**************************************************************/
SELECT cu.UserName, el.Module, e.Name, el.*
FROM eventLog.EventLog el (nolock)
INNER JOIN eventLog.Event e (nolock)
ON e.EventId = el.EventId
INNER JOIN CoreUser cu (nolock)
ON cu.CoreUserId = el.ActionBy
WHERE el.CreatedOn > DATEADD(DAY, -30, GETUTCDATE())
AND el.Module LIKE 'Dashboard'
--AND LocationGroupID IN (SELECT LocationGroupID
--FROM LocationGroup WHERE Name LIKE '%AirWatch%')
Role compare script
This script shows you two roles side by side for comparison.
/**************************************************************
Update the values in @Role1 and @Role2
to compare two roles. If you need to find
why a custom role is missing use the next
script.
**************************************************************/
DECLARE @Role1 AS nvarchar(50);
DECLARE @Role2 AS nvarchar(50);
SET @Role1 = 'AirWatch Administrator';
SET @Role2 = 'System Administrator';
SELECT re.ResourceID, c.Name AS 'Module', re.Name, m1.Allow AS 'Role1 Allowed', m2.Allow AS 'Role2 Allowed'
FROM Resource re
INNER JOIN Category c (nolock)
ON c.CategoryID = re.CategoryID
INNER JOIN Mode m1 (nolock)
ON re.ResourceID = m1.ResourceID
INNER JOIN Role r1 (nolock)
ON r1.RoleId = m1.RoleId
INNER JOIN Mode m2 (nolock)
ON re.ResourceID = m2.ResourceID
INNER JOIN Role r2 (nolock)
ON r2.RoleID = m2.RoleID
WHERE r1.Name = @Role1 AND r2.Name = @Role2
ORDER BY Module, ResourceID
Missing Custom Role script
This script shows you two roles side by side for comparison.
/**************************************************************
Update the values in the variables to check
if resources are missing from the role who is
supposed to have higher priviledges.
**************************************************************/
DECLARE @HigherRole AS nvarchar(50);
DECLARE @LowerRole AS nvarchar(50);
SET @HigherRole = 'Role with higher privileges';
SET @LowerRole = 'Role with less privileges';
SELECT m.ModeID, r.Name, re.Name, m.Allow
FROM Mode m (nolock)
INNER JOIN Role r (nolock)
ON r.RoleId = m.RoleId
INNER JOIN Resource re
ON re.ResourceID = m.ResourceID
WHERE r.Name LIKE @LowerRole AND m.Allow = 1 AND
m.ResourceID IN
(SELECT ResourceID FROM Mode m
INNER JOIN Role r
ON r.RoleId = m.RoleId
WHERE r.Name LIKE @HigherRole AND
m.Allow = 0
)
### Status of Events stuck in ‘Processing’ or ‘Failed’ to ‘Ready for Processing’
UPDATE adp.AdpExportTracking
SET [Status] = 2 -- 2 = Completed Event
WHERE [Status] = 1; -- 2 = Pending Event
-- This script updates the status of Error Exports in adp.ADPExportTracking table
SET NOCOUNT ON;
BEGIN TRY
IF OBJECT_ID(N'adp.ADPExportTracking') IS NOT NULL
BEGIN
UPDATE adp.ADPExportTracking
SET [Status] = 2
WHERE [Status] = -1 -- -1 = Failed Status
END
END TRY
BEGIN CATCH
DECLARE @error_severity INT,
@error_state INT,
@error_message NVARCHAR(2048);
SELECT @error_severity = ERROR_SEVERITY(),
@error_state = ERROR_STATE(),
@error_message = ERROR_MESSAGE();
RAISERROR(@error_message, @error_severity, @error_state);
END CATCH
GO
VMware Tunnel in SQL
ws1tunnel.TunnelConfiguration
Column | Description | Type of Data |
---|
Uuid | UUID based primary Key | uniqueidentifier |
Name | Name of the Tunnel Configuration | nvarchar(255) |
Enabled | If Tunnel Configuration is enabled or not | bit |
PacFileSupportEnabled | If PAC File Upload support is enabled or not | bit |
OrganizationGroupUuid | UUID based Organization Group Id | uniqueidentifier |
VpnVersion | Vpn Config version | nvarchar(255) |
VpnVersionUtcDateTime | VpnConfigVersionUtcDateTime | nvarchar(255) |
CreatedBy | Core user who created this record | uniqueidentifier |
CreatedOn | Created on Date | datetime |
ModifiedBy | Core user who modified this | uniqueidentifier |
ModifiedOn | Modified Date | datetime |
ws1tunnel.TunnelClientConfiguration
Column | Description | Type of Data |
---|
Uuid | UUID based primary Key | uniqueidentifier |
TunnelConfigUuid | Uuid of Tunnel Configuration | uniqueidentifier |
AuthenticationType | Authentication Type | int |
CertificateSource | VPN Certificate Source | nvarchar(255) |
CertificateAuthorityUuid | VPN CA Id. | uniqueidentifier |
CertificateTemplateUuid | Airwatch VPN CA Template Id. | uniqueidentifier |
ScepCertificateTemplateUuid | SCEP Certificate Template ID for Client Certificate for Tunne | uniqueidentifier |
ProfileScepCertificateTemplateUuid | the Airwatch CA template id for SCEP template with key 4096 | uniqueidentifier |
CreatedBy | Core user who created this record | uniqueidentifier |
CreatedOn | Created on Date | datetime |
ModifiedBy | Core user who modified this | uniqueidentifier |
ModifiedOn | Modified Date | datetime |
ws1tunnel.TunnelServerConfiguration
Column | Description | Type of Data |
---|
Uuid | UUID based primary Key for the table | uniqueidentifier |
TunnelConfigUuid | Uuid of Tunnel Configuration | uniqueidentifier |
AwCertificateThumbprint | VPN Certificate Thumbprint | nvarchar(255) |
AwIdentifier | Unique string for AWCM to identify an installation of AirWatch for VPN | nvarchar(1024) |
VpnIdentifier | Unique string for AWCM to identify an installation of VPN server | nvarchar(1024) |
TunnelConfigurationType | Whether VPN is in a standard or endpoint-relay configuration. | int |
EndpointHost | VPN server endpoint Host name | nvarchar(255) |
RelayEndpointPort | VPN server port | int |
VpnServerHostName | VPN server Host name | nvarchar(255) |
VpnServerPort | VPN server port | int |
PublicSslVPN | Whether to use a public SSL certificate for vpn. | bit |
LogLevel | VPN logging level | int |
ApiViaProxy | Whether to enable API and AWCM outbound calls via proxy. | bit |
AccessLogMode | AccessLogMode (rsyslog or file) | int |
EnableAccessLog | Vpn EnableAccessLog | bit |
VpnServerConnectionType | VPN server connection type | int |
SyslogHost | Syslog Hostname | nvarchar(255) |
SyslogPort | Syslog Port | int |
CreatedBy | Core user who created this record | uniqueidentifier |
CreatedOn | Created on Date | datetime |
ModifiedBy | Core user who modified this | uniqueidentifier |
ModifiedOn | Modified Date | datetime |
ws1tunnel.ConfigCertificateMapping
Column | Description | Type of Data |
---|
Uuid | UUID based primary Key for the table | uniqueidentifier |
TunnelConfigUuid | Reference Identifier to TunnelConfiguration table | uniqueidentifier |
CertificateUuid | UUID for Certificate | uniqueidentifier |
CertificateType | Certficate type | int |
IsActive | Actively used certificate | bit |
CreatedBy | User ID that created this record | uniqueidentifier |
CreatedOn | DateTime when the record was created in the DB | datetime |
ModifiedBy | User ID that modified this record | uniqueidentifier |
ModifiedOn | DateTime when the record was last modified in the DB | datetime |
ws1tunnel.TunnelTestConnectionStatus
Column | Description | Type of Data |
---|
Uuid | UUID based primary Key for the table | uniqueidentifier |
TunnelConfigUuid | Reference Identifier to TunnelConfiguration table | uniqueidentifier |
TestConnectionState | Current state of Test connection | nvarchar(20) |
TestConnectionInitiatedTime | Time at which test connection was initiated | datetime |
AWCMReachable | AWCM is Reachable or not | bit |
AWCMConnectionFailureReason | Reason for AWCMC not being reachable | nvarchar(50) |
TestConnectionResult | Test connection was successful or not | bit |
CreatedBy | Created by for the Test connection status record | uniqueidentifier |
CreatedOn | Created on for the Test connection status record | datetime |
ModifiedBy | Modified by for the Test connection status record | uniqueidentifier |
ModifiedOn | Modified on for the Test connection status record | datetime |
WS1 UEM DB Important Tables
Each application uploaded to AirWatch would have its own row in the deviceApplication.Application table. Each column contains specific attributes for that object. For example, an application in the deviceApplication.Application table would have a column for Name, PackageID, LocationGroup, etc.
Device Management Tables
dbo.DeviceOperatingSystem: OS versions supported by AirWatch Console in current version
dbo.DeviceModelInfo: Device models (for Apple) or Android vendors supported by AirWatch Console in current version
User Management Tables
dbo.CoreUser: Admins – UserName, TimeZone, LastLogin, etc.
mobileManagement.EnrollmentUser: End users – UserName, LocationGroupID, DisplayName, UserPrincipleName, etc.
mobileManagement.CurrentDeviceEnrollmentUser: Ties enrolled devices to enrollment users
dbo.UserGroup: FriendlyName,DN,RootLocationGroupID, domain, etc
MAM Tables
deviceApplication.Application: Internal applications
deviceApplication.RecommendedExternalApplication: All things public application
deviceApplication.VPPLicensePool: SmartGroup, Allocated licenses, Redeemed licenses, etc.
deviceApplication.ApplicationGroup: For blacklists/whitelists/required lists of apps
smartGroup.SmartGroup: info about the Smart Group
smartGroup.AWEntitySmartGroupAssignmentMap: Maps a Smart Group to the App/Book that uses it for assignment
MEM Tables
mobileEmailGateway.MEMConfig
Contains attributes related to the Email Configuration, including pre-7.1 sets of System Codes
mobileEmailGateway.MEMDevice
Contains all the attributes for each device, whether managed or unmanaged
mobileEmailGateway.MEMDeviceActivity
Contains the status and activity info for each device
mobileEmailGateway.MEMConfigProfile
Contains associations between MEMConfig and EAS device profiles
mobileEmailGateway.MEMDeviceConfig
Contains associations between MEMConfig and MEMDevice. When an enrolled device receives an EAS profile, MEMDevice will receive corresponding MEMConfig
mobileEmailGateway.MEMDeviceDiagnostic
Contains diagnostics data for devices
mobileEmailGateway.{all the rest}
Email Policy configurations, such as attachment encryption or OS policies
MCM Tables
enterpriseContent.Content
All settings pertaining to uploaded content – offlineViewing, allowEmail, Name, etc.
enterpriseContent.ContentRepository
Details for repositories – Name, Link, AuthenticationUsername, User/Admin Repository
enterpriseContent.ContentMap
Links DeviceID, EnrollmentUserID, and ContentVersionID
enterpriseContent.ContentVersion
BlobID, Version, Size, Author, etc
MDM Tables
dbo.Device
Device centric information – OS, Friendly Name, Serial Number, Phone Number, Enrollment Date, etc.
Profiles
DeviceProfile.DeviceProfile
Settings for profiles – Name, Description, DevicePlatformID, ProfileAssignmentTypeID, etc.
deviceProfile.DeviceProfileVersion
Shows historic versions of a profile and their modified dates
deviceProfile.DeviceProfileDevicePool
Maps devices to profiles and profileVersions
deviceProfile.deviceProfileSettingValue
Maps actual profile settings to profileVersions
Telecom
dbo.Plan
Telecom plan details
dbo.PlanAssignmentRule
Assignment rule details
dbo.CellCard
Cellular details
MobileDataUsage, CellCardUsage
Interrogator tables
Compliance
DeviceComplianceStatus
Compliance status of devices
Policy
List of policies
Device Samples
ManagedAppList,ProfileListSample, SecurityInformationSample
Interrogator tables
Reports
dbo.Report
Info on all available reports
dbo.ReportSubscription
Created subscriptions
Miscellaneous tables
dbo.Certificate
Thumbprint, SubjectName, Validity Dates, etc.
dbo.LocationGroup
OG info – LocationGroupID, Name, CreatedOn, CreatedBy, MDMSettingsInheritable
deviceCommandQueue.DeviceQueue / deviceQueueAudit
Current commands queued / already executed commands for devices
dbo.SystemCode
All settings and default values configurable in the AirWatch Console - Groups & Settings
EventLog.EventLog
Where console Event Logs pull from
Example of deleting “wipe device” commands from Message Queue
This is needed in case of accidental mass wipe created as result of incorrect compliance policy etc.
SELECT * FROM deviceCommandQueue.DeviceQueue WHERE commandid = 10;
DELETE * FROM deviceCommandQueue.DeviceQueue WHERE commandid = 10;
WS1 UEM DB Monitoring
Recommendations for monitoring on the WS1 UEM database
Monitor |
Descritpion |
Data Files |
Monitor and alert for resizing when free space in data files drops below 10%. |
Transaction Logs |
Monitor and resize if free space in log drops below 10%. |
Waiting Tasks |
Waiting tasks in the SQL activity monitor must be under 10 on average. Ideally waiting tasks should be between 0 and 2 when compared to 20,000 batch requests per second. |
Index Rebuild |
Monitor for fragmentation between 10% and 29%. Reorganize with an update of statistics. Indexes with fragmentation greater than 29% should be rebuilt. |
SQL Server CPU |
Monitor sustained high CPU utilization (Over 90% for a 15 minute duration). |
SQL Server Job History |
Monitor failed SQL Server Agent Jobs (in particular, AirWatch Jobs). |
SQL Server Page Life Expectancy |
Monitor SQL Server Page Life Expectancy (dropping below 3000). |
SQL Server Disk Space |
Monitor disk space usage on all Data and Log Drives for ‘AirWatch’ and ‘tempdb’ Databases. |
SQL Server Disk Queuing |
Monitor Disk Queuing on all Data and Log Drives for ‘AirWatch’ and ‘tempdb’ Databases. Check Disk Queue Length via Task Manager > Performance > Resource Monitor > Dist Tab > Storage. It should average between 2 and 4. It could increase or decrease, but on average it should be between those values. |
Page Life Expectancy |
Page Life Expectancy is an indication of whether the database server has memory pressure. The expected number is over 1,000 (seconds). If it is low, this is a first indicator of memory pressure. This may not be an issue if: |
|
- The PLE is increasing over time. If it is increasing, but is still less than 1,000, then that is a sign of a memory pressure. |
|
- After an index maintenance job, the PLE can be low. This needs to be monitored for a few hours to see if it goes up. |
Index Fragmentation Level |
A high fragmentation level means data retrieval becomes less efficient and reduces database performance. Run the defragmentation job on a nightly basis. The script below shows the fragmentation level (in percent) against all the tables. The recommended fragmentation level is less than 30% when the page size is more than 1,000. |
SELECT OBJECT_NAME(object_id), index_id, index_type_desc, index_level,
avg_fragmentation_in_percent, avg_Page_space_used_in_percent, page_count
FROM sys.dm_db_index_physical_stats(DB_ID(N'AirWatch'), null, null,
null, 'SAMPLED') ORDER BY avg_fragmentation_in_percent DESC
If the database is highly fragmented, it is recommended that you perform an index reorganize or rebuild.
Health Checks
Synthetic transactions are the strongest indicator of a healthy AirWatch environment. They can mimic end user actions (for example, enrollment) and report if there are issues. Many different use cases could be considered, and high-use scenarios should be tested with synthetic transactions. An example synthetic transaction could be:
- Navigate to the AirWatch Console.
- Log in using credentials.
- Navigate to Hub > Reports & Analytics > Reports > List View.
- Run a report.
- Log out.
Typically, a tool like Keynote or AlertSite would be used to generate and monitor synthetic transactions.
WS1 UEM Installation
Articles in section
Installation
❗️All Windows systems, which will be used to deploy AirWatch / Workspace One UEM, in localization settings, there should be language = US-EN.
❗️Special attention to Regional Settings on floating point identifier: it must be a dot, not a comma!
❗️All Windows systems, which will be used to deploy AirWatch / Workspace One UEM, should have all current patches applied. For example, early versions of Win2012R2 have broken ASP.NET and break AirWatch installation.
Legend
- BE - (BackEnd server) WS1 UEM Admin Console
- FE - (FrontEnd server) WS1 UEM Device Services
- SQL - Microsoft SQL Database Server
- UEM - AirWatch / Workspace One UEM
Database Deployment
❗️See first SQL Recommendations page before production or semi-production deployment.
- Install SQL
- Login to the SQL server, launch SQL Management Studio;
- Create a new database. In database settings apply General → Autogrowth / Maximize → File Growth → In Megabytes = 128;
- Choose collation: Options → Collation → SQL_Latin1_General_CP1_CI_AS;
- SQL 2008 and MS SQL 2008R2 are not supported anymore. For MS SQL 2016 choose Options → Compatibility Level = 2014
❗️Issues currently detected with installing Workspace ONE UEM up to version 1909 in Microsoft SQL 2017. Services do not start after install, console does not launch.
As recommended by Microsoft for SQL 2017, services should use the element to improve startup performance. Using this element can also help avoid delays that can cause a time-out and the cancellation of the service startup. See Microsoft KB article.
- Create a user in Mixed-mode SQL (non-domain), with sysadmin permissions for server and db_owner for database. Gice the user permissions for msdb: SQLAgentUserRole, SQLAgentReaderRole, db_datareader roles. Do not forget to cancel password expiration for this user;
- If there is not Internet on DB server - download Microsoft .NET Framework 4.6.2 from Microsoft website for English Windows on separate computer and copy to this server (during setup WorkspaceONE_UEM_DB_XX.YY.Z.K_Setup tries to download the framework itself, with no Internet it may hang the installation process for some time);
- Copy WorkspaceONE_UEM_DB_XX.YY.Z.K_Setup files to the server, launch it. Copy to **C:\Distr**;
❗️Do NOT launch installer from C:\Users\Documents and Settings\Downloads etc folders - long path may cause unpacking error.
- Enter “localhost” in install wizard, login and password of the SQL user, and choose the UEM database;
- ❓️ If the database is created on AlwaysOn Cluster - turn on Using SQL AlwaysON Availability Groups option;
- Wait for installer to end (10-15min). Install progress can be seen as log file growth in c:\AirWatch\AirWatch 1811\Database\AWDatabaseLog.txt* (the log will grow up to 2.3Mb when the installation will finish);
- Check the installation, use SQL Management Studio to launch a script:
select * from dbo.DatabaseVersion;
The answer should be the UEM version number.
❓️ For AlwaysOn cluster - do not forget to clone the database Jobs on the other cluster nodes!
Device Services Front-End (FE) Server
- Enter Windows Server Manager and check the following roles/features (double-check official doc for feature list):
- Web Server (IIS)
- Web Server (IIS) → Web Server → Common → Static Content, Default Document, Directory Browsing, HTTP Errors, HTTP Redirection
- Web Server (IIS) → Web Server → Performance → Dynamic Content Compression
- Web Server (IIS) → Web Server → ASP
- Web Server (IIS) → Web Server → ASP.NET 4.5
- Web Server (IIS) → Web Server → Security → IP & Domain Restrictions
- Web Server (IIS) → Web Server → Health & Diagnostics → Request Monitor
- Web Server (IIS) → Web Server → Application Development → Server Side Includes
- .NET Framework 4.5 → WCF → HTTP Activation
- Message Queuing
- Telnet Client
❗️ DON NOT turn on Web Server (IIS) → Web Server → Common → WebDav Publishing - this will lead to multiple bugs in managing iOS devices
-
❓️ If there is not Internet on FE server - download and install NET Framework 4.6.2 (Microsoft .NET Framework 4.6.2 (Offline Installer) for Windows 7 SP1, Windows 8.1, Windows Server 2008 R2 SP1, Windows Server 2012 and Windows Server 2012 R2). Reboot server;
-
❓️ If there is not Internet on FE server - download and install URL Rewrite Module 2.0 (https://go.microsoft.com/?linkid=9722532) for IIS. Old version of Rewrite Module 2.0 provided on this page as attachment in case of need;
-
Upload an external certificate in PFX format with private key. Password used to protect the certificate MUST be 6+ characters long. Short password will lead to problems with AWCM Java keystore! Install the certificate into Local Machine account, leaving Automatic Detect option for certificate type. Also install any root and intermediate certificates of the certificate trust chain. Subject Alternative Name of the certificate MUST contain the external DNS name of the server!
-
Check correct start config of IIS - use browser to go to http://127.0.0.1/ (start page of IIS must be present)
-
Go to IIS admin console, bind the certificate: in sites tree choose Default Web Site → Bindings menu → Add.., choose https, in SSL Certificates list choose the certificate from previous step. Enter the external DNS name of the server, which is written in the certificate.
❗️Port binding is needed ONLY for Device Service and Console Service.
- Launch installer WorkspaceONE_UEM_Application_X.X.X.X_Full_Install. Choose Continue setup without importing/exporting config file;
- In modules selection choose only Device Services, select This feature will not be available for Admin Console, continue installation;
❗️For AirWatch 9.2.2+: during installation, AirWatch installer deploys SQL Native Client, which may not have enough time to initialize during the work of the wizard. During SQL check, an error may be generated, that SQL is not found. Press Cancel and reboot the server, then re-launch the setup process.
- Enter SQL data: in full database name, only enter the server name, do not enter SQL Instance name;
- Specify the DNS name for reaching the server by HTTPS from outside and inside. Do not choose SSL Offload - it is much easier to make all connections as HTTPS and then edit configuration;
⭐️ Instead of choosing different DNS names and then have issues with AWCM, I recommend to enter the same external name for Device Services and Web Console (check Same as above? option). After this, make an alias on the local DNS server, or use the hosts file on Admin Console/BE server to alias the external name of Directory Services/FE to an internal IP address.
- Choose Default Web Site as install target;
- Leave AWCM listening IP as 0.0.0.0 since it is installed locally, and port 2001 for connection. Install the PFX certificate and enter its’ password;
❗️The PFX certificate MUST be created with Export All Properties option! Or the Java keytool will not be able to import it into awcm.keystore, and it will not give errors in the log! But AWCM will not work!
- Choose Implicit Clustering (do not cluster AWCM);
- Wait for install completion. AirWatch Certificate Installation Wizard will open, click Next and choose SQL Authentication. If Internet is accessible, a code must be entered. For offline installation, click Get File and save the *.plist fiel on disk;
- Go to my.workspaceone.com: My Workspace One menu → My Company → Certificate Signing Portal → Authorize Install → Generate a token (for Internet access);
OR
- My Workspace One menu → My Company → Certificate Signing Portal → Authorize Install → Upload Your File (for offline), and upload *.plist file.
- Save the certs.plist answer file and upload it in the installation wizard, thus ending the installation.
❗️AirWatch (WOne UEM 1909) services may not start due to timeout error on Windows 2008-2012.
Increase Timeout time in Windows registry: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control → ServicesPipeTimeout=180000
External link: https://kb.vmware.com/s/article/50105044?lang=en_US
Admin Console Back-End (BE) Server
- Enter Windows Server Manager and check the following roles/features (double-check official doc for feature list):
- Web Server (IIS)
- Web Server (IIS) → Web Server → Common → Static Content, Default Document, Directory Browsing, HTTP Errors, HTTP Redirection
- Web Server (IIS) → Web Server → Performance → Dynamic Content Compression
- Web Server (IIS) → Web Server → ASP
- Web Server (IIS) → Web Server → ASP.NET 4.5
- Web Server (IIS) → Web Server → Security → IP & Domain Restrictions
- Web Server (IIS) → Web Server → Health & Diagnostics → Request Monitor
- Web Server (IIS) → Web Server → Application Development → Server Side Includes
- .NET Framework 4.5 → WCF → HTTP Activation
- Message Queuing
- Telnet Client
❗️ DON NOT turn on Web Server (IIS) → Web Server → Common → WebDav Publishing - this will lead to multiple bugs in managing iOS devices
- ❓️ If there is not Internet on FE server - download and install NET Framework 4.6.2 (Microsoft .NET Framework 4.6.2 (Offline Installer) for Windows 7 SP1, Windows 8.1, Windows Server 2008 R2 SP1, Windows Server 2012 and Windows Server 2012 R2). Reboot server;
- ❓️ If there is not Internet on FE server - download and install URL Rewrite Module 2.0 (https://go.microsoft.com/?linkid=9722532) for IIS. Old version of Rewrite Module 2.0 provided on this page as attachment in case of need;
- Check correct start config of IIS - use browser to go to http://127.0.0.1/ (start page of IIS must be present)
- Configure the certificate on IIS - for Admin Console on BE a self-signed certificate may be used:
- Enter IIS Admin Console, choose Server Certificates, and in the right column menu choose Create Self-Signed Certificate;
- Enter a name for the certificate, type = Web Hosting, click ОК;
- Go to IIS admin console, bind the certificate: in sites tree choose Default Web Site → Bindings menu → Add.., choose https, in SSL Certificates list choose the certificate from previous step.
❗️Port binding is needed ONLY for Device Service and Console Service.
- Launch installer WorkspaceONE_UEM_Application_18.11.0.3_Full_Install. Choose Continue setup without importing/exporting config file;
- In modules selection choose only the Admin Console, choose This feature will not be available for Device Services, continue the installation;
❗️For AirWatch 9.2.2+: during installation, AirWatch installer deploys SQL Native Client, which may not have enough time to initialize during the work of the wizard. During SQL check, an error may be generated, that SQL is not found. Press Cancel and reboot the server, then re-launch the setup process.
- Enter SQL data: in full database name, only enter the server name, do not enter SQL Instance name;
- Specify the FQDN name for HTTPS access on Admin Console from the inside. Do NOT use a short name of DNS alias. Choose an External DNS name for access via HTTPS on Device Services server. Check the absence of space characters before or after the names. An error in this form may be corrected only by re-installing UEM!
- Choose Default Web Site as the install target;
- In Company Profile choose the company name and installation type = Production;
❗️AirWatch (WS1 UEM 1909+) services may not start due to timeout error on Windows 2012R2+.
Increase Timeout time in Windows registry: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control → New 32-Bit DWORD: ServicesPipeTimeout, Decimal=200000 (Decimal=60000 too small, put more!)
External link: https://support.microsoft.com/en-us/kb/922918
Manually running services: under Supplemental Software → QueueSetup locate and run the InstallQueues.Bat file.
- Check the installation:
- Enter the UEM console. Use Login: administrator, Password: airwatch. Choose a new password, choose a PIN-code and secret questions/answers pairs.
⭐️ Hardening of the IIS web-server for AirWatch/UEM Device Services is described in this article.