Статьи по установке, настройке и решению проблем с Workspace ONE Access (бывший Identity Manager).
WS1 Access Installation
Remove AD from WS1 Access DB
Config and troubleshooting Elasticsearch in WS1 Access
Configuring Mobile SSO for iOS
Simple packet sniffer on Python 2.x for WS1 Access
Managing certificates for WS1 Access
Resetting passwords for WS1 Access Users
Configuring SQL DB for Access
Identity Manager Appliance
Enterprise Connector
Increasing memory of services on connector:
Xmx1g entry to Xmxng where n is the maximum heap memory you want to allocate. Example: Xmx5gNetwork ports for connector: Inbound & outbound TCP443 (many uses); Outbound TCP389, TCP636, TCP3268, TCP3269 (LDAP); Outbound TCP88, UDP88, TCP464, TCP135, TCP445 (Kerberos, Directory Sync); Outbound TCP53, UDP53 (DNS); Outbound TCP5555 (RSA SecurID); Outbound UDP514 (Syslog).
Intelligent Hub This is a client simultaneously for WS1 UEM and WS1 Access (Corp apps marketplace + Hub Services: people search, notifications)
Identity Manager on Windows Server (Old, uses only Cloud KDC)
! For IDM on Windows do NOT use non-English localized Windows versions. Workaround: change the regional number setting for decimal to use a period “.” instead of a comma “,”. ! For IDM on Windows shutdown IIS to free up port TCP80. It is not used, but it is needed for IDM install.
Workspace One Client Mobile App App Bundle ID: com.air-watch.appcenter
External links:
Most common command is to build PFX-file from PEM files:
openssl pkcs12 -export -out certificate.pfx -inkey privkey.pem -in fullchain.pem -certfile cert.pemWS1 Access always has to be signed with corporate or trusted certificates. If Access is clustered, sign the load-balanced name with external trusted certificate and the 3 nodes - with certs from corporate CA.
❗️The order of certificates is important! First server, then intermediate, then root.
######### The cert request in idm01.domain.local.inf file
[Version]
Signature= "$Windows NT$"
[NewRequest]
Subject = "CN=idm01.domain.local,OU=IT,O=Horn_n_hooves,L=Moscow,S=Moscow,C=RU"
KeySpec = 1
KeyLength = 2048
Exportable = TRUE
MachineKeySet = TRUE
UseExistingKeySet = FALSE
ProviderName = "Microsoft RSA SChannel Cryptographic Provider"
ProviderType = 12
RequestType = PKCS10
KeyUsage = 0xa0
FriendlyName = "vdm" ; needed for Horizon Connection Server only!
[EnhancedKeyUsageExtension]
OID=1.3.6.1.5.5.7.3.1
[RequestAttributes]
CertificateTemplate = WebServerExportable2008
[Extensions]
; If your client operating system is Windows Server 2008, Windows Server 2008 R2, Windows Vista, or Windows 7
; SANs can be included in the Extensions section by using the following text format.Note 2.5.29.17 is the OID for a SAN extension.
2.5.29.17 = "{text}"
_continue_ = "dns=idm01.domain.local&dns=idm01&dns=idm02.domain.local&dns=idm02&dns=idm03.domain.local&dns=idm03&dns=idm.domain.local"
##################################################################################Bat script to submit certificate request:
set srvname=idm01.domain.local
cd C:\temp
Certreq -New -f %srvname%.inf %srvname%.req
Certreq -submit %srvname%.req %srvname%.crt
Certreq -accept %srvname%.crt
Certutil -exportpfx -p 12345 %srvname% "%srvname%.pfx"In order to copy and paste the private key from PFX certificate for vIDM, you need a decrypted version of the key. Use OpenSSL to obtain this key:
openssl pkcs12 -in idm01.domain.local.pfx -nocerts -out idm01.domain.local_encrypted.key
openssl rsa -in idm01.domain.local_encrypted.key -out idm01.domain.local_decrypted.keyOpen idm01.domain.local_decrypted.key with a text editor and copy the key from there. After inserting certificates, click OK to restart WS1 Access web service.
External Links:
❗️The admin user password must be at least 12 characters in length.
Connect to WS1 Access Appliance by SSH, run command:
passwd sshuserLog in to the URL as a root user: https://ApplianceFQDN:8443/cfg/changePassword
Access WS1 Access using ssh (sshuser password needed), do su and use this command to reset password:
/usr/sbin/hznAdminTool setOperatorPassword --pass newsecretpassword/usr/sbin/hznAdminTool setSystemAdminPassword --pass newsecretpasswordWS1 Access works on VMware PhotonOS variation of Linux. It has a Single User Mode.
init=/bin/bash to the line;passwd and for sshuser: passwd sshuser;
❗️Follow password policies on new passwords!shutdown -h -P now;
❗️VMware Tools do NOT work in Single User Mode, so shutdown from vCenter will NOT work;To login into the DB, get the PostgreSQL password: first login to console with SSH.
cat /usr/local/horizon/conf/db.pwdCopy password, then login with it:
psql saas horizon❗️ Database schema name must be ‘saas’, cannot be changed. ❗️Collation must be ‘Latin1_General_CS_AS’, could be changed but change not recommended. ❗️The server role used to grant server-wide security privileges is set to public. The database role membership is db_owner.
Microsoft SQL Database Using Local SQL Server Authentication Mode for Workspace ONE Access (replace values in brackets < > ):
CREATE DATABASE saas
COLLATE Latin1_General_CS_AS;
ALTER DATABASE saas SET READ_COMMITTED_SNAPSHOT ON;
GO
BEGIN
CREATE LOGIN <loginusername> WITH PASSWORD = N'<password>';
END
GO
USE <saasdb>;
IF EXISTS (SELECT * FROM sys.database_principals WHERE name=N'<loginusername>')
DROP USER [<loginusername>]
GO
CREATE USER [<loginusername>] FOR LOGIN [<loginusername>]
WITH DEFAULT_SCHEMA=saas;
GO
CREATE SCHEMA saas AUTHORIZATION <loginusername>
GRANT ALL ON DATABASE::saas TO <loginusername>;
GO
ALTER ROLE [db_owner] ADD MEMBER <loginusername>;
GOSQL local user jdbc:sqlserver://<DB_VM_IP_ADDR>;DatabaseName=saas
jdbc:sqlserver://<DB_VM_IP_ADDR>\INSTANCE_NAME:PORT;DatabaseName=saas (you can remove the instance name if default)
AD domain user jdbc:jtds:sqlserver://<DB_VM_IP_ADDR>:1433/saas;integratedSecurity=true;domain=LAB.LOCAL;useNTLMv2=true
Multi-site, SQL Always On
jdbc:sqlserver://
Back and continue buttons become greyed out and unclickable.
See resolution link
Check the result of the following DB query, If DB is locked, you should see an entry with the locked value set to 1 or TRUE.
SELECT * FROM DATABASECHANGELOGLOCK
if locked=TRUE then run the below UPDATE statement to release/reset the DB lock.
USE [DBNAME]
GO
UPDATE [saas].[DATABASECHANGELOGLOCK]
SET [LOCKED] = 0
,[LOCKGRANTED] =NULL
,[LOCKEDBY] = NULL
WHERE ID = 1
GORestart horizon-workspace service on each node one after another. ❗️DO NOT restart all nodes of WS1 Access cluster simultaneously.
Update the Connector if there is more than 1 sync and Authentication
UPDATE “Connector” SET “isDirectorySyncEnabled”=‘false’ WHERE “host”=‘<auth_connector_hostname>’
Check if IDM certificate is present in the console from DB side
SELECT * from dbo.coreuser WHERE isactivedirectoryuser=0
SELECT * from dbo.certificate WHERE certificatethumbprint like '%%'
SELECT * from dbo.UserLink WHERE coreuserid = < username >
SELECT * from dbo.role WHERE roleid =3
List of UUID with Super Admin access
SELECT strUsername, strFirstName, StrLastName,strExternalId, stremail, uuid FROM slesdb.saas.Users U (nolock) INNER JOIN
(SELECT * FROM Slesdb.saas.ACS_RuleSetAssociation (nolock)
WHERE ruleSetId LIKE
(SELECT id FROM slesdb.saas.ACS_RuleSet (nolock)
WHERE name LIKE 'Super Admin)) as A
On U.uuid = A.SubjectUUID
ORDER BY strUsername
Update email address of the user or admin
UPDATE sles.saas.Users
SET strEmail = 'value'
WHERE username LIKE 'value'
Update the value of the Identity Provider with correct connector name
UPDATE slesdb.saas.IdentityProviders
SET strDescription = (
SELECT host FROM slesdb.saas.Connector WHERE id = (
SELECT idConnector FROM slesdb.saas.IdpJoinConn WHERE idIdentityProvider = (
SELECT id FROM slesdb.saas.IdentityProviders WHERE strFriendlyName LIKE '%Workspace%')))
Update the attribute column to make it mandatory or non-mandatory
SELECT id, * FROM slesdb.saas.userattributedefinition WHERE idorganization IS NOT NULL AND ownerUuid IS NULL AND strName LIKE '<attribute to be updated>
Then run:
UPDATE slesdb.saas.userattributedefinition SET bIsRequired = 0 WHERE id = <id identified above>
Connector Sync Validation
SELECT id, uuid, tenantID, host, domainJoined, createdDate, oAuth2ClientId, isDirectorySyncEnabled FROM saas.Connector;
SELECT idSyncProfile, directoryConfigId, syncConnectors FROM saas.DirectorySyncProfile
⭐️ FROM saas.Connector isDirectorySyncEnabled = 1 means connector is the connector set as Sync. If not, you can update:
UPDATE saas.connector SET isDirectorySyncEnabled=0 WHERE id=1;
⭐️ FROM saas.DirectorySyncProfile syncConnector = saas.Connector uuid value. You can update if needed as well:
SET isDirectorySyncEnabled=0 WHERE id=1;
update saas.DirectorySyncProfile SET syncConnectors='["12345678-abcd-1234-1234-0123a678b78"]' WHERE idSyncProfile=1;
Steps:
Access saas database of IDM and delete association of IdP with Connector.
If database of IDM is deployed as inherent PostgreSQL, then use SSH to connect to the primary appliance. ❗️Database on primary appliance is in read/write mode.
Connect to database:
su - postgres /opt/vmware/vpostgres/current/bin/psql vcacChange schema to saas:
SET SCHEMA 'saas';Use SQL Management Studio to connect to DB server, open saas database.
Find Connector table (example shown for MS SQL Server 2014) and ‘disconnect’ it from AD:
SELECT * FROM saas.saas.Connector;
-- Check the ID of your AD connector
UPDATE saas.saas.Connector
SET domainJoined = NULL
WHERE id = N; -- set N to the ID of your AD connector❗️You cannot delete the Connector in SQL - it has dependencies from foreign keys. But all you need to do is find the ID of BuiltIn IdP and disconnect it:
SELECT * FROM saas.saas.IdentityProviders;
SELECT * FROM saas.saas.IdpJoinConn;
-- Check the ID of your IdP defore doing this!!
DELETE FROM saas.saas.IdpJoinConn
WHERE idIdentityProvider = N; -- set N to the ID of your IdP❗️After adding a new AD do not forget to follow the guide backwards: do not forget to add “Password (Cloud Directory)” back to Policies!
❗️The BuiltIn IdP must be associated back with the new Connector. This can be done using the console, or it can also be done in SQL:
SELECT * FROM saas.saas.IdpJoinConn;
-- Double-check the ID of your IdP and NEW Connector defore doing this!!
INSERT INTO saas.saas.IdpJoinConn
VALUES (X, Y) -- set X to the ID of your IdP, set Y to the ID of ConnectorElasticsearch in VMware Workspace ONE Access is used for storing records on:
These records are collected on the local SAAS node and are being sent to RabbitMQ instance on the local node once per second. They are then picked up and sent to Analytics Service on local node, which then sends the records for storing in Elasticsearch. RabbitMQ acts as a buffer, so that if Elasticsearch is down, the records are not lost. It also decouples the generation rate from the storing rate, so if records are coming in faster than they can be stored, they can be queued up and left in RabbitMQ.
RabbitMQ previously WAS clustered, so records generated on one node could be picked up and processed by any other node in the cluster. This meant that if the local Elasticsearch instance is having issues, another node could grab the message and process them. Now RabbitMQ is NOT clustered (it had complex node stop/start order requirements to maintain a stable cluster, and maintaining a RabbitMQ cluster was more hassle than it was worth), so it is possible for the records generated on one node to start piling up in RabbitMQ. This is less robust, but makes it easier to spot when a specific node is having a problem.
Services connected to Elasticsearch:
Elasticsearch is a distributed, replicated data store, which means each node does not have full copy of the data. In a 3-node cluster, each node will have 2/3 of the data, so that all 3 nodes combined provides 2 full copies, and you can lose one node without losing any data. This distribution and replication is achieved by splitting each index into 5 shards and each shard has a primary copy and a replica (backup) copy. All ten shards for an index are then distributed across all three nodes so that no single node has both the primary and replica copy of any one shard, or all the shards for one index.
Audit data and sync logs are stored in a separate index for each day. The index is versioned based on the document structure so that you can seamlessly migrate from an old format to a new format. The audit indexes have the format "v<version>_<year>-<month>-<day>'. For example, version 4 of the document structure will be "v4_2022-03-04".
Search data (users, groups, applications) used by the admin UI for search and autocomplete, and the PeopleSearch feature, is stored in a separate index called "v<version>_searchentities", and currently uses version 2 of that document structure.
⭐️ The indices can be seen in the /db/elasticsearch/horizon/nodes/0/indices directory or listed with useful stats via: curl localhost:9200/_cat/indices?v
A cluster has a master node which has the additional job of coordinating the distribution of data and farms out the search and document retrieval requests to the other nodes based on which node has which shard. Any node can be the master node, the nodes choose the master themselves based on the makeup of the cluster. If the master node disappears, one of the remaining nodes is chosen as the new master.
Elasticsearch uses a list to determine how many nodes it should be able to see before a master can be elected and the cluster formed (and the other nodes its should be reaching out to). This is done to prevent a split-brain situation, where due to network issues not all the nodes can see each other, so form independent clusters, which require resetting the errant node and losing its data. So when WS1 Access tells Elasticsearch there should be 3 nodes, it knows it must see at least 2 nodes to form a cluster.
❗️If Elasticsearch is up before WS1 Access (horizon-workspace service), there will be timeout failures in the logs, which can result in the cluster getting stuck. If the other nodes in the cluster are not up, there will be error messages about trying to add them to the cluster and then removing them again because they did not respond.
❗️Due to rules and regulations in some countries and companies around data retention of audit data, by default historical data is not deleted.
Elasticsearch maintains a very complex series of lookup tables in memory that allows it to determine which documents match a particular query in just a couple of milliseconds, whether there is one document to query or a billion.
By default, to reduce memory footprint, the number of days (indices) allowed for Elasticsearch to keep in memory (or open) is limited to 90 days.
Older indices are closed to remove them from memory and are no longer searchable. Once a day, just after midnight (00:30am), the retention policy is executed, and any open index that is older than the 90 days is closed.
Also the open indices that are no longer being written to (any open index older than the current day) are optimized: As documents come in, Elasticsearch writes them to lots of small files on disk. This lets it store documents quickly, but slows down retrieval and consumes a huge amount of open file descriptors. When an index is no longer being written to (for example, because it is for yesterday and no dated records for yesterday should be coming in anymore to write to it), Elasticsearch takes all those little files and combines them into one big file. This reduces the number of file descriptors needed and speeds up document retrieval for older documents.
When the retention policy runs, there will be messages in the analytics-service log saying it has started, and for each open index, whether it was closed because it is now past the cut-off date, or was optimized (the optimization is blindly optimizing all open, older-than-today indices, because if it is already optimized, Elasticsearch does nothing).
com.vmware.idm.analytics.elasticsearch.ElasticSearchHttpStorageAdapter - Executing analytics retention policy, cutoff date is 2019-04-07
com.vmware.idm.analytics.elasticsearch.ElasticSearchHelper - Closed index: v4_2019-04-06
com.vmware.idm.analytics.elasticsearch.ElasticSearchHelper - Optimizing index v4_2019-05-02
com.vmware.idm.analytics.elasticsearch.ElasticSearchHelper - Optimizing index v4_2019-06-28
com.vmware.idm.analytics.elasticsearch.ElasticSearchHttpStorageAdapter - Analytics retention policy completed.Elasticsearch will be out of sync when the WS1 Access nodes are with different times. The easy way of checking is from the System Diagnostics Page -> Clocks - it should show all the clock times.
Check using CLI on all nodes: watch -n 1 date
Due to default policy of never deleting old data, the /db filesystem usage will continue to grow until it runs out of space.
Check the free disk space for the /db filesystem with the command: df -h
An indicator that something is wrong is if the size of pending documents in the analytics queue is growing. RabbitMQ is used for other types of messages, which go in their own queues, but the analytics queue is the only one that does not have a TTL, so it is the only one that can grow. This is done so that its’ messages never get deleted and the audit records never get lost. Typically the analytics queue should have just 1-2 messages waiting to be processed, unless something is wrong or there is a temporary spike in records being generated (during a large directory sync the queue might temporarily grow, if the system cannot keep up).
Check queue size on each node with CLI: rabbitmqctl list_queues | grep analytics
If the number reported is larger than 100, its time to investigate why.
⭐️ This is also reported as the AuditQueueSize value in WS1 Access health API: /SAAS/API/1.0/REST/system/health/
Overall cluster health:
curl http://localhost:9200/_cluster/health?prettyCheck that the node counts (number_of_nodes and number_of_data_nodes) equals the size of the cluster and the state is green (except for single nodes, whose state will always be yellow)
GREEN = good, there are enough nodes in the cluster to ensure at least 2 full copies of the data spread across the cluster;
YELLOW = ok, there are not enough nodes in the cluster to ensure HA (single-node cluster will always be in the yellow state);
RED = bad, unable to query existing data or store new data, typically due to not enough nodes in the cluster to function or out of disk space.
Check that nodes agree on which ones are actually in the cluster and who the master is (to ensure you don’t have errant nodes, or split-brain with two clusters instead of one).
Run this on each node and verify the output is the same for every node:
curl http://localhost:9200/_cluster/state/nodes,master_node?prettyThis information is also reported in the “ElasticsearchNodesCount”, “ElasticsearchHealth”, “ElasticsearchNodesList” values in the health API, but that will only be coming from one of the nodes as picked by the gateway: /SAAS/API/1.0/REST/system/health/
Other useful commands:
curl http://localhost:9200/_cluster/health?pretty=true
curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
curl http://localhost:9200/_cluster/state/master_node,nodes?pretty
curl http://localhost:9200/_cluster/stats?pretty=true
curl http://localhost:9200/_nodes/stats?pretty=true
curl http://localhost:9200/_cluster/state?pretty=true
curl http://localhost:9200/_cat/shards | grep UNASSIGNED
curl -XGET ‘http://localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason’ | grep UNASSIGNED Determine the growth rate of data nodes can store. Use that to either schedule disk size increase, or determine how many days of data with their usage they can keep.
After a few weeks of running the system, look at the size of each index and determine the average day’s size using:
curl localhost:9200/_cat/indices?vExtrapolate how many days nodes have space for. For safety, use 50-60% of space as the benchmark. You can change the data retention policy to automatically delete old data so that the filesystem usage is capped. This will not interrupt service and is done by a configuration change that must be done on each node, setting the maxQueryDays to the number of days to keep data. The new policy will take affect at 00:30 am:
analytics.deleteOldData=true (default is false, close the older indices)
analytics.maxQueryDays=90 (default is 90)service horizon-workspace restartcurl -XPOST http://localhost:9200/_all/_openWS1 Access can take a long time to come up, because the first node has the default configuration of only expecting one node in the cluster and decides it is the master. Then it checks what indices it has and discovers a bunch a shards it cannot find. When the second node comes up, it is told by the first node that the first node is the master, but the first node is stuck because of the missing shards, so refuses to act like a master. The same thing happens when the third node comes up. This situation can be fixed by restarting Elasticsearch on the first node. This can be completely avoided by changing the configuration, so that on restart all the nodes know to expect more nodes in the cluster, so they wait and only elect a master when the second node comes up.
On each node, edit /opt/vmware/elasticsearch/config/elasticsearch.yml to add these lines, which tell the cluster to not form until at least 2 nodes are in communication, and to wait up to 10 minutes for a third node to come online before deciding to start copying shards between the two nodes that are there to ensure HA. It will also prevent accidentally starting Elasticsearch more than once on a node:
discovery.zen.minimum_master_nodes: 2
gateway.recover_after_nodes: 2
gateway.expected_nodes: 3
gateway.recover_after_time: 10m
node.max_local_storage_nodes: 1❗️❗️ NOTES: ❗️❗️
. /usr/local/horizon/scripts/hzn-bin.inc && /usr/local/horizon/bin/curl -v -k -XPUT -H "Authorization:HZN <cookie>" -H "Content-Type: application/vnd.vmware.horizon.manager.systemconfigparameter+json" https://localhost/SAAS/jersey/manager/api/system/config/SearchCalculatorMode -d '{ "name": "SearchCalculatorMode", "values": { "values": ["REINDEX"] } }'When you try to check the elasticsearch cluster health you get:
{
"error" : "MasterNotDiscoveredException[waited for [30s]]",
"status" : 503
}You can enable additional logging by editing /opt/vmware/elasticsearch/config/logging.yml and adding the following line after the “org.apache.http: INFO” one.
com.vmware: DEBUG OR com.vmware: TRACE
Causes:
Stale/incorrect node entries stored in the DB being reported to Elasticsearch. Look at the contents of the “ServiceInstance” DB table. If there are any wrong entries manually delete them. Check the “status” value is 0. Any other value is indicates an issue with that node that needs to be looked at (1 = Inactive, 2 = error, 3 = invalid master key store, 4 = pending removal). When the issues with the ServiceInstance table have been addressed, within 30 seconds the elasticsearch plugin on all the nodes should now get just the correct records and sort itself out. If it hasn’t done that within a few minutes, restart elasticsearch on each node (service elasticsearch start).
Mis-configured multi-DC setup, resulting in the service instance REST API reporting incorrect nodes to elasticsearch. On the affected nodes check the output of:
/usr/local/horizon/bin/curl -k https://localhost/SAAS/jersey/manager/api/system/clusterInstancesIf it does not show the entries you expect, look in the “ServiceInstance” table in the DB and check the “datacenterId” column value is correct for the nodes (so they are identified as part of the correct per-DC cluster. Check the “status” value is 0. Any other value indicates an issue with that node that needs to be looked at (1 = inactive, 2 = error, 3 = invalid master key store, 4 = pending removal).
❗️ DO NOT DELETE UNASSIGNED SHARDS
You need to address the underlying cause, then the unassigned shards will be resolved by the cluster on its own. It is extremely rare for unassigned shards to be a real issue needing direct action (eg. filesystem corruption, accidental data deletion).
One of the nodes is down, unreachable or broken. Find out which one is in trouble by getting the list of nodes and seeing which one is missing:
curl http://localhost:9200/_cluster/state/nodes,master_node?prettyDo not worry about unassigned shards in the health API output, getting the node back in the cluster will allow the master to sort out who has what shard and within minutes the unassigned shards will be resolved and the cluster green.
Causes:
If the node is down, bring it back up.
Check the node is reachable from the other nodes in the cluster. If it isn’t, fix the networking/firewall issue so that it can rejoin the cluster.
On the missing node, check the disk space is ok. If it’s full, either:
a. increase the size of the filesystem, OR
b. free up disk space to get the cluster back to green by manually deleting the older indices, based on their date. To do that, first stop Elasticsearch on all the nodes
elasticsearch service stop. Then delete the directories for the oldest indices in the /db/elasticsearch/horizon/nodes/0/indices directory on the node with disk space issues. When there is at least 500MB-1GB free space again on that node, delete the exact same ones on the other two nodes. Finally, restart Elasticsearch on each node service elasticsearch start and the cluster should go back to green and then you can adjust the retention policy.
At least two of the nodes are broken, the analytics queue in RabbitMQ will start to grow.
Causes:
[discovery.zen ] [Node2] failed to send join request to master [Node1],
reason [RemoteTransportException[[Node1]][internal:discovery/zen/join]]; nested:
ElasticsearchIllegalStateException[Node [Node1]] not master for join request from [Node2]Determine the master node from the log (eg Node2 in the example above) or with:
curl http://localhost:9200/_cluster/state/nodes,master_node?pretty
Restart Elasticsearch on the master node service elasticsearch restart . One of the other two nodes should immediately become the master (re-run the curl on the other nodes again to verify) and the old master will re-join the cluster as a regular node when it comes up.
The analytics queue size in RabbitMQ should usually only show 0-5 messages when the system is functioning correctly. If you see larger values, this means messages were not able to be delivered to elasticsearch.
Causes:
curl -XPOST http://localhost:9200/_all/_openYou can use following command to check the indices status:
curl http://localhost:9200/_cat/indicesThe messages should now be able to get processed and the queue size should drop.
"parse_exception", then there is a malformed document.You can clear the entire queue by executing rabbitmqctl purge_queue <queue>.
Note: This will cause the loss of any other audit events, sync logs and search records that were also queued up, so the search index must be rebuilt to ensure the search/auto-complete functionality continues to work fully.
You can also use the management UI to remove just the bad message. Go to the queue in the Management UI and select the “Get messages” option with re-queue set to false to pull just the first message off the queue (which should be the bad message). Once the bad message is cleared, the others should get processed successfully.
a. Enable firewall access to the RabbitMQ management UI:
- Edit /usr/local/horizon/conf/iptables/elasticsearch to add “15672” to the ELASTICSEARCH_tcp_all entry, => ELASTICSEARCH_tcp_all="15672"
- Re-apply the firewall rules by executing the /usr/local/horizon/scripts/updateiptables.hzn script.
b. You can only access the RabbitMQ Management UI using the default credentials of guest/guest on localhost, so you will have to create a new admin user in RabbitMQ. To create a user called “admin” with the password “s3cr3t”:
rabbitmqctl add_user admin s3cr3t
rabbitmqctl set_user_tags admin administrator
rabbitmqctl set_permissions -p / admin ".*" ".*" ".*"c. In a browser, access the RabbitMQ Management UI by going to http://host-or-ip-of-node:15672
d. For security, re-disable remote access to the RabbitMQ Management UI by re-doing the a) step, but removing the 15672 port: ELASTICSEARCH_tcp_all="", then re-apply the firewall rules with the updateiptables.hzn script.
This means RabbitMQ is in a bad state. Causes:
"disk_free_limit" and "disk_free" values to verify:rabbitmqctl statusSee cause #3 in the “Elasticsearch cluster health yellow” section for the steps to free up disk space.
RabbitMQ is not able to start, i.e. rabbitmqctl status gives a “nodedown” error and you see this error in /db/rabbitmq/log/startup_log:
BOOT FAILED ... {error, {not_a_dets_file,
"/db/rabbitmq/data/rabbitmq@<YOUR_HOSTNAME>/recovery.dets"}}}This will require deleting the recovery file and starting RabbitMQ again:
a. Stop all RabbitMQ processes by doing kill -9 on all processes found with ps -ef | grep -i rabbit, until none remain.
b. rm /db/rabbitmq/data/rabbitmq@<YOUR_HOSTNAME>/recovery.dets
c. rabbitmq-server -detached &
Once RabbitMQ is running again, workspace should be able to connect to it.
Note, it may take up to 10 minutes before re-connection is tried. Look in horizon.log for messages from RabbitMQMessageSubscriber and RabbitMQMessagePublisher. If you see messages that indicate they were shutdown instead of connected or there are ProviderNotAvailableExceptions in the log, then workspace must be restarted service horizon-workspace restart to recreate them.
❗️❗️ DO NOT USE UNLESS NOTHING ELSE HELPS ❗️❗️
Stop elasticsearch on all nodes, delete the data in elasticsearch and RabbitMQ, restart the elasticsearch process on each node, and trigger a re-index. This will result in the loss of the historic sync logs and audit data.
NOTE: If this is a Multi-Site setup:
Step 1. Stop elasticsearch on every node in the site:
service elasticsearch stop
Step 2. When stopped on all nodes, clear the data, by doing the following on each node:
rm -rf /db/elasticsearch/horizon
rm -rf /opt/vmware/elasticsearch/logs
//so we start with fresh logs to make it easier to see what is happening.Get the analytics queue name(s) and purge them. On the primary site there should be two analytics RabbitMQ queues that need purging, one for 127.0.0.1 ("-.analytics.127.0.0.1") and one for the DR site (as configured via the analytics.replication.peers setting in the runtime-config.properties file, “-.analytics.dr.site.com”):
rabbitmqctl list_queues | grep analytics
rabbitmqctl purge_queue <analytics-queue-name>Step 3. Verify the data retention policy configuration on each node is the same by looking at the /usr/local/horizon/conf/runtime-config.properties file and the “analytics.maxQueryDay” and “analytics.deleteOldData” values. If any node is different, then change them to match and restart WS1 Access on the node that did not match with:
service horizon-workspace restart
Wait for WS1 Access to be back up before restarting another and before going on to step 4.
Step 4. Add the additional config to Elasticsearch from the above chapter “Avoid full cluster restart issues” preventative measure to help it recover from a full cluster restart, by adding these lines to /opt/vmware/elasticsearch/config/elasticsearch.yml on each node if missing:
discovery.zen.minimum_master_nodes: 2
gateway.recover_after_nodes: 2
gateway.expected_nodes: 3
gateway.recover_after_time: 10mStep 5. Restart elasticsearch on each node (preferably quickly)
service elasticsearch start
Step 6. Verify the cluster is functioning fully by running the following on each node and making sure the results match (ie, nodes are 3, same master and the node list is the same) and health is green:
curl http://localhost:9200/_cluster/state/nodes,master_node?pretty
curl http://localhost:9200/_cluster/health?prettyIf the output for those two commands do not match on all nodes stop, do not proceed to step 7. Gather the logs in /opt/vmware/elasticsearch/logs for each node.
STEP 7. Trigger a re-index of the search data.
To manually force a re-index, choose either the zero-down time option if you’re comfortable finding the HZN cookie value, or the down-time option by modifying the value directly in the DB.
The re-index should only take a few minutes. You can verify it started by looking in the workspace horizon.log for a message like:
com.vmware.horizon.search.SearchCalculatorLogic -
Keep existing index. Search calculator mode is: REINDEXZero down-time
Copy the value of the HZN cookie.
ssh into a node make the following REST API call, replacing <cookie_value> with the HZN cookie value obtained from the browser.
. /usr/local/horizon/scripts/hzn-bin.inc && /usr/local/horizon/bin/curl -k -XPUT -H "Authorization:HZN <cookie_value>" -H "Content-Type: application/vnd.vmware.horizon.manager.systemconfigparameter+json" https://localhost/SAAS/jersey/manager/api/system/config/SearchCalculatorMode -d '{ "name": "SearchCalculatorMode", "values": { "values": ["REINDEX"] } }'After initiated “REINDEX”, you can change the log to confirm it is started
cat /opt/vmware/horizon/workspace/logs/horizon.log | grep REINDEXWith down-time
Stop horizon-workspace on all nodes: service horizon-workspace stop
Edit the DB:
UPDATE "GlobalConfigParameters" SET "strData"='REINDEX' WHERE "id"='SearchCalculatorMode';Restart horizon-workspace on all nodes: service horizon-workspace start
Verify progress
The reindex should start within a couple of seconds. Verify the reset was successful by looking for the message “Forcing a REINDEX of the search index” in the logs.
Reindexing can take hours, depending on the size of the data. You can monitor its progress by looking in the logs for “SearchCalculatorLogic” statistic messages, which show the counts of the objects its currently processing (typically it will be 5000 at a time when doing a re-index). When the all counts reach zero, it means it has finished. eg:
2020-02-28T05:54:08,208 WARN (pool-37-thread-2) [ACME;-;-;-]
com.vmware.horizon.common.datastore.BaseCalculator - ++SearchCalculatorLogic$
$EnhancerBySpringCGLIB$$3e4c8f8c [ 0] 0.000000/Object ([ 0] 0.000000/Group, [ 0] 0.000000/Users,
[ 0] 0.000000/ResourceDb)Re-Doing the Elastic Search Index.
Symptom: in case the user/group search is not working properly.
Resolution: Stop workspace service (on all nodes):
service horizon-workspace stop
update saas.GlobalConfigParameters set "strData"=-1 where "id"= 'LatestUserEntitlementVersion’;
For Search:
update saas.GlobalConfigParameters set "strData"=-1 where "id"= 'LatestSearchVersion’; For User Group association:
update saas.GlobalConfigParameters set "strData"=-1 where "id"= 'LatestUserGroupVersion’;service horizon-workspace startExternal Links:
❗️❗️ Do NOT use UAG in front of WS1 Access with MobileSSO for iOS scenario!
ADMINISTRATOR CONFIGURATION
USER ENROLLMENT
USER APP LAUNCH
Login Flow Steps:
Given L4 Load Balancer at 1.2.3.4 and a DNS domain of example.com and a realm of EXAMPLE.COM:
kdc.example.com. 1800 IN AAAA ::ffff:1.2.3.4 kdc.example.com. 1800 IN A 1.2.3.4 _kerberos._tcp.example.com. IN SRV 10 0 88 kdc.example.com. _kerberos._udp.example.com. IN SRV 10 0 88 kdc.example.com.
❗️ There MUST be an IPv6 AAAA entry - iPhone requires it to work correctly.
AAAA entry may have to be converted to IPv6 format: :::::ffff:102:304 ⭐️ Use web-tool: https://www.ultratools.com/tools/ipv4toipv6
To test the DNS settings, you can use the dig command (built-in to Mac and Linux) or NSLOOKUP on Windows.
Here is what the dig command looks like for DNS records:
dig SRV _kerberos._tcp.example.com
dig SRV _kerberos._udp.example.com
# You may wish to define the name server to use with dig by using the following command:
dig @ns1.no-ip.com SRV _kerberos._tcp.example.comUse Google Toolbox to check the SRV entries:
https://toolbox.googleapps.com/apps/dig/#SRV/_kerberos._tcp.example.com
The result must contain string of type:
;ANSWER
_kerberos._tcp.example.com. 3599 IN SRV 10 0 88 krb.example.com.Use NC to check UDP/Kerberos services:
nc -u -z kdc.example.com 88
# Answer:
#> Connection to kdc.example.com port 88 [udp/kerberos] succeeded!Use nslookup to check the SRV entries:
nslookup -q=srv _kerberos._tcp.vmwareidentity.eu
Server: 10.26.28.233
Address: 10.26.28.233#53
Non-authoritative answer:
_kerberos._tcp.vmwareidentity.eu service = 10 0 88 kdc.vmwareidentity.eu.
The mobile device profile for the Cloud KDC feature must include the following:
Specify a list of app bundle ids:
The CA OCSP responder must be accessible on the Internet so that it can be accessed by the KDC. The URL for the OCSP responder is provided in the client certificate in the usual manner.
WS1 UEM has confirmed that the AW CA will meet this requirement.
Port 88 must be open TCP+UDP. Use packet sniffer to check packets from device.
For a single Workspace ONE environment or primary domain:
/etc/init.d/vmware-kdc init --realm MY-IDM.EXAMPLE.COM
# Restart horizon-workspace service:
service horizon-workspace start
# Start KDC service:
service vmware-kdc startFull instructions link
Login to the console (or SSH and SU to root) and do the following: a. Stop the vmware-kdc service b. Force reinitialization of the KDC service (note the use of the ‘–force’ switch at the end) c. Restart the vmware-kdc service d. Restart the horizon-workspace service
⭐️ Last 2 steps can be replaced with rebooting the appliance.
service vmware-kdc stop
/etc/init.d/vmware-kdc init --realm EXAMPLE.COM --subdomain idm.example.com --force
service vmware-kdc start
service horizon-workspace restart
#Or just type reboot to restart the appliance.This is useful for multiple Workspace ONE environments (Prod, UAT, DEV). Remember that realm must be all CAPs and subdomain is all lowercase.
/etc/init.d/vmware-kdc init --force --realm SUBDOMAIN.IDM.DOMAIN.COM --subdomain subdomain.idm.customer.comservice horizon-workspace startservice vmware-kdc startFlags: –force (force init erasing previous initialized KDC) –realm (by convention, this is all caps, this is the Kerberos realm and is usually the DNS domain of the customer. It can also be a DNS sub domain if the customer needs to have multiple Kerberos realms for multiple systems –subdomain (this is not related to subdomain in the conventional sense. This should be your WS1 Access FQDN that the end users use)
Please be aware that this procedure is unnecessary if the vmware-kdc init command is run with the correct values in the first place. If a reinitialization is necessary, then there are two better options.
If the KDC CA cert doesn’t have to be maintained, then just run vmware-kdc init --force with the right arguments, or
If the KDC CA cert does need to be maintained (so that the WS1 UEM profile doesn’t have to be updated), then copy the kdc-cacert.pem, kdc-cert.pem, and kdc-certkey.pem files from /opt/vmware/kdc/conf to a temporary area, make them readable by the horizon user, and then run vmware-kdc init --force but use the arguments that allow the certs to be passed in (–kdc-cacert, –kdc-cert, and –kdc-certkey).
How to update kdc subdomain without regenerating certificates
Navigate to the kdc-init.input file and change the subdomain there to the new value
service vmware-kdc updateservice vmware-kdc restart/etc/init.d/vmware-kdc kadminaddprinc -clearpolicy -randkey - requires preauth HTTP/(insertnewsubdomain)ktadd -k kdc-adapter.keytab HTTP/(insertnewsubdomain)listprincsWindows VIDM uses the cloud KDC which only requires the appliance to reach outbound on 443 and 88. The clients also need to be able to reach outbound on 443 and 88.
REALM used in WS1 UEM and in Windows VIDM is: OP.VMWAREIDENTITY.COM
What is not working?
If the device cannot login, why is the login failing?
Is it failing for all devices or just a subset or one device?
What version of iOS is being used?
Can iOS device reach the KDC? (use sniffer to check)
Are the DNS SRV entries set up correctly?
Is the subdomain value for the on-premises KDC set correctly?
If using Hybrid KDC, does registration fails to save the adapter configuration?
Is the AirWatch tunnel being used?
Is REALM domain in KDC and DNS all CAPITAL LETTERS, and subdomain all lower letters?
If you are using {Enrollment User} as the Kerberos principal in the device profile and {Enrollment User} is mapped to the sAMAccountName, and the sAMAccountname does not match the first portion of the user’s UPN – which is everything before the @ sign – then mobile SSO will fail.
For certificate authentication to work (Android SSO included), UserPrincipalName value in SAN needs to be set to user’s UPN or Email, in user’s certificate or SAN must have user’s email if UPN is missing.
When creating the certificate template in ADCS - use the KerberosAuthentication template with the following modifications: change the template name, change Subject Name to accept it from the request, add a new “Kerberos Client Authentication” auth policy (oid=1.3.6.1.5.2.3.4), add the AirWatch service account to the list of users that can use the certificate.
❗️The authentication policy oid must be exactly as specified
Built-in Kerberos adapter log: /opt/vmware/horizon/workspace/logs/horizon.log KDC logs: /var/log/messages log and grep for ‘pkinit’ to see the errors returned.
grep -i 'pkinit' /var/log/messagesThis may return errors like vmw_service_matches_map: “HTTP/idm.test.ws@TEST.WS” is not a match for the service_regex and Server not found in Kerberos database. Basically, either DNS is not correct or built-in KDC Server is not properly initialized.
When having issues troubleshooting Mobile SSO for iOS, install the GSS Debug Profile on the device to retrieve more verbose Kerberos logs.
Link: https://developer.apple.com/services-account/download?path=/iOS/iOS_Logs/Enterprise_SSO_and_Kerberos_Logging_Instructions.pdf ❗️Action requires Apple Dev Account!
There is no tcpdump on WS1 Access, and there is no possibility to install it due to dependency problems (written & tested on SUSE Ent Linux version of WS1 Access).
Use Python2 to write your own sniffer of packets for Access-vIDM (fresh build on GitHub):
# Filename = sniffer.py
# Packet sniffer script 0.3
# Made by Alexei Rybalko for vIDM-Access Server
# Based on SUSE Ent. Linux 11 with python2
# Usage:
# python sniffer.py 192.168.1.1
# Will sniff any packets going from or coming into IP=192.168.1.1, includes ping-ICMP/TCP/UDP
import socket, sys
from struct import *
if __name__ == "__main__":
s = socket.socket(socket.AF_PACKET, socket.SOCK_RAW, socket.ntohs(0x0003))
if (not sys.argv[1]):
print("Enter IP Address to filter packets from!")
sys.exit(0)
while True:
packet = s.recvfrom(65565)[0]
eth_header = packet[:14]
eth = unpack('!6s6sH', eth_header)
eth_protocol = socket.ntohs(eth[2])
if eth_protocol == 8: # IP
ip_header = packet[14:34]
iph = unpack('!BBHHHBBH4s4s', ip_header)
ttl = iph[5]
protocol = iph[6]
s_addr = socket.inet_ntoa(iph[8])
d_addr = socket.inet_ntoa(iph[9])
#print "Source IP: " + s_addr
#print "Destination IP: " + d_addr
if (s_addr == sys.argv[1]) or (d_addr == sys.argv[1]): # IP Address only the one provided as argument to script
if protocol == 6: # TCP
tcp_header = packet[20:40]
tcph = unpack('!HHLLBBHHH', tcp_header)
source_port = tcph[0]
dest_port = tcph[1]
print("--TCP--")
print "Source port: " + str(source_port)
print "Destination port: " + str(dest_port)
elif protocol == 1: # ICMP
icmp_header = packet[20:24]
icmph = unpack('!BBH', icmp_header)
icmp_type = icmph[0]
code = icmph[1]
checksum = icmph[2]
print("--ICMP--")
print "Type: " + str(icmp_type)
print "Code: " + str(code)
elif protocol == 17: # UDP
udp_header = packet[20:28]
udph = unpack('!HHHH', udp_header)
source_port = udph[0]
dest_port = udph[1]
print("--UDP--")
print "Source port: " + str(source_port)
print "Destination port: " + str(dest_port)
else:
print('Unknown Protocol!')