Questions related to DNS and service discovery regularly come up during deployments of Identity Management (IdM) in Red Hat Enterprise Linux in a trust configuration with Active Directory. This blog article will shed some light of this aspect of the integration.
We will start with a description of the environment. Let us say that the Active Directory environment consist of four servers split in pairs between two datacenters in two different geographic regions with low bandwidth and high latency network between them. For the sake of this article it does not really matter whether there are several domains or not. Of cause factoring in the hierarchy of domains and what replication agreements these servers have matters but for discovery and affinity conversation we will assume that all four mentioned servers represent one AD domain and replication is working properly so the content of the servers is the same. We will also introduce four IdM servers, two in each datacenter. There is a one-way trust relationship established between the IdM and Active Directory forests (IdM trusts AD). Finally there is a Linux client in each datacenter. This Linux client represents all the Linux clients in the datacenter assuming they are configured in the same way. The following diagram represents this setup.
The horizontal lines between AD and IdM servers denote replication between those servers and yellow arrows show one-way trust.
The questions about affinity and discovery arise mostly in the setups like this which is in fact quite popular. So how exactly can I configure the client to prefer local servers? The question is actually more complex than that. Let us look under the hood and see what communication is actually going on.
On the next diagram we will remove replication and trust lines and arrows to make diagram cleaner and more readable but, please, keep in mind that they are still there.
The first diagram shows Kerberos protocol communication that is used for authentication of the users as well as for authentication of the client itself. As you see clients can talk to either of the AD servers and to either of IdM servers. It is also important to understand that there is a client component that runs on IdM servers themselves. Those “embedded” clients are used to pull identity information from Active Directory so those clients also need to be able to connect to the right AD servers.
In addition to the communication over Kerberos protocol there is a communication over LDAP protocol. The following diagram shows the details.
The difference is that the client only talks LDAP to IdM servers however the “embedded” client on the IdM server can talk to either of the AD servers.
So the task of preventing the communication to go over the low latency network to another datacenter boils down to the following list of items:
- Make sure that client talks to local IdM servers over LDAP
- Make sure that client talks to local IdM servers over Kerberos
- Make sure that client talks to local AD servers over Kerberos
- Make sure that embedded client on the IdM talks to local AD servers over Kerberos
- Make sure that embedded client on the IdM talks to local AD servers over LDAP
So to accomplish the goal LDAP and Kerberos configuration needs to be considered on the clients and on the IdM servers.
There is yet one more factor that we have not brought up yet. Active Directory has a notion of sites. An Active Directory client usually detects the presence of site definition by looking up information in LDAP and then resolving the actual servers using DNS. If a site is configured DNS lookup will return the list of local AD servers for client to connect to. Let us assume that AD server in each of the two datacenters form a site.
There is a similar capability in IdM but only if the IdM DNS is used. Now we have all the permutations of the setup established and can jump to the configuration options.
Configuring LDAP on the client
IdM DNS is used
Lookup of the corresponding services can be done automatically based on the DNS. Clients by default will use this method unless it is explicitly overwritten during the ipa-client installation by providing failover parameters in the command line or if the SSSD configuration is modified after the installation. See next section for more details about altering the configuration and not using DNS discovery.
If the default configuration is chosen and not overridden and IdM DNS is used then the client can take advantage of the feature called DNS locations. It is similar in concept to the AD sites. The client is expected to have IdM DNS configured in the resolv.conf. If locations are configured in IdM the client will always receive the local servers in the DNS lookup for the IdM service when it asks for SRV records. For more information on how to configure this feature see Linux Domain Identity, Authentication, and Policy Guide.
IdM DNS is not used
If IdM DNS is not used the chosen DNS server would not have appropriate information about the IdM server topology and thus service discovery method can’t be used. Clients would have to be explicitly configured with the failover information. This is done by explicitly configuring SSSD on the client to use primary and secondary server configuration. One can read more about the details of this configuration in sssd-ldap man page (look for Failover section). Besides changing SSSD configuration one can use additional arguments during ipa-client installation to provide an explicit list of primary and secondary servers the client should connect to.
Configuring Kerberos on the client
There are two parts of the Kerberos configuration. One is related to the communication with IdM servers and another to communication with AD servers.
For communication with IdM servers the client will use same information that is described in the previous section so the discovery or explicit configuration would apply to the Kerberos communication too. However communication with AD requires special handling. There is currently no way for the client to automatically discover the right AD servers to talk to. So the only way for the clients to limit them to the specific AD servers is to alter krb5.conf configuration to add AD realm information and explicitly list the AD servers to talk to. For more information how to do it see krb5.conf man page.
There is an outstanding feature request that the SSSD project is aware about. It is on the roadmap but so far, as of the time of writing, the automatic discovery of the AD servers in the same location is not yet implemented.
Configuring embedded client on the server for Kerberos and LDAP
The embedded client on the IdM server is configured as a client of the Active Directory server. For several releases it already has been capable of discovering AD sites and using them. However to do the discovery the first lookup might hit an AD server in the remote location that it not accessible or is reachable over the slow network. In such situations for the embedded client not to give up too early the dns_resolver_timeout should be increased. For clarity the man pages were recently updated to better explain the meaning and use of this timeout. These changes did not make Red Hat Enterprise Linux 7.4 but will be included in the next minor release. There is also a plan to increase the default value of the timeout in the next community release which might be included into next Red Hat Enterprise Linux minor release.
Before 7.4 the client on the IdM server used an automated discovery algorithm to determine the right AD site and AD servers to use. In some cases autodiscovery did not return the right information. There are different reasons why wrong information might be returned. In real deployments the servers are added and removed and in some cases the discovery would return an AD server that was decommissioned or completely firewalled.
To overcome this issue a new feature has been added in recent community release and made its way into Red Hat Enterprise Linux 7.4. It allows explicit overrides of some of otherwise automatically discoverable information. The following page in the SSSD site describes the related changes in a lot of details. The corresponding changes have been made to the SSSD man pages. Unfortunately a separate chapter of the Red Hat documentation that helps with configuration of the failover and domain discovery in the complex setups like this did not make the documentation available from Red Hat web site at the moment of writing. The corresponding team is actively working on it so by the time you read this you most likely will be able to find a corresponding chapter in the Windows Integration Guide.
If you are running IdM version that is still not updated to 7.4 you can use the same failover related overrides with AD server as the ones described in the sssd-ldap man pages for IdM servers. Those settings will explicitly pin client on the IdM server to the specific set of AD servers you chose without respect to sites. This, however, might not work well in the cases when you have a complex hierarchy of domains since there is no way to express which preferred servers belong to which domain. The general recommendation would be to upgrade your IdM server to 7.4 and use ability to pin client running on the server to a specific AD site.
As you can see getting the failover configuration right is not simple. But now you have all the moving parts covered and can make sure that all the service in one datacenter only talk to the services in that same datacenter.
As always I will be looking forward to your questions and comments as well as suggestions regarding next blog topics.