Troubleshooting one of the most important and versatile parts of the Windows PKI world is a fairly complex process, since it involves a plethora of prerequisites in order for it to work correctly. Most of the times, consultants and administrators create a lab before a PKI deployment where autoenrollment usually works relatively easy – especially if the lab does not resemble the true production environment in anything other than the domain names.
In a very recent Windows PKI lab deployment for a major customer, we had a fairly lengthy autoenrollment troubleshooting session. The lab setup was a close-match replica of the production environment, as the Active Directory has been recreated using a backup of an actual production domain controller, as well as restored Exchange servers. The case was interesting: as it always seems to happen, everything was in place for autoenrollment to work, yet the process would not fire up at all.
To make a long story short, and after finding that no firewall, virtualization or general networking issues were involved, we finally tracked the autoenrollment failure down to two issues: the one is frequently seen in large organizations and has to do with GPO (Group Policy Object) precedence and settings’ overlapping. In our case, a GPO with no autoenrollment configured would prevail in the OU that the auto-enrolled users would reside and thus would not push the relevant settings to the clients. The second issue, however, was much more complex to target: as the environment has been set up many years ago, many administrators and third-party software packages have made a lot of changes in the production. One of the changes we found out that created problems, was the membership of two built-in groups: the Users and the Certificate Service DCOM access group. In the first one, the default membership should consist of Authenticated Users, Domain Users and Interactive (it did not, as you can imagine). In the second one, the Authenticated Users group should exist as a member (again, it was not – the list was empty).
We should say that in cases of autoenrollment failures, one should focus on:
- Certificate template security – make sure your users/computers have Read, Enroll and Autoenroll permissions and that the Authenticated Users group has not been deleted (it should be there with Read-only permissions).
- GPO precedence – make sure that you create a separate, enforced GPO to enable autoenrollment or, at least, that the GPOs applied to your users/computers finally lead to an enabled state for autoenrollment (RSOP is a great client-side tool to discover such issues).
- Group membership – the Issuing CA’s computer account should be a member of the Cert Publishers group in the intended enrollment domain; also, the Users and the Certificate Service DCOM Access built-in groups should have the membership detailed previously in this post.
- Server-side certificate issuance errors – a poorly configured certificate template (for example, one that requires an e-mail address in order for certificates to be issued when some user accounts may not have an e-mail address in AD) could lead to a certificate issuance request that is left in a pending or failed status, as seen in the Certification Authority console. Do check this by manually initiating a certificate request through an MMC console or a Web enrollment page (if configured) and make sure that the manual enrollment method actually succeeds in creating the intended certificate.
- Networking issues – blocked ports (RPC, HTTPS etc.) as well as problematic networking software or hardware (drivers etc.) could lead to problems in autoenrollment, Again, check if a manual request can succeed.
In June 2013, Microsoft issued an update that makes the update of CTLs (Certificate Trust Lists) easier in disconnected environments. For the purposes of automatic updating, Microsoft considers any environment that does not have access to the Windows Update site as “disconnected.”
The new update enables Windows PKI administrators to:
- change the update location from the predefined Windows Update URL to an intra-organizational shared folder that is reachable from disconnected clients
- selectively disable/enable updating of either trusted or untrusted CTLs
- create a custom set of trusted root certificates and distribute it via Group Policy.
The relevant knowledge base article can be found here, and its supporting documentation here. It is worth noting that the update is included in Windows Server 2012R2 and Windows 8.1, and does not apply to Windows XP and/or Windows Server 2003.
On August 13th, Microsoft released a security advisory update (2862973 – Update for Deprecation of MD5 Hashing Algorithm for Microsoft Root Certificate Program), applicable to Windows Vista/2008 and later client and server operating systems. According to the associated TechNet article (Protecting Against Weak Cryptographic Algorithms) the functionality is already built into Windows Server 2012R2 and Windows 8.1 preview editions and should be included when the final bits of 2012R2 and 8.1 ship. Essentially, this update gives administrators control on logging and/or blocking weak algorithm-based certificates that are known to be vulnerable to multiple attacks. More detailed information can be found on the link above.
In this second part of our discussion on Credential Roaming, we are going to discuss about its relation to key archival as well as offer some practical alternatives to credential portability.
As we saw in the previous blog post, the capability of “transporting” private keys in Active Directory is one of the basic pillars of the process’s ability to roam credentials. However, this should in no way be seen as a good way of backing them up for future restores. The procedure that should be used for key backup is the Windows PKI process of Key Archival and there is no way of replacing this with credential roaming’s private key “upload” process. In fact, even though private keys do reside inside the AD database, there is no known or supported way of extracting them in any form, in case of disaster recovery procedures. Moreover, if the user deletes his certificate along with his private key from one of his profiles, this deletion will propagate into all other profiles where the certificate has roamed to.
Even though it has already been mentioned in the previous blog post, it is worth noting that an easy and quick workaround for credential roaming is the use of roaming user profiles, which (of course) involves a lot more administrative costs and changes in relation to the activation of credential roaming in ΡΚΙ. In accordance to this, the concurrent use of user profile roaming and credential roaming is not supported.
Another workaround (although not as automated as Credential Roaming) is the storage of certificates and private keys in a smart card. This solution does make the user’s credentials available in every PC that he logs on to, however it assumes smart card hardware availability (this can also be a USB-based token) and also assumes a planned or existing smart card infrastructure, which does come with high deployment and administrative costs. However, a carefully planned and executed smart card deployment can furthermore be used for other applications, like VPN remote access, local computer and RDP user login etc. In general, and independently of the fact that a smart card deployment is a good alternative to credential roaming, the use of smart cards is recommended in all medium- & high-security PKI deployments, as many security officers are not happy with the storage of private leys inside an operating system (inside non-dedicated devices, in general).
Credential roaming is the ability of a Windows Server PKI infrastructure to support roaming certificates from a Windows computer to another Windows computer, according to where a user is logged on. The way it works is as follows: when a user requests a certificate, its local operating system generates a private and public key and, using a secure channel between the computer and the CA, it uploads the private key and the signed request to the CA. The private key is securely stored in Active Directory and can later be downloaded to any PC that a domain user logs on to.
All this is fine and dandy and, based on specs, it is the perfect solution for users that tend to use a multitude of computers for their business use. However, setting up the infrastructure to support it is not as easy as one would have thought. In this post, we will take a look at some of the disadvantages and difficulties of credential roaming. In the post to follow, we will make some more general observations as well as offer some alternatives to credential roaming. So, to cut a long story short, credential roaming:
- Demands a complicate (but not very time-consuming) procedure of installation and use, as well as many scenarios to be tested to make sure it works as expected.
- Extends the physical size of the Active Directory database, according to the issued certificates to be roamed. If the service is extended to a few hundred or thousands of certificates, we may have an AD database which will grow to a few hundred MBs larger. Accordingly, the AD backup/restore time will be bigger, the AD database will become more fragmented etc. Microsoft Support has been involved in many support cases where a sudden increase of the Active Directory database has been attributed to the activation of credential roaming.
- Demands Windows XP SP3 or Windows XP SP2 with a specific update, Windows Vista , 7 or 8. It does not support other operating systems (for example, mobile OSs).
- Needs difficult and time-consuming troubleshooting techniques for Windows XP (Vista/7 use CAPI2 logging, so it’s easier there).
- Increases the organization’s attack vector. In case we don’t use enhanced security measures at the PCs that will use private key actions (i.e., lock workstation, Bitlocker protection, users’ security training etc.), using multiple points where certificates are accessible and used, we multiply the possibility of someone extracting the private key(s).
- Poses problems in network EFS encryption. Under normal circumstances, credential roaming does not work as expected, due to the nature of the logon that it supports (credential roaming: local logon, network shares: network logon). The specific limitation is described in KB907247 (in Credential roaming will not be used when using EFS to encrypt files on a file server). EFS (and EFS using credential roaming) has not been designed to work with network shares, but only for local encryption. Some workarounds exist, such as using Roaming profiles instead of credential roaming (KB837359), use Offline Files and local encryption of the cache, use of web folders (in Remote EFS Operations on File Shares and Web Folders).