I have been keenly interested in the information coming out as a result of the ongoing investigation of the recent SolarWinds cybersecurity breaches, particularly the vectors used to impersonate identity providers for services like email. I have spent the better part of the last 10 years focusing my expertise on cloud and hybrid identity solutions like, Azure AD, ADFS, AD Connect and Azure AD B2C. While some of the topics have been complex, I always seek to explain these systems in terms the anyone could understand. The video below speaks to an airport analogy that I have used for many years to broker understanding of the concepts.
If you comprehend the basics of ticket or token based identity systems there are two assumptions that are fundamentally required.
- The shared secret between the resource (or service provider) and the identity (or token) provider is secure and is not known by any other parties (imposters). In most identity systems this would be the token signing certificate.
- The issuance of the token (or ticket) cannot happen without valid authentication (proving that you are who you say you are) inside of the identity provider.
When an outside party is able to steal the certificates with private keys that are critical to establishing the validity of the identity provider, they are able to mimic the valid identity provider (IdP). Because the resource (or service) provider, in this case the email application, trusts implicitly the tokens issued by the identity provider and verifies their authenticity using these shared certificates they are fully susceptible to counterfeit tokens if the imposter is using valid certificates for signing and/or encrypting tokens.
Now for the tricky part, how do we begin to remedy this issue (permanent remediation is an illusion). First and foremost you will need to:
- Revoke old certificates for the identity provider (optional)
- Change certificate (with private key) in the identity provider
- Secure the certificate credentials where access is controlled, monitored and audited
- Share the new certificate (with public key) with all of your resource partners
- Confirm access is being controlled by the new credentials successfully
Now for a little guidance for each step.
Revoke old certificates for the identity provider (optional)
This step is dependent on the Certificate Authority that issued your certificate. This can identified by opening the certificate file and viewing
In this case, we use GoDaddy as our certificate authority so the commensurate guidance on revoking a certificate can be found at: //www.godaddy.com/help/revoke-a-certificate-4747
Determine your certificate provider and revoke the certificate so that it will appear in the certificate authorities certificate revocation list (CRL) and will thereby become unusable to imposters. S expressed in the article above, you don’t necessarily have to revoke the certificate, but can re-key it as expressed below.
Change certificate (with private key) in the identity provider
The changing of the key can effectively result in the same results for imposters using a stolen certificate with private key. In our case the procedure is found at: //www.godaddy.com/help/rekey-my-certificate-4976
Please do not underestimate the securing of the environment of the IdP and in the case of ADFS use best practices guidance: //docs.microsoft.com/en-us/windows-server/identity/ad-fs/deployment/best-practices-securing-ad-fs
There are also certificate guidance that matters, especially if you use the same cert for SSL and Token Signing which is often the case: //docs.microsoft.com/en-us/windows-server/identity/ad-fs/technical-reference/ad-fs-and-keyspec-property
Next we need to think about making sure we have strong measures for protecting, managing and monitoring these critical credentials, we might say “keys to the Kingdom”.
Secure the certificate credentials where access is controlled, monitored and audited
I have often seen certificate files containing private keys stored on servers in casually protected directories. Granted, these files are almost always password protected their access, but monitoring who has accessed or copied them to other places is of paramount importance. This can be remedied by use of a key vault.
There are many key vault technologies available out there. I will mention Azure Key Vault as one such solution. It has the ability securely store these credentials and provide role based access controls (RBAC) that leverages your existing user authentication and authorization. There are also controls for monitoring and alerting, if desired.
The following articles and videos are useful for developing a quick understanding:
- Managing certificates in Azure Key Vault (video): //www.youtube.com/watch?v=D2fi1mvI-NM
- Why use Azure Key Vault?: //docs.microsoft.com/en-us/azure/key-vault/general/overview#why-use-azure-key-vault
- Import a certificate: //docs.microsoft.com/en-us/azure/key-vault/certificates/tutorial-import-certificate
- Import a certificate FAQ: //docs.microsoft.com/en-us/azure/key-vault/certificates/import-cert-faqs
- Monitor and Manage certificates: //docs.microsoft.com/en-us/azure/key-vault/certificates/create-certificate-scenarios
- Certificate Access control: //docs.microsoft.com/en-us/azure/key-vault/certificates/certificate-access-control
- (Preview) Azure Key Fault RBAC (video): //www.youtube.com/watch?v=oYzFWOrZMKc
- Lockdown Azure Key Vault using firewall rules and VNETS: //www.youtube.com/watch?v=zTU3a_X40PU
- Why use Azure AD Connect Health?: //www.youtube.com/watch?v=zTU3a_X40PU
Advanced concept of keeping certificates updated regularly supporting these actions in a lifecycle that can be automated to some degree: //docs.microsoft.com/en-us/azure/key-vault/certificates/tutorial-rotate-certificates
Share the new certificate (with public key) with all of your resource partners
This is perhaps the most cumbersome part of our exercise! With some Identity Providers this is easier than others. The attack vector for the SolarWinds breach was based on SAML protocol so I will focus on some guidance on SAML partners using Microsoft’s ADFS platform as an identity provider.
Federation Partners with applications that use your identities, in other words you are the identity provider and your partner needs to update their trust definition in order to authenticate your users in their application (Relying Party). ADFS and some other IdP’s have the capability to provide information over the network via a federation metadata URL that provides easy access for updating the trust, sometimes even automatically. Inside of an ADFS system it would look like this: //docs.microsoft.com/en-us/azure/active-directory/hybrid/how-to-connect-fed-saml-idp#use-the-tool-to-verify-that-single-sign-on-has-been-set-up-correctly
Non-automated processes of doing this involve you have to send your certificate with the public key (not the private key!!!) to your federation partner for them to perform the update. These events are best coordinated so you can immediately test and verify the implementation.
Sample 3rd Party (BMC) manual implementation of ADFS SAML IdP: //docs.bmc.com/docs/ars9000/configure-adfs-as-an-idp-for-saml-authentication-602727782.html
Manage ADFS Certificates from Azure AD Connect: //docs.microsoft.com/en-us/azure/active-directory/hybrid/how-to-connect-azure-ad-trust
For more technical folks that want to develop a deep understanding of the mechanics, I would recommend reading blog by Michelle Ferrari: //techcommunity.microsoft.com/t5/core-infrastructure-and-security/adfs-monitoring-a-relying-party-for-certificate-changes/ba-p/259391
Be sure that you understand this portion most of all because it can result in clients inability to authenticate. It is my suspicion (unconfirmed) that something similar is the source of the big Google outage the other day, although Google is not saying what entirely happened we do know that it happened shortly after the SolarWinds news broke.
Confirm access is being controlled by the new credentials successfully
If you are using ADFS as your IdP, you can monitor the successful establishment of trusts, but you must have a valid user that can access the application to confirm access.
- Tool to Verify SAML 2.0 IdP: //docs.microsoft.com/en-us/azure/active-directory/hybrid/how-to-connect-fed-saml-idp#use-the-tool-to-verify-that-single-sign-on-has-been-set-up-correctly
- Troubleshooting certificate in ADFS: //docs.microsoft.com/en-us/windows-server/identity/ad-fs/troubleshooting/ad-fs-tshoot-certs
- Broader troubleshooting with ADFS: //docs.microsoft.com/en-us/windows-server/identity/ad-fs/troubleshooting/ad-fs-tshoot-overview
In conclusion, there is much remediation to begin and many considerations to contemplate as more evidence is produced regarding the depth and breadth of the SolarWinds breach. I wanted to provide a concise set of considerations that begin to get back on track. Especially when we consider the magnitude of a high jacked identity provider credential.