Microsoft ADFS
Integration manual for ReadonlyREST Enterprise with the on-premises Active Directory Federated Services Single Sign-on from Microsoft.
How to Connect ROR Enterprise with SAML and ADFS
ReadonlyREST (ROR) Enterprise allows for complex authentication and authorization configurations with Kibana and Elasticsearch. When Elasticsearch is combined with Kibana, a data visualization dashboard, the combination provides a powerful way to ingest logs and analyze data.
To access that data, many enterprises manage users in a central directory. This directory could be an Active Directory (AD) instance, in the case of a Windows-centric environment, or a cloud directory provider, such as Google Cloud Identity, in a cloud-based environment. Instead of integrating these services directly into a product, an abstraction layer such as SAML can provide authentication and authorization and tie into different back ends as necessary.
ReadonlyREST provides a free Elasticsearch plugin that provides advanced authentication options. When it is combined with the ReadonlyREST Enterprise plugin for Kibana, integrating SAML authentication into the authentication process becomes easy.
This article will walk through the process of setting up an entire environment in order to demonstrate how the ReadonlyREST free and Enterprise plugins integrate with Active Directory Federation Services (AD FS) to provide SAML authentication.
In this tutorial, you will learn how to:
Provision Azure Virtual Machines to host Active Directory, Elasticsearch, and Kibana
Install and configure Active Directory (AD) Services
Provision sample AD users
Install and configure Active Directory Certificate Services (AD CS)
Install and configure Active Directory Federation Services (AD FS)
Install and configure ElasticSearch and the ReadonlyREST Free Plugin
Install and configure Kibana and the ReadonlyREST Enterprise Plugin
Provisioning Azure Virtual Machines to Host Active Directory, Elasticsearch, and Kibana
Any Windows Server 2016 Virtual Machines (VM) can be used for this process; however, in this demonstration, the Microsoft Azure environment will be used to provision and host the VMs.
You can name your VMs whatever you would like. This article will refer to the names listed below for consistency.
Virtual Machine 1: lc-win2019-02
Roles: Active Directory, AD Certificate Services, AD Federation Services, DNS
Memory: 4GB
Virtual Machine 2: lc-win2019-03
Roles: Elasticsearch, Kibana
Memory: 8GB
These Azure VMs will be Pay-As-You-Go and Spot Instances for affordability. The example shown below is for the Elasticsearch and Kibana VM which will be duplicated for the Active Directory VM but will have 4GBs of memory instead of 8GB.
Please note that Azure Spot Instances cannot be resized after creation.
Provisioning Virtual Machines
Log into the Azure portal using a Pay-As-You-Go subscription.
Create a new virtual machine.
If you do not already have a resource group created to serve as a home for the VMs, select Create new and create the resource group.
Name your virtual machine appropriately, and choose the details for your instance, as shown in the example below.
You can use the default hard drive sizes and Standard HDD disks for this environment.
The default Networking options will also work here.
As will the default Management options.
No additional Advanced options are necessary.
If you would like to tag your VMs for later categorization and tracking, you can do so here.
Finally, create the VM.
After this VM has been created, create one more to host the Active Directory and related services. In the end, you should have two VMs as outlined above.
Installing and Configuring Active Directory (AD) Services
After the two VMs have been provisioned, the next step is to set up directory services on the first VM, lc-win2019-02.
Installing Active Directory Services
Once you are logged into lc-win2019-02, choose Add Roles and Features on the Server Manager screen.
Select Role-based or feature-based installation.
Select the correct server from the server pool.
Select Active Directory Domain Services, and add the additional features as prompted.
No additional features are necessary since the default options work.
Click Next on the Active Directory Domain Services informational screen.
Finally, select Restart the destination server automatically if required. Click Yes when prompted, and then click Install.
Once installation has finished, click on Close.
Configuring Active Directory Services
If DNS has not been installed already, the role installation screen may pop up in the middle of the Active Directory installation. Installation instructions for the DNS role are shown after the Configuring Active Directory Services section below.
Click on Promote this server to a domain controller, which will allow you to see the Deployment Configuration screen.
Name the domain. In this case, use ad.lc-test.local.
This name was arbitrarily chosen. Using a subdomain such as “ad” instead of your actual domain (i.e., [lc-test.local](http://lc-test.local\ is recommended.
Select Windows Server 2016 as the functional level. For the domain controller capabilities, choose Domain Name System (DNS) server. Set a Directory Services Restore Mode (DSRM) password.
The following DNS Options warning message can be disregarded:
Set the NetBIOS domain name, which is usually the short name prior to the host name (e.g., AD), and click Next.
Use the default paths, and click on Next.
On the Review Options screen, click Next if everything looks correct.
On the Prerequisites Check screen, click Install.
You’ll see a number of warnings related to the fact that this is a test environment. They can be safely ignored.
When you click Close, you will have a successful configuration.
Click Close on the restart prompt.
Configuring the Domain Name Services (DNS) Role
Click on the DNS Services role, add the additional features as requested, and click Next.
If you are using DHCP for the server (this is not recommended for a production service), then you will see the validation warning shown below. It can be disregarded. Click on Continue and Install.
Click Next on the Features screen, since no additional features are necessary.
Click Next on the informational DNS Server screen.
Click Install on the Confirmation screen.
Select Restart the destination server automatically if required, and click Install. Finally, click Close, and you will have a successful installation.
Joining Computers to the Domain
Next we need to join the second server—the one hosting Elasticsearch and Kibana—to the domain.
Open an RDP connection to the second server. Then, open Notepad as an Administrator, and open the file C:\Windows\System32\drivers\etc\hosts.
Add the IP address and hostnames for the domain controller.
These will reflect the IP addresses and hostnames you chose for your configuration:
FQDN: 10.0.0.5 - ad.lc-test.local
NetBIOS: 10.0.0.5 - ad
Additionally, you will need to change your network adapter DNS to point to your domain server; in this case, it is 10.0.0.5.
If the system restarted, open an RDP connection, and then open the System screen under the Control Pane and select Advanced System settings.
Click on Change to add this server to the domain.
Enter the domain (e.g., ad.lc-test.local).
Click on OK, and enter the credentials of the account that has privileges enabling it to add the domain.
Restart the server after joining it to the domain.
Provisioning Sample AD Users
For testing purposes, it can be useful to provision additional users within the Active Directory. The following PowerShell script, which should be run on the domain controller, will make this easy. Note that we are setting the mail attribute which will be used for the SAML username.
Import-Module -Name 'ActiveDirectory'
$Domain = 'ad.lc-test.local' $OU = 'CN=Users,DC=ad,DC=lc-test,DC=local'
$Users = @{ "TestUser1" = "testPass1" "TestUser2" = "testPass2" "TestUser3" = "testPass3" "TestUser4" = "testPass4" "TestUser5" = "testPass5" }
$Users.GetEnumerator() | ForEach-Object { $Name = $_.Key $Password = $_.Value
$Params = @{ "Name" = $Name "Path" = $OU "AccountPassword" = (ConvertTo-SecureString -AsPlainText $Password -Force) "Enabled" = $True "DisplayName" = $Name "PasswordNeverExpires" = $True "CannotChangePassword" = $True "EmailAddres" = "$Name@$Domain" }
New-ADUser @Params }
Installing Active Directory Certificate Services (AD CS)
Click on Active Directory Certificate Services, and add the additional features as prompted.
Since no additional features are necessary, allow defaults, and click on Next.
On the Active Directory Certificate Services informational screen, click Next to continue.
Select the Certification Authority role services, and click Next.
Select Restart the destination server automatically if required, and click on Install.
Finally, click on Close when the installation has been completed.
Configuring Active Directory Certificate Services
Click on Configure Active Directory Certificate Services.
Select the default administrative user for AD CS credentials.
Under Role Services, check Certification Authority, and click Next.
Select Enterprise CA, and click on Next.
Click on Root CA, then click on Next.
Click on Create a new private key, and then click on Next.
Select RSA#Microsoft Software Key Storage Provider, a default key length of 2048, and a hash algorithm of SHA256. Click Next.
Use the defaults given for the CA Name, and click on Next.
Select a validity period of 5 years, and click on Next.
Leave the default database locations in place, and click on Next.
Click on Configure and Close when the configuration process has been completed.
Installing and Configuring Active Directory Federation Services (AD FS)
Previously, it was recommended that AD FS should not be installed on the same server as the DC because IIS was installed as part of that process. As of 2012, this recommendation has changed, since AD FS does not use IIS anymore. Now, mounting AD FS and DC on the same server is advised for domains under 1000 users.
Provisioning SSL Certificate Templates
Open the Certification Authority MMC snapin, right-click on Certificate Templates, and click on Manage.
Select Duplicate Template on the Web Server template.
Enter “SSL Certificates” in the box labeled Template display name on the General tab.
On the Security tab, click on Enroll and Allow for Authenticated Users, and, finally, Apply the configuration.
Right-click on Certificate Templates → New → Certificate Template to Issue.
Select SSL Certificates from the Certificate Template list, and click OK.
Provisioning the SSL Certificate
Open the Certificates MMC snapin for the Local Computer. Navigate to Personal → Certificates, and right-click to open All Tasks → Request New Certificate.
Click Next on the Before you Begin screen.
Click Next on the Select Certificate Enrollment Policy screen.
Select SSL Certificates, and click on More information is required to enroll for this certificate. Select Click here to configure settings to configure the certificate.
Add the following details on the Certificate Properties Subject screen, then click OK:
Subject Name
Common Name: CN=lc-win2019-02.ad.lc-test.local
Alternative Name
DNS: lc-win2019-02.ad.lc-test.local
DNS: enterpriseregistration.ad.lc-test.local
Click on Enroll to request the certificate, then click Finish.
Setting up Active Directory Federation Services
Select the Active Directory Federation Services role, and click Next.
Click Next on Select Features, since no additional features are needed.
Click Next on the Active Directory Federation Services informational screen.
Check Restart the destination server automatically if required, and click Install and Close when the installation has completed.
Creating a Group Managed Service Account and Adding a KDS Key
It is best to use a gMSA (group Managed Service Account) instead of a traditional sMSA (standalone Managed Service Account). The primary difference between the two is that, in a gMSA, the Windows operating system manages the password for the account instead of relying on the administrator to do it.
Before we can select the gMSA, however, we need to add a KDS Root Key. To avoid non-blocking warnings later in the process, this key should be added with an effective date of 10 hours prior to the current date and time.
Open a PowerShell session as an Administrator, and run the following command to add the KDS root key:
Add-KdsRootKey -EffectiveTime ((Get-Date).AddHours(-10))
If you do not take this step, you will see the following error when attempting to add the gMSA account:
To create a gMSA account to use with the AD FS service, use the PowerShell script provided below. If you get an “access denied” error when running Install-ADServiceAccount, you may need to restart the server first.
$Name = 'sa_adfs'
$Params = @{ "Name" = $Name "DNSHostName" = 'lc-win2019-02.ad.lc-test.local' "PrincipalsAllowedToRetrieveManagedPassword" = 'lc-win2019-02$' "ServicePrincipalNames" = 'http/lc-win2019-02.ad.lc-test.local' }
$ServiceAccount = New-ADServiceAccount @Params
Install-ADServiceAccount -Identity $Name
Add-ADComputerServiceAccount -Identity 'lc-win2019-02' -ServiceAccount $ServiceAccount
Configuring Federation Services
Click on the link that says Configure the federation service on this server.
Select Create the first federation server in a federation server farm, and click Next.
On the Connect to Active Directory Domain Services screen, leave the default user selected, then click on Next.
Select the previously created SSL Certificate, and enter “LC Test” for the Federation Service Display Name. Click Next.
On the Specify Service Account screen, click on Select to use an existing account and locate the sa_adfs service account that was previously created. Click Next.
Select Create a database on this server using Windows Internal Database, and click Next.
Click on Next under Review Options.
Verify the Pre-requisite Checks, and click on Configure.
Click on Close, and restart the server.
The warnings shown below can be disregarded for this test instance:
Once the server has restarted, open an Administrative PowerShell session, and run the following command to enable the IdP Signon Page:
Set-ADFSProperties -EnableIdPInitiatedSignonPage $True
Verify that AD FS metadata is being returned by navigating to the following URL:
https://{FQDN of AD FS Server}/adfs/fs/federationserverservice.asmx
Setting Up ReadonlyREST Relying Trust
Open the AD FS MMC snapin, right-click on the Relying Party Trusts folder, and select Add Relying Party Trust.
Select Claims aware, and click Start.
Choose Enter data about the relying party manually, and click Next.
Enter a Display Name (in this case, “ror”), and click Next.
It’s not necessary to specify a token encryption certificate, so click Next to continue.
Select the option Enable support for the SAML 2.0 SSL service URL, and enter:
The saml_adfs will change depending on the name chosen in the configuration of the kibana.yml file.
Enter the Relying party trust identifiers, in this case, “ror.” This will match the Issuer in the Kibana configuration. Click Next when you are done with this step.
On the Access Control Policy screen, select Permit everyone, and click Next.
Click Next to finish adding the trust.
Verify that Configure claims insurance policy for this application is selected, and click on Close.
Configuring Claims
Though we have not yet configured claims for Kibana, the metadata for the SAML configuration in Kibana would look similar to the following, if you were able to view it:
The important section to note concerns the claims issuance policy. We need to return a NameID format in the form of an emailAddress by entering the following code**:**
<NameIDFormat> urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress </NameIDFormat>
Therefore, we need two rules: one to pull back the LDAP attribute from the Active Directory, and another to transform that data into the correct format.
Click on Add Rule on the Edit Claim Issuance Policy for ror screen.
For the first rule, choose Send LDAP Attributes as Claims, and click Next.
Choose the AD Attribute to return—in this case, the email address—and click Finish on the Configure Rule screen of the Add Transform Claim Rule Wizard.
Claim rule name: LDAP Email
Active Store: Active Directory
LDAP Attribute: E-Mail-Addresses
Outgoing Claim Type: E-Mail Address
Click on Add Rule, then choose the Transform an Incoming Claim claim rule template, and click on Next.
Enter the transformation details as listed below, and, on the Edit Rule screen, click on OK.
Claim rule name: Email Transform
Incoming claim type: E-Mail Address
Outgoing claim type: Name ID
Outgoing name ID format: Email
Pass through all claim values: Selected
Click OK to save the rules.
Please note that the order of the rules on the Edit Claim Issuance Policy screen is important.
Updating Relying Party Trusts
Navigate to the Relying Party Trusts folder, right-click on the ror trust, and select Properties.
Click on the Endpoints tab, select the SAML Assertion Consumer Endpoints, and click on Edit.
Click on Set the trusted URL as default, and change the Index to 1 from 0. Click OK.
On the Endpoints screen, click on Add SAML, and enter the SAML Logout details as follows:
Endpoint Type: SAML Logout
Binding: POST
Trusted URL: https://{IP Address of Kibana Server}:5601/ror_kbn_sso_saml_adfs/notifylogout
Installing and Configuring Elasticsearch and the ReadonlyREST Free Plugin
Installing Elasticsearch
Elasticsearch will be installed on the lc-win2019-03 server provisioned with 8GB of RAM in Azure.
Locate a recent download of Elasticsearch, and install the MSI package.
At the time this article was written, the most recent version available was 7.6.2; however, you may want to check for more updated versions as they become available.
Launch the downloaded installer and click Next on the Locations screen, leaving the defaults in place.
Use the defaults on the Service screen, and click Next.
Use the defaults on the Configuration screen, and click Next.
No additional plugins are necessary; therefore, click Next.
Leave the X-Pack licenses set to Basic, and click on Install.
Click on Exit.
Installing the Elasticsearch Plugin
Navigate to the ReadonlyREST Plugin download page to enter your details. You will receive the download link in your email. Make sure to choose the Free Elasticsearch Plugin that matches your Elasticstack version.
Download the plugin, open an Administrative command prompt, and navigate to the Elasticsearch program directory. Run the plugin installation by entering the following:
cd "C:\Program Files\Elastic\ElasticSearch\7.6.2\bin"
elasticsearch-plugin.bat install file:///C:/Users/lc-admin.AD/Downloads/readonlyrest-1.19.4_es7.6.2.zip
Navigate to the C:\ProgramData\Elastic\ElasticSearch\config directory, and create the file readonlyrest.yml.
Open the readonlyrest.yml file in Notepad to run this very basic configuration that configures the following two different access control rules: 1. “::KIBANA-SRV::”—this rule allows the Kibana server to authenticate to Elasticsearch using digest authentication with the username “kibana” and password “kibana.” 2. “ADFS Users”—this rule uses the ror_kbn_auth method which allows SAML authenticates to succeed.
Create a random 256-character signature_key. This key will be shared between Kibana and Elasticsearch.
Please note that the kbn1 identifier must match in the ror_kbn_auth and ror_kbn sections; however, any names can be used for them.
Restart Elasticsearch Windows Service. This can be done in the Services MMC snapin.
Installing and Configuring Kibana and the ReadonlyREST Enterprise Plugin
Installing Kibana
Locate a recent download of Kibana, and download the zip package. At the time this article was written, the most recent version was 7.6.2. You may want to check for more updated links as they become available.
Extract the Kibana installation. Note that this is a rather large file. If you have trouble with the default Windows zip extractor, you may want to try a tool such as 7-Zip.
Move the extracted folder to C:\kibana. This may require you to rename the folder.
Open an administrative command prompt, and navigate to the Kibana directory to run the kibana.bat batch file and start Kibana.
Once Kibana has started, navigate to [http://localhost:5601](http://localhost:5601\ to verify that Kibana is functional.
Creating a Self-Signed Certificate for Kibana
It is necessary to make Kibana operate under SSL for AD FS to perform SAML authentication.
The easiest way to generate a self-signed certificate using the required format is to use OpenSSL. A Windows version of this tool available for download is located here.
If Microsoft Visual C++ 2017 Redistributables (64-bit) is not already installed, click Yes to download the installation and run the installer first.
Accept the license agreement, and click on Install.
Back on the OpenSSL installation, click on I accept the agreement, then click on Next.
Click Next on the Destination Location screen.
Click Next on the Select Start Menu Folder screen.
Select The Windows system directory on the Additional Tasks screen, and click Next.
Click on Install.
Click on Finish.
Open an administrative command prompt, and run the following command to create the certificates in the specific X509 PEM format that Kibana requires:
"C:\Program Files\OpenSSL-Win64\bin\openssl.exe"
req -x509 -sha256 -nodes -days 730 -newkey rsa:2048 -keyout localhost-key.pem -out localhost.pem -subj "/C=US/ST=IL/L=Bloomington/O=lc-test/CN=10.0.0.6"
Change the subj to one that is more indicative of your installation. Make sure the CN={IP Address} matches the accessible IP of your Elasticsearch/Kibana server.
Locate the newly created pem certificates and copy them to C:\kibana\ssl_cert.
The ssl_cert directory will need to be created first. For our purposes here, it has been arbitrarily named.
Restart Kibana by entering Ctrl-C in the running command prompt window and then re-running kibana.bat.
Installing the ReadonlyREST Enterprise Plugin
Navigate to the ReadonlyREST Plugin download page to enter your details. You will get the download link in your email. Making sure to choose the Enterprise Kibana Plugin and match it with your Elasticstack version.
The email that you receive will contain installation instructions. The link will be time-limited, as shown below.
Navigate to C:\kibana\config, and locate the kibana.yml configuration file.
Open the kibana.yml file in Notepad and update it with the following details:
elasticsearch.username—This field matches the first part (pre-colon) of the auth_key in the readonlyrest.yml Elasticsearch configuration file.
elasticsearch.password—This field matches the second part (post-colon) of the auth_key in the readonlyrest.yml Elasticsearch configuration file.
elasticsearch.ssl.verificationMode—Set the value to “true” to ignore SSL errors. This is useful when working in a test environment.
xpack.security.enabled—This must be disabled for ReadonlyREST to work.
[server.host](http://server.host—By default, this will be [localhost](http://localhost\; but, we need to use a routable address, which, in this case, is the 10.0.0.6 IP of this server.
server.ssl.enabled—This is used to turn on SSL and respond to https.
server.ssl.certificate—This is the location of the public key certificate.
server.ssl.key—This is the location of the private key for the certificate.
readonlyrest_kbn.logLevel—The value is set to “debug” to enable troubleshooting in the console.
readonlyrest_kbn.clearSessionOnEvents—This clears the session on a successful login event.
readonlyrest_kbn.auth.signature_key—This must match the 256-character value in the signature_key attribute of the readonlyrest.yml Elasticsearch configuration file.
readonlyrest_kbn.auth.saml_adfs.buttonName—This is the name of the login button on the login screen of Kibana.
readonlyrest_kbn.auth.saml_adfs.enabled—This enables the SAML SSO configuration.
readonlyrest_kbn.auth.saml_adfs.type—For AD FS, this must be “saml.”
readonlyrest_kbn.auth.saml_adfs.issue —This is the unique identifier that was defined in the AD FS Relying Party Trust configuration, in this case, “ror.”
readonlyrest_kbn.auth.saml_adfs.protocol—AD FS requires https.
readonlyrest_kbn.auth.saml_adfs.entryPoint—This is the entry point for AD FS—https://{AD FS Server}/adfs/ls.
readonlyrest_kbn.auth.saml_adfs.logoutUrl—This is the logout call to AD FS— https://{AD FS Server}/adfs/ls?wa=wsignout1.0.
readonlyrest_kbn.auth.saml_adfs.kibanaExternalHost—This is the address and port without the protocol preceding (i.e., https).
readonlyrest_kbn.auth.saml_adfs.usernameParameter—This configuration is only doing authentication, and it must match the nameID parameter.
Opening the Firewall Port
To allow the AD FS server to talk to Kibana, we need to open the 5601 port on the Kibana server since [localhost](http://localhost\ is not routable.
Open the Windows Firewall with Advanced Security screen, and add a new rule under Inbound Rules. Choose Port.
Add the specific local port of 5601, and click Next.
Select Allow the connection, and click Next.
Choose all profiles (the default), and click Next.
Name the rule “Kibana,” and click Finish.
Demonstration
Navigate to your Kibana URL (https://10.0.0.6:5601\ using Chrome or Firefox. Do not use IE or the SSO button may not show up.
Click on ADFS, the button configured in the kibana.yml file, and log in with one of the created AD users. Use the defined mail attribute on the AD account (an email address).
With a successful login, the Kibana screen will appear, and you will see your SAML authenticated user in the lower right corner.
Conclusion
ReadonlyREST combined with Elasticsearch and Kibana opens a world of advanced authentication and authorization options to you. Though only a basic configuration was outlined here, many more useful configuration options are available. You can find out more information about these advanced configurations in the ReadonlyREST documentation and in the ROR forums.
Last updated