Here is the content converted to the new article format:
Summary
This is practical how to, on configuring the external token provider functionality within Dremio UI with KeyCloak acting as the authorization server/identity provider.
Reported Issue
This article came about from a customer reporting issue where the following error was seen: WARN c.d.s.u.SimpleAccessTokenService - Token authentication failed, error parsing the token: Illegal base64 character 2e when testing External Token Providers with Dremio v24.3.4.
Overview
As of v24.3.0, Dremio Software supports the use of external token providers so it is possible to use a JSON Web Token (JWT) issued by an OpenID Connect (OIDC)-conformant authorization server to establish ODBC/JDBC (non-Arrow Flight) connections to Dremio. The authorization server can be any JWT provider, including an identity provider (IdP).
In this example, we will be using the open source solution KeyCloak to use as the authorization server when we configure this as our external token provider in the Dremio UI.
NOTE: This configuration example will be done using a local on-premise instance of Dremio, however the external token provider configuration is relevant for any Dremio deployment type.
Relevant Versions Tools and Integrations
This working example applies to Dremio versions 24.3.0 and later.
Steps to Resolve
This article assumes KeyCloak is installed and running with user accounts already added in KeyCloak. Full background on KeyCloak can be found at www.keycloak.org, and https://github.com/keycloak.
1. Start by connecting to the KeyCloak Admin/Dashboard on port 8080. In this example KeyCloak is running locally so we will open http://localhost:8080/admin/master/console/.
2. For the purpose of simplicity we will be using the master realm which is the default when standing up a KeyCloak instance.
3. On KeyCloak Admin/Dashboard select the Clients tab on the navigation bar, and Create Client. Name your client (i.e. "dremio") and on the next screen be sure to select/toggle on Client Authentication and click Next. This is required for the generation of the Client Secret for the HTTP POST call we will make to get our JWT later on.
4. On the next screen we can leave all fields as default, since in the external token provider model Dremio does not consume the authorization code from the front end call during SSO so we can just click Save here.
5. This will now take you to the newly created Client and the client details screen
6. From the Client scopes tab, click on the <client_name>-dedicated scope from the list. The name should match up with the same name given to the created client above. In our case this is dremio-dedicated.
7. Then click Configure a new mapper and from the list select the 'Audience' mapping. For the Name field, give the mapper any name and then click Save. In this example we will name this 'dremio'. For the Included Client Audience field, click the drop down and select the client that was previously created. In our case our client is 'dremio' so we will select this. NOTE: Keep a note of this name as we will need this to populate the Audience value for our External Token Provider setup in the Dremio UI.
--------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------
8. Now start a fresh instance of Dremio v24.3.x. In our example we are using a local instance of Dremio with no SSO authentication or Wire Encryption enabled so our dremio.conf is a default one as below.
NOTE: This configuration setup for an External Token Provider will also work with existing installations of Dremio.
paths: {
# the local path for dremio to store data.
local: ${DREMIO_HOME}"/data"
# the distributed path Dremio data including job results, downloads, uploads, etc
dist: "pdfs://"${paths.local}"/pdfs"
}
services: {
coordinator.enabled: true,
coordinator.master.enabled: true,
executor.enabled: true,
flight.use_session_service: true
}
9. Since we have configured a local instance of Dremio with no Identity Provider for authentication, we need to make sure the new user we create in Dremio UI after the EULA page matches the same user we used to login to the KeyCloak Admin/Dashboard. In this example we are using admin/admin as the username/password account for KeyCloak so we will create the dremio first time user as 'admin'. NOTE: If using an identity provider such as LDAP (ad.json)/AzureAD (azuread.json) or even KeyCloak (oauth.json) for user authentication, then the user must match up to the preferred_username value (or any other username mapping claim from KeyCloak) within the JWT issued by KeyCloak.
10. Once the new user is created in the 'Welcome to Dremio', 'Create Admin Account' page, click on the Settings cog icon on the lower left of the Dremio UI navigation bar and select the External Token Providers tab.
11. Now click Add Provider and fill in the following details
- Name: We will use 'keycloak'
- Audience: We will use 'dremio' which is the same name we gave to the Audience mapper on Step #7
- User Claim Mapping: Since we are using KeyCloak, the user claim mapping field that identifies the Dremio username of the JWT user is 'preferred_username'. So this is what we will use.
- Issuer URL: We can get this from the issuer field on the KeyCloakhttp://<host>:<port>/realms/master/.well-known/openid-configurationURL. In our example this is: 'http://localhost:8080/realms/master'
- JWKS URL (optional): We can leave this blank as Dremio will pick this up automatically
12. Now click Add. NOTE: If any of the above details are incorrect, you will get the following error pop-up: INVALID_ARGUMENT: Could not lookup OpenID configuration for the provided issuer.
13. Assuming all went well adding the provider, if we now edit the provider, we can notice that Dremio has looked-up and added the JWKS URL.
14. The next step is to now get a JWT token. In this example to keep things simple we will be using cURL/Postman to hit the correct KeyCloak endpoint (with the relevant credentials and information in the HTTP POST command) to get our JWT.
If using cURL, we issue the below POST command to our local KeyCloak instance:
curl --location 'http://localhost:8080/realms/master/protocol/openid-connect/token' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'grant_type=password' \
--data-urlencode 'client_id=dremio' \
--data-urlencode 'client_secret=8afcZoR0RTBrZ580mL3B6XIyTZ663VDb' \
--data-urlencode 'username=admin' \
--data-urlencode 'password=admin'
NOTE: client_credentials can also be used instead of grant_type depending on the customer environment and setup.
If using Postman, we will issue the below POST request with field values against our local KeyCloak instance. NOTE: We got the client_secret from our configured KeyCloak client from Step #3.
--------------------------------------------------------------------------------------------------------------------
15. Now take a copy of the returned access_token value.
16. Open up a any third-party SQL client such as DBeaver and create a database connection with Dremio using the following JDBC connection details:
- JDBC URL: jdbc:dremio:direct=localhost:31010;token_type=jwt
- Username: We will leave as blank
- Password: We will copy in the JWT (access_token) we got from step #15.
17. NOTE: We set the property token_type to jwt in our connection string per Step #4. of the Dremio Documentation: https://docs.dremio.com/current/sonar/security/authentication/external-token
18. Now click Test Connection and you should get a Connected message from DBeaver like the below.
19. At the same time the following should be logged in the Dremio coordinator server.log.
2024-05-22 11:19:59,294 [UserServer-1] INFO com.dremio.ConnectionLog - [0160faf1-44cd-4a12-a69c-c76b64d48825] Connection opened.
Endpoint: 127.0.0.1:55376
Protocol Version: 5
Record Type: DREMIO
Record Formats: DREMIO_23_0, DREMIO_1_4, DREMIO_0_9
Support Complex Types: true
Name: Dremio JDBC Driver
Version: 24.2.2-202309281602050954-7d5e877f (24.2.2)
Application: 14560@LAPTOP-MJMBCBJM
User Properties:
useEncryption=false
direct=localhost:31010
token_type=jwt
We have now successfully set up and configured an External Token Provider for the Dremio SQL/JDBC endpoint.
Common Challenges
Add additional content.
Additional Resources
https://docs.dremio.com/24.3.x/sonar/security/authentication/external-token
https://support.dremio.com/hc/en-us/articles/12776377653147-Configuring-KeyCloak-as-an-SSO