Objective
This article helps to troubleshoot the Ranger-Based Hive table access issues.
Description
Access Denied Error is a generic error message the Ranger client reports against a failure while accessing the Ranger-Based Authorized dataset. In Dremio, the error will be captured in the coordinator server.log. In the log, if you see the error stack with the line "failed to refresh policies. Will continue to use last known version of policies (-1)" means the Ranger Client in Dremio cannot fetch the policies from the ranger server. This can be verified using the ranger hive policy cache property. Refer to this article Verify Ranger Policies on Dremio Coordinator .
What Next
After confirming that Dremio is unable to download the policies, please investigate these areas to narrow down the issue.
1.Please ensure that the Dremio service user has been granted permission to download policies in Ranger. To do this, go to the Ranger UI and check if the Dremio service user ID is listed under the "ranger policy.download.auth.users" property in the Hive Service Name section. If it is not listed, please grant access to the Dremio service user.
or
2. Please ensure that Dremio is connected to the correct and active Ranger server. It is possible for a source to be added to Dremio even with an incorrect entry in the Ranger host details, resulting in a false positive. This may lead to an "access denied" error when attempting to query the dataset. Currently, we have DX-21988 in our backlog.
or
3. If the Ranger Server is secured with SSL, please verify the SSL certificates mentioned in the policymgr-ssl.xml are valid. Refer to the article Troubleshooting Ranger-Hive SSL Credential Provider Errors
or
4. Verify if the config files have white spaces in the Property or Value sections.
The four checks mentioned above only apply to the Dremio configuration. However, it is possible that there may be issues with the customer's environment. Therefore, it is recommended to verify the following from the infrastructure side.
1. To check if the Coordinator box can connect to the Ranger server and download policies, we can use the curl command. This is similar to how we verify the policy cache on the coordinator through the UI property.
if Ranger and Dremio is Undescured
$ curl --negotiate -u : -v -X GET http://ranger:6080/service/plugins/policies/download//<RANGER_HIVE_PLUGIN_SERVICE_NAME>
If Ranger is secured with SSL
$ curl --negotiate -u : -vk -X GET "https://<RANGER-ADMIN_NODE>:6182/service/plugins/secure/policies/download/<RANGER_HIVE_PLUGIN_SERVICE_NAME>"
If Kerberos is enabled, try after a kinit with the Dremio service user
$ kinit -kt <DREMIO_KEYTAB> <DREMIO_KERBEROS_USER_PRINCIPAL>
If the command above yields the following output, then there are no issues in the network and the Dremio service user permissions between the coordinator and ranger server.
{"serviceName":"cm_hive","serviceId":5,"policyVersion":97,"policyUpdateTime":1637315110528,"policies":[{"id":7,"guid":"417b5732-42c0-4a98-94d4-fe14c4086335","isEnabled":true,"version":3,"service":"cm_hive","name":"all - global","policyType":0,"policyPriority":0,"description":"Policy for all - global","isAuditEnabled":true,"resources":{"global":{"values":["*"],"isExcludes":false,"isRecursive":false}},"policyItems":[{"accesses":[{"type":"select","isAllowed":true},{"type":"update","isAllowed":true},{"type":"create","isAllowed":true},{"type":"drop","isAllowed":true},{"type":"alter","isAllowed":true},{"type":"index","isAllowed":true},{"type":"lock","isAllowed":true},{"type":"all","isAllowed":true},{"type":"read","isAllowed":true},{"type":"write","isAllowed":true},{"type":"repladmin","isAllowed":true},{"type":"serviceadmin","...
2. Verify whether the Ranger URL is an LB or a direct Ranger Server FQDN. Most of the cases the customers use LB and there will be a firewall that prevents the Dremio connection. You can see the error stack as "Connection Refused" or "Timeout" in Dremio log as well as in the Curl response.
Conclusion
Please take a look at the scenarios mentioned above to identify whether the issue is in Dremio Software Configuration or Infrastructure. If Dremio can read Ranger policies, then the error "failed to refresh policies. Will continue to use last known version of policies (-1)" will not appear in the server.log. To address a permissions error, please make sure that the appropriate privileges are granted to both the service and the logged-in user.