Unable to format any PDS on a specific Source: "Unexpected Error occured" – Dremio Support

ISSUE:

We have seen cases wherein users observe issues Promoting/Formatting any Dataset PDS on a particular Source (like for ADLS/S3 source)

Below is a reference screenshot for a related issue: when trying to promote a Parquet Table on ADLS Source we were getting a "Unexpected error occurred" on Dremio UI:

Screenshot

POSSIBLE CAUSE:

The above error is very generic, and the detailed actual error should be referred from the Dremio `server.log` ($DREMIO_HOME/log/server.log)

Firstly, from server.log we can see a REFRESH DATASET triggered for this new dataset:

INFO  c.d.service.jobs.LocalJobsService - The SQL query REFRESH DATASET "ADLS"."raw"."general"."system"."sys_user_group_2023.parquet" will be submitted on the same thread

However, further trace indicates issues as Dremio is unable to find the Role/Owner:

[Fabric-RPC-Offload18894] INFO  c.d.exec.maestro.FragmentTracker - Fragment 1b7db611-2866-03fb-51df-26b2ca454900:0:0 failed, cancelling remaining fragments.
[Fabric-RPC-Offload18893] INFO  c.d.exec.work.foreman.AttemptManager - 1b7db611-2866-03fb-51df-26b2ca454900: State change requested RUNNING --> FAILED, Exception com.dremio.common.exceptions.UserRemoteException: SYSTEM ERROR: StatusRuntimeException: INVALID_ARGUMENT: SetOwner could not find role or active user 'da51213-33ac-42e8-810d-740a8a363745' of type 'USER_OWNER'

SqlOperatorImpl WRITER_COMMITTER
Location 0:0:2
Fragment 0:0

[Error Id: bff91cbf-f8df-403f-931a-0e063cb7c981 on azc1drexedl01.corp.bcimc.com:0]

  (io.grpc.StatusRuntimeException) INVALID_ARGUMENT: SetOwner could not find role or active user 'da51213-33ac-42e8-810d-740a8a363745' of type 'USER_OWNER'
    io.grpc.stub.ClientCalls.toStatusRuntimeException():262
    io.grpc.stub.ClientCalls.getUnchecked():243
    io.grpc.stub.ClientCalls.blockingUnaryCall():156
    com.dremio.service.catalog.DatasetCatalogServiceGrpc$DatasetCatalogServiceBlockingStub.addOrUpdateDataset():222
    com.dremio.exec.store.iceberg.model.FullMetadataRefreshCommitter.addOrUpdateDataSet():130

* The above error generally indicates that the reported user 'da51213-33ac-42e8-810d-740a8a363745' is deleted/unavailable in the external user system (AD/LDAP) or IdP(Identity Provider side) and most likely is the Owner of the Source/Folder (a user who configured the Source becomes the Owner)

- Object ownership is a security feature used to control access to an object as well as oversee who has that control. In Dremio, each object MUST have an owner, and may have only one owner. Ownership is automatically granted to the user who creates the object. Please refer to below document for more details on Dremio objects ownership:

Ref[1]: Ownership: https://docs.dremio.com/software/security/rbac/rbac-structure/#ownership

Within Dremio, we can verify if the Users/Groups exists:

- To confirm the reported user, query the user table and share the output if any entry:

select * from sys.users where user_id = 'da51213-33ac-42e8-810d-740a8a363745'

SOLUTION:

If the above query returns nothing, it means the user/role is no longer available or might be deleted. In this case we can fix it by updating the Ownership at the Source level or Folder level from Dremio side:

- 1. We can first test by updating the ownership on one of the Folders:

GRANT OWNERSHIP ON FOLDER "ADLS"."raw"."general"."system" TO USER dremio_superowner

- 2. If the PDS formatting now works, we can also GRANT ownership on the Source level itself:

GRANT OWNERSHIP ON SOURCE "ADLS"."raw"."general"."system" TO USER dremio_superowner

Reference[2]: GRANT: https://docs.dremio.com/software/sql-reference/sql-commands/rbac/

* Please Note: In order to avoid this issue, it is recommended to use a service account as the Owner (permanent user/role), thereby ensuring it wouldn't be removed.

Related articles