Summary
This article helps troubleshoot the "INVALID_ARGUMENT: Unable to get query profile. Profile not found for the given queryId." exception that can be reported in the Dremio logs.
Reported Issue
The following warning is displayed in the logs:
[grpc-default-executor-*] WARN c.d.service.jobs.LocalJobsService - Exception while fetching job profile for job QUERY_UUID
io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Unable to get query profile. Profile not found for the given queryId.
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java)
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java)
at com.dremio.service.jobtelemetry.JobTelemetryServiceGrpc$JobTelemetryServiceBlockingStub.getQueryProfile(JobTelemetryServiceGrpc.java)
at com.dremio.service.jobs.LocalJobsService.getProfileFromJob(LocalJobsService.java)
at com.dremio.service.jobs.LocalJobsService.getJobDetails(LocalJobsService.java)
at com.dremio.service.jobs.Chronicle.handleUnaryCall(Chronicle.java)
at com.dremio.service.jobs.Chronicle.getJobDetails(Chronicle.java)
at com.dremio.service.job.ChronicleGrpc$MethodHandlers.invoke(ChronicleGrpc.java)
at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java)
at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java)
at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java)
at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java)
at io.grpc.Contexts$ContextualizedServerCallListener.onHalfClose(Contexts.java)
at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java)
Relevant Versions
This can happen in all Dremio releases.
Troubleshooting Steps
1. Check if the warning is consistent, either for the same query or for multiple queries at the same time. If it's consistent, then the storage needs to be investigated, to find out why Dremio is unable to fetch the profile or if the profile is missing. If the behaviour is not consistent, then this can be caused by a race condition.
2. One can collect a thread dump when the issue reproduces, to further investigate it, check if there are multiple threads blocked in waiting for query profiles.
3. Check if the original query wrote the results to disk (e.g. /var/lib/dremio/pdfs/results/QUERY_UUID). Does the file exist on the filesystem? Does it contain data? If not, check the permissions of the directory, make sure the Dremio user can write to it.
4. Is the storage on a distributed storage type? If yes, try to move it to local storage, to see if the issue reproduces.
Cause
This warning log is coming from getJobDetails API, which tries to fill up profile information and make a call to getQueryProfile RPC in JTS (Java Transaction Services). In rare conditions, if the query is fresh and profile has not been added, we get this warning. This is generally a timing issue with updating the job profile. This should not influence anything or be related to why the query is re-run when the results get downloaded.
The profile update for planning and executor profile was done every 5 sec in the Dremio releases prior to v25, but we changed it to 10 sec because it was causing load on JTS while fetching profile for long running query. The profile also get updated as soon as query is completed in the recent releases.
Steps to Resolve
Follow the troubleshooting steps. In the future releases, Dremio is working on decreasing the number of profiles being retrieved.