How can I use the API to automate something I can do in the UI? – Dremio Support

Overview

If you have a large number of operations or configurations that you need to do, it can be time-consuming and error-prone to have to manually carry them out using the UI. Scripting such operations can save a lot of time and effort.

Automation can also help if there are certain tasks that you want to run on a scheduled basis - again, via scripting.

This article discusses how to script exactly the same commands that the UI runs.

Note: Some of the methods described here use the v2 API. This API is no longer fully supported and there is no guarantee that these calls will not change or be removed completely in future versions.

Applies To

All versions of Dremio

Details

The basic process we'll follow is to use the browser's in-built Developer Tools to identify which API calls the UI is sending, and use the URLs and body content of those calls to create our own curl commands, which can then be put into a script.
We will also be using jq to extract specific pieces of information from the server response for later use.

Why shouldn't I just use the API documentation?

The Dremio API is powerful, but with that power comes complexity and potential confusion. Nothing in this article supercedes or overrides the API documentation, it's simply a way to understand how some of the APIs work through generating your own examples.

Authentication

First, we need our script to be able to send authenticated commands. To do this, we run an initial curl command with a username and password, and retrieve an authentication token that we will use in later commands.
The curl command for this is :

curl $DREMIO_BASE_PATH/apiv2/login -k -H 'Content-Type: application/json' -d"{\"userName\":\"$USERNAME\",\"password\":\"$PASSWORD\"}" -s | jq -r ".token"

where $DREMIO_BASE_PATH is something like : http://dremio_server:9047
and $USERNAME and $PASSWORD are environment (or script) variables.

To put that into a script, and save the authentication token for later re-use, the full script line would be:

DREMIO_AUTH_TOKEN=_dremio$(curl $DREMIO_BASE_PATH/apiv2/login -k -H 'Content-Type: application/json' -d"{\"userName\":\"$USERNAME\",\"password\":\"$PASSWORD\"}" -s | jq -r ".token")

Extracting the command from the browser

As an example to work through, we'll look at Formatting / Promoting a PDS, and to set up the example first add the Sample Source dataset.

In the UI, drill down through the Datasets page, and click on the Format File icon for the zips.json dataset. This brings up the Dataset Settings page.

Before clicking on Save, bring up the Developer Tools for your browser. In Firefox and Chrome this is done by pressing press F12 or going to the menu option More Tools -> Developer Tools

In the Developer Tools, click on the Network tab.

Now, in the main Dremio UI, click Save.

The first line that gets logged in the Dev Tools console is a PUT command. Click on it, and the right hand panel will open up with the details of that call.

From the Headers tab in this right-hand panel, we can see the full URL and the API call used. In this case it is :

PUT http://dremioserver:9047/apiv2/source/Samples/file_format/samples.dremio.com/zips.json

In the Dev Tools console, Click on the Request tab, and click the Raw button to display the raw json, and we see that the body of the PUT request was:

{"type":"JSON"}

Turning the Dev Tools information into a curl command

We can now take the data from the Developer Tools console and turn it into the following curl command:

curl -X POST -k -H 'Content-Type: application/json' -H "Authorization: $DREMIO_AUTH_TOKEN" $DREMIO_BASE_PATH/apiv2/source/Samples/file_format/samples.dremio.com/zips.json -d'{"type":"JSON"}'