Connecting to Tencent COS storage from Dremio Software

Overview

Tencent's Cloud Object Storage (COS) ( https://www.tencentcloud.com/products/cos ) is a distributed storage service that uses similar APIs to Amazon S3 storage.

Dremio can connect to COS to use it as a source for your data.

Applies To

All Versions of Dremio.
Note: Connections to Tencent COS do not undergo QA testing, and it is not guaranteed that future versions of Dremio (version 25+ at the time of writing) will work with COS as described here.
Customer should satisfy themselves as to the correct functioning of connections to COS.

Details

Setting up a connection to COS is similar to setting up a connection to an Minio S3-compatible source as detailed here using the S3 template in the Dremio UI.

In the General tab,

• Select AWS Access Key
• In the AWS Access Key box, put your Tencent Cloud User's SecretID
• In the AWS Access Secret, put your Tencent Cloud User's SecretKey

In the Advanced Options section for an S3 source,

• Check Enable compatibility mode.
• In Connection Properties, add the property fs.s3a.path.style.access and set the value to true
• In Connection Properties, add the propertyfs.s3a.endpoint and set the corresponding server endpoint value - e.g.   cos.eu-frankfurt.myqcloud.com
• In Allowlisted Buckets add the names of buckets that you want to be able to connect to - e.g.  dremio-test-1234567890

Other options can be enabled or disabled as explained in the documentation for connecting to Minio S3 storage