Destinations

Databricks

What will you learn in this article? In this article you will learn how to connect Databricks to Erathos. Step by Step To add Databricks as a target in Erathos’ data movement platform, the following informations are necessary : Server Hostname, HTTP path and Access token. You can read the tutorial bellow on how to find these credentials. The first step is to create a new SQL Warehouse. To do so, select the “SQL Warehouse” option found inside “New”: Name your SQL warehouse and define the configurations accordingly to your use, for example the cluster size, auto stop, scaling and type. When the configuration is done, click on Create. Enter the Connection details tab to obtain the Server Hostname and HTTP path credentials. To identify the Catalog Name select Catalog in the left bar. The list of available catalogs will be exhibited. Choose one of the options and copy the value, you will need it later to complete the Erathos platform form. In case you don't now your AWS Account ID, enter the databricks accounts page and select the Cloud resources option in the side bar. In the Role ARN column you can find the numeric ID between iam:: and :role. There are two possible ways for supported authentication: Personal Access Token and OAuth. You can follow the steps down below to obtain both informations. Remember:only one of these ways is necessary to establish the connection. Personal Access Token: In the Connection Details tab, in the selected SQL Warehouse, select the "Create a personal access token" option to create a new Access Token: Select the Generate new token button and name the new token for identification purposes. Warning: copy the token value, it will not be displayed again. OAuth In order to iniciate you must create a new Service Principal. To do so, navigate to the databricks accounts inicial page and click on User Management in the side bar. After that, select the Service Principals tab and click on the Add service principal button. In the service principal creation page, make sure to name it so you can identify the user that will be connectes to Erathos. Click on Add to complete the creation. Now, returning to the list, click on the created service principal. Click on the Generate secret button to create new values for Client ID and Client Secret (the Secret value will not be exhibited again, so make sure to copy and paste it somewhere safe) Next, you will need to confirm the workspace's access permissions for the new service principal. To do so, select Workspaces in the side bar and click on the chosen workspace in the list of loades workspaces. In the configurations, select the Permissions tab and click on Add permissions. A text box will be exhibited, in which in must search the service principal's given name. Then, select the Admin permission level and click on save to finish. Erathos Platform Fill in the Erathos platform Databricks connection form using the credentials obtained for Server Hostname, HTTP path, Catalog Name, AWS Account ID and Access Token (or Client ID and Client Secret) Preencha o cadastro do Databricks na plataforma Erathos, usando as informações obtidas para Server Hostname, HTTP path, Catalog Name, AWS Account ID e Token de Acesso (ou Client ID e Client Secret).

PostgreSQL Azure

The initial requirement to use PostgreSQL on Azure as a destination is to enable the extension that allows reading files from Blob Storage. First, navigate to the Azure portal and locate your database. On the main page of the database, expand the Settings option in the sidebar and select Server parameters. On the loaded screen, filter the configuration parameters by "azure.extension," and then click on the value list in the VALUE column of the azure.extension row in the table. Find and select the AZURE_STORAGE option from the available list, and then click Save at the top of the page to apply the new configuration to the list of allowed extensions. After adding the extension to the allowed list, execute the following query to install the additional functionalities related to Storage: CREATE EXTENSION azure_storage; To finalize, grant admin permission for the extension to the user of the credential that will be used on the platform with the following query: GRANT azure_storage_admin TO erathosuser; Firewall To establish a connection from the database to the Erathos platform without the need for an SSH tunnel through a machine that has access to the database, you will need to add the IP address 35.212.87.86 as an allowed rule in the firewall. Navigate to the main page of the chosen database and select the Networking option within Settings. Scroll to the Firewall rule name table and choose a name of your preference to identify our platform. Then, add the platform's IP as the Firewall rule name and End IP address. Apply the new rule by clicking Save at the top of the page.