Connecting the Blob Storage with Databricks
--- Create the resource group and the blob storage in Azure.
--- Open the Databricks account.
--- In the Azure portal, search for the Azure Microsoft Entra ID.
--- Find out the Application registration section in the list of options.
--- Click on New Registrations and create a new registration.
--- Copy the IDs 1 and 1,2 as shown in the above image.
--- Then click on the certificates & secrets as marked with red colour in the above image.
--- Copy the key as shown in the image below.
--- Now it's time to assign the roles to the applications.
--- Open the storage account u would like to connect to Databricks.
--- Click on the IAM in the list of sections. Then click on Add Role.
--- Add the Application ID created.
--- Now open Databricks and use the code below to connect it to the Blob Storage.
Code:
spark.conf.set("fs.azure.account.auth.type.<storage-account>.dfs.core.windows.net", "OAuth")
spark.conf.set("fs.azure.account.oauth.provider.type.<storage-account>.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set("fs.azure.account.oauth2.client.id.<storage-account>.dfs.core.windows.net", "<application-id>")
spark.conf.set("fs.azure.account.oauth2.client.secret.<storage-account>.dfs.core.windows.net", service_credential)
spark.conf.set("fs.azure.account.oauth2.client.endpoint.<storage-account>.dfs.core.windows.net", "https://login.microsoftonline.com/<directory-id>/oauth2/token")
Note:
Application ID: The ID copied after registering the Application.
Service_credential: The key created at certificate & Secrets.
Directory_ID: The ID copies along with the application ID.
--- Click on Run
Comments
Post a Comment