In the same way that is possible to read and write data from snowflake inside databricks, its also possible to use databricks with query federation against diverse SQL engines, including snowflake. The current supported engines are:

• PostgreSQL
• MySQL
• Snowflake
• Redshift
• Synapse
• SQL Server

We are going to demonstrate how it works with Snowflake. We will first create a table in databricks, it can be a delta table stored in the data lake, or an unmanaged table pointing to a set of files (External Table) or anything in between.

Creating the required resources

I will go to databricks and run the following:

CREATE OR REPLACE TABLE default.DATABRICKS_ALBERT(
NAME STRING,
SEX STRING);

INSERT INTO default.DATABRICKS_ALBERT(NAME, SEX) VALUES ('Albert','Male');

This process can either be done from the Data Engineering persona or the SQL persona. I created it first from the data engineering part.

Now we go to snowflake and create a second table which we want to join:

CREATE TABLE SANDBOX.DEFAULT.DATABRICKS_FEDERATED(
NAME STRING,
AGE INTEGER);

INSERT INTO SANDBOX.DEFAULT.DATABRICKS_FEDERATED (NAME, AGE) VALUES ('Albert', 37);

And it gets created succesfully:

Now we can stay in the data engineering persona or go to databricks SQL and we create the virtual table that will create the link with the table in snowflake. This will do the mapping:

CREATE TABLE MY_TABLE
USING snowflake
OPTIONS(
dbtable 'YourTable',
sfURL 'yourURL.snowflakecomputing.com', --You can use privatelink if you have one
sfUser 'YourUser',
sfPassword 'YourPassword',
sfDatabase 'YourDB', --You can use a secret scope like: secret('scope_name', 'pwd_entry'),
sfSchema 'YourSchema',
sfWarehouse 'YourSnowflakeWarehouse'
);

In the Databricks SQL:

In The Data Engineering:

Glueing it up together

If you use Databricks SQL I couldn’t make it work with the STARTER ENDPOINT so be sure to use a normal WAREHOUSE to avoid any errors. In my case i created an XS warehouse, but now i can run a query fetching data from both tables:

select a.name, a.sex, b.age
FROM default.DATABRICKS_ALBERT a , default.SNOWFLAKE_ALBERT b
where a.name=b.name

And in the Data Engineering Persona:

And of course you can use spark.sql with python or any other language to query the table as well:

Or directly with Dataframes treating the federated table as any other table in the catalog:

Hope this clears your way and helps you integrate data from different sources without having to use a virtual metadata layer.

The post Databricks query federation with Snowflake. Easy and Fast! appeared first on Albert Nogués.

The purpose is to use it a sa PoC to show to colleagues, managers, prospective customers or just to have fun and play

It’s publicly available on my github repo and any collaboration is welcome. You can fork it, improve it, send PR’s and do whatever you want!

The first version will run solely on azure. The objective is to show the following technologies/disciplines:

* Infrastructure as a Code (IaaC), by using Terraform

* Cloud architecture anc Cloud Ops by using an azure cloud environment

* Data Engineering by using a Spark powered Databricks Notebook and an ADF Pipeline (future)

* DevOps to trigger some pipelines based on changes (future)

* Basic Security concepts (keyvault, service principals, least privileged rbac accesses…)

* FinOps keeping the costs at minimum and choosing the proper tools for the job

* Reporting and Dashboarding on data in the platform

* Data management: We will use an adls storage account and azure sql db

TOOLS:

* Terraform to deploy all the infra as a code

* Azure Cloud to host our resources

You have the code plus all the information on my github repo:

https://github.com/anogues/ProjectZ

The post Smallest Analytical Platform Ever! appeared first on Albert Nogués.

But an easy way to just copy notebooks between workspaces can be implemented easily with Azure DevOps.

We are going to use the git repos capability of Azure Databricks, so when a new code change is commited in a notebook an Azure DevOps pipeline will trigger the transport copy of the workbook from the first Databricks workspace (in our case a NonProd workspace) to the target one, again, in our case, the Prod workspace.

To achieve this we will use some more components from the Azure ecosystem, including the use of Keyvaults to keep all our secrets stored safely. The list of prerequisites is the following:

• Two Databricks workspaces, one our source workspace (NonProd) and another, our Production one.
• An Azure Keyvault (or two if we want to segregate the environments)
• Azure Databricks repository configured at least in our source workspace, so when the change is commited we can triger the pipeline that will fetch the notebook and transport it to the prod workspace
• Access to Azure DevOps (Something similar can be implemented with Github + Github Actions)

Lets see how to implement it. First we need to make sure git repos in enabled our source workspace. We can verify by login with an admin privileged user to our workspace and make sure the option is checked as follows:

Secondly, we go to Azure DevOps services and we create a new project. I’ve called it DatabricksCICD but feel free to call it whatever you need:

Once created we take the details for cloning our repo and copying them. We go to our databricks workspace and then we look for the Repos option on the left, and add a new repository. We need to paste the url to clone our newle Azure DevOps created repository:

Once we linked our Databricks workspace with our DevOps repo, now we can create a new notebook. In the same Repos section, click on the down arrow to create a new notebook, as shown below:

The content of the notebook, you can put anything you want. I’m writing a print(“Hello from Albert”) statement. We will not run it, we just want to show it’s possible to transport it. Once done, click on the save now in the revision tab:

Then click on the left in the main branch button, from there we will be commiting the changes to our repository:

If we go back now to our Azure DevOps project we should see the file has been commited to the repository. This ends the first part of this tutorial.

In the second blog entry we will see how to trigger the pipeline after a modification of this notebook and passing the credentials of the second workspace to be able to deliver the changed notebook to our Production (target) workspace.

The post Implementing CI/CD in Databricks with Azure DevOps (Part 1) appeared first on Albert Nogués.

Introduction

But what are cluster policies?

Cluster policies are basically a json file with some parameters that we use to allow (or not) users to select certain things when creating a cluster. Not only users select (or deselect) but we can force some parameters of the cluster as well by default.

The purpose of using cluster policies is not only standarizing of forcing certain specific configurations but also to limit human error that can cost the copany lots of money by capping certain parameters to only allow specific machine sizes, maximum number of nodes, or the cluster timeout.

To be able to use cluster policies, you need to have a Premium workspace. And of course, be an admin of the workspace to be able to define them.

Format of a cluster policy and it’s elements

The format of a policy, as we said, its a json document:

interface Policy {
  [path: string]: PolicyElement
}

The type of policy elements we can use to control its quite long, and you have them explained in the official databricks documentation:

• Fixed policy
• Forbidden policy
• Limiting policies: common fields
• Allow list policy
• Block list policy
• Regex policy
• Range policy
• Unlimited policy

In this article, we will create a simple policy, that will preconfigure a cluster based on a specific machine size, we will restrict the maximum number of nodes to 5, and that will autotag the cluster with a specific key that we will define in the policy.

However there are endless possibilities so I recommend you to have a look at the official documentation to have yourself an idea of all the parameters you can configure here.

Creating a custom cluster policy

Ok, lets start! To create our first policy we need to log in into our workspace, go to the compute section and click on the cluster policies tab:

Then, there if we have rights (i.e. we are administrators of the workspace) we should see a buton called Create Cluster Policy. Once clicked we will see something similar to Fig 2:

Here we have to concentrate on three things. The first one, is the policy name. This is the name your users will see, so I recommend to choose a meaningful name. For example multinode small cluster, or something like that.

Once the name is selected, we need to actually define the policy. This is a json area content box. In my sample case i want a to create a policy that chooses by default a machine with a small/medium size sku and maximum can autoscale to 5 nodes. Also i want to tag the cluster with a fixed string. To go further on this example, i will give the user a choice of two machines for the nodes, while the driver will be restricted to a specific vm sku. I will also set up auto termination to 10 minutes to make sure i am not paying for something not in use. My policy will look like the following:

{
  "node_type_id": {
    "type": "allowlist",
    "values": [
      "Standard_D8d_v4",
      "Standard_D16d_v4"
    ],
    "defaultValue": "Standard_D8d_v4"
  },
  "driver_node_type_id": {
    "type": "fixed",
    "value": "Standard_D8d_v4",
    "hidden": true
  },
  "autoscale.min_workers": {
    "type": "fixed",
    "value": 1,
    "hidden": true
  },
  "autoscale.max_workers": {
    "type": "range",
    "maxValue": 5,
    "defaultValue": 2
  },
  "autotermination_minutes": {
    "type": "fixed",
    "value": 10,
    "hidden": true
  },
  "custom_tags.project": {
    "type": "fixed",
    "value": "Albert"
  }
}

Once defined i will click on save. If the format of the policy is ok, the cluster policy will be created. I can then go to the permissions tab and assign it to users or groups. By default the policy is only assigned to be used by the admin group:

We will leave it as it is as we are just playing arround.

We can now try to create a cluster using this policy:

Select the newly created policy, then in the worker type by default in the policy we are choosing the machine sku Dtandard_D8d_v4 but we are allowing the user to chose a bigger one, and in the autoscaling we can choose up to 5 nodes. If we try to input 6 we will get an error: “Max Workers cannot be more than 5” and will not let us go through the cluster creation.

The driver type and the default cluster timeout are not seen as we dont allow the user to change it, they are set by default and we choose to mak them hidden. By removing the hidden attribute we will let the users to see the values but not modifying it:

"driver_node_type_id": {
    "type": "fixed",
    "value": "Standard_D8d_v4",
    "hidden": true
  },
...
  "autotermination_minutes": {
    "type": "fixed",
    "value": 10,
    "hidden": true
  },

And the last needed to be check is in the advanced options. We wanted to autotag the cluster with the project tag with a set string. If we uncollapse the advanced options we can verify how this tag is applied:

And if we go to our cloud provider and we check the machines created as part of the cluster we will see how the tag has also been propagated to the cloud resources. This will allow us to do cost analysis from the provider cloud portal.

Next Steps

There is also another very good article about defining the strategy when implementing policies. As this process can lead to some errors its important not to enforce the policies directly in a production workspace before testing. Databricks has created a reference methodology that i think it makes perfect sense when implementing these policies, starting by testing the policy, then create a barebone policy to do some tagging, then deploy the real policy but without enforcing it to any user (so it can only be selected optionally) and once tested everything is working as expected then enforce it to the users.

By using this framework you can simply correct all aspects that may be wrong before creating a potential issue to the final users.

There is also an interesting section in that article which is a worry usually for all of us that work for large user base. This is the tag enforcement of resources, very important when crosscharging between teams or different parts of the organization is required. In order to be able to input the costs to the proper project, department or other body, you need to enforce tagging of your resources. This is a very important topic when working on the cloud and while its a bit difficult to implement a proper tagging policy in databricks as you have mainly to rely on freeform text or some regex expression, you can try to build an effective system to crosscharge your projects or departments in the use of your databricks workspace(s).

This set of best practices and challenges is documented here and for me its an important resource for any databricks administrator.

Have fun!

The post Databricks cluster policies at a glance. The easy way! appeared first on Albert Nogués.

Why we may want a private endpoint? Thats a good question. For oth security and performance. Just like using TSCM Equipment for optimal safety and security. We dont want the traffic going outside to the internet to return again back to the azure datacenter if the resource we are trying to reach is already there. So with a PrivateLink the traffic will stay inside the Azure backbone network avoiding reaching the internet. More information about private endpoints here and here.

Though its possible to create private endpoints to connect to services in other subcriptions we will use the same subscription and the West Europe Region in this article. The goal is to connect to both a AzureSQL database using private connectivity and to a datalake using private connectivity as well.

Creating a Private Endpoint for AzureSQL and integrating in the databricks vnet

For this, i created a Databricks workspace and selected to use an already existing VNET, so this way I can add a new subnet for my private endpoints. One of the good things of doing this way is that NICs between subnets see each other and are reacheable (unless we block it with a network security group) but by default traffic is open within the VNET. So I can create a Private endpoint in a specific subnet of the same VNET that hosts the databricks subnets.

Bear in mind that it’s not possible to add a private endpoint to a subnet managed by databricks. So the two subnets we created, when we deployed the databricks workspace (Bot public and private) should not be modified. We will create a new one as shown in the screen below:

Once defined properly the VNET, we are going to create the private endpoint to reach our AzureSQL through it.

First we need to go to the Azure Portal, find our AzureSQL Server, and click on the left menu called Private Endpoint Connections and click on the plus sing on top to create a new one. We just need to select the subscription, the resource group the nameof the private endpoint and the region. We can fill it as shown in the following picture:

The second step requires a bit more of information, here we will define which resource we try to target with our private endpoint. As expected we need to find our AzureSQL Server here. We fill the combo boxes as usual

The third screen is the most important one. We need to select the VNET and Subnets that will host our private endpoint. In this case we want to use databricks so we need to use the VNET we created for databrickks, and then the subnet we created specifically to host the private endpoints.

Another important step here is to integrate it with the DNS. If we dont integrate it, when we use the AzureSQL hostname provided by azure we will still access through the public endpoint. By integrating it in the DNS. the dns queries over the public endpoint in this private zone, will resolve to the private IP of the NIC of the Private Endpoint

If we chose no for the DNS integration then we will have to add static entries in the /etc/hosts or somewhat, or use the private IP instead of the hostname when connecting to the AzureSQL server. To simplify we choose to integrate it.

Once created we should see the private endpoint available. If you look at the right its implemented though a NIC (Network Interface card), and by clicking on it, we can find it and see the ip address assigned:

Test the AzureSQL DB Endpoint from Databricks

Now we have it ready. We can still see from outside that VNET, that our old server still resolves to a public ip, as it was the case before even inside databricks. We can ping it for testing purposes:

C:\Users\Albert>ping azure-sql-server-albert.database.windows.net

Haciendo ping a cr4.westeurope1-a.control.database.windows.net [104.40.168.105] con 32 bytes de datos:

As you can see we have a public ip, but lets try to ping it inside the cluster:

So it’s working. Its using the private ip instead of the public one. Our last step is to see if we can fetch the data from the database:

Creating an AzureDataLake PrivateEndpoint and saving our data to the DataLake through it.

We are not done yet! We can still complicate matters and create a private endpoint as well to save data to our datalake.

I’ve created an ADLS Gen 2 storage account, and going back to databricks I see by default it’s using public access:

But we can implement a Private Endpoint as well, and route all the traffic through the azure datacenter itself. Lets see how to do it. For achieving this, we go to our ADS Gen2 storage account, and on the left we click again in Networking, and the second tab is called Private Endpoint connections. We click the plus button to create a new one, and basically we follow the same steps as before with a subtle difference

The difference with a storage account is that we need to chose which api we want to create the private endpoint for. We can use the blob, the table, the queue, the file share and the dfs (DataLake) endpoint (And also the static website!).

We will use the dfs endpoint, and again we will place it in the Private Endpoint subnet of our Databricks vnet. Something like this:

After a few minutes our private endpoint will be ready to be used. We can go again to see the NIC and check the private ip or go directly to databricks and ping the storage account url to see if now it’s resolving to our private endpoint:

Test the ADLS Gen2 SA endpoint from Databricks

If we have the IAM credential Passthrough enabled in our cluster and we have permisison to write to the datalake, now we should be able to write there without going through the internet:

So this is the end of the tutorial. We created two private endpoints, one for AzureSQL Database and Another for our DataLake and used bot them from Databricks. We also confirmed we are effectively using them by pinging the hostnames of both resources and seeing a change from the public ip to the private one.

Happy Data pojects!

The post Using Azure Private Endpoints with Databricks appeared first on Albert Nogués.

There are a few requirements however to be able to use redshift spectrum. First, the data loaded will be like an external table in other systems (RDBMS) or like hive external tables. So this means that we can forget about updating or deleting our data. We will only get read only capability, but usually thats we are looking for (think in historical data from already closed exercices, archived data and so on), where the cost of storing it in s3 even in a cool tier can ve several orders of magnitude cheaper than keeping it inside redshift.

There are other technical limitations tough. For example, the s3 bucket has to be in the same region as our redshift cluster, so plan in advance or move the data to another s3 bucket. There are other requirements like permissions and so on, you can read it here.

Before starting we need to attach a role to our redshift cluster that grants access to read s3 buckets. If you followed my previous article on redshift you will already have this role, otherwise check how to do it here.

If you followed my previous article, apart of the s3 read only permisison, you need to add the glue catalog permission to create a table. For this, modify the role (or if you’re creating a new one, add the following permission: AWSGlueConsoleFullAccess

For the sake of this exercise I’ve load part of the TPC-DS customer data into a s3 bucket i made public on the Paris region. Then we will create an external customer table pointing to the data in the bucket (which is in fact compressed in gz) and we will query that data from redshift.

Once the bucket data is loaded, we are ready to go back to our redshift cluster. I’ve used this file for the tests. We will create an external schema and an external customers table. For this we need first to copy the arn of the role we created with the s3 read access. To get that let’s go to our cluster, then click on properties and later on click in Copy Amazon Resource Name (ARN):

Once we have the arn of our role, then run the folloowing in your editor. It will create the external schema in a new spectrum database (feel free to choose any other database name if you need)

create external schema spectrum_albertnogues 
from data catalog 
database 'spectrum' 
iam_role 'arn:aws:iam::742123541312:role/Redshift_Albertnogues.com'
create external database if not exists;

Then we are ready to create our external table:

create external table spectrum_albertnogues.customer
(
  c_customer_sk int4,                 
  c_customer_id char(16),             
  c_current_cdemo_sk int4 ,   
  c_current_hdemo_sk int4 ,   
  c_current_addr_sk int4 ,    
  c_first_shipto_date_sk int4 ,                 
  c_first_sales_date_sk int4 ,
  c_salutation char(10) ,     
  c_first_name char(20) ,     
  c_last_name char(30) ,      
  c_preferred_cust_flag char(1) ,               
  c_birth_day int4 ,          
  c_birth_month int4 ,        
  c_birth_year int4 ,         
  c_birth_country varchar(20) ,                 
  c_login char(13) ,          
  c_email_address char(50) ,  
  c_last_review_date_sk int4
)
row format delimited
fields terminated by '|'
stored as textfile
location 's3://redshift-spectrum-albertnogues/customers/';

As you can see it’s not really 100% the same customers table from the TPC-DS data. This is because external tables do not support primary keys, not null syntax and some other keywords that do not make sense in external tables. Make sure you used the right separator and selected the right format of data (compression is determined by the file extension).

Then, as you will see the table creation takes virtually nothing. This is because in fact, the data is not loaded, its only a shortcut to our s3 data. But to make sure it’s working we can run a few queries to our new table:

select count(*) from spectrum_albertnogues.customer;

And to make sure data is properly loaded, we can query it as well:

select * from spectrum_albertnogues.customer LIMIT 5;

As a last example, if you run my previous blog post on redshift, we can now join our new external customers table with our web returns fact table, to see how many objects these users returned depending on their origin country, to see if there is any pattern or not.

select c_birth_country as customer_birth_country, sum(wr_return_quantity) as qty_returned_total
 from web_returns,
      date_dim,
     spectrum_albertnogues.customer
 where wr_returned_date_sk = d_date_sk 
   and d_year =2002
   and wr_returning_customer_sk = c_customer_sk
 group by c_birth_country
 order by 2 desc;

Of course the query should take a bit more since the data is not inside redshift, and needs to be fetched from s3 but we can get our output quite fast and easily:

Thats it!. For more advanced topics like performance improvements when querying from s3 and others, you can read here and here. I recommend basically partitioning, a splittable file format like parquet compressed with snappy or some other codec, small files (but not very small), add as many filers as you can to avoid retrieving unnecesary data, and if you use a columnar format like parquet, you will avoid fetching unused columns in your select statement. This is important (and can have a huge impact in costs) because now it’s the time of discussing the price of all this.

Bear in mind that AWS charges 5 dollars for each Terabyte of data scanned from s3 with spectrum, so make sure you do the right usage on this (only data infrequently queried) and do not overestimate these charges as they grew quickly.

For expert advice or project requests you can contact me here. Happy Querying!

The post Use Redshift Spectrum to query infrequently used data on S3 appeared first on Albert Nogués.

Before start, we need a few prerequisites, these are the following:

1. Installing terraform, azure-cli and azure-storage-queue
2. Create a service principal for deploying the resources with terraform
3. Creating the terraform tf file with all the components required to be deployed. These include:
   • A Resource Group
   • A Storage account
   • A queue inside the previous storage account
4. Run the terraform deployment
5. Write the python code to send a message and retrieve it from the queue created in the storage account (Part II)

Let’s start!

Before starting to work with terraform, we need to install terraform. We can do it easily downloading it from the official website and placing it in the path of our sysvars. We can download the latest packaged binary from here.

To install azure-cli and azure-storage-queue we can use python + pip. If you dont have python, you can download it from the official website here. Make suire you install pip as well as part of the installation process.

Once python is setup we can run the following to install the packages and it’s dependencies:

pip install azure-cli azure-queue-storage

After installing the packages we are good to start moving to point 2. For the service principal creation we can use the recently installed azure-cli. The call to create the spn is as follows:

az ad sp create-for-rbac --name="SPForTerraform" --role="Contributor" --scopes="/subscriptions/ourSubscriptionId"

Make sure you replace ourSubscriptionId by your azure subscription. In case you dont know where to obtain it you can follow the following procedure or this one. The spn we will create will be named with the –name parameter and will have the contributor role. This means this spn will have access to everything but to manage users. So we can create any sort of resouce with in. This process may take a few seconds, after a while we should see something like this:

Changing "SPForTerraform" to a valid URI of "http://SPForTerraform", which is the required format used for service principal names
Creating a role assignment under the scope of "/subscriptions/xxxxxx"
  Retrying role assignment creation: 1/36
  Retrying role assignment creation: 2/36
  Retrying role assignment creation: 3/36
The output includes credentials that you must protect. Be sure that you do not include these credentials in your code or check the credentials into your source control. For more information, see https://aka.ms/azadsp-cli
{
  "appId": "xxxxxx",
  "displayName": "SPForTerraform",
  "name": "http://SPForTerraform",
  "password": "yyyyyy",
  "tenant": "zzzzzz"
}

Make sure you protect these, as anybody getting them will be able to access your tenant and start deploying resources in it.

Now, we can move to the thid point which is creating the terraform file. We can start now our preferred editor and create a *.tf file with the following content:

provider "azurerm" {
subscription_id = "kkkkkk"
client_id = "xxxxxx"
client_secret = "yyyyyy"
tenant_id = "zzzzzz"
features {}
}

terraform {
  required_version = ">= 0.13"
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
    }
  }
} 

The first block initializes the provider azurerm from terraform, while the second one initializes the terraform file with the required azurerm providrer, and tells terraform where to obtain it and which is the minimum version required. client_id is the same as the appID returned by the az cli when the spn was created.

Then we need another three blocks, one for each resource we want to deploy:

resource "azurerm_resource_group" "rg1" {
name = "TerraformRG"
location = "West Europe"
tags = { Owner = "Albert Nogues" }
}

resource "azurerm_storage_account" "sacc1" {
  name                     = "teststorageacc1"
  resource_group_name      = azurerm_resource_group.rg1.name
  location                 = azurerm_resource_group.rg1.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_storage_queue" "queue1" {
  name                 = "queue1"
  storage_account_name = azurerm_storage_account.sacc1.name
}

The way terraform blocks work is the following:

resource “name_of_the_azurerm_resource” “our_alias”{

… parameters and vars…

}

We can use then the alias to select things from other blocks we defined previously. So for example, in our example, the storage account, takes the location from the location field defined in the resource group, this way we ensure we create the storage accoun in the same azure location than the resource group (even it’s not mandatory). An example of how to define an storage account can be found in the official documentation here.

The same with the queue, which is created in the storage_account_name referred by the previous block. We can also put tags to the resources as shown in the resource group block.

Once we are ready, we can launch terraform to create our resources. For security purposes (and if working in multiple environments) it’s usually not good having in the code the client_id, secret_id, tenant_id and subcription_id. Terraform can read system variables in execution time. These can be set (windows) or exported (linux) with these names:

ARM_SUBSCRIPTION_ID
ARM_CLIENT_ID
ARM_CLIENT_SECRET
ARM_TENANT_ID

Once all ready we can trigger our creation script. We can navigate where our .tf form is and launch it:

terraform plan

Which will give us the changes needed in our azure subscription to acommodate the resources, and it should show us three changes: one for the resource group, another for the storage account and another for the queue:

terraform.exe plan

azurerm_resource_group.rg: Refreshing state... [id=/subscriptions/kkkkkk/resourceGroups/TerraformRG]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # azurerm_resource_group.rg1 will be created
  + resource "azurerm_resource_group" "rg1" {
      + id       = (known after apply)
      + location = "westeurope"
      + name     = "TerraformRG"
      + tags     = {
          + "Owner" = "Albert Nogues"
        }
    }

  # azurerm_storage_account.sacc1 will be created
  + resource "azurerm_storage_account" "sacc1" {
      + access_tier                      = (known after apply)
      + account_kind                     = "StorageV2"
      + account_replication_type         = "LRS"
      + account_tier                     = "Standard"
      + allow_blob_public_access         = false
      + enable_https_traffic_only        = true
      + id                               = (known after apply)
      + is_hns_enabled                   = false
      + large_file_share_enabled         = (known after apply)
      + location                         = "westeurope"
      + min_tls_version                  = "TLS1_0"
      + name                             = "teststorageacc1"
      + primary_access_key               = (sensitive value)
      + primary_blob_connection_string   = (sensitive value)
      + primary_blob_endpoint            = (known after apply)
      + primary_blob_host                = (known after apply)
      + primary_connection_string        = (sensitive value)
      + primary_dfs_endpoint             = (known after apply)
      + primary_dfs_host                 = (known after apply)
      + primary_file_endpoint            = (known after apply)
      + primary_file_host                = (known after apply)
      + primary_location                 = (known after apply)
      + primary_queue_endpoint           = (known after apply)
      + primary_queue_host               = (known after apply)
      + primary_table_endpoint           = (known after apply)
      + primary_table_host               = (known after apply)
      + primary_web_endpoint             = (known after apply)
      + primary_web_host                 = (known after apply)
      + resource_group_name              = "TerraformRG"
      + secondary_access_key             = (sensitive value)
      + secondary_blob_connection_string = (sensitive value)
      + secondary_blob_endpoint          = (known after apply)
      + secondary_blob_host              = (known after apply)
      + secondary_connection_string      = (sensitive value)
      + secondary_dfs_endpoint           = (known after apply)
      + secondary_dfs_host               = (known after apply)
      + secondary_file_endpoint          = (known after apply)
      + secondary_file_host              = (known after apply)
      + secondary_location               = (known after apply)
      + secondary_queue_endpoint         = (known after apply)
      + secondary_queue_host             = (known after apply)
      + secondary_table_endpoint         = (known after apply)
      + secondary_table_host             = (known after apply)
      + secondary_web_endpoint           = (known after apply)
      + secondary_web_host               = (known after apply)

      + blob_properties {
          + cors_rule {
              + allowed_headers    = (known after apply)
              + allowed_methods    = (known after apply)
              + allowed_origins    = (known after apply)
              + exposed_headers    = (known after apply)
              + max_age_in_seconds = (known after apply)
            }

          + delete_retention_policy {
              + days = (known after apply)
            }
        }

      + identity {
          + principal_id = (known after apply)
          + tenant_id    = (known after apply)
          + type         = (known after apply)
        }

      + network_rules {
          + bypass                     = (known after apply)
          + default_action             = (known after apply)
          + ip_rules                   = (known after apply)
          + virtual_network_subnet_ids = (known after apply)
        }

      + queue_properties {
          + cors_rule {
              + allowed_headers    = (known after apply)
              + allowed_methods    = (known after apply)
              + allowed_origins    = (known after apply)
              + exposed_headers    = (known after apply)
              + max_age_in_seconds = (known after apply)
            }

          + hour_metrics {
              + enabled               = (known after apply)
              + include_apis          = (known after apply)
              + retention_policy_days = (known after apply)
              + version               = (known after apply)
            }

          + logging {
              + delete                = (known after apply)
              + read                  = (known after apply)
              + retention_policy_days = (known after apply)
              + version               = (known after apply)
              + write                 = (known after apply)
            }

          + minute_metrics {
              + enabled               = (known after apply)
              + include_apis          = (known after apply)
              + retention_policy_days = (known after apply)
              + version               = (known after apply)
            }
        }
    }

  # azurerm_storage_queue.queue1 will be created
  + resource "azurerm_storage_queue" "queue1" {
      + id                   = (known after apply)
      + name                 = "queue1"
      + storage_account_name = "teststorageacc1"
    }

Plan: 3 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

Once we confirm this is the change we want and all is in order we can run terraform apply:

terraform.exe apply
...
...
...
Plan: 3 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

We say yes and let’s go!

It may happen that there is some error, like a taken name, in this case we will get an error:

Error: Error creating Azure Storage Account "teststorageacc1": storage.AccountsClient#Create: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="StorageAccountAlreadyTaken" Message="The storage account named teststorageacc1 is already taken."

  on terraformTest.tf line 48, in resource "azurerm_storage_account" "sacc1":
  48: resource "azurerm_storage_account" "sacc1" {

We can fix it and rerun again, and the resources that were created already will not be modified (like in this case the resource group).

azurerm_storage_account.sacc1: Creating...
azurerm_storage_account.sacc1: Still creating... [10s elapsed]
azurerm_storage_account.sacc1: Still creating... [20s elapsed]
azurerm_storage_account.sacc1: Creation complete after 21s [id=/subscriptions/kkkkkk/resourceGroups/TerraformRG/providers/Microsoft.Storage/storageAccounts/teststorageacc1anogues]
azurerm_storage_queue.queue1: Creating...
azurerm_storage_queue.queue1: Creation complete after 0s [id=https://teststorageacc1anogues.queue.core.windows.net/queue1]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Now we can either call the cli or browse through the azure portal to confirm that the storage account was created and the queue inside it:

az storage account list
[
  {
    "accessTier": "Hot",
    "allowBlobPublicAccess": false,
    "azureFilesIdentityBasedAuthentication": null,
    "blobRestoreStatus": null,
    "creationTime": "2020-12-23T18:30:49.007107+00:00",
    "customDomain": null,
    "enableHttpsTrafficOnly": true,
    "encryption": {
      "keySource": "Microsoft.Storage",
      "keyVaultProperties": null,
      "requireInfrastructureEncryption": null,
      "services": {
        "blob": {
          "enabled": true,
          "keyType": "Account",
          "lastEnabledTime": "2020-12-23T18:30:49.085263+00:00"
        },
        "file": {
          "enabled": true,
          "keyType": "Account",
          "lastEnabledTime": "2020-12-23T18:30:49.085263+00:00"
        },
        "queue": null,
        "table": null
      }
    },
    "failoverInProgress": null,
    "geoReplicationStats": null,
    "id": "/subscriptions/kkkkkk/resourceGroups/TerraformRG/providers/Microsoft.Storage/storageAccounts/teststorageacc1anogues",
    "identity": null,
    "isHnsEnabled": false,
    "kind": "StorageV2",
    "largeFileSharesState": null,
    "lastGeoFailoverTime": null,
    "location": "westeurope",
    "minimumTlsVersion": "TLS1_0",
    "name": "teststorageacc1anogues",
    "networkRuleSet": {
      "bypass": "AzureServices",
      "defaultAction": "Allow",
      "ipRules": [],
      "virtualNetworkRules": []
    },
    "primaryEndpoints": {
      "blob": "https://teststorageacc1anogues.blob.core.windows.net/",
      "dfs": "https://teststorageacc1anogues.dfs.core.windows.net/",
      "file": "https://teststorageacc1anogues.file.core.windows.net/",
      "internetEndpoints": null,
      "microsoftEndpoints": null,
      "queue": "https://teststorageacc1anogues.queue.core.windows.net/",
      "table": "https://teststorageacc1anogues.table.core.windows.net/",
      "web": "https://teststorageacc1anogues.z6.web.core.windows.net/"
    },
    "primaryLocation": "westeurope",
    "privateEndpointConnections": [],
    "provisioningState": "Succeeded",
    "resourceGroup": "TerraformRG",
    "routingPreference": null,
    "secondaryEndpoints": null,
    "secondaryLocation": null,
    "sku": {
      "name": "Standard_LRS",
      "tier": "Standard"
    },
    "statusOfPrimary": "available",
    "statusOfSecondary": null,
    "tags": {},
    "type": "Microsoft.Storage/storageAccounts"
  }
]

And then confirming the storage account has been created we can check the queue inside:

az storage queue list --account-name teststorageacc1anogues

There are no credentials provided in your command and environment, we will query for the account key inside your storage account.
Please provide --connection-string, --account-key or --sas-token as credentials, or use `--auth-mode login` if you have required RBAC roles in your command. For more information about RBAC roles in storage, visit https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-cli.
Setting the corresponding environment variables can avoid inputting credentials in your command. Please use --help to get more information.
[
  {
    "approximateMessageCount": null,
    "metadata": null,
    "name": "queue1"
  }
]

And we confirmed all is ready for our second part!

It’s time to move to azure-storage-queue python library for our 5th step. There are different ways to authenticate to the queue. Either we use the QueueServiceClient or the QueueService. the differences between both are in their official documentation here.

The easiest way albeit not the recommended one is to use the queue key, or even better, we can grab the connection string directly from the queue keys section. Be aware that this allows to do virtually all with that queue. So don’t lose this key, and if you do, rotate them and generate new ones.

Just open a new python file and to write a sync message use the following code:

from azure.storage.queue import QueueClient

queue = QueueClient.from_connection_string(conn_str="DefaultEndpointsProtocol=https;AccountName=teststorageacc1anogues;AccountKey=xxxxxx;EndpointSuffix=core.windows.net", queue_name="queue1")

queue.send_message("Hello World!")

This will send a Hello World message to our queue. If we print the output received we should see some debug info like the following:

{'id': 'a489914f-cc7e-4504-9feb-33b0efce6587', 'inserted_on': datetime.datetime(2020, 12, 24, 19, 21, 20, tzinfo=datetime.timezone.utc), 'expires_on': datetime.datetime(2020, 12, 31, 19, 21, 20, tzinfo=datetime.timezone.utc), 'dequeue_count': None, 'content': 'Hello World!', 'pop_receipt': 'AgAAAAMAAAAAAAAA727vbgja1gE=', 'next_visible_on': datetime.datetime(2020, 12, 24, 19, 21, 20, tzinfo=datetime.timezone.utc)}

Now if we do not believe the message has arribed properly to the queue, we can go to the azure portal and check by ourselves. For this, we find ou storage account, click in queues and on queue1 (or whatever name you choose in terraform) and you should see the message along with the arrival time and the expiry time (by default, 7 days from now on)

Let’s see how to retrieve that message from python now. Remember that storage account queues do not delete the processed messages so If you do not delete them they will stay until expiration (even they will stay “hidden” for 90 seconds by default unless you pass other parameter) , so make sure your call deletes the message once processed (in this case read):

messages = queue.receive_messages()

for msg in messages:
    print(msg.content)
    queue.delete_message(msg)

And this will output our message:

Hello World!

If you want to only see your message without actually “reading it” you can use the peek function. This will not mark your message as “seen” and will be returned by any other consumer reading the queue or any retry you do.

If we go back to the azure portal and check the queue we should see the message is not there anymore. We can also delete the messages from the azure portal,a s well as create them directly there.

To improve our code a little bit we can use SAS keys instead of the storage account master keys. You can generate a SAS token directly from python with the following code:

from datetime import datetime, timedelta
from azure.storage.queue import QueueServiceClient, generate_account_sas, ResourceTypes, AccountSasPermissions

sas_token = generate_account_sas(
    account_name="<storage-account-name>",
    account_key="<account-access-key>",
    resource_types=ResourceTypes(service=True),
    permission=AccountSasPermissions(read=True),
    expiry=datetime.utcnow() + timedelta(hours=1)
)

queue_service_client = QueueServiceClient(account_url="https://<my_account_name>.queue.core.windows.net", credential=sas_token)

But you still need to have your credentials inside the code. So the safest way to manage it is to create the SAS token from somewhere else with a caducity not long in the future and use this in your code, or use environment variables for passing sensitive information to your code. With the os library in python you can read them and use inside your code, without having to be written there and exposing them in your code or git.

For more information you can check microsoft official documentation here, the quickstart guide here or the pypi page here.

The post Deploying an Azure Storage Account with a queue with terraform and python appeared first on Albert Nogués.