<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>DevOps Archives - Albert Nogués</title>
	<atom:link href="https://www.albertnogues.com/category/devops/feed/" rel="self" type="application/rss+xml" />
	<link>https://www.albertnogues.com/category/devops/</link>
	<description>Data and Cloud Freelancer</description>
	<lastBuildDate>Sat, 07 May 2022 08:38:17 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	

<image>
	<url>https://www.albertnogues.com/wp-content/uploads/2020/12/cropped-cropped-AlbertLogo2-32x32.png</url>
	<title>DevOps Archives - Albert Nogués</title>
	<link>https://www.albertnogues.com/category/devops/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Smallest Analytical Platform Ever!</title>
		<link>https://www.albertnogues.com/smallest-analytical-platform-ever/?utm_source=rss&#038;utm_medium=rss&#038;utm_campaign=smallest-analytical-platform-ever</link>
		
		<dc:creator><![CDATA[Albert]]></dc:creator>
		<pubDate>Sat, 07 May 2022 08:38:12 +0000</pubDate>
				<category><![CDATA[Azure]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Databricks]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Spark]]></category>
		<category><![CDATA[SQL]]></category>
		<category><![CDATA[databricks]]></category>
		<category><![CDATA[git]]></category>
		<category><![CDATA[python]]></category>
		<category><![CDATA[spark]]></category>
		<guid isPermaLink="false">https://www.albertnogues.com/?p=1440</guid>

					<description><![CDATA[<p>I&#8217;ve started working on some of my free time in a project to build the smallest useful analytics platform on the cloud (starting with azure). The purpose is to use it a sa PoC to show to colleagues, managers, prospective customers or just to have fun and play It&#8217;s publicly available on my github repo &#8230; </p>
<p>The post <a href="https://www.albertnogues.com/smallest-analytical-platform-ever/">Smallest Analytical Platform Ever!</a> appeared first on <a href="https://www.albertnogues.com">Albert Nogués</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p>I&#8217;ve started working on some of my free time in a project to build the smallest useful analytics platform on the cloud (starting with azure).</p>



<p>The purpose is to use it a sa PoC to show to colleagues, managers, prospective customers or just to have fun and play</p>



<p>It&#8217;s publicly available on my github repo and any collaboration is welcome. You can fork it, improve it, send PR&#8217;s and do whatever you want!</p>



<p>The first version will run solely on azure. The objective is to show the following technologies/disciplines:</p>



<p>* Infrastructure as a Code (IaaC), by using Terraform</p>



<p>* Cloud architecture anc Cloud Ops by using an azure cloud environment</p>



<p>* Data Engineering by using a Spark powered Databricks Notebook and an ADF Pipeline (future)</p>



<p>* DevOps to trigger some pipelines based on changes (future)</p>



<p>* Basic Security concepts (keyvault, service principals, least privileged rbac accesses&#8230;)</p>



<p>* FinOps keeping the costs at minimum and choosing the proper tools for the job</p>



<p>* Reporting and Dashboarding on data in the platform</p>



<p>* Data management: We will use an adls storage account and azure sql db</p>



<p>TOOLS:</p>



<p>* Terraform to deploy all the infra as a code</p>



<p>* Azure Cloud to host our resources</p>



<p>You have the code plus all the information on my github repo:</p>



<p><a href="https://github.com/anogues/ProjectZ">https://github.com/anogues/ProjectZ</a></p>



<p></p>
<p>The post <a href="https://www.albertnogues.com/smallest-analytical-platform-ever/">Smallest Analytical Platform Ever!</a> appeared first on <a href="https://www.albertnogues.com">Albert Nogués</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Implementing CI/CD in Databricks with Azure DevOps (Part 1)</title>
		<link>https://www.albertnogues.com/implementing-ci-cd-in-databricks-with-azure-devops-part-1/?utm_source=rss&#038;utm_medium=rss&#038;utm_campaign=implementing-ci-cd-in-databricks-with-azure-devops-part-1</link>
		
		<dc:creator><![CDATA[Albert]]></dc:creator>
		<pubDate>Sat, 30 Apr 2022 15:10:29 +0000</pubDate>
				<category><![CDATA[Azure]]></category>
		<category><![CDATA[Databricks]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Spark]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[databricks]]></category>
		<category><![CDATA[git]]></category>
		<category><![CDATA[spark]]></category>
		<guid isPermaLink="false">https://www.albertnogues.com/?p=1416</guid>

					<description><![CDATA[<p>There are many ways to implement some CI/CD with Databricks. We can use Azure DevOps, Github+Github Actions or any other combination of tools, including the dbx tool. But an easy way to just copy notebooks between workspaces can be implemented easily with Azure DevOps. We are going to use the git repos capability of Azure &#8230; </p>
<p>The post <a href="https://www.albertnogues.com/implementing-ci-cd-in-databricks-with-azure-devops-part-1/">Implementing CI/CD in Databricks with Azure DevOps (Part 1)</a> appeared first on <a href="https://www.albertnogues.com">Albert Nogués</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p>There are many ways to implement some CI/CD with Databricks. We can use Azure DevOps, Github+Github Actions or any other combination of tools, including the <a href="https://dbx.readthedocs.io/en/latest/templates/python_basic.html#project-file-structure" target="_blank" rel="noreferrer noopener">dbx tool</a>.</p>



<p>But an easy way to just copy notebooks between workspaces can be implemented easily with Azure DevOps.</p>



<p>We are going to use the git repos capability of Azure Databricks, so when a new code change is commited in a notebook an Azure DevOps pipeline will trigger the transport copy of the workbook from the first Databricks workspace (in our case a NonProd workspace) to the target one, again, in our case, the Prod workspace.</p>



<p>To achieve this we will use some more components from the Azure ecosystem, including the use of Keyvaults to keep all our secrets stored safely. The list of prerequisites is the following:</p>



<ul class="wp-block-list"><li>Two Databricks workspaces, one our source workspace (NonProd) and another, our Production one.</li><li>An Azure Keyvault (or two if we want to segregate the environments)</li><li>Azure Databricks repository configured at least in our source workspace, so when the change is commited we can triger the pipeline that will fetch the notebook and transport it to the prod workspace</li><li>Access to Azure DevOps (Something similar can be implemented with Github + Github Actions)</li></ul>



<p>Lets see how to implement it. First we need to make sure git repos in enabled our source workspace. We can verify by login with an admin privileged user to our workspace and make sure the option is checked as follows:</p>



<figure class="wp-block-image size-large is-resized"><img fetchpriority="high" decoding="async" src="https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD1-924x1024.png" alt="" class="wp-image-1418" width="687" height="760" srcset="https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD1-924x1024.png 924w, https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD1-271x300.png 271w, https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD1-768x851.png 768w, https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD1-54x60.png 54w, https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD1.png 1072w" sizes="(max-width: 687px) 100vw, 687px" /><figcaption>Fig 1. Make sure that github repos is enabled in our workspace.</figcaption></figure>



<p>Secondly, we go to <a href="https://azure.microsoft.com/en-us/services/devops/" target="_blank" rel="noreferrer noopener">Azure DevOps services</a> and we create a new project. I&#8217;ve called it DatabricksCICD but feel free to call it whatever you need:</p>



<figure class="wp-block-image size-large"><img decoding="async" width="1024" height="714" src="https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD2-1024x714.png" alt="" class="wp-image-1419" srcset="https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD2-1024x714.png 1024w, https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD2-300x209.png 300w, https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD2-768x536.png 768w, https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD2-86x60.png 86w, https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD2.png 1531w" sizes="(max-width: 1024px) 100vw, 1024px" /><figcaption>Fig 2. Create a new repo and initialize it</figcaption></figure>



<p>Once created we take the details for cloning our repo and copying them. We go to our databricks workspace and then we look for the Repos option on the left, and add a new repository. We need to paste the url to clone our newle Azure DevOps created repository:</p>



<figure class="wp-block-image size-large"><img decoding="async" width="1024" height="319" src="https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD3-1024x319.png" alt="" class="wp-image-1420" srcset="https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD3-1024x319.png 1024w, https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD3-300x93.png 300w, https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD3-768x239.png 768w, https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD3-1536x478.png 1536w, https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD3-2048x637.png 2048w, https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD3-193x60.png 193w" sizes="(max-width: 1024px) 100vw, 1024px" /><figcaption>Fig 3. Cloning our Azure DevOps Repository</figcaption></figure>



<p>Once we linked our Databricks workspace with our DevOps repo, now we can create a new notebook. In the same Repos section, click on the down arrow to create a new notebook, as shown below:</p>



<figure class="wp-block-image size-full"><img loading="lazy" decoding="async" width="889" height="373" src="https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD4.png" alt="" class="wp-image-1421" srcset="https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD4.png 889w, https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD4-300x126.png 300w, https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD4-768x322.png 768w, https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD4-143x60.png 143w" sizes="auto, (max-width: 889px) 100vw, 889px" /><figcaption>Fig 4.  Creating a new notebook.</figcaption></figure>



<p>The content of the notebook, you can put anything you want. I&#8217;m writing a print(&#8220;Hello from Albert&#8221;) statement. We will not run it, we just want to show it&#8217;s possible to transport it. Once done, click on the save now in the revision tab:</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1024" height="73" src="https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD5-1024x73.png" alt="" class="wp-image-1423" srcset="https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD5-1024x73.png 1024w, https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD5-300x21.png 300w, https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD5-768x55.png 768w, https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD5-1536x110.png 1536w, https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD5-2048x146.png 2048w, https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD5-600x43.png 600w" sizes="auto, (max-width: 1024px) 100vw, 1024px" /><figcaption>Fig 5. Saving our changes to the notebook.</figcaption></figure>



<p>Then click on the left in the main branch button, from there we will be commiting the changes to our repository:</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1024" height="314" src="https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD6-1024x314.png" alt="" class="wp-image-1424" srcset="https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD6-1024x314.png 1024w, https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD6-300x92.png 300w, https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD6-768x235.png 768w, https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD6-1536x470.png 1536w, https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD6-2048x627.png 2048w, https://www.albertnogues.com/wp-content/uploads/2022/04/DatabricksCICD6-196x60.png 196w" sizes="auto, (max-width: 1024px) 100vw, 1024px" /><figcaption>Fig 6. Pushing our notebook to the DevOps repo</figcaption></figure>



<p>If we go back now to our Azure DevOps project we should see the file has been commited to the repository. This ends the first part of this tutorial.</p>



<p>In the second blog entry we will see how to trigger the pipeline after a modification of this notebook and passing the credentials of the second workspace to be able to deliver the changed notebook to our Production (target) workspace.</p>
<p>The post <a href="https://www.albertnogues.com/implementing-ci-cd-in-databricks-with-azure-devops-part-1/">Implementing CI/CD in Databricks with Azure DevOps (Part 1)</a> appeared first on <a href="https://www.albertnogues.com">Albert Nogués</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
