TWIL: October 9, 2022
I learned a lot this past week! From Azure Data Lake, Azure Synapse Analytics and Azure Databricks to Elastic Databases in Azure SQL, there are several interesting articles. I’m also highlighting a few podcasts episodes, namely the latest from .NET Rocks about GitHub Copilot, and Kusto Detective Agency. Enjoy!
The Azure Podcast
Episode 440: Azure and SAP – Better Together!
The team catches up with Holger Bruchelt to get an update SAP and why we now say “SAP and Azure” and not “SAP on Azure”.
Towards Data Science
Episode 127: The emerging world of ML sensors
Today, we live in the era of AI scaling. It seems like everywhere you look people are pushing to make large language models larger, or more multi-modal and leveraging ungodly amounts of processing power to do it. But although that’s one of the defining trends of the modern AI era, it’s not the only one. At the far opposite extreme from the world of hyperscale transformers and giant dense nets is the fast-evolving world of TinyML, where the goal is to pack AI systems onto small edge devices.
Episode 1814: GitHub Copilot with Michelle Mannering
GitHub Copilot is here! Are we all going to lose our jobs? While at NDC in Oslo, Carl and Richard talked to Michelle Mannering about how Copilot helps you write code – emphasis on help! Michelle explains that Copilot is able to take your explanations of what code needs to be written to find examples of that code for you to take advantage of. It’s still up to you to break down the problem well enough, but you spend less time fussing with syntax. This is especially powerful when calling into unfamiliar APIs or coding in languages you have less experience with. The conversation dives into how the developer ecosystem continues to evolve with these new tools, so that we can do more faster!
Episode 861: Sonic Pi Live Coding Music Synth with Sam Aaron
Sonic Pi is a new kind of musical instrument which enables exciting new learning pathways in the classroom! Not only can you create music quickly and “live code” your music to change when performing, but you can also use Sonic Pi as a way to learn coding in a more creative way rather than focusing on abstract concepts or working with data.
Azure Data Lake
Overview of Azure Data Lake Storage for cloud-scale analytics
The Azure Data Lake is a massively scalable and secure data storage for high-performance analytics workloads. You can create storage accounts within a single resource group for cloud-scale analytics. We recommend provisioning three Azure Data Lake Storage Gen2 accounts within a single resource group.
Data lake zones and containers
It’s important to plan your data structure before you land it into a data lake. Having a plan helps you use security, partitioning, and processing effectively.
Disaster recovery and storage account failover
Microsoft strives to ensure that Azure services are always available. However, unplanned service outages may occur. If your application requires resiliency, Microsoft recommends using geo-redundant storage, so that your data is copied to a second region. Additionally, customers should have a disaster recovery plan in place for handling a regional service outage. An important part of a disaster recovery plan is preparing to fail over to the secondary endpoint in the event that the primary endpoint becomes unavailable.
Data governance with Profisee and Microsoft Purview
After cataloging enterprise data sources, it may be determined that there are multiple sources of customer data. To be effective, master data should be merged, validated, and corrected in Profisee, by using governance definitions, insights, and expertise that are detailed in Purview. In this way, Purview and Profisee form the foundation for governance and data management, and they maximize the business value of data in Azure.
Azure Databricks Best Practices
The Azure Databricks documentation includes a number of best practices articles to help you get the best performance at the lowest cost when using and administering Azure Databricks.
Modern analytics architecture with Azure Databricks
This solution outlines a modern data architecture that achieves these goals. Azure Databricks forms the core of the solution. This platform works seamlessly with other services such as Azure Data Lake Storage, Azure Data Factory, Azure Synapse Analytics, and Power BI.
Azure Synapse Analytics
Customer 360 with Azure Synapse and Dynamics 365 Customer Insights
This solution combines Azure Synapse Analytics with Dynamics 365 Customer Insights, to build a comprehensive view that presents your customer data and to provide the best customer experience.
Data flows in Azure Synapse Analytics
Data flows are visually designed data transformations in Azure Synapse Analytics. Data flows allow data engineers to develop data transformation logic without writing code. The resulting data flows are executed as activities within Azure Synapse Analytics pipelines that use scaled-out Apache Spark clusters. Data flow activities can be operationalized using existing Azure Synapse Analytics scheduling, control, flow, and monitoring capabilities.
Azure Synapse Lakehouse Sync
Azure Synapse Lakehouse Sync provides an easy solution to synchronizing modeled Gold Zone data from your data lake, to your Synapse Analytics Data Warehouse. Through a series of Databricks notebooks and Synapse Analytics pipelines, it offers a working example of how to continually synchronize your tables.
Machine Learning operations maturity model
The purpose of this maturity model is to help clarify the Machine Learning Operations (MLOps) principles and practices. The maturity model shows the continuous improvement in the creation and operation of a production level machine learning application environment. You can use it as a metric for establishing the progressive requirements needed to measure the maturity of a machine learning production environment and its associated processes.
Scaling out with Azure SQL Database
You can easily scale out databases in Azure SQL Database using the Elastic Database tools. These tools and features let you use the database resources of Azure SQL Database to create solutions for transactional workloads, and especially Software as a Service (SaaS) applications.
What is Marketing Attribution?
According to Wikipedia, in marketing attribution, or marketing attribution modelling is the identification of a set of user actions towards a goal, or a conversion, which we’ll refer to as touchpoints and then the assignment of a value to each of these touchpoints. The goal of marketing attribution is to get insights into what touchpoint or combination of touchpoints influences the individual towards a goal completion or conversion.
Kusto Detective Agency
Learn Kusto Query Language by solving cases as a detective. Really cool idea!
Have an awesome week!
Photo by Sander Sammy on Unsplash