I had an intense learning week on topics like Azure Data Factory, Azure Cosmos DB, Azure Synapse Analytics and Data Governance. Also, I’d like to highlight the AI Portugal Podcast, an awesome Portuguese podcast focused on AI topics with surprising depth and knowledge. Finally, some news on ChatGPT that bring some interesting (even if a bit pessimistic) points of view to the impact of AI in our future. Happy reading!

---

Podcasts

Hanselminutes

Episode 875: Ben West is not waiting for Diabetes Tech to catch up
Ben West has been at the forefront of the DIY diabetes management movement. Ben and the community’s work on Nightscout, an open-source platform for continuous glucose monitoring, has revolutionized the way people with diabetes manage their condition. Ben, along with a dedicated community of developers, has been working tirelessly to empower individuals to take control of their diabetes, giving them the tools they need to live healthier, more independent lives.

Episode 876: Game Emulation with former MAME programmer Aaron Giles
Scott talks to pro­gram­mer, mu­si­cian, web de­vel­o­per, and gra­phic de­sign­er Aaron Giles about the state of game emulation. Aaron has worked for Microsoft, Connectix, LucasArts, contributed to the MAME project for over 17 years, and even ran the project for 6  years! We chat about his current project, DREAMM, which is a Windows-based emulator for classic LucasArts SCUMM adventure games, from Maniac Mansion through The Curse of Monkey Island and everything in-between!

.NET Rocks

Episode 1830: Sustainable Open Source with Sarah Novotny
How do you create a sustainable open-source ecosystem? Carl and Richard talk to Sarah Novotny about how the open-source community continues to evolve. Sarah talks about how the vast majority of software utilizes open-source code and should be contributing to those projects. Those contributions can be financial or development resources, whatever makes more sense for the organization. The conversation also explores the nature of those contributions when the project has smaller and larger audiences, is purely for developer consumption, or the greater public. Open-source software is diverse, so the support for open-source software also needs to be diverse!

Building the Future AI Portugal Podcast

Portuguese podcast born out of the annual Building the Future event where technologies, ideas and initiatives that transform our world are discussed, with a particular focus on Artificial Intelligence. The episodes of this podcast are spoken in portuguese, as are their descriptions.

Guia de como aprender IA
No episódio de hoje decidimos focar a nossa conversa nas diferentes formas, ferramentas e recomendações para os nossos ouvintes que estão a molhar os pés em IA, ou que já são experientes em IA e conhecem outras pessoas que querem começar ou crescer nesta área e os querem aconselhar. É um tópico que nos têm falado que devíamos abordar e por isso estamos aqui a fazer isso mesmo

Filosofia da Mente e Consciência (com convidado Paulo Castro)
Vamos hoje dedicar o episodio de hoje a falar sobre Filosofia da Mente e Consciência. Existe imenso debate e até relativamente pouco consenso sobre a definição formal de consciência no que toca a IA, foram desenhados vários testes formais, muitos deles já provados ultrapassados e outros completamente inatingíveis. As máquinas ditas inteligentes de hoje estão cada vez mais proficientes em imitar os humanos e em enganar estes testes e não tanto em realmente ter esta dita consciência de uma forma mais completa.

Explainable AI
O tópico do episódio de hoje é o Explainable AI, também conhecido como XAI. Este é o nome da área de estudo dentro da IA que trata a criação de modelos de IA cujos processos de decisão que dão origem às predições e ações conseguem ser entendidas por humanos. No fundo é o oposto das conhecidas abordagens de Black Box, em que apenas temos acesso aos outputs das predições do modelo sem nenhum insights sobre como é que o modelo chegou a essa conclusão.

---

Azure Data Factory

Copy activity in Azure Data Factory and Azure Synapse Analytics
In Azure Data Factory and Synapse pipelines, you can use the Copy activity to copy data among data stores located on-premises and in the cloud. After you copy the data, you can use other activities to further transform and analyze it. You can also use the Copy activity to publish transformation and analysis results for business intelligence (BI) and application consumption.

Copy Data tool in Azure Data Factory and Synapse Analytics
The Copy Data tool eases and optimizes the process of ingesting data into a data lake, which is usually a first step in an end-to-end data integration scenario. It saves time, especially when you use the service to ingest data from a data source for the first time.

---

Azure Cosmos DB

Performance Improvements in Azure Cosmos DB Kafka Connector

Azure Cosmos DB Kafka connector had been updated recently and several new features were delivered as part of the v1.6.0 release based on the feedback and requirements of several customers implementing it as part of their application modernization journey. In this blog post – I will share details about those improvements and introduce you to the new connector features and use cases they are targeted to solve.

---

Azure Synapse Analytics

Synapse – Data Lake vs. Delta Lake vs. Data Lakehouse
As a data engineer, we often hear terms like Data Lake, Delta Lake, and Data Lakehouse, which might be confusing at times. In this blog we’ll demystify these terms and talk about the differences of each of the technologies and concepts, along with scenarios of usage for each.

---

Data Governance

The Power of Data Catalog
Data catalogs enable organizations to optimize their data management processes and improve performance. In this article, we’ll explore the power of data catalogs and how they can benefit organizations. We’ll cover what a data catalog is, the benefits of using a data catalog, how to implement one, the benefits of data catalogs for data governance and data security, as well as best practices for developing a data catalog. We’ll also discuss the challenges of implementing a data catalog.

A Comprehensive Framework for Data Quality Management
In this article, we discuss the definition of Data Quality, five dimensions of Data Quality including Accuracy, Completeness, Timeliness, Consistency and Uniqueness. We walk through the calculation of the metrics, the high-level implementation, the process of managing Data Quality and the responsibilities of all parties involving in this process.

---

Architecture

Astronomer: Modern Data Orchestration
Astro is a cloud solution that helps you focus on your data pipelines and spend less time managing Apache Airflow, with capabilities enabling you to build, run, and observe data all in one place.

Event-Driven Architecture Visuals
Awesome collection of small bite-sized visuals about event-driven architectures.

Good Tech Things
Good Tech Things are comics, videos, flowcharts, and infographics handcrafted by Forrest Brazeal.

---

Artificial Intelligence

AutoML + GPT-3: a Match made in Heaven for Data Science Success
Welcome to my blog about how Automated Machine Learning (AutoML) can be enhanced by integrating it with OpenAI, or more specifically GPT-3. Using Azure Cloud as the platform for discussing and demonstrating examples of the integration, I will delve into ways in which OpenAI can augment AutoML. Let’s dive in!

---

News

By posting these links, I’m not endorsing these articles or stating that I share the opinions of their authors. Even if I don’t agree with everything that is written, I found the topics and approaches interesting.

ChatGPT could make these jobs obsolete: ‘The wolf is at the door’
Artificial intelligence is here, and it’s coming for your job. So promising are the tool’s capabilities that Microsoft — amid laying off 10,000 people — has announced a “multiyear, multibillion-dollar investment” in the revolutionary technology, which is growing smarter by the day. And the rise of machines leaves many well-paid workers vulnerable, experts warn.

Rogue AI ‘could kill everyone,’ scientists warn as ChatGPT craze runs rampant
While artificial intelligence systems might make lives exponentially easier, they could also have a sinister side effect — making us go extinct. That’s right, researchers are deeming rogue AI an “existential threat to humanity” that needs to be regulated like nuclear weapons if we are to survive.

Inteligência Artificial: quem é o verdadeiro artista?
[Portuguese Article] Há 25 anos discutíamos Photoshop, hoje discutimos sistemas como DALL-E, Stable Diffusion ou Midjourney. E continuamos a confundir criatividade com autoria.

---

Have a fantastic week!