It’s becoming harder and harder to keep track of all the new products, frameworks and features being announced and released every week. In the past two weeks, I’ve been focused on Microsoft Fabric, Power BI and Microsoft Purview. Also, I’m highlighting a set of articles on Generative AI, Semantic Kernel and the awesome AutoGen framework. Have fun!

---

Podcasts

Lex Fridman Podcast

Episode 399: Jared Kushner: Israel, Palestine, Hamas, Gaza, Iran, and the Middle East
Jared Kushner is a former Senior Advisor to President Donald Trump and author of Breaking History, a book about his role in the Middle East peace process and other foreign policy issues. This episode covers a wide range of topics, such as the Israel-Gaza war, the Abraham Accords, Trump 2024, Iran, North Korea, China, Russia, Ukraine, Saudi Arabia, and more.

---

Microsoft Fabric

Manage Apache Spark libraries in Microsoft Fabric
Each workspace comes with a preinstalled set of libraries available in the Spark run-time that you can use immediately in a notebook or Spark job definition. We refer to these as built-in libraries. Based on your scenarios and specific needs, you can include other libraries.

How to create custom Spark pools in Microsoft Fabric
In this document, we explain how to create custom Apache Spark pools in Microsoft Fabric for your analytics workloads. Apache Spark pools enable users to create tailored compute environments based on their specific requirements, ensuring optimal performance and resource utilization.

Microsoft Fabric October 2023 update
We have a lot of features this month including updates to the Item type icons, Keyword-Based Filtering of Tenant Settings, On-object Interaction updates, Eventstream Kafka Endpoints and many more. Continue reading for more details on our new and updated features!

Announcing: Fabric Warehouse publishing full DML to Delta Lake Logs
We are excited to announce that the Data Warehouse now publishes all Inserts, Updates and Deletes for each table to their Delta Lake Log in OneLake! Our vision is to break down data silos and make it really easy to share data from your Data Warehouses with other teams who use different services without having to create copies of your data in different formats.

---

Power BI

Power BI Project (PBIP) and Azure DevOps build pipelines for continuous integration
Combining Fabric Git Integration with Azure DevOps, enables you to connect a workspace to a branch in an Azure DevOps repository and automatically synchronizes between them. Integrating the PBIP format with Azure DevOps lets you use Azure Pipelines to automate Continuous Integration/Continuous Deployment (CI/CD) pipelines. These pipelines process the PBIP metadata files and apply a series of quality checks to your development before deploying it to the production system.

---

Microsoft Purview

Injecting lineage and attributes into Microsoft Purview
Automatic lineage collection in Microsoft Purview is currently only supported by certain ADF activities and Synapse activities. No doubt this will expand over time, however you may wish to capture lineage from other engines such as Synapse Spark or Databricks. Whilst there is a Spark based lineage collector, as well as the Azure Databricks to Purview Lineage Connector based on Open Lineage, you can alternatively inject your own lineage programmatically — for every transformation or process which creates a new dataset there will need to be some additional code run in order to create the relationship or linkage between inputs and outputs.

---

Generative AI

Access Control in Generative AI applications with Azure Cognitive Search
We’ve seen huge interest from organizations that want to use Azure OpenAI service to access Large Language Models (LLMs) in combination with their own data. Allowing these applications to access your organization’s knowledge base allows inclusion of data relevant to the conversation, creating a richer and more useful experience. However, this introduces new problems if the Generative AI application isn’t aware of any access control requirements. We’ve recently updated the Cognitive Search OpenAI Demo to allow user login and access control, which enables the Generative AI application to tailor responses on a per-user basis.

---

Semantic Kernel

Introduction to Microsoft Semantic Kernel: Supercharge Your LLM Application (Data Visualization App -Python Tutorial)
Welcome to a journey into the realm of the Microsoft Semantic Kernel! In this tutorial, we will delve into the capabilities and integration of the Microsoft Semantic Kernel, an open-source Software Development Kit that makes it easy to combine LLMs from multiple services such as OpenAI, AzureOpenAI, and Hugging Face.

Semantic Kernel Memory — A new open source AI project from Microsoft
In the ever-evolving landscape of artificial intelligence and natural language processing, the ability to efficiently index vast datasets and query them with natural language has become a paramount necessity. Enter Semantic Memory (SM), an open-source service and plugin that specializes in this very task, taking data-driven applications to new heights. In this blog post, we will delve into the world of Semantic Memory, exploring its capabilities, use cases, and potential impact.

---

AutoGen

Introducing AutoGen
AutoGen is a framework to develop LLM-powered applications using multiple agents that are able to communicate and orchestrate among each others. You can literally think about it as a think tank full of experts, each one with a different domain of knowledge, yet all of them extremely capable in advanced reasoning, working together to solve a complex, multidisciplinary task.

AutoGen is Mindblowing: 4 Features that Make AutoGen the State-of-the-art Framework for Creating AI Agents.
This blog post presents AutoGen, a framework for creating LLM applications with multiple AI agents that can interact with each other and external APIs. It explains how AI agents can plan, execute, and communicate to solve complex tasks and highlights four features that make AutoGen the leader among AI-agent frameworks: complete flexibility, human participation, multi-agent conversations, and flexible autonomy. The page also demonstrates an example of how AutoGen can autonomously solve a non-trivial coding task involving plotting the dollar to euro exchange price.

Microsoft’s Groundbreaking Release: AutoGen — An AI Game Changer
Microsoft recently introduced AutoGen, a groundbreaking AI framework that promises to reshape the landscape of AI applications. This significant unveiling was accompanied by a massive presentation outlining its integration with Microsoft’s existing AI offering, Copilot.

Pioneering the Future of Web Scraping with Intelligent AI Agents: Unleash the Power of AutoGen
Welcome to the future of web scraping, where the fusion of advanced AI agents and web data extraction is not only possible but remarkably accessible. In this article, we embark on an exciting journey into the realm of AutoGen — a revolutionary framework that empowers developers and enthusiasts to create intelligent AI agents, capable of conversing, collaborating, and seamlessly integrating with humans and tools.

---

Prompt Engineering

A New Prompt Engineering Technique Has Been Introduced Called Step-Back Prompting
Step-Back Prompting (STP) is a new prompting technique that enables Large Language Models (LLMs) to perform abstractions and reasoning by paraphrasing a question to a more generic one. This blog post provides some examples of STP and the principles behind them. It also shows how to create a prompt for the final answer using the stepback question and answer. It reports the performance of STP on various tasks and datasets, and compares it with other prompting techniques such as Chain-of-Thought (COT) and Retrieval Augmented Generation (RAG).

---

Interesting Stuff

14 habits that make you more focused than 98% of people
Alex Mathers shares 14 habits that can make anyone more focused and productive. The habits include knowing the difference between being overloaded and overwhelmed, asking how to do more with less, avoiding multitasking and doing one thing at a time, writing down goals every day, relying on natural focus instead of stimulants, prioritising enjoyment and finding ways to have fun, understanding the glycemic index and eating low GI foods, avoiding cheap dopamine sources like porn and video games, stopping making everything personal and focusing on the next step, identifying a purpose and writing it down, using accountability partners or coaches, loving boredom and staying with it until creativity follows, having a routine and sticking to it, and tracking key metrics and results.

Building immersive experiences with Microsoft Mesh
Microsoft Mesh, now in public preview, is a new 3D immersive experience for the workplace, helping virtual meetings and events feel more like face-to-face connections. Now you can bring your 3D world building skills to the global workforce, helping users connect like never before. With Mesh toolkit you can create fully customized, multi-user 3D experiences – giving you a powerful, imaginative medium for addressing business needs and delighting users.

---

Have a brilliant week!