Written by George Moore andAdamAbdelhamed, Group Principal Program Managers in the Copilot in Azure team
Introduction
Copilot in Azure is an intelligent assistant designed to help you design, operate, optimize and troubleshoot your Azure environment. This article provides a technical deep dive into the Copilot in Azure architecture and data flows.
Copilot in Azure is designed to make IT administrators, developers, data professionals and financial operations more efficient with their interactions across all Azure services. Because Copilot is aware of the live data in the Azure resource graph and real-time service telemetry, it can carefully craft responses which are deeply customized and relevant to you. This can help to automate many common manageability tasks, enabling operators to manage larger collections of cloud or on-prem assets with less effort.
In addition, because there are many different role types in a typical organization, the interactions between different roles are just as important as the specific data available to each role. We are thinking deeply about these role interactions to ensure that team effectiveness is facilitated by Copilot. As an example, the developer role interactions between Copilot in Github (for code generation) and Copilot in Azure (for app deployment and testing):
Architectural Overview
The Copilot in Azure architecture consists of three major components: 1) the Frontend user interface, 2) an Orchestration layer across all Azure services, and 3) the underlying AI infrastructure:
Frontend
The Frontend provides a common, consistent user interface to Copilot in Azure. It implements the conversation window, provides multimodal responses to questions (text, charts, illustrations), and collects user feedback. It also understands the current navigation context so the user can ask questions in a natural manner over the resources displayed on the current portal page.
Orchestration
The Orchestration layer is the heart of Copilot in Azure. Because Azure has hundreds of different services, this layer generates a deep semantic understanding of the user’s question using the Large Language Model, reasons over all Azure resources and resource types, and then dispatches the question to the relevant domain-specific plugins. Those plugins use their service-specific graph or observability data to answer the user’s question. For example, questions relating to YAML editing results in the Orchestrator calling the AKS plugin to invoke the built-in YAML Editor:
The Orchestrator can also reason over highly complex, multistep questions, such as “Please show all VMs running with less than 10% utilization which have been deployed in the last hour in Europe”. Copilot in Azure will then invoke the Azure Resource Graph plugin to query the graph and produce the results:
By pressing the "Run" button in the Copilot, you can then easily run the query in the Graph Explorer:
AI Infrastructure
The AI Infrastructure used by Copilot in Azure is the same Azure OpenAI (AOAI) infrastructure which is commercially available to any developer. The same functions, methodologies and architectural best practices described in this blog can be readily used by anyone to build their own advanced AI applications.
When the user asks a question to the AI, this resulting data flow is used to provide the answer:
Step 1: Metadata about the current navigational context is gathered to assist AOAI in building a semantic understanding of the user’s question. For example, if you are on an Azure Kubernetes deployment page in the portal, metadata about the current resource (version, node pools, node sizes, etc) are gathered by the portal and added as grounding context to the prompt for the AOAI infrastructure.
Step 2: The Frontend calls the Orchestration layer with the full prompt from Step 1. The fully grounded prompt is then pre-processed for Responsible AI. Assuming everything is correct, the prompt is then injected into AOAI, which reasons over the list of domain-specific plugins and returns the best match plugin for this question. If the user’s question is off-topic or semantically malformed, a friendly error message is returned in the conversation window.
Step 3: Orchestration then calls the selected plugin from the Plugin Store.
Step 4: The domain-specific plugin runs. It can fetch and combine data from many sources available within the user’s security context. For example, the “Docs & Learn” plugin performs a RAG-pattern query across the corpus of documentation in docs.microsoft.com, while the Azure Compute plugin can report on your Virtual Machine details from the Azure Graph.
Step 5: The plugin responds with the answer to the user’s question as a result of the queries executed in Step 4. A final Responsible AI pass is then performed over the resulting answer, which is then pushed to the Frontend.
Steps 6-7: The response payload from the plugin is sent to Portal for rendering a nicely formatted response with rich graphics. If the user has made a request for a change to their environment, the Portal will prompt the user for confirmation before proceeding with the requested change to the Azure Graph in Step 7.
On Premises AI Manageability
Copilot in Azure can also provide AI-enhanced manageability over the millions of bare metal Linux, VMWare vSphere, Windows Server and Azure Stack HCI servers running on prem. Each of these servers can be cloud-connected via Azure Arc, which results in their control plane and observability state being replicated to the Azure cloud. This enables Copilot in Azure to assist in the manageability of these remote servers using the same exact orchestration patterns as described above. As an example, an IT admin can easily understand the real-time status of their remote fleet of on-prem servers:
Responsible AI
AI fairness, reliability, safety, privacy, security, inclusiveness, transparency and accountability are all key parts of Microsoft’s Responsible AI principles. Copilot in Azure is designed to enforce these principles at multiple levels:
Technical safety controls within the AI model
Copilot in Azure uses Content Safety to prioritize questions about Azure and to not engage in unrelated topics. The following example shows the polite way it declines an off-topic conversation:
Copilot in Azure won’t answer off-topic questions, such as this example involving blueberry pancakes
Technical safety controls between the AI model and the underlying Azure Graph
No Elevation of Privilege: The portal frontend and orchestration layers run in the user’s current authentication and authorization security context, which means Copilot in Azure can only access data which is available to the currently authenticated user.
Permission is required for changes: Most questions to the AI are about the state of your Azure resources, with the AI generating authoritative and contextualized answers. However, if there is a request which would result in changes to your environment, Copilot in Azure always stops and requests permission before proceeding. Under no circ*mstances would Copilot in Azure make changes to your Azure environment without your knowledge.
Learn more
Azure Build 2024 Infrastructure Blog
Copilot in Azure website
Copilot in Azure documentation
George Moore is one of the original co-founders of the Azure Engineering Team in 2006. He designed and built large portions of the compute infrastructure for Azure over the last 18 years. He is presently focused on bringing Copilot in Azure AI experiences to existing on-prem servers.
