Demystifying Azure Monitor Agent

Azure, Azure Arc, Azure Monitor Leave a comment

Introduction

I’ve started to get more and more questions from customers about the Azure Monitor Agent, and how it’s different from the Microsoft Monitoring Agent, or Log Analytics Agent as it’s also known as.
I will try to cover installation, configuration and troubleshooting in this post. This is however limited to servers, so I will only briefly cover client operating systems as this is not my expertise.

You might previously have used the Microsoft Monitoring Agent, which was just an installer you downloaded. With the Azure Monitoring Agent you will instead use Azure VM extensions to install the agent. But Jesper, how do I install this on my on-premises servers then? Don’t worry, Azure Arc is here to help you. Currently it is a requirement that you onboard your servers to Azure Arc, if you want to use Azure Monitor Agent. In my opinion Azure Arc is a no-brainer, so I don’t see a any reasons why you wouldn’t want to do this.

After you have connected your on-premises VMs to Azure Arc, they will show up in Azure Portal and can be managed from there, just like a regular Azure VM. This means you can install extensions, and manage many other aspect that I won’t dive too much into in this post. But with Azure Arc, you can:

  • [Preview] Connect via SSH to Linux servers
  • [Preview] Install Windows Admin Center and manage Windows servers through Azure Portal – including RDP!
  • Utilize the power of Defender for Cloud
  • Apply Azure Policies to your servers – install agents, deploy machine configurations and such
  • Leverage Automanage, which you should really look into
  • Do inventory and change tracking
  • Manage Updates for both Windows and Linux – and it integrates directly with Azure Update Management Center
  • And of course: Monitor your servers and collect performance data and logs!

See? No reason to not onboard your servers right away!

Installation

As briefly mentioned above, the Azure Monitor Agent is installed through Azure VM Extensions when it comes to servers. For client OS’ there is actually an executable to download and install on your client. The Agent is supported for the following Operating Systems:

  • Windows 10 & 11
  • Windows Server 2022, 2019, 2016, 2012 R2, 2012, 2008 R2 SP1
  • AlmaLinux 8
  • Amazon Linux 2
  • CentOS 7 & 8
  • CBL-Mariner 2.0
  • Debian 9, 10 & 11
  • OpenSUSE 15
  • Oracle Linux 6, 7 & 8
  • Red Hat Enterprise Linux Server 7, 8, 8.6, 9
  • Rocky Linux 8
  • SUSE Enterprise LInux Server 12. 15 (SP1, SP2, SP3, SP4)
  • Ubuntu 16.04, 18.04, 20.04 & 22.04

For an updated list, please refer to this page: https://learn.microsoft.com/en-us/azure/azure-monitor/agents/agents-overview

To install the Azure Monitor Agent extension, navigate to your Azure Arc server and select Extensions in the left menu:

On the Extensions blade your can see the already installed Extensions, and for this server I have MDE.Windows extension installed, which is Microsoft Defender for Endpoint. In the top you can Add a new extension:

And on the next blade you can select the different extensions available. Select the Azure Monitor Agent for Windows (if you have a Windows VM of course, otherwise Linux), and select Next:

If you have proxy settings you need to, remember to do it on the next page. I will come back to Data Collections Rules later, so skip that part for now, and just select Review + Create, and then Create:

If you click the “Submitting deployment” popup, you will be taken to the deployment page, which shows the progress:

After deployment navigate to your Azure Arc server again and look at Extensions:

As you can see, the AzureMonitorWindowsAgent extension is now installed. For some reason currently there is an update available, but you can just update it or wait for it to be automatically updated, since the extension is Enabled for Automatic Upgrade. If you want to update the extension manually, just tick of the extension, and select Update and then yes:

How can I see if the agent is installed on the server?

If you’re in doubt that the agent has been installed, you can also see this on the server itself. Unlike what you might think, you can’t see this is Control Panel -> Programs and Features. There you’ll just see the Azure Connected Machine Agent (Azure Arc). Instead open up File Explorer and browse to C:\Resources\Directory where you’ll find AMADataStore.<servername> (in my case AMADataStore.ARCDC01:

In this folder you’ll find different resources for the Azure Monitor Agent. In Configuration folder for example, you can find some log files the agent:

In C:\Resources\Directory\AMADataStore.ARCDC01\mcs folder you’ll find more information like the configuration from Data Collection Rules etc. If there is no Data Collection Rule associated with the server, you’ll see this in the MonAgentHost logfile:

So let’s dive into these now

Is the agent running?

There is a few commands to help you troubleshoot the Azure Monitoring Agent. First you can check if the Azure Connected Machine agent is healthy:

azcmagent show

This will show you if the Extension service is running:

You can also check that all required endpoints are reachable by the agent, by using this command:

azcmagent check

To see if the Azure Monitor Agent is running, find the process named MonAgentCore:

Get-process -Name MonAgentCore

If the agent is running, you should see the process:

Data Collection Rules

A bit of history first: With Microsoft Monitoring Agent, the configuration for which data to collect, was decided on the Log Analytics Workspace which the agent was connected to. This was a workspace wide setting, so all servers got the same configuration, and collected the same logs and performance data. Not the best design, I think we can agree..

Anyways, with Azure Monitoring Agent we now have Data Collection Rules, or DCRs. The name is pretty explanatory: It’s a configuration which tells your agent what data should be collected.

For example you might have different teams, who want to collect different data. The Application team might be interested performance data and logs from the application running on the server, and will create a Data Collection Rule for that and send the data to their Log Analytics Workspace. The security team don’t need that data, so instead they configure their own DCR and collect data to their Log Analytics Workspace, which is probably also used with Azure Sentinel. This diagram from Microsoft explains it very well:

Let’s see how to create a DCR. In the Azure Portal, navigate to Azure Monitor:

In the left menu find and select Data Collection Rules, and then Create:

Configure which platform the DCR is used on, and of course the other settings:

On the resources blade you can select which machines to associate this Data Collection Rule to. I’ll wait with that, and show later.

Next is “Collect and deliver” blade, where we configure which data sources to collect, and where to send it. Select “Add data source”:

Then select if you want to collect Performance Counters or Windows Event Logs – I’ll go with Event Logs, and select the Security logs “Audit success” and “Audit failure”:

On the destination page, I’ve selected Azure Monitor Logs and then the Log Analytics Workspace I want to send the event logs to. It’s important to note that you can send data to Log Analytics Workspaces in a different subscription here, it does not have to be in the same subscription. When you’re done, select “Add data source” in the bottom:

After that select Review and create, and then Create.

When the creation is done, navigate to Azure Monitor again, find your Data Collection rule and select “Resources” on the left menu, then “Add” in the top:

Next select the server you want to assign the Data Collection Rule to – this can of course be both Azure VMs and Azure Arc servers.

Shortly after you click Apply, the configuration will be deployed to the server. It’s usually a matter of 1-2 minutes, but might take a bit longer. You can also see this on the server itself.

First you can see in the C:\Resources\Directory\AMADataStor.<servername>\Configuration\MonAgentHost.log logfile you can see that the DCR is now associated and has been downloaded:

In the C:\Resources\Directory\AMADataStore.<servername>\mcs folder you’ll see that a mcsconfig.xml file is now created. The contents of this file explains what data should be collected, and where it should be sent:

<?xml version="1.0" encoding="utf-8"?>
<MonitoringManagement version="1.0" namespace="AzureMonitorAgent" timestamp="2014-03-09T00:00:00.0000000Z">
  <Management eventVolume="Medium" disableNodeDiagnostics="true">
       <Identity type="TenantRole" />
    <AgentResourceUsage diskQuotaInMB="10000" />
  </Management>
  <Events>
    <HeartBeats>
      <HeartBeat diskQuotaInMB = "32" retryTimeout = "PT10M" retentionInDays = "29" duration = "PT2M" sampleRateInSeconds = "60" storeType = "Local" eventName = "MaHeartBeats" />
    </HeartBeats>
    <WindowsEventLogSubscriptions duration="PT1M" startTimeDeltaInSeconds="-900">
      <Subscription eventName="c13285865487085541030_546419284751770569" storeType="Local" query="Security!*[System[(band(Keywords,13510798882111488))]]">
        <Column name="PublisherId" type="mt:wstr" defaultAssignment="00000000-0000-0000-0000-000000000000">
          <Value>/Event/System/Provider/@Guid</Value>
        </Column>
        <Column name="TimeCreated" type="mt:wstr" >
          <Value>/Event/System/TimeCreated/@SystemTime</Value>
        </Column>
        <Column name="PublisherName" type="mt:wstr" defaultAssignment="">
          <Value>/Event/System/Provider/@Name</Value>
        </Column>
        <Column name="Channel" type="mt:wstr" defaultAssignment="">
          <Value>/Event/System/Channel</Value>
        </Column>
        <Column name="LoggingComputer" type="mt:wstr" defaultAssignment = "" >
          <Value>/Event/System/Computer</Value>
        </Column>
        <Column name="EventNumber" type="mt:wstr" defaultAssignment = "0" >
          <Value>/Event/System/EventID</Value>
        </Column>
        <Column name="EventCategory" type="mt:wstr" defaultAssignment = "1" >
          <Value>/Event/System/Task</Value>
        </Column>
        <Column name="EventLevel" type="mt:wstr" defaultAssignment="4">
          <Value>/Event/System/Level</Value>
        </Column>
        <Column name="UserName" type="mt:wstr" defaultAssignment = "N/A" >
          <Value>/Event/System/Security/@UserID</Value >
        </Column>
        <Column name="RawXml" type="mt:wstr" defaultAssignment="">
          <Value>/</Value>
        </Column>
        <Column name="EventDescription" type="mt:wstr" defaultAssignment="">
          <Value>GetEventMetadata("Description")</Value>
        </Column>
        <Column name="RenderingInfo" type="mt:wstr" defaultAssignment="">
          <Value>/Event/RenderingInfo</Value>
        </Column>
        <Column name="EventRecordId" type="mt:wstr" defaultAssignment="">
          <Value>/Event/System/EventRecordID</Value>
        </Column>
      </Subscription>

    </WindowsEventLogSubscriptions>
  </Events>
  <EventStreamingAnnotations>
    <EventStreamingAnnotation name ="^c13285865487085541030_546419284751770569$">
      <OMS>
        <Content>
          <![CDATA[ <Config channelProtocol="ods" workspaceName="<workspacename>" workspaceId="<workspaceid>" solutionType="LogManagement" logType="GENERIC_EVENT_BLOB" globallyUniqueChannelId="dcr-a6ea61938b314f409447bd8577819198.ods-<workspaceid>" endpoint="<workspaceid>.ods.opinsights.azure.com" tokenBased="true" /> ]]>
        </Content>
      </OMS>
    </EventStreamingAnnotation>
    <EventStreamingAnnotation name ="^c13285865487085541030_hb$">
      <OMS>
        <Content>
          <![CDATA[ <Config channelProtocol="ods" workspaceName="<workspacename>" workspaceId="<workspaceid>" solutionType="LogManagement" logType="HEALTH_ASSESSMENT_BLOB" globallyUniqueChannelId="dcr-a6ea61938b314f409447bd8577819198.ods-<workspaceid>" endpoint="<workspaceid>.ods.opinsights.azure.com" tokenBased="true" /> ]]>
        </Content>
      </OMS>
    </EventStreamingAnnotation>

  </EventStreamingAnnotations>
</MonitoringManagement>

As you can see in the highlighted text, this is the DCR we just configured to collect events in the Security event log. And within a few minutes after associating the DCR I also got data in my Log Analytics Workspace:

You can also see that the agent by default sends a Heartbeat to the workspace:

That’s it for now, I hope the information was useful. The next post will show how you can use XPath queries to select only specific events in an event log.

Leave a Reply