Have PowerShell trigger an action when CPU or memory usage reaches certain values

From Svendsen Tech PowerShell Wiki
Jump to: navigation, search

This is an example of real world problem solving - or rather problem patching - or working around problems (hopefully temporarily in many cases) - or possibly a bunch of scenarios I haven't considered. Possibly many "permanent-use scenarios" as well.

A client approached me with a need for a temporary workaround for an issue they had where CPU usage would suddenly spike - and the workaround was (possibly at 3 AM for someone on call) to run "iisreset", as dirty as that may seem. Users were then happy again for a while.

While working on isolating the root cause, they wanted a script to automatically run iisreset when CPU usage reached 70 % or higher based on a set of sample intervals over the last five minutes. I spent some hours writing this and later decided to write a more generic version to share with the world, since my life is empty and such. :P

The sample code below will allow you to perform an action, such as run "iisreset", run any cmd/batch script, executable file, any PowerShell code, and really do anything a computer running the script/service is capable of doing.

In my sample code I just use "cmd /c echo 1" and check the exit code. To run PowerShell code, just replace it, and you probably want a try/catch statement to catch errors and warn you about them, like I do myself in the sample code, with Send-MailMessage. You can add other Send-MailMessage statements to send errors on the various conditions, such as when the condition/code/script is triggered. Or you could Splunk / log (beyond the built-in logging) / whatever.

This is meant to be adapted to suit your needs.

It should work with PowerShell version 2 and up (default on Server 2008 R2 / Windows 7). Tested on 2012 R2 and Windows 10.



Example run manually in the console

Powershell-cpu-memory-usage-measurement-trigger-service-example.png

As text:

PS C:\temp> .\testservice.ps1
VERBOSE: [2018-03-17 02:39:31] Starting PowerShell Service.
VERBOSE: [2018-03-17 02:39:38] Only 1 samples so far. Need 10. Collect more values.
VERBOSE: [2018-03-17 02:39:45] Only 2 samples so far. Need 10. Collect more values.
VERBOSE: [2018-03-17 02:39:52] Only 3 samples so far. Need 10. Collect more values.
VERBOSE: [2018-03-17 02:39:59] Only 4 samples so far. Need 10. Collect more values.
VERBOSE: [2018-03-17 02:40:06] Only 5 samples so far. Need 10. Collect more values.
VERBOSE: [2018-03-17 02:40:13] Only 6 samples so far. Need 10. Collect more values.
VERBOSE: [2018-03-17 02:40:20] Only 7 samples so far. Need 10. Collect more values.
VERBOSE: [2018-03-17 02:40:27] Only 8 samples so far. Need 10. Collect more values.
VERBOSE: [2018-03-17 02:40:34] Only 9 samples so far. Need 10. Collect more values.
VERBOSE: [2018-03-17 02:40:41] CPU usage alert triggered! CPU usage for the samples was
 6.69877037857689 % (trigger value: 3). Running command and emptying counter array.
VERBOSE: [2018-03-17 02:40:43] Successfully ran cmd.
VERBOSE: [2018-03-17 02:40:43] Memory usage alert triggered! Memory usage for the sampl
es was 8.2791 GB (trigger value: 10000). Running command and emptying counter array.
VERBOSE: [2018-03-17 02:40:44] Successfully ran cmd.
VERBOSE: [2018-03-17 02:40:44] Average CPU usage percent for 10 samples was 6.70 %.
VERBOSE: [2018-03-17 02:40:44] Average free memory for 10 samples was 8,477.81 MB.
VERBOSE: [2018-03-17 02:40:55] Only 1 samples so far. Need 10. Collect more values.
VERBOSE: [2018-03-17 02:41:02] Only 2 samples so far. Need 10. Collect more values.
VERBOSE: [2018-03-17 02:41:09] Only 3 samples so far. Need 10. Collect more values.
VERBOSE: [2018-03-17 02:41:16] Only 4 samples so far. Need 10. Collect more values.
VERBOSE: [2018-03-17 02:41:23] Only 5 samples so far. Need 10. Collect more values.
VERBOSE: [2018-03-17 02:41:30] Only 6 samples so far. Need 10. Collect more values.
VERBOSE: [2018-03-17 02:41:37] Only 7 samples so far. Need 10. Collect more values.
VERBOSE: [2018-03-17 02:41:44] Only 8 samples so far. Need 10. Collect more values.
VERBOSE: [2018-03-17 02:41:51] Only 9 samples so far. Need 10. Collect more values.
VERBOSE: [2018-03-17 02:41:58] CPU usage alert triggered! CPU usage for the samples was
 5.61043619484128 % (trigger value: 3). Running command and emptying counter array.
VERBOSE: [2018-03-17 02:41:59] Successfully ran cmd.
VERBOSE: [2018-03-17 02:41:59] Memory usage alert triggered! Memory usage for the sampl
es was 8.2763 GB (trigger value: 10000). Running command and emptying counter array.
VERBOSE: [2018-03-17 02:42:00] Successfully ran cmd.
VERBOSE: [2018-03-17 02:42:00] Average CPU usage percent for 10 samples was 5.61 %.
VERBOSE: [2018-03-17 02:42:00] Average free memory for 10 samples was 8,474.98 MB.
VERBOSE: [2018-03-17 02:42:11] Only 1 samples so far. Need 10. Collect more values.
VERBOSE: [2018-03-17 02:42:18] Only 2 samples so far. Need 10. Collect more values.

Installing it as a service

To run it, you could do it manually or set it up as a scheduled task that triggers on server startup (only trigger/launch it once, it runs indefinitely), but I will also demonstrate how to set it up as a service with nssm.exe (non-sucking service manager). It'll run under the SYSTEM account by default. To stop it, simply stop the service (Stop-Service -Name YourServiceName).

First I change this line in the sample code script, so I get a new log I can check to see if anything happens and gets logged.

[String] $LogFile = "C:\temp\TestingThePowerShellService.txt"

This does not currently exist.

Then I take the command from the commented-out code at the top and adapt it a bit. This is on Windows 10 with the latest Win 10 Creators Edition build of nssm.exe as per 2018-03-17.

This screenshot demonstrates the full process and verification that it works (as text below).

Powershell-as-a-service-cpu-memory-trigger-nssm.png

The same as text:

PS C:\temp> $PowerShellServicePath = "c:\temp\testservice.ps1"

PS C:\temp> $ServiceName = "PowerShellTestService"

PS C:\temp> Start-Process -FilePath "C:\temp\nssm_x64.exe" -NoNewWindow -Wait `
>>     -ArgumentList "install $ServiceName ""C:\Windows\system32\WindowsPowerShell\v1.0\powershell.exe"" ""-NoProfile""
""-ExecutionPolicy Bypass"" ""-NonInteractive"" ""-File $PowerShellServicePath"" "
Service "PowerShellTestService" installed successfully!

PS C:\temp> Get-Service PowerShellTestService | Start-Service

PS C:\temp> get-date

Saturday, March 17, 2018 3:11:16 AM


PS C:\temp> Get-Content .\TestingThePowerShellService.txt
[2018-03-17 03:11:08] Starting PowerShell Service.

PS C:\temp> # wait a while ...

PS C:\temp> Get-Content .\TestingThePowerShellService.txt
[2018-03-17 03:11:08] Starting PowerShell Service.
[2018-03-17 03:12:19] CPU usage alert triggered! CPU usage for the samples was 7.20894380917347 % (trigger value: 3). Running command and emptying counter array.
[2018-03-17 03:12:20] Successfully ran cmd.
[2018-03-17 03:12:20] Memory usage alert triggered! Memory usage for the samples was 7.9962 GB (trigger value: 10000). Running command and emptying counter array.
[2018-03-17 03:12:21] Successfully ran cmd.

PS C:\temp> Get-Date

Saturday, March 17, 2018 3:12:25 AM

PS C:\temp>

PS C:\temp> Get-Service PowerShellTestService | Stop-Service

PS C:\temp> .\nssm_x64.exe remove PowerShellTestService confirm
Service "PowerShellTestService" removed successfully!

PS C:\temp> Get-Service PowerShellTestService
Get-Service : Cannot find any service with service name 'PowerShellTestService'.
At line:1 char:1
+ Get-Service PowerShellTestService
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (PowerShellTestService:String) [Get-Ser
   vice], ServiceCommandException
    + FullyQualifiedErrorId : NoServiceFoundForGivenName,Microsoft.PowerShell.Command
   s.GetServiceCommand

PS C:\temp>

Customization with pre-defined variables

These variables near the top of the script allow you to change some "parameters" easily (you don't want parameters for a script of this type normally, unless you build some supadupa framework of a weird, esoteric nature).

Change them as necessary and experiment to see how long the overhead is, this is good to do in the console with the frequent, verbose-timestamped messages before setting it up as a service, if that's what you choose to do.

# Set according to needs... adapt further, etc.
# Set one to false to not measure that.
# It makes no sense to set both to false, then you have a very strange sleep NOOP
# script...
[Bool] $MeasureCpu = $True
[Bool] $MeasureMemory = $True

[String] $SmtpServer = "smtp.internal.example.com"
[String] $LogFile = "C:\temp\TestingThePowerShellService.txt"
[Decimal] $CpuTriggerValue = 80.0 # in percent
[Decimal] $MinimumFreeMemoryTriggerValueMB = 900

# The same intervals are used for both CPU and memory measurements, rename the
# script and run two in parallel for different intervals... or rewrite yourself
[Int32] $SleepSeconds = 20
[Int32] $SampleCount = 15
[Int32] $CatchCount = 0
[Int32] $HistoryMinutes = 5 # [Math]::Floor($SleepSeconds * $SampleCount / 60) + 3

Code

Source code:

#requires -version 2
[CmdletBinding()]
Param()

# MIT license.
# Copyright (c) 2018. Svendsen Tech. Joakim Borger Svendsen, 
# 2018, March.
# Compatible with PowerShell version 2.

<#
To install as a service with nssm.exe (non-sucking service manager...) running as the SYSTEM account,
adapt the command below (paths, other).

$PowerShellServicePath = "c:\temp\testservice.ps1"
$ServiceName = "PowerShellTestService"
Start-Process -FilePath "C:\temp\nssm_x64.exe" -NoNewWindow -Wait `
    -ArgumentList "install $ServiceName ""C:\Windows\system32\WindowsPowerShell\v1.0\powershell.exe"" ""-NoProfile"" ""-ExecutionPolicy Bypass"" ""-NonInteractive"" ""-File $PowerShellServicePath"" "

#>

$Host.CurrentCulture.NumberFormat.NumberDecimalSeparator = '.'

# Set according to needs... adapt further, etc.
[Bool] $MeasureCpu = $True
[Bool] $MeasureMemory = $True

[String] $SmtpServer = "smtp.internal.example.com"
[String] $LogFile = "C:\temp\TestingThePowerShellService.txt"
[Decimal] $CpuTriggerValue = 80.0 # in percent
[Decimal] $MinimumFreeMemoryTriggerValueMB = 900

# The same intervals are used for both CPU and memory measurements, rename the
# script and run two in parallel for different intervals... or rewrite yourself
[Int32] $SleepSeconds = 20
[Int32] $SampleCount = 15
[Int32] $CatchCount = 0
[Int32] $HistoryMinutes = 5 # [Math]::Floor($SleepSeconds * $SampleCount / 60) + 3

function Write-Log {
    param(
        [string] $Message,
        [switch] $Verbose = $true)
    "[$([DateTime]::Now.ToString('yyyy\-MM\-dd HH\:mm\:ss'))] $Message" | Add-Content -LiteralPath $LogFile -Encoding UTF8
    Write-Verbose -Message "[$([DateTime]::Now.ToString('yyyy\-MM\-dd HH\:mm\:ss'))] $Message" -Verbose:$Verbose
}

$CpuData = @()
$MemoryData = @()
Write-Log -Message "Starting PowerShell Service."

# Simple service...
while ($True) {
    
    $DoBreak = $False
    while ($True) {
        if ($MeasureCpu) {
            try {
                $CpuUsageData = Get-Counter -Counter '\Processor(_Total)\% Processor Time' -ErrorAction Stop
                $DoBreak = $True
            }
            catch {
                $DoBreak = $False
                Write-Log -Message "Get-Counter failed to retrieve a cooked CPU usage value. Retrying in 5 seconds"
            }
        }
        if ($MeasureMemory) {
            try {
                $MemoryUsageData = Get-Counter -Counter '\Memory\Available Bytes' -ErrorAction Stop
                $DoBreak = $True
            }
            catch {
                $DoBreak = $False
                Write-Log -Message "Get-Counter failed to retrieve a cooked memory usage value. Retrying in 5 seconds"
            }
        }
        Start-Sleep -Seconds 5
        if ($DoBreak) {
            break
        }
        <#else {
            Send-MailMessage ... or increase a counter and then mail, or whatever
        }#>
    }
    # Now we have a cooked value or two that we will add to an array or two that contain(s) values for the last $HistoryMinutes
    # in custom PSObjects with a Timestamp from the counter data itself.
    if ($MeasureCpu) {
        $CpuData += New-Object -TypeName PSObject -Property @{
            DateTime = $CpuUsageData.Timestamp
            CookedValue = $CpuUsageData.CounterSamples.CookedValue
        }
    }
    if ($MeasureMemory) {
        $MemoryData += New-Object -TypeName PSObject -Property @{
            DateTime = $MemoryUsageData.Timestamp
            CookedValue = $MemoryUsageData.CounterSamples.CookedValue
        }
    }

    # Filter out elements older than $HistoryMinutes.
    $CpuData = @($CpuData | Where-Object {
        $_.DateTime -gt [DateTime]::Now.AddMinutes(-1 * $HistoryMinutes)
    })
    $MemoryData = @($MemoryData | Where-Object {
        $_.DateTime -gt [DateTime]::Now.AddMinutes(-1 * $HistoryMinutes)
    })

    # We should have at least $SampleCount samples before we act... 5 minutes in sample version.
    if ($MeasureCpu) {
        if (($Count = $CpuData.Count) -lt $SampleCount) {
            Write-Verbose -Verbose -Message "[$([DateTime]::Now.ToString('yyyy\-MM\-dd HH\:mm\:ss'))] Only $Count samples so far. Need $SampleCount. Collect more values."
            continue
        }
    }
    if ($MeasureMemory) {
        if (($MemCount = $MemoryData.Count) -lt $SampleCount) {
            Write-Verbose -Verbose -Message "[$([DateTime]::Now.ToString('yyyy\-MM\-dd HH\:mm\:ss'))] Only $Count samples so far. Need $SampleCount. Collect more values."
            continue
        }
    }
    # We have $SampleCount or more records. Process these and run command
    # (iisreset in this example) if necessary.
    # Calculate average and see if it's higher than $CpuTriggerValue or $FreeMemoryTriggerValue, if
    # it is, run iisreset and reset the $CpuData array, so we start over.
    $AverageCpuUsage = $CpuData |
        Measure-Object -Property CookedValue -Average |
        Select-Object -ExpandProperty Average
    $AverageMemoryUsage = $MemoryData |
        Measure-Object -Property CookedValue -Average |
        Select-Object -ExpandProperty Average
    if ($MeasureCpu -and $AverageCpuUsage -ge $CpuTriggerValue) {
        Write-Log "CPU usage alert triggered! CPU usage for the samples was $AverageCpuUsage % (trigger value: $CpuTriggerValue). Running command and emptying counter array."
        $ErrorActionPreference = "Stop"
        while ($True) {
            $ProcessResult = Start-Process -FilePath cmd -NoNewWindow -Wait -ErrorAction Stop -PassThru -ArgumentList '/c', 'echo 1'
            if ($ProcessResult.ExitCode -eq 0) {
                Write-Log -Message "Successfully ran cmd."
                
                # You can add a Send-MailMessage here if you want. Send-MailMessage -To whatever@whatever.org -From ...
                
                $CpuData = @()
                break
            }
            else {
                # Avoid insane amounts of spam in case it hangs...
                $CatchCount++
                if ($CatchCount -gt 1) {
                    if ($CatchCount -gt 99) {
                        $CatchCount = 0
                    }
                }
                else {
                    Send-MailMessage -SmtpServer $SmtpServer -Cc 'joakim@example.com' -From 'PS_Service@example.com' `
                        -To 'DistributionList@example.com' -Subject "Failed to iisreset on $Env:ComputerName! $(Get-Date)" `
                        -Body @"
PowerShell Service failed to run command on $Env:ComputerName!
$(Get-Date). Will try again in about $SleepSeconds seconds. After 100 repeated failures, you will get a new mail..."
"@
                }
            }
        }
        $ErrorActionPreference = "Continue"
    }
   if ($MeasureMemory -and ($AverageMemoryUsage/1MB) -le $MinimumFreeMemoryTriggerValueMB) {
        Write-Log "Memory usage alert triggered! Memory usage for the samples was $('{0:N4}' -f ($AverageMemoryUsage / 1MB)) MB (trigger value: $MinimumFreeMemoryTriggerValueMB MB). Running command and emptying counter array."
        $ErrorActionPreference = "Stop"
        while ($True) {
            $ProcessResult = Start-Process -FilePath cmd -NoNewWindow -Wait -ErrorAction Stop -PassThru -ArgumentList '/c', 'echo 1'
            if ($ProcessResult.ExitCode -eq 0) {
                Write-Log -Message "Successfully ran cmd."
                
                # You can add a Send-MailMessage here if you want. Send-MailMessage -To whatever@whatever.org -From ...
                
                $MemoryData = @()
                break
            }
            else {
                # Avoid insane amounts of spam in case it hangs...
                $CatchCount++
                if ($CatchCount -gt 1) {
                    if ($CatchCount -gt 99) {
                        $CatchCount = 0
                    }
                }
                else {
                    Send-MailMessage -SmtpServer $SmtpServer -Cc 'joakim@example.com' -From 'PS_Service@example.com' `
                        -To 'DistributionList@example.com' -Subject "PowerShell service failed to iisreset on $Env:ComputerName! $(Get-Date)" `
                        -Body @"
PowerShell Service failed to run command on $Env:ComputerName!
$(Get-Date). Will try again in about $SleepSeconds seconds. After 100 repeated failures, you will get a new mail..."
"@
                }
            }
        }
        $ErrorActionPreference = "Continue"
    }

    Write-Verbose -Verbose -Message "[$([DateTime]::Now.ToString('yyyy\-MM\-dd HH\:mm\:ss'))] Average CPU usage percent for $Count samples was $('{0:N2}' -f $AverageCpuUsage) %."
    Write-Verbose -Verbose -Message "[$([DateTime]::Now.ToString('yyyy\-MM\-dd HH\:mm\:ss'))] Average free memory for $Count samples was $('{0:N2}' -f ($AverageMemoryUsage/1MB)) MB."
    Start-Sleep -Seconds $SleepSeconds

    # Avoid memory leaks, especially on PS versions before 4 or 5 (not sure which version (apparently) got better at it).
    [System.GC]::Collect()

}