68578: Acronis Cyber Protect: Application-aware backup of machine with Exchange fails with "Windows error: (0x80131501) Exception has been thrown by the target of an invocation"

    Last update: 29-04-2021

    Symptoms

    Application-aware backup of a machine with Exchange fails with:

    Windows error: (0x80131501) Exception has been thrown by the target of an invocation.

    An application-aware backup of a machine with Exchange used to work but started failing with this error message, after an unclean shutdown (e.g. due to power loss or hardware-caused reboot, or reboot of underlying hypervisor if the Exchange server is running inside a VM).

    Additional symptoms

    In such situations, Exchange typically keeps throwing quite a few exceptions in its frontend/OWA/IIS parts, and also appears to still be "catching up", and there are events showing its throwing exceptions when the backup agent tries to use the usual PowerShell cmdlets to query for applications and for the details of the Exchange DBs/mailboxes during the app discovery stage and metadata gathering at the start of backup.

    Most relevant error events examples:

    ======
    The description for Event ID 4999 from source MSExchange Common cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

    If the event originated on another computer, the display information had to be saved with the event.

    The following information was included with the event:

    9816
    E12
    c-RTL-AMD64
    15.01.2176.009
    w3wp#MSExchangePowerShellAppPool
    M.E.C.ComponentConfig.Transport
    M.E.T.V._ParallaxSchemaProvider_SchemaProvider_.Register_Microsoft_Exchange_Transport_VariantConfig_JetDatabase
    S.Reflection.TargetInvocationException
    e3ae-dumptidset
    15.01.2176.009
    True
    False
    w3wp#MSExchangePowerShellAppPool

    4/21/2021 2:48:19 AM
    ======

    That's the PowerShell query failing.

    The corresponding backup stages from the agent have these in their mms log:

     

    Additionally, at times there can be warnings about low memory/resource depletion, such as those below, mostly happening in the minutes/hours after the sudden reboot, when the Exchange server is actively trying to heal itself. See the example below (details are likely to look different on different machines with different memory sizes and sizes of the Exchange DB and data):

    ====
    Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: Microsoft.Exchange.Store.Worker.exe (764) consumed 1283129344 bytes, noderunner.exe (5068) consumed 1235542016 bytes, and w3wp.exe (6788) consumed 719433728 bytes.
    4/20/2021 1:31:07 PM
    Resource Exhaustion Diagnosis Events
    ===
    Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: Microsoft.Exchange.Store.Worker.exe (764) consumed 1283035136 bytes, noderunner.exe (5068) consumed 1231429632 bytes, and w3wp.exe (6788) consumed 752402432 bytes.
    4/20/2021 1:25:59 PM

    ====

    Cause

    Exchange does not always cleanly recover after unclean shutdowns. Important processes on the Exchange side which are ultimately responsible for handling the application discovery calls made by the backup agent may get stuck, or fail to start, or process pools may get exhausted in Exchange, making it unresponsive to requests coming via PowerShell from the backup agent.

    Solution

    It is often sufficient to reboot the (virtual) machine hosting the Exchange server once more and let it finish its self-healing processes (might take tens of minutes or one or more hours). Application-aware backups should run fine after that.

    If the same errors keep occurring, the Exchange server needs to be fixed before the backups can continue. Contact your Exchange administrator and/or Microsoft Exchange Support for available troubleshooting if necessary.

    Tags: