I cannot count how many times that I’ve been called in the middle of the night to troubleshoot a mail flow problem. Typically, the calls come when you are not thinking about Exchange, and not in the “work” mindset. As I mentioned in a previous blog post, each administrator should have a mental checklist (or paper) that will be used whenever issues arise. This is essential to be a well-prepared administrator and to help ensure a timely recovery of mail services.
In extremely regulated environments, the use of journaling, transport rules, and even incoming/outgoing disclaimers become critical due to legal disclosures and data preservation. With that being said, it can be very frustrating when a function that just always works – does not. So, if transport agents are not working in your environment, and they did earlier in the day, where do you begin troubleshooting?
When you configure an application to perform a certain function, and that action does not behave as expected, your patience can definitely be stretched. Thankfully, there is a logging method that can be used to examine what changes each transport agent makes to the message as it flows through the transport pipeline.
Pipeline tracing allows the administrator to configure Exchange to create message snapshot files for a user that displays what happens during each step of the message categorization process. Specifically, pipeline tracing will create a copy of the message prior to a transport agent making a change, and then a copy as each agent is triggered during an SMTP on event. A copy will be made even if a transport agent is triggered, lets say during OnSubmittedMessage, but does not actually modify the message. This monitoring can only be enabled for a specific user and should only be turned on while troubleshooting an issue.
When pipeline tracing is turned on, transport agents that run within the Transport and Mailbox Delivery Transport service will log what specific changes are made to a message, in separate message snapshot files. Each monitored message will be created in a new folder under the MessageSnapshots main folder. These folders will be named after the GUID of each tracked message. This will allow the administrator to examine the logs of each transport agent that was triggered during a specific SMTP on event.
In order to see what the transport pipeline looks like on your Exchange 2013 server, the Get-TransportPipeline cmdlet can be executed. This displays the list of SMTP on events and what transport agents will be triggered at each event.
You can use the Get-MailboxTransportService cmdlet to show the configuration for the Transport service on a Mailbox or Edge Transport server. Additionally, you can use the Get-MailboxTransportService cmdlet to see the Mailbox Transport service configuration for a Mailbox server.
By default, pipeline tracing is disabled on each Exchange 2013 server. We can verify this by using the Get-TransportService | *pipelineTracing* cmdlet.
We see that the PipeLineTracingEnabled value is set to false. In order to turn on pipeline tracing for all Exchange 2013 servers, we first have to set the PipelineTracingSenderAddress. This is the address that we are going to use in our troubleshooting efforts (i.e. the user that the transport agents are not running against). We use the Get-TransportService | Set-TransportService -PipelineTracingSenderAddress firstname.lastname@example.org cmdlet to globally set the value on all Exchange 2013 servers.
Once this value is set, you can specify the PipelineTracingPath if so desired using the
Get-TransportService | Set-TransportService -PipelineTracingPath "c:\Logs\Pipeline” cmdlet. For this example, I will leave the PipelineTracingPath set to the default location.
Now that we have set the PipelineTracingSenderAddress, and optionally the PipelineTracingPath, we can actually turn on pipeline tracing for all our Exchange 2013 servers. We accomplish this using the Get-TransportService | Set-TransportService -PipelineTracingEnabled $true cmdlet.
Now, I will send an email message to a recipient from email@example.com and then examine the PipelineTracingPath to see the logs. For this example, I looked on the Exchange 2013 Mailbox server that serves this user. The logs for this test email can be found under the C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\Hub\PipelineTracing\MessageSnapshots\4f6cef5f-8726-4570-8e17-5575ff46cf64 folder.
The Routing000x.eml files that are stored under the MessageSnapShots folders will correspond to the SMTP on events that occur on the server as a message passes through the transport pipeline.
Using the results from the Get-TransportPipeline cmdlet earlier, we can notice that the Transport Rule Agent is configured to fire on the OnResolvedMessage SMTP event. We can see this in the routing files as well.
Once you have examined the logs, and found the reason for your problem, the pipeline tracing should be turned off. This can be accomplished on all your Exchange 2013 servers by using the cmdlets below:
Get-TransportService | Set-TransportService –PipelineTracingEnabled $false
Get-TransportService | Set-TransportService –PipelineTracingSenderAddress
Hopefully, this article will get you thinking about your mental checklist and what options are available when the transport agents stop working!