Archive for Troubleshooting

Finding messages stuck in outboxes

Sometimes you encounter problems with the transport queues or with mailflow that are difficult to track down. Maybe your mail.queue file is ballooning in size even though you’ve replaced it with a fresh one. Maybe outbound mail isn’t going out as quickly as it used to.

One diagnostic in this situation is to determine whether any of the mailboxes have messages sitting in the Outbox. Sometimes large messages in the outbox can create loops if they are too large but you’ve removed all max-send-size limits. Sometimes the message in the Outbox is corrupted and is continually resubmitted to queuing.

The following script will look at all the mailboxes for messages in the Outbox and will compile a report of the mailboxes with items in their outboxes. Once you have the report, you can look at certain mailboxes with OWA and delete any problem items you find.

Get-Mailbox -ResultSize Unlimited | Get-MailboxFolderStatistics | Where-Object {$_.Name -eq "Outbox" -and $_.ItemsInFolder -gt '0' } | Select-Object Identity, FolderType, ItemsinFolder, FolderSize | Export-CSV "C:\Outbox.csv"

Bulk Public Folder Rename and Cleanup Script

While doing a migration to Exchange 2010 a few years ago I ran into a situation where lots of the public folders would not replicate from 2007 due to the inclusion of characters in the name that exchange doesn’t like, but outlook allows, including (and most difficult to troubleshoot) a trailing space in the name.

unfortunately this client had over 15,000 individual public folders they had created over years of time with data they couldn’t loose, with folder names including shipment dates (and the “/” that went along with it) and many other things. Read more

Exchange 2013 Messsage Tracking “Console”

Exchange 2013 lacks the traditional Exchange Management Console that you are probably familiar with, and with it went the message tracking console that many administrators relied on to troubleshoot mail flow problems.  Sure there is now the Delivery Reports tool in the web Exchange Control Panel, but it’s just not quite the same, especially since it limits you to searching within a specific mailbox.

ECP Mail Flow

Luckily there is something similar to the Message Tracking Center in Powershell. Read more

Restore Mailbox after Mailbox Clean has removed Disconnected Mailbox

We have recently had an issue where we needed to bring back a mailbox and export it to .PST after the Exchange mailbox cleanup agent deleted the disconnected mailbox.

First we had to bring the files back from tape.  We used CommVault, and did a Restore to Non-Exchange Location <Out of Place, No Recover> – The database will be restored to the specified location and the logs will not be replayed.

We now have the files restored on a different drive than the original mailbox database was on,  all we need to do now is create a new Recover Database, and then point it to the files that were restored.

Once the files are there, you will need to bring them into a state that allows for you to mount them.  There are a couple of ways to do this.  Instead of giving the details here, you can go to this BLOG for details: http://www.mikepfeiffer.net/2010/04/getting-an-exchange-database-into-a-clean-shutdown-state-using-eseutil/

In this case, I was getting a JET error Operation terminated with error -1216 (JET_errAttachedDatabaseMismatch, An outst anding database attachment has been detected at the start or end of recovery, bu t database is missing or does not match attachment info) after 16.661 seconds.

Because I am only worried about creating a .PST file of a old mailbox, and do not wish to restore any other data, I did the Hard Recovery.  If I were restoring this database to be moved back into production, I would trouble shoot the error first.  I am only using the hard recovery option because this database is not going back into a production environment, and the loss of the log files is an acceptable risk.  NOTE: This process could take a long time.  The mailbox database I ran the command on was 103 GB in size, and it took 2395.255 seconds to run the command.

Once you can run the eseutil -mh on the database and have it return a Clean Shutdown state, then you are ready to create your recovery database, mount it, and then export the mailbox you are looking for.

To create the mailbox database, you need to use the new-mailboxdatabase command with the -recover switch.  Here is the command I used because the .edb file and log files were restored to s:\restore\ location.

new-mailboxdatabase -Recover -Name RDB1 – Server prexmb01 -EdbFilePath s:\restore\ex10-ms03.edb -LogFolderPath s:\restore\logfiles

You will also need to create a recovery mailbox, it cannot be in the recovery database.  I have a TempDB database that I use for things like this an legal discovery if needed.  I just created a recovery mailbox for that user.  It will be needed in the next step when you use the new-mailboxrestorerequest command.

You can find more details by going to Microsoft’s site.  I will use the commands I used in below.

First, you need to get the MailboxGuild.  I used the following command.

Get-MailboxStatistics -Database RDB1 | Where { $_.DisplayName -eq “USERNAME” } | Format-List LegacyDN, DisplayName, MailboxGUID, DisconnectReason

You need the following info from the display:

MailboxGuid      : f5b9e238-1095-4a96-be24-1ab89e939506

Then you can use the new-mailboxrestorerequest to get the email from the recovery database and import it into the temp database.

New-MailboxRestoreRequest -SourceDatabase “RDB1″ -SourceStoreMailbox f5b9e238-1095-4a96-be24-1ab89e939506 -TargetMailbox recovermailbox -AllowLegacyDNMismatch

Notice that I added the -AllowLegacyDNMismatch so that you don’t have to have the new mailbox you are restoring to have the same LegacyExchangeDN or X500 proxy address as the mailbox you are recovering.

You can check on the progress of the restore by using the Get-mailboxrestorerequest command.   It should go from a status of Queued, to InProgress, and then to Completed.

At this point, you can then use the new-exportmailboxrequest to create a .PST file.  You can find more details on this command in a previous blog post that I did.

To export the mailbox, you then use the new-mailboxexportrequest command.  I used the following command:

New-MailboxexportRequest -Mailbox USERNAME -FilePath “\\mkiefferex\exld$\USERNAME_10-30-2010_1505.pst” -MRSServer prexchu01

You can check on the status of the export request by using the get-mailboxexportrequest.  The status should change from Queued to InProgress then to Completed.

Once it has completed, you can then copy the .PST to a file share, CD or DVD to give to the user that requested the mailbox restore.  NOTE:  I LEARNED THIS THE HARD WAY.  If you try to copy the file and it says the file is still in use, DO NOT terminate or close the connection that is accessing the file.  If you go and look at open files on the computer the .PST file was copied to, you will see the file is open by the client access computers admin user.  If you close the connection, it WILL damage the .PST file and you will not be able to use it.

Clean-Up – Don’t leave stuff that is unneeded in your Exchange Org.

When you are done, make sure you do the following.

  1. Delete the Recovery Mailbox Database.
  2. Delete the files that were used for the restore and the Recovery Mailbox Database.
  3. Delete the user you recovered the email to.

Foward to DL fails

This one is not that big of a deal, and is a DUH on my part, but it may be something someone else runs into so I am going to post it.

Senario: User has a shared mailbox (SB1) that he wants to receieve email, and then foward it to his Personal mailbox (PM1) and an external user (EU1).

Solution: you need to setup a contact for EU1 that points to the users external email address. Setup a distribution group that contains the contact for EU1 as well and PM1. Then go into the mailbox settings and set it up so that the Delivery of email to SB1 is fowarded to the distribution group. That will make it so that if someone sends an email to SB1, it will go to SB1, PM1, and EU1.

Problem: But here is the problem. The forward worked correctly for internal emails, but did not work for external emails.   I had no idea that the little checkbox on the Distribution list that requires authentication works for forwards as well.  So if you want email from an outside address to foward correctly, you have to take off the check box for “Require that all senders are authenticated” on the distribution groups Message Delivery Restrictions settings.

To verify that this is the issue, you will want to check the message tracking logs and see if you can find one that shows the failure.  Then look for a “Recipient Status of 550 5.7.1 RESOLVER.REST.AuthRequired; authentication required”

To do this, I used the following command.

get-transportserver | get-messagetrackinglog -messagesubject "PUT SUBJECT HERE" -start 8/2/2012 -EventID Fail | FL | more

I always put a get-transportserver before any message tracking log commands.  We have several HUB servers, and this command will then search the traking logs for ALL of the HUB servers.

Note:  To make sure you don’t confuse the heck ot of your end users you will want to make the DL you created for the forward, as well as the Contact to the external user hidden from the address book.

I hope that is now clear as mud.  Comment if you have questions, or suggestions on how to do this.  I don’t like using rules in the actual mailbox because they are harder to maintain than the ability to use the EMC and powershell.

new-mailboxexportrequest fails – TimeoutErrorTransientException

Sometimes when you do a new-mailboxexportrequest you receive the following error:
The call to 'net.tcp://prexchu01/Microsoft.Exchange.MailboxReplicationService PREXCHU01.******** (14.2.247.1 caps:0 7)' timed out.

Error details: This request operation sent to net.tcp://prexchu01/Microsoft.Exchange.MailboxReplicationService
did not receive a reply within the configured timeout (00:01:00). The time allotted to this operation may have been a portion of a longer timeout. This may be because the service is still processing the operation or because the service was unable to send a reply message. Please consider increasing the operation timeout (by casting the channel/proxy to IContextChannel and setting the OperationTimeout property) and ensure that the service is able to connect to the client.

+ CategoryInfo : NotSpecified: (0:Int32) [New-MailboxExportRequest], TimeoutErrorTransientException

+ FullyQualifiedErrorId : C857038,Microsoft.Exchange.Management.RecipientTasks.NewMailboxExportRequest

I have found that it is because the CAS server is located behind a hardware load balancer. We are using a BIGIP load balancer for all our CAS/HUB functions. If you do a get-mailboxdatabase on the database of the mailbox you are trying to move, you will probably see that the RpcClientAccessServer is set to point to the VIP of the CAS pool instead of the actual CAS server.

I have found two ways to fix this, the first (not recommended) is to do a set-mailboxdatabase and set the RpcClientAccessServer to the UNC for the CAS server you are going to use for the export request. Make sure you change it back after you make the change.

The second way, and the way I recommend is to modify the HOSTS record on the CAS server so that the UNC found in the RpcClientAccessServer points to the IP address of that CAS server. This is a suggestion that Microsoft gives on there site (KB2675690).

Also, make sure that you have setup the file share you are creating the PST file on correctly by following instructions found on the new-mailboxexportrequest instructions.

The FilePath parameter specifies the network share path of the .pst file to which data is exported, for example, \\SERVER01\PST Files\exported.pst.

You need to grant read/write permission to the group Exchange Trusted Subsystem to the network share where you’ll export or import mailboxes. If you don’t grant this permission, you’ll receive an error message stating that Exchange is unable to establish a connection to the target mailbox.

Also, make sure you use the -MRSSever switch so that it is using the CAS server that you modified the Hosts file on.

Then make sure you remove or comment out the entry in the HOSTS file after the new-mailboxexportrequest finishes.

Script – Check Message Queue

note: following the transfer of this domain to the new owners, per user requests this article was recovered from the internet archive wayback machine, but may not be complete.

Here is a simple script I wrote that checks the message queue and then sends an email if the message queue goes over a specified limit. Read more