I would like to share some Exchange 2010 hybrid migration facts with you that we figured out.
First, again many thanks to Michael Van Horenbeeck! He helped me discuss this with a customer. I’m always very happy to work with him. And many thanks to Ben Winzenz and Jeff Kizner as well, I’m very grateful for your help.
In short: a customer is trying to keep about 65k mailboxes in sync to ensure a short cutover time. We are using a maximum of 1,500 mailboxes per batch, 5 batches per week, and switching 7,500 mailboxes with an overall data of about 5TB per week. For some technical details, we are using Azure ER (800 Mbit) for migration with 4 TMG as a proxy and some kind of F5 load balancing in between, PAW is activated, and two migration endpoints with each 100 sync/complete in parallel. We did some networking measuring and move request statistics and we had an average migration velocity of 18.6GB/h for batches starting the first incremental sync (0% to 95%) which is great. Of course, the migration velocity depends on the number of batches, mailboxes, mailbox items, network workload, etc.
We suspected if we keep a large number of mailboxes in sync might affect throttling or the overall migration velocity.
Another important part is how long MRS will sustain the synced mailboxes. The documentation https://support.office.com/en-us/article/Manage-migration-batches-in-Office-365-d164b35c-f624-4f83-ac58-b7cae96ab331 figures if a migration batch has no administrator interaction (nobody touching it) for 90 days, it will be automatically stopped. 30 days later, again if no administrator interaction, it would then be deleted. We have had some batches longer in a synced state for testing purposes, but we don’t want to fall into this limit in our production “migration” environment.
However, our problem was the delay in between when the request gets marked as complete, and when the batch status on the portal or Get-MigrationBatch gets updated and shows complete. We are using the status of the migration batch because there is some automated logic behind the migration for different pre and post hybrid migration tasks.
An example: a batch was started (CompleteAfter) 01/29 00:01 AM and showed completed in the portal and with Get-MigrationBatch at 01/29 13:00 PM. Nearly at the same time the email report from Exchange Online was sent as well.
I did some research with Ben’s help with the following cmdlets:
$moves = Get-MoveRequest -batchname “MigrationService:20180130T000100_MB_X-X” | Get-MoveRequestStatistics
$moves | select identity, completeafter, finalsynctimestamp, completiontimestamp
Interesting to see that the CompletionTimestamp flag for every move request (1,863 mailboxes in the batch) was finished only about 2 hours later.
Identity CompleteAfter FinalSyncTimestamp CompletionTimestamp
——– ————- —————— ——————-
XXXXX 1/29/2018 12:17:19 AM 1/29/2018 2:20:15 AM 1/29/2018 2:22:08 AM
XXXXX 1/29/2018 12:17:19 AM 1/29/2018 2:20:35 AM 1/29/2018 2:23:23 AM
XXXXX 1/29/2018 12:17:19 AM 1/29/2018 2:20:41 AM 1/29/2018 2:22:40 AM
XXXXX 1/29/2018 12:17:19 AM 1/29/2018 2:20:51 AM 1/29/2018 2:23:03 AM
XXXXX 1/29/2018 12:17:19 AM 1/29/2018 3:13:16 AM 1/29/2018 3:14:24 AM
Another approach from Ben was to check the activity on the migration endpoints:
Get-MigrationEndpoint | fl *active*
ActiveMigrationCount : 0
ActiveIncrementalSyncCount : 0
ActiveMigrationCount : 0
ActiveIncrementalSyncCount : 0
20180212T000100_MB_X-X Syncing ExchangeRemoteMove 96
20180221T000100_MB_X-X Synced ExchangeRemoteMove 98
20181231T000100_MB_X-X Syncing ExchangeRemoteMove 1322
20180215T210000_MB_X-X Synced ExchangeRemoteMove 5
20180213T210000_MB_X-X Synced ExchangeRemoteMove 5
20180214T210000_MB_X-X Syncing ExchangeRemoteMove 14
20180221T000100_MB_X-X Synced ExchangeRemoteMove 101
20180207T114420_MB_X-X Completed ExchangeRemoteMove 1
20180221T000100_MB_X-X Syncing ExchangeRemoteMove 98
20180221T000100_MB_X-X Syncing ExchangeRemoteMove 95
20180221T000100_MB_X-X Syncing ExchangeRemoteMove 94
20180208T093847_MB_X-X Syncing ExchangeRemoteMove 1
But it seems this is not accurate enough.
If I remember correctly, FastTrack is using move requests instead of batch migrations (or migration service) as well. In my customers case we have to use the migration batch and therefore the migration service as our migration plan. But this is only based on the organizational migration strategy from the customer, which means add, delete, change switch date and time, etc. must be done a lot for some batches. And every batch contains a single business unit so that all of them will be switched at the same date and time. Other features like health checking, reports, etc. are also very important for us.
At least Jeff gave us some interesting information: While MRS is actively processing moves and updating its status in real time with moving the data, migration service processes its queue serially. So as long as it has batches and migration users in its queue, it has to work its way through them all. Spending a slice of time on each one. Following on that, the status of the migration batch itself will only change once all of the migration users in the batch have been updated. Though its improved, managing status on large scale migrations using get-migrationbatch and get-migrationuser is generally a slow process.
I have some stats for you as well:
Name : MB_X-X
EndTime : 01.02.2018 12:53:50
MigrationDuration : 12 day(s) 11:27:39
MailboxCount : 287
TotalGBTransferred : 129,73
PercentComplete : 95,00
MaxPerMoveTransferRateGBPerHour : 2,41
MinPerMoveTransferRateGBPerHour : 0,00
AvgPerMoveTransferRateGBPerHour : 0,47
MoveEfficiencyPercent : 74,83
AverageSourceLatency : 4.056,92
IdleDuration : 0,00%
SourceSideDuration : 31,34%
DestinationSideDuration : 13,02%
WordBreakingDuration : 6,09%
TransientFailureDurations : 16,01%
OverallStallDurations : 26,70%
ContentIndexingStalls : 0,00%
HighAvailabilityStalls : 0,00%
TargetCPUStalls : 26,70%
SourceCPUStalls : 0,00%
MailboxLockedStall : 0,00%
ProxyUnknownStall : 0,00%
Thanks to everyone who helped me with this scenario, highly appreciated!