This blog post covers a summary of the session from Paul Collinge and Jeff Mealiffe about a recommended network architecture to get the most out of Office 365.
The enterprise connectivity challenge is that most customers are using a lot of expensive network equipment for the outgoing and incoming network traffic to and from the Internet. For example, proxy servers, WAN accelerator, secure web gateway, intrusion prevention system, etc. All of this network and connectivity equipment is expected because all things outside is unknown and untrusted.
But this model doesn’t fit with the cloud world of Office 365 and causes various connectivity problems.
Issues with the traditional model for Office 365 traffic for Exchange Online explained in more detail:
The graph in the picture shows the average latency of 5ms to Exchange Online for a single user behind a proxy server. Multiplicate this value with your users will increase the latency to a much higher value in some enterprise organizations. You could optimize and buy some more proxy servers to handle the load, but this will not reduce the latency your users to Exchange Online.
Skype for Business & Microsoft Teams
Issues with the traditional model for Office 365 traffic Skype for Business & Microsoft Teams explained in more detail:Skype and Teams using the UDP protocol for media traffic and proxies often delay frames on the way through, adding jitter and latency.
Skype Network test – Direct vs standard TCP Proxy
SharePoint Online & OneDrive for Business
Issues with the traditional model for Office 365 traffic SharePoint Online & OneDrive for Business explained in more detail:Proxy latency to the egress may be high as well as for the other services. Main problem is that SharePoint Online and OneDrive for Business are using the same anycast IP address for all connections meaning NAT scalability is reduced if using port oversubscription.
Office 365 – Centralized network egress
These values are from the demo session at Ignite with centralized network egress for a customer with 80,000 users and 100 locations worldwide:
Office 365 Connectivity Principles
The following picture shows the Office 365 connectivity principles and recommendations for your enterprise organizations network:
New URL and IP categories and web services API
New, priority driven endpoint taxonomy: http://aka.ms/ipurlblog for easier customer network optimization.
Focus on the high impact endpoints first which are described in the “Optimize” section. Then follow the guidance for all other endpoints.
Office 365 Egress connections
Egress Office 365 data connections as close to the user as practical with matching DNS resolutions. This is very important how your users connect and has impact to the overall user experience and connection latency.
- The Microsoft Global Network is a fast, globally available network with 100k mils of fiber in 130+ locations
- 130+ global edge nodes reaching 63% of the Global GDP within 25ms
- Peering relationships with 2700+ ISPs in 190+ locations
- Connects 35+ Office 365 Datacenter locations
- Fully software defined and managed by Microsoft
Office 365 peering locations can be found at http://aka.ms/8075.
Application level Security for Optimize endpoints
With the new endpoints it is much better to asses the risk connection to Office 365 endpoints without traffic inspection. Microsoft is doing a great job for securing their environment and hopefully every organization will be able to egress locally and get the best result from their network and connectivity.
Modern Network Architecture Example
Branch Office – Traditional Approach:
SDWAN for direct Office 365 – Reduced/Removed MPLS load/costs:
SDWAN for Direct Office 365 through regional Egress:
SDWAN for egress through Secure Web Gateway:
General benefits of local egress:
- Data/Applications/Services are increasingly moving to the cloud and no longer live in the corporate network, it therefore doesn’t make sense to backhaul all traffic to a central egress
- Central security/egress stacks are very expensive to uplift for cloud services, may still not be optimal and may need continual uplift for future services
- Cloud security elements often replicate security delivered at the egress
- Allows an enterprise to me more agile to an increasingly fast paced word
- Consider the shift in software update distribution, often best delivered by local CDN
- Centralized management for all remote egress devices
- MPLS costs are often much higher than local internet connectivity
Office 365 – Local and Direct network egress
These values are from the demo session at Ignite with local and direct network egress for a customer with 80,000 users and 100 locations worldwide:
Skype for Business Online / Microsoft Teams Network Connectivity
SIP traffic is used to initiate the media sessions and the media traffic itself (voice, video, screensharing, etc.). The signaling traffic (SIP) is always going to connect to a pool of resources that are actually in the location of the Office 365 tenant. For example, if your tenant is located in North America, your users are always getting connected to North America, regardless of how your users are located in the world.Important to understand is that if you are having a 1-to-1 conversation or 1-to-1 media session and if you are on the same corporate network without any barriers in the way, that’s a peer-to-peer conversation from a media perspective. This means there is no performance improvements regarding the egress to apply here because it is all in your local network.However, if these two users are in two different networks then we have a different story.For example, one user is located in Germany, the other user in Spain. Both users are on completely different networks. In order to have a conversation between both users, the connection is going to the Media Relay where the tenant location is.
There is a new option in Microsoft Teams for 1-to1 calls for both users (as well as for a subset of users for Sykpe for Business Online):Instead of going to the location where the tenant was created (North America), the users are connection to a Transport Relay via Anycast routing closest to the user’s physical location. This means the Transport Relay listens to the same IP address around the world and based on the user’s location you will be directed to the most comfortable Transport Relay. You will get a much better end user experience as a result.
SharePoint Online / OneDrive for Business Network Connectivity
SharePoint now uses the same behavior as Microsoft Teams, namely connect to the same IP globally via Anycast routing.SharePoint Online & OneDrive for Business Connection Process:
A user somewhere in EMEA wants to connect to a SharePoint team site and the tenant is still located in North America.
- The first thing is that the client goes to DNS and resolves the tenantname.sharepoint.com address
- DNS answers back with the Anycast IP address for SharePoint / OneDrive for Business
- The client will do a connection via TCP 443 to that Anycast IP address and using Anycast routing. That will ensure to get the least costly path to get the client to one of the service front doors that are responsible for managing this traffic.(Step 2 is Anycast IP address, this is an issue in the slide)
The file performance will increase dramatically and Anycast is now rolled out globally and customers following the network principles will instantly see the benefit without any customer side change required:
Exchange Online Network Connectivity
Exchange Online has changed the connectivity model as well. The change here is moving from a system that was based on Geo DNS, locating your users where Microsoft believe your users be physically located, based on the source IP of the DNS request coming in to Microsoft. Is has also changed to an Anycast routing model to get the users connected to the front door as quickly as possible. The Geo DNS behavior does still exist as a backup and Microsoft may occasionally use that so make sure you are resolving DNS as close as possible to where your traffic egresses.
The Anycast model in Exchange Online works as follow:
- The first thing is that the client goes to DNS and resolves the outlook.office365.com address
- DNS answers back with a CNAME called Outlook.ms-acdc.office.com (acdc refers to “anycast dns on café”)
- DNS capacity (DNS services) running on café servers in Exchange Online across their datacenters. Because Microsoft can now know that the DNS requests that come in to the café servers are coming from a close physical location. Then the DNS on the café servers will provide a response that is appropriate for that location
- There are a few IP addresses in front of the café capacity servers (load balanced VIPs) for high availability
- The client picks the first of the list and opens a https connection (TCP 443) to café and gets proxied back to the mailbox server that’s responsible for their active database where the mailbox is hosted.
Optimizing Exchange inbound flows
Hybrid connectivity frequently involves cloud to on-premises flows, like for free/busy requests, cross-premises mailbox permissions, calendar sharing, etc. The Exchange general best practices guidance for inbound flows can be found here: http://aka.ms/JustALoadBalancer.Egress location should be close to target infrastructure to minimize latency. Also, minimize impact of security controls on inbound traffic to “enough to mitigate threats”. For example, it makes no sense that your MRS endpoint is in North America but you are migration users from Germany to Exchange Online.
This guidance for Exchange Hybrid also applies to SharePoint Hybrid.
Office 365 Multi-Geo
Multi-geo is currently available for Exchange Online and OneDrive, coming soon for SharePoint Online and Groups as well. This allows you to have a single tenant that is spread out to multiple regions. You can change the location of your workload data on a per-user basis to any of the available multi geos today. The data will be relocated on the backend for those users in an asynchronous way invisible for the users.It is really about data residency and not about performance. Local/regional egress model is critical for multi geo customers to avoid performance degradation. The performance may increase, but only if the connectivity guidance is followed.Learn more: http://aka.ms/Multi-Geo
Network Bandwidth Estimation for Office 365
The biggest question from customer is always: how much bandwidth do I need for Office 365? Well, this question can’t be answered in general. It really depends on the workloads being used and the figure is variable per customer profile. Using an average may lead to poor planning decisions because not every organization works equal the average, even with the same number of seats and workloads.
The traditional approach is to use calculators for each workload:
- Network and Migration Planning for Office 365
- Exchange Client Network Bandwidth calculator
- Skype for Business Bandwidth Calculator
- Teams Bandwidth Planner
- Technical Case Study from Microsoft IT
There is a beta solution on how to monitor network traffic based on Azure Service Map & Log Analytics:How does it work?
- SaaS is disrupting network and security. Re-envision how you connect and protect your enterprise. Expand your existing network and security boundary with the connect of the SaaS tenant.
- User experience is paramount for enterprise success with SaaS. Best user experience can be provided by embracing local and direct Internet escape. Don’t forget DNS name resolution path.
- SaaS will evolve and attempt to move closer to the user. Make it work for you by minimizing the (hidden) private network backhaul.
- Use Office 365 connectivity principles to keep network strategy for your company SaaS optimal and future proof.
- Send this presentation to your network and security team. It’s their decisions that hold the keys to a delightful Office 365 end user experience!