Hybrid Networking

Table of Contents

Site-to-Site VPN connection 1.1

Site-to-site VPN connection for HA

  • By default, when you deploy one Azure VPN gateway, there are two instances of the gateway configured in an active/standby configuration.
  • This standby instance provides some redundancy but is not quite highly available, as it might takes a few minutes for the second instance to come online and reconnect to the destination of the VPN.
  • For this lower level of redundancy, you can choose whether the VPN is regionally redundant or zone-redundant.
    • If you choose to use a Basic public IP address, then the VPN that you configure can only be regionally redundant. If you have a need for a zone-redundant configuration, then you must use a Standard public IP address with the VPN gateway.

While the above scenarios might provide some redundancy, they wouldn’t be considered highly available. There would be some amount of downtime in minutes for the Azure VPN gateways to fail over between the active and standby instances. There is also a single point of failure with the single on-premises gateway devices for that side of the connection.

Above figure enhances this configuration by adding two gateway devices for the on-premises network. However, the Azure side still only has redundancy instead of high availability.

Note: A VPN can be used to connect two VNets, a VNet to on-premises, a VNet to another public cloud, or a vendor/partner organization.

The best of both worlds would be to configure redundancy on both sides of the VPN. In this case, that would be deploying two gateway devices on-premises, as well as two Azure VPN gateway instances in your Azure region.

With two active instances each with two connections, there will be four total VPN tunnels between the sites. Because ECMP is required, network traffic will be distributed across all four active connections.

S2S VPN Components

  1. Gateway Subnet
    • Dedicated subnet, subnet range typically /27
    • Name: GatewaySubnet
  2. VPN Gateway
    • 2 Managed VMs that manage VNet VPN connection
    • Multiple SKUs available
  3. Local Network Gateway
    • Remote network configuration
    • VNet-to-VNet connections do not use local network gateway
  4. VPN Connection
    • Tunnel between VPN and local gateway
      • Authenticate via shared secret
    • Can have multiple tunnels per VPN gateway
    • Can connect to multiple peer networks
    • All tunnels share available SKU bandwidth

Highly Available cross-premises

To provide better availability for your cross premises connections, there are a few options available:

  1. Multiple on-premises VPN devices
  2. Active-active Azure VPN gateway
  3. Combination of both

Multiple on-premises VPN devices

  • 2 VPN tunnels
  • Protects against on-premises hardware failure
  • Azure VPN gateway in active-standby mode
  • Requires local network gateway for each on-premises connection
  • Requires BGP to advertise on-premises prefixes

You can use multiple VPN devices from your on-premises network to connect to your Azure VPN gateway.

This configuration provides multiple active tunnels from the same Azure VPN gateway to your on-premises devices in the same location. There are some requirements and constraints:

  1. You need to create multiple S2S VPN connections from your VPN devices to Azure.
    • When you connect multiple VPN devices from the same on-premises network to Azure, you need to create one local network gateway for each VPN device, and one connection from your Azure VPN gateway to each local network gateway.
  2. The local network gateways corresponding to your VPN devices must have unique public IP addresses in the “GatewayIpAddress” property.
  3. BGP is required for this configuration.
    • Each local network gateway representing a VPN device must have a unique BGP peer IP address specified in the “BgpPeerIpAddress” property.
  4. You should use BGP to advertise the same prefixes of the same on-premises network prefixes to your Azure VPN gateway, and the traffic will be forwarded through these tunnels simultaneously.
  5. You must use Equal-cost multi-path routing (ECMP).
  6. Each connection is counted against the maximum number of tunnels for your Azure VPN gateway, 10 for Basic and Standard SKUs, and 30 for HighPerformance SKU.

The Azure VPN gateway is still in active-standby mode, so the same failover behavior and brief interruption will still happen.

Active-Active VPN gateways

  • 2 Azure VPN gateway tunnels to a single on-premises device
  • Protect against Azure VPN gateway failure/maintenance

You can create an Azure VPN gateway in an active-active configuration, where both instances of the gateway VMs will establish S2S VPN tunnels to your on-premises VPN device.

  • Each Azure gateway instance will have a unique public IP address, and each will establish an IPsec/IKE S2S VPN tunnel to your on-premises VPN device specified in your local network gateway and connection.
  • Note that both VPN tunnels are actually part of the same connection. You will still need to configure your on-premises VPN device to accept or establish two S2S VPN tunnels to those two Azure VPN gateway public IP addresses.
  • Because the Azure gateway instances are in active-active configuration, the traffic from your Azure virtual network to your on-premises network will be routed through both tunnels simultaneously, even if your on-premises VPN device may favor one tunnel over the other.
  • For a single TCP or UDP flow, Azure attempts to use the same tunnel when sending packets to your on-premises network. However, your on-premises network could use a different tunnel to send packets to Azure.
  • When a planned maintenance or unplanned event happens to one gateway instance, the IPsec tunnel from that instance to your on-premises VPN device will be disconnected. The corresponding routes on your VPN devices should be removed or withdrawn automatically so that the traffic will be switched over to the other active IPsec tunnel. On the Azure side, the switch over will happen automatically from the affected instance to the active instance.

Dual-redundancy

Active-Active VPN gateways for both Azure and on-premises networks

  • Active-Active Azure VPN gateway
  • 2 on-premises connection points
  • Full mesh of 4 redundant connection
  • Can also be used in VNet-to-VNet configurations

The most reliable option is to combine the active-active gateways on both your network and Azure.

  • Here you create and set up the Azure VPN gateway in an active-active configuration, and create two local network gateways and two connections for your two on-premises VPN devices.
  • The result is a full mesh connectivity of 4 IPsec tunnels between your Azure virtual network and your on-premises network.
  • All gateways and tunnels are active from the Azure side, so the traffic will be spread among all 4 tunnels simultaneously, although each TCP or UDP flow will again follow the same tunnel or path from the Azure side.
  • Even though by spreading the traffic, you may see slightly better throughput over the IPsec tunnels, the primary goal of this configuration is for high availability.

This topology will require two local network gateways and two connections to support the pair of on-premises VPN devices, and BGP is required to allow the two connections to the same on-premises network.

Gateway SKUs

When you deploy an Azure VPN gateway, one of the required options is to select the SKU of the gateway. This SKU determines the maximum number of tunnels, connections, network throughput, and whether BGP and zone-redundancy are supported. Depending on whether you select an active/standby or active/active configuration, that can limit which SKUs you can deploy to implement that configuration.

  • Note that in general, the difference between generations is that Generation1 has a maximum of 1.25 Gbps throughput, and Generation2 offers at least 1.25 Gbps of throughput.
  • BGP is supported for every SKU except Basic.
  • The maximum number of connections per VPN gateway is limited to 30. If you need to configure more than 30 separate connections, then you must use Virtual WAN.
  • Finally, you can resize to a different SKU, as long as it is within the same generation.

Policy-based VPN vs Route-based VPN

Policy-based

  • Static routing
  • IKEv1 only
  • No Point-to-Site VPN
  • Connect to single site
  • Most backwards compatible
  • Basic gateway SKU only

Route-based

  • Dynamic routing
  • IKEv2
  • Support Point-to-Site VPN
  • Connect to multiple sites

The main point of consideration for choosing policy-based versus route-based is simply the vendor, device type, and operating system version of the remote device that you plan to connect to the Azure VPN gateway.

The basic difference between the two configuration types is that policy-based uses static routes based on address prefixes. Therefore, each subnet must be included in the VPN configuration. Route-based VPNs use a route table to direct network packets to the appropriate tunnel connection. The individual interface then performs encryption or decryption using wildcards as the traffic selector.

Configuration

  1. Create a virtual network
  2. Create a VPN gateway
  3. Create a local network gateway
  4. Create a VPN connection

Create VPN Gateway

The virtual network gateway is the primary Azure resource that you create to communicate with the target.

As part of creating the virtual network gateway, you must select a virtual network to associate the gateway with. This virtual network must have available address space to create an additional and dedicated subnet specifically for the gateway, named GatewaySubnet.

To create a virtual network gateway resource, follow these steps:

  • Name: vpn_to_onprem
  • Region: East US
  • Gateway type: VPN
  • VPN Type: Route-Based
  • SKU: VpnGw2
  • Generation: Generation2
  • Virtual network: VNet1 (select a virtual network that is defined in your Azure subscription, or create a new one.)
  • Gateway subnet address range: 10.0.255.0/24 (This range will be used to create the GatewaySubnet in the virtual network.)
  • Public IP address: Create new
  • Public IP address name: vpn_pip
  • Enable active-active mode: Disabled
  • Configure BGP: Disable

Create Local Network Gateway

The local network gateway is a resource that you create in Azure but is a logical object that represents the on-premises network that you are connecting.

To create a local network gateway in your Azure subscription, follow these steps:

  • Name: onprem_network
  • Endpoint: IP address or FQDN (Fully Qualified Domain Name).
    • IP address: If you have a static public IP address allocated from your Internet service provider for your VPN device, select the IP address option. This is the public IP address of the VPN device that you want Azure VPN gateway to connect to.
    • FQDN: If you have a dynamic public IP address that could change after certain period of time, often determined by your Internet service provider, you can use a constant DNS name with a Dynamic DNS service to point to your current public IP address of your VPN device.
  • Address Space: refers address range(s) that represents the internal IP address scheme of the on-premises environment. You can add multiple address space ranges. Make sure that the ranges you specify here don’t overlap with ranges of other networks that you want to connect to.
  • If necessary, select Configure BGP settings and provide the BGP peer ID. This is required if you plan to implement a highly available deployment with multiple VPN gateways.
  • Create.

Create a Connection

After the gateway has been deployed, you must also configure a connection that is associated with the local network gateway.

To create a connection, follow these steps:

  • On the virtual network gateway, click the Connections blade.
  • From the Connections page, click Add.
  • Name: azure_to_onprem
  • Connection Type: Site-to-site (IPsec)
  • Virtual network gateway: select the gateway that you previously configured.
  • Local network gateway: select the local network gateway that you previously configured.
  • Shared key: Enter a character string to secure the tunnel with. Both endpoints of the VPN tunnel must use the same key.
  • Use Azure Private IP Address: unchecked
  • Enable BGP: unchecked
  • IKE Protocol: IKEv2
  • OK.

Verify the VPN connection

Initially, the connection might show a status of Not connected until you either configure the destination side of the VPN or customize the policy that is used to complete the tunnel.

Note that when you are configuring the connection, there is a Connection type dropdown menu that has three options for configuring a connection:

  • VNet-to-VNet
  • Site-to-site (IPsec)
  • ExpressRoute

A VNet-to-VNet gateway is used when network peering is not possible, or not desired for encryption requirements. The same virtual network gateway is also used to configure an ExpressRoute circuit, which provides a dedicated connection to an on-premises environment.

Create and configure an IPsec/IKE policy

Azure uses a default set of IKE parameters, depending on whether you configured a policy-based or route-based type of gateway.

A custom IKE policy can be created for each connection that you define for a virtual network gateway. To create a custom policy, follow these steps:

  • On the virtual network gateway, click the Connections blade.
  • Click the name of the connection that was previously created. For example, click azure_to_onprem.
  • From the connection, click the Configuration blade.
  • IPsec/IKE policy: toggle the button to Custom.

VPN gateway connectivity issues

When you are troubleshooting connectivity issues, the first steps are to verify the existing configuration and ensure that it matches the configuration of the on-premises or target device. The most common mismatches could be:

  • Pre-shared key
  • Peer IP addresses
  • Subnets match, if using policy-based
  • IKE policy

You can also troubleshoot the status of a connection by using the logging features of the virtual network gateway. This requires that the Microsoft.Insights resource provider be registered with the subscription.

VPN Log Types

  • GatewayDiagnosticLog
    • Configuration changes
    • Maintenance events
  • TunnelDiagnosticLog
    • Tunnel state change
    • Connects/disconnects
  • RouteDiagnosticLog
    • Route changes (static, BGP)
  • IKEDiagnosticLog
    • IKE control events
    • Useful for troubleshooting IPsec tunnel handshake
  • P2SDiagnosticLog
    • Point-to-Site VPN activity

Point-to-Site VPN connection 1.2

Virtual network gateways can be used to connect site-to-site locations with IPsec, as well as distributed connections using a variety of connection types and authentication methods. These distributed connections are point-to-site connections that allow various client devices to securely access Azure resources through a VPN.

When you are planning to deploy an Azure VPN gateway for point-to-site connections, you use the same virtual network gateway that is deployed when planning a site-to-site VPN.

Supported tunnel protocol types for point-to-site VPNs:

  1. Secure Socket Tunneling Protocol (SSTP) (Windows only)
  2. OpenVPN (Open standard, supports all OS, desktop and mobile)
  3. IKEv2 (macOS, Linux, Android, Windows 10 and newer, Windows Server 2016 and newer)

SSTP is a VPN tunnel that uses TLS 1.2 and is supported only with Windows ­operating ­system clients. The scalability is limited at 128 connections per gateway.

OpenVPN also uses TLS, but it has more connection support, depending on the ­gateway SKU, and broader operating system support including:

  • Android
  • Apple iOS
  • Windows
  • Linux
  • macOS 10.13 and above

The point-to-site IKEv2 VPN uses an IPsec VPN and supports macOS 10.11 and higher operating system devices.

How are P2S VPN clients authenticated?

Before Azure accepts a P2S VPN connection, the user has to be authenticated first. There are three mechanisms that Azure offers to authenticate a connecting user.

Supported authentication types for point-to-site VPNs:

  1. Certificate
  2. Radius Server
  3. Azure AD

RADIUS authentication

If you plan to use a RADIUS server for authentication, you can choose whether the RADIUS server is deployed on-premises or in an Azure virtual network. If the RADIUS server is located on-premises, then the existing connection must be an IPsec site-to-site VPN. Using ExpressRoute with a RADIUS point-to-site VPN is not supported.

Adding a point-to-site connection can be configured after you deploy a virtual network gateway.

To configure the point-to-site with RADIUS, follow these steps:

  1. Navigate to the gateway resource that you deployed.
  2. On the virtual network gateway, click the Point-to-site configuration blade.
  3. On the Point-to-site configuration page, click Configure now.
  4. Address pool: 192.168.1.0/24 (rovide an IP address range for the client VPN connections)
  5. Tunnel type: OpenVPN (SSL)
  6. Authentication type: RADIUS authentication
  7. Primary Server IP address: 172.16.1.5 (enter the IP address of the RADIUS server)
  8. Primary Server secret: a1b2c3 (enter the shared secret that has also been configured on the RADIUS server)
  9. Click Save.
  10. After clicking Save, click the Download VPN client button to assist with client configuration.

Certificate-based authentication

Another authentication option for point-to-site VPNs is to use certification-based authentication. To use certification authentication, you will need the public key (.CER file) for the root certificate. As part of the configuration process, you will need to specify the public key. After uploading the root certificate, the gateway will generate a client certificate that is used with the VPN client configuration.

To create a point-to-site VPN configuration using certificate-based authentication, follow these steps:

  1. On the virtual network gateway, click the Point-to-site configuration blade.
  2. On the Point-to-site configuration page, click Configure now.
  3. Address pool: 192.168.1.0/24
  4. Tunnel type: OpenVPN (SSL)
  5. Authentication type: Azure certificate
  6. Root certificates Name: RootCert (provide a name for the certificate)
  7. Public certificate data, paste in the value of the public certificate.
  8. Click Save.

Generate and export certificates for point-to-site

Create a self-signed root certificate

From a computer running Windows 10 or later, or Windows Server 2016, open a Windows PowerShell console with elevated privileges.

The following example creates a self-signed root certificate named ‘P2SRootCert‘ that is automatically installed in ‘Certificates-Current User\Personal\Certificates‘. You can view the certificate by opening certmgr.msc, or Manage User Certificates.

$cert = New-SelfSignedCertificate -Type Custom -KeySpec Signature `
-Subject "CN=P2SRootCert" -KeyExportPolicy Exportable `
-HashAlgorithm sha256 -KeyLength 2048 `
-CertStoreLocation "Cert:\CurrentUser\My" -KeyUsageProperty Sign -KeyUsage CertSign

Generate a client certificate

Each client computer that connects to a VNet using Point-to-Site must have a client certificate installed. You generate a client certificate from the self-signed root certificate, and then export and install the client certificate. If the client certificate isn’t installed, authentication fails.

You may generate multiple client certificates from the same root certificate. When you generate client certificates, the client certificate is automatically installed on the computer that you used to generate the certificate. If you want to install a client certificate on another client computer, you can export the certificate.

New-SelfSignedCertificate -Type Custom -DnsName P2SChildCert -KeySpec Signature `
-Subject "CN=P2SChildCert" -KeyExportPolicy Exportable `
-HashAlgorithm sha256 -KeyLength 2048 `
-CertStoreLocation "Cert:\CurrentUser\My" `
-Signer $cert -TextExtension @("2.5.29.37={text}1.3.6.1.5.5.7.3.2")

Export the root certificate public key (.cer)

After you create a self-signed root certificate, export the root certificate .cer file (not the private key).

  1. To get the certificate .cer file, open Manage user certificates.
    • Locate the self-signed root certificate, typically in “Certificates – Current User\Personal\Certificates“, and right-click. Click All Tasks -> Export. This opens the Certificate Export Wizard.
  2. In the wizard, click Next.
  3. Select No, do not export the private key, and then click Next.
  4. On the Export File Format page, select Base-64 encoded X.509 (.CER)., and then click Next.
  5. For File to Export, browse to the location to which you want to export the certificate. For File name, name the certificate file. Then, click Next.
  6. Click Finish to export the certificate.
  7. In Notepad, copy everything between the BEGIN CERTIFICATE and END CERTIFICATE text — this is the public key.

Export the client certificate

When you generate a client certificate, it’s automatically installed on the computer that you used to generate it. If you want to install the client certificate on another client computer, you need to first export the client certificate.

  1. To export a client certificate, open Manage user certificates.
  2. In the Certificate Export Wizard, click Next to continue.
  3. Select Yes, export the private key, and then click Next.
  4. On the Export File Format page, leave the defaults selected. Make sure that Include all certificates in the certification path if possible is selected. This setting additionally exports the root certificate information that is required for successful client authentication. Without it, client authentication fails because the client doesn’t have the trusted root certificate. Then, click Next.
  5. On the Security page, you must protect the private key. If you select to use a password, make sure to record or remember the password that you set for this certificate. Then, click Next.
  6. On the File to Export, Browse to the location to which you want to export the certificate. For File name, name the certificate file. Then, click Next.
  7. Click Finish to export the certificate.

On the VPN Gateway, Paste the Public Certificate Data

  1. Select the virtualNetworkGateway resource.
  2. Settings, click Point-to-site configuration.
  3. In the Root certificates section, in the Name field, type p2spublickey.
  4. In the Public certificate data field, paste in the public key you copied from Notepad.
  5. Click Save.

Azure AD authentication

If you plan to use Azure AD as the authentication method with a point-to-site VPN, then you must use the OpenVPN connection type. This configuration also requires using the Azure VPN client and does not support built-in operating system or third-party VPN clients.

To configure Azure AD authentication, follow these steps:

  1. From the Azure portal, search for and select Azure Active Directory.
  2. From Azure AD, click the Properties blade.
  3. In the Overview blade, copy the tenant ID and save it to a text editor for use later.
  4. On the virtual network gateway, click the Point-to-site configuration blade.
  5. Address pool: 192.168.1.0/24
  6. Tunnel type: OpenVPN (SSL)
  7. Authentication type: Azure Active Directory
  8. Azure AD Tenant: https://login.microsoftonline.com/tenantID/ (enter the URL with the tenant ID you copied)
  9. Audience: enter one of the following strings, depending on the Azure cloud you are using:
    • Enter 41b23e61-6c1e-4545-b367-cd054e0ed4b4 for Azure Public
    • Enter 51bb15d4-3a4f-4ebf-9dca-40096fe32426 for Azure Government
    • Enter 538ee9e6-310a-468d-afef-ea97365856a9 for Azure Germany
    • Enter 49f817b6-84ae-4cc0-928c-73f27289b3aa for Azure China 21Vianet
  10. Issuer: https://sts.windows.net/tenantID/
  11. Click Save.
  12. After you save the configuration, a link will appear to Grant administrator consent for Azure VPN client application below your configuration details. Click this link, and then sign in using the global administrator account of your tenant.
  13. Select the checkbox next to Consent on behalf of your organization, and then click Accept.

Implement VPN client configuration file

After you have configured a point-to-site configuration with the desired connection type and authentication method, you can download the client configuration file from the point-to-site configuration page.

Azure ExpressRoute 1.3

ExpressRoute lets you extend your on-premises networks into the Microsoft cloud over a private connection with the help of a connectivity provider. With ExpressRoute, you can establish connections to Microsoft cloud services, such as Microsoft Azure and Microsoft 365.

ExpressRoute connections don’t go over the public Internet. This allows ExpressRoute connections to offer more reliability, faster speeds, consistent latencies, and higher security than typical connections over the Internet.

Azure ExpressRoute provides a hybrid connectivity across your on-premises environment and an Azure region by using a dedicated circuit through a third-party telecommunications provider.

ExpressRoute circuits are commonly compared to MPLS circuits, although they ­function slightly differently. However, the general concept is the same: a telecom provider terminates connectivity—typically fiber but could be Ethernet—at your on-premises location. After configuring the software-defined connection in your Azure subscription, you have a dedicated, private circuit from the virtual network that you associate with the configuration and your ­on-premises network.

ExpressRoute relies on the telecommunications provider for the physical connectivity between your on-premises location and the Azure region that you plan to connect to. Therefore, different Azure regions support different telecommunication vendors and vary by Azure region and geography.

ExpressRoute connectivity models

You can create a connection between your on-premises network and the Microsoft cloud in four different ways:

  1. CloudExchange Co-location
  2. Point-to-point Ethernet Connection
  3. Any-to-any (IPVPN) Connection
  4. ExpressRoute Direct

Connectivity providers may offer one or more connectivity models. You can work with your connectivity provider to pick the model that works best for you.

Co-located at a cloud exchange

Useful if you already have a presence in same co-location datacenter as service provider

Colocated providers can normally offer both Layer 2 and Layer 3 connections between your infrastructure, which might be located in the colocation facility, and the Microsoft cloud. For example, if your datacenter is colocated at a cloud exchange such as an ISP, you can request a virtual cross-connection to the Microsoft cloud.

Point-to-point Ethernet connections

Connect from your own Data Center or office

You can connect your on-premises datacenters/offices to the Microsoft cloud through point-to-point Ethernet links. Point-to-point Ethernet providers can offer Layer 2 connections, or managed Layer 3 connections between your site and the Microsoft cloud.

Any-to-any (IPVPN) networks

You can integrate your WAN with the Microsoft cloud. IPVPN providers (typically MPLS VPN) offer any-to-any connectivity between your branch offices and datacenters. The Microsoft cloud can be interconnected to your WAN to make it look just like any other branch office. WAN providers typically offer managed Layer 3 connectivity.

Direct

Directly connect your equipment to Microsoft edge router

You can connect directly into the Microsoft’s global network at a peering location strategically distributed across the world. ExpressRoute Direct provides dual 100 Gbps or 10-Gbps connectivity, which supports Active/Active connectivity at scale.

Choose between provider and direct model (ExpressRoute Direct)

Choosing between a “regular” ExpressRoute circuit and an ExpressRoute Direct circuit is ­primarily based on two decision points:

  • Do you have a bandwidth requirement of more than 10 Gbps?
  • If yes, do you have, or can you acquire, the required hardware to use an ExpressRoute Direct circuit?

ExpressRoute Direct gives you the ability to connect directly into Microsoft’s global network at peering locations strategically distributed around the world. ExpressRoute Direct provides dual 100 Gbps or 10-Gbps connectivity, which supports Active/Active connectivity at scale.

To compare the ExpressRoute options, a regular circuit provides up to 10 Gbps of connectivity through the service provider and can be terminated on-premises with Ethernet or MPLS connection options. An ExpressRoute Direct circuit can provide up to 100 Gbps connectivity but requires single-mode fiber terminations with a router-pair at the on-premises location.

The full ExpressRoute Direct requirements for the routing hardware include:

  • Dual 10 Gigabit or 100-Gigabit Ethernet ports only across router pair
  • Single-mode Long Reach (LR) fiber connectivity
  • IPv4 and IPv6
  • Transmission size of 1500 bytes

There are requirements at the switching layer, including:

  • One 802.1Q tag or two Tag 802.1Q (QinQ) tag encapsulation
  • If using QinQ, you must be able to configure a Microsoft-specified VLAN ID tag
  • Ethertype must be 0x8100
  • Must support multiple BGP sessions (VLANs) per port and device
  • IPv4 and IPv6 connectivity
    • For IPv6, no extra subinterface will be created. An IPv6 address will be added to the existing subinterface.

Optionally, you can also configure Bidirectional Forwarding Detection (BFD) support, which is configured by default on all Private Peerings on ExpressRoute circuits.

Another consideration between the two options would be pricing. A regular ExpressRoute premium circuit in the United States at 10 Gbps and 100 TB of data transfer per month would be approximately $9,000 per month. Upgrading the circuit to ExpressRoute Direct with 100 Gbps connectivity increases the price to over $55,000 per month. Note that these cost estimates are examples and vary by geography, subscription type, and actual data usage.

Service Provider Direct
Partner with ExpressRoute-specific service provider Use any service provider
50 Mbps - 10 Gbps connections 5 Gbps - 100 Gbps connections
- Ideal for massive data ingestion
Connect over service provider's network
- May share hardware with other tenants
Physically isolated network hardware
- Ideal for regulation requirements

Circuit SKUs

Azure ExpressRoute offers three different circuit SKUs:

  1. Local
  2. Standard
  3. Premium

A key feature of Local is that a Local circuit at an ExpressRoute peering location gives you access only to one or two Azure regions in or near the same metro. In contrast, a Standard circuit gives you access to all Azure regions in a geopolitical area and a Premium circuit to all Azure regions globally.

ExpressRoute Local

While you need to pay egress data transfer for your Standard or Premium ExpressRoute circuit, you don’t pay egress data transfer separately for your ExpressRoute Local circuit. In other words, the price of ExpressRoute Local includes data transfer fees. ExpressRoute Local is a more economical solution if you have massive amount of data to transfer and you can bring your data over a private connection to an ExpressRoute peering location near your desired Azure regions.

Compared to a Standard ExpressRoute circuit, a Local circuit has the same set of features except:

  • Scope of access to Azure regions as described above
  • ExpressRoute Global Reach isn’t available on Local

Gateway SKUs

When you create a virtual network gateway, you need to specify the gateway SKU that you want to use. When you select a higher gateway SKU, more CPUs and network bandwidth are allocated to the gateway, and as a result, the gateway can support higher network throughput to the virtual network.

Determining which ExpressRoute SKU to use depends on the workload and goals of the design. The following SKUs are available for ExpressRoute gateways:

  • Standard
  • High performance
  • Ultra performance
  • ERGW1AZ
  • ERGW2AZ
  • ERGW3AZ

The following upgrades are supported:

  • Standard to High Performance
  • Standard to Ultra Performance
  • High Performance to Ultra Performance
  • ErGw1Az to ErGw2Az
  • ErGw1Az to ErGw3Az
  • ErGw2Az to ErGw3Az
  • Default to Standard

There is no feature difference for FastPath or circuit connection support for Standard SKU and ERGW1AZ SKU. The difference between these two SKUs is that the AZ SKU supports Availability Zones if the region you deploy the gateway to also supports Availability Zones.

ExpressRoute Global Reach

By default, when you configure ExpressRoute with a cross-region design, your on-premises offices can talk to the regions that are connected to the circuit. However, without Global Reach, you cannot use the ExpressRoute circuit to communicate directly between on-premises locations. For example, referring to Figure, the Los Angeles office and the New York office would not be able to communicate by default.

If you would like to use the ExpressRoute circuits to enable communication between Los Angeles and New York through ExpressRoute, Global Reach will enable this. However, if the two office locations that you are trying to connect are across geopolitical regions, then you must be using the Premium add-on to the ExpressRoute circuit. For example, this might be between offices in the United States and Canada.

In the above example, with the addition of ExpressRoute Global Reach, your San Francisco office (10.0.1.0/24) can directly exchange data with your London office (10.0.2.0/24) through the existing ExpressRoute circuits and via Microsoft’s global network.

To add Global Reach to an existing ExpressRoute circuit, follow these steps:

  1. Search for ExpressRoute, then select the ExpressRoute circuits result.
  2. From the circuit, click the Peerings blade.
  3. Select the Azure private peer that has been provisioned.
  4. On the Azure private peering page, click Add Global Reach.
  5. In the Add Global Reach flyout page, provide a name, select a second circuit, and provide a /29 IPv4 subnet to use with the private peer.

ExpressRoute FastPath

ExpressRoute FastPath is a performance-enhancing feature that is available only on the top-tier SKUs: Ultra performance and ERGW3AZ. Additionally, if you plan to use IPv6, then you can only use the ERGW3AZ SKU.

ExpressRoute FastPath optimizes the network routing so that traffic destined to virtual machines is delivered directly and bypasses the virtual network gateway, lowering the overall latency.

To enable FastPath on an existing ExpressRoute circuit, follow these steps:

  1. Search for ExpressRoute, then select the ExpressRoute circuits result.
  2. Select a circuit that you have already configured and want to enable FastPath on.
  3. Click the Connections blade, and then click the existing connection.
  4. On the specific connection, click the Configuration blade.
  5. Select the checkbox next to FastPath, and then click Save.

ExpressRoute Peering

As part of the ExpressRoute configuration, you have the option of configuring Azure private peering, Microsoft peering, or both. Azure private peering provides connectivity from your on-premises environment to the Azure subscriptions and resources that you link the circuit to.

In rare circumstances, you might also want Microsoft 365 services, such as Microsoft Exchange Online and Microsoft SharePoint Online, to be accessible over the ExpressRoute circuit. This scenario uses Microsoft peering, but it is not recommended by Microsoft and should be used only if your organization has strict requirements for network traffic. Microsoft 365 was designed to be securely accessed over the public internet, and still requires public internet access for certain services even if Microsoft peering over ExpressRoute is configured.

Azure private peering

The private peering domain is considered to be a trusted extension of your core network into Microsoft Azure. You can set up bi-directional connectivity between your core network and Azure virtual networks (VNets). This peering lets you connect to virtual machines and cloud services directly on their private IP addresses.

You can connect more than one virtual network to the private peering domain.

To configure Azure private peering, the provider status of the ExpressRoute circuit must display Provisioned. As part of this provisioning, some ExpressRoute providers will manage the routing as part of the circuit, and these steps would not be necessary.

If the provider does not manage the routing, then you’ll need to follow these steps:

  1. Search for ExpressRoute, then select the ExpressRoute circuits result.
  2. Select a circuit that you have already configured and want to configure private peering for.
  3. Click the Peerings blade, and then click Azure private.
  4. On the Private peering page, configure the Peer ASN, protocol, subnets, and VLAN ID.
  5. Click Save.

Microsoft peering

Microsoft 365 was created to be accessed securely and reliably via the Internet. Because of this, we recommend ExpressRoute for specific scenarios.

Connectivity to Microsoft online services (Microsoft 365 and Azure PaaS services) occurs through Microsoft peering. We enable bi-directional connectivity between your WAN and Microsoft cloud services through the Microsoft peering routing domain. You must connect to Microsoft cloud services only over public IP addresses that are owned by you or your connectivity provider and you must adhere to all the defined rules.

To configure Microsoft peering, the provider status of the ExpressRoute circuit must display Provisioned. Note that Microsoft peering requires the use of public IP addresses but can use either public or private AS numbers.

To configure Microsoft peering, follow these steps:

  1. Search for ExpressRoute, then select the ExpressRoute circuits result.
  2. Select a circuit that you have already configured and want to configure private peering for.
  3. Click the Peerings blade, and then click Microsoft peering.
  4. On the Microsoft peering page, configure the Peer ASN, protocol, subnets, advertised prefixes, and VLAN ID.

Peering Comparison

Configuration

Create ExpressRoute Circuit

Search for ExpressRoute and then select Create.

You can view the properties of the circuit by selecting it. On the Overview page for your circuit, you’ll find the Service Key. Provide the service key to the service provider to complete the provisioning process. The service key is unique to your circuit.

To use the ExpressRoute circuit, it must be in the following state:

  • Provider status: Provisioned
  • Circuit status: Enabled

Create ExpressRoute Private Peering

Preequisites

  • You must have an active ExpressRoute circuit.
    • To configure peering(s), the ExpressRoute circuit must be in a provisioned and enabled state.

To create Azure private peering:

  1. Configure Azure private peering for the circuit. Make sure that you have the following items before you continue with the next steps:
    • A pair of subnets that aren’t part of any address space reserved for virtual networks.
      • One subnet will be used for the primary link, while the other will be used for the secondary link.
      • From each of these subnets, you’ll assign the first usable IP address to your router as Microsoft uses the second usable IP for its router.
    • A valid VLAN ID to establish this peering on. Ensure that no other peering in the circuit uses the same VLAN ID. For both Primary and Secondary links you must use the same VLAN ID.
    • AS number for peering.
    • You must advertise the routes from your on-premises Edge router to Azure via BGP when you configure the private peering.
    • Optional – An MD5 hash if you choose to use one.

Create ExpressRoute Gateway

  1. Create a gateway subnet.
  2. Create virtual network gateway.

Create the gateway subnet

The Name for your subnet is automatically filled in with the value ‘GatewaySubnet‘. This value is required in order for Azure to recognize the subnet as the gateway subnet. Adjust the autofilled Address range values to match your configuration requirements. We recommend creating a gateway subnet with a /27 or larger (/26, /25, and so on.). If you plan on connecting 16 ExpressRoute circuits to your gateway, you must create a gateway subnet of /26 or larger.

Create the virtual network gateway

An ExpressRoute gateway is still a virtual network gateway, only with different options selected.

To create a virtual network gateway for ExpressRoute, search for virtual network gateways, then select the Virtual network gateways result.

Connect a VNet to an ExpressRoute circuit

Similar to creating a VPN, creating the virtual network gateway as an ExpressRoute gateway does not create the connection for the ExpressRoute circuit. After you create the gateway, you must also create a connection to link the circuit to a virtual network.

Route advertisement

Some telecommunication companies that offer ExpressRoute services also offer a managed routing service as part of the deployment. If this is the case, then you don’t need to do anything additional to configure route advertisements. However, if you need or plan to configure the route advertisement manually, then you need to meet the following requirements.

For IPv4, the requirements include:

  • A single /29 or two /30 subnets for the interfaces
    • If you use a /29, it is separated into a /30 anyway
    • The first /30 is for the primary link
    • The second /30 is for the secondary link
    • The first available IP address must be used on your hardware
    • The second available IP address will be used by Azure
  • For Azure private peering, the addresses used can be public or private
  • These ranges must not conflict with any virtual network in Azure you plan to connect to

Similar requirements exist if you plan to use IPv6, including:

  • A single /125 subnet or two /126 subnets for the interfaces
    • If you use a /125, it is split into two /126 subnets
    • The first available IP address must be used on your hardware
    • The second available IP address will be used by Azure
  • For Azure private peering, the addresses used can be public or private
  • These ranges must not conflict with any virtual network in Azure you plan to connect to

If you plan to use Microsoft peering, then you must use public IP addresses, which is required for BGP sessions. You must also be able to verify ownership of the addresses through the Internet Routing Registry. The same requirements above apply to Microsoft peering but are only supported with public IP addresses.

After you determine which peering type and protocol that you will use, you can plan route aggregation and default routes. Azure ExpressRoute private peering supports up to 4,000 individual IPv4 prefixes, and 100 IPv6 prefixes that will be advertised. If you enable the premium SKU, then you can increase the IPv4 limit up to 10,000 prefixes.

You can also configure the ExpressRoute circuit to be the default route for Azure virtual networks. This will ensure that all traffic from Azure then flows through ExpressRoute to your on-premises hardware before being routed to the internet.

For BGP and AS numbers, Microsoft uses AS number 12076 for its AS number. You can connect to individual communities for various services within the region that you configure the ExpressRoute circuit in.

Encryption over ExpressRoute

ExpressRoute on its own is a private, dedicated connection through your selected telecommunication provider to an Azure region. However, that does not mean that the traffic is encrypted. You have two options for configuring encryption over ExpressRoute:

  1. A site-to-site VPN using a virtual network gateway
  2. A site-to-site VPN using Azure Virtual WAN

Implement BFD

Bidirectional Forwarding Detection (BFD) is useful to accelerate the realization that one of the ExpressRoute links has failed. The BGP keep-alive time is typically configured at 60 seconds, and the hold-time configured at 180 seconds. This means it can take up to three minutes for the service to realize that one of the ExpressRoute links has failed. With BFD, you can lower the keep-alive times as low as 300 milliseconds, allowing failovers to occur in under a second.

BFD is configured by default for all new ExpressRoute private peer connections. After making the configuration change on the on-premises hardware, you only need to reset the ExpressRoute private peer connection to renegotiate the connection with BFD.

To reset an existing connection, follow these steps.

  1. Search for ExpressRoute, then select the ExpressRoute circuits result.
  2. Click the Peerings blade, and then click Azure private.
  3. On the Private peering page, clear the checkbox next to Enable IPv4 Peering, click Save.
  4. Select the checkbox next to Enable IPv4 Peering, and then click Save.

Chapter summary

  • Virtual network gateways can be deployed in either zone-redundant or zonal configurations.
    • For zone-redundant virtual network gateways, you must use a Standard SKU public IP address.
    • The virtual network gateway SKU determines the throughput and maximum number of site-to-site tunnels.
    • Policy-based VPNs use static routes based on address range prefixes.
    • Route-based VPNs use a route table to direct packets.
    • When configuring a virtual network gateway, the virtual network must have a subnet named GatewaySubnet.
    • A local network gateway is an Azure resource that represents the destination address range.
  • Point-to-site VPNs support SSTP, OpenVPN, or IKEv2 connection types.
    • The virtual network gateway SKU determines the throughput and maximum number of point-to-site connections.
    • Point-to-site VPN tunnels can use RADIUS, certificate, or Azure AD for authentication.
      • When using Azure AD for authentication, you must use the Azure VPN client.
  • ExpressRoute provides a private, direct connection from your on-premises environment to Azure.
    • The ExpressRoute SKU determines the bandwidth and connection limit for the circuit.
    • ExpressRoute Global Reach requires the Premium add-on and provides connectivity to multiple Azure regions through one ExpressRoute circuit and is configured on the Private peer.
    • ExpressRoute FastPath bypasses the virtual network gateway and communicates directly with virtual machines and is configured on the circuit connection.
    • Private peering provides access from your on-premises network to Azure.
    • Microsoft peering provides access from your on-premises network to Microsoft 365 services, but it is not recommended.
    • ExpressRoute route advertisement is configured by using BGP.

Thought experiment

  • Contoso Mortgage is a financial services company that has two primary datacenters in Chicago and New York.
  • The datacenters have a redundant 5 Gbps connection between them.
  • The company also has smaller branch offices with site-to-site VPN tunnels from the branch office to each datacenter.
  • The IT and executive staff have software-defined VPNs on mobile devices for remote connectivity to each datacenter.

Contoso plans to move their primary application from the on-premises datacenters to the Azure cloud. The network must support the migration and on-going connectivity similarly to the current on-premises design. In the event that one region or connectivity provider has a severe outage, the on-premises datacenter must still be able to communicate with Azure.

For each goal, identify the Azure service to use and incorporate it into the existing architecture.

  1. What should you use to provide hybrid connectivity to each datacenter?
  2. What should you use to provide hybrid connectivity to each branch office?
  3. How should the services be deployed to ensure highly available connectivity?
  4. What should the IT and executive staff use to connect to Azure remotely?

Thought experiment answers

  • The desired design based on the requirements of Contoso Mortgage would include ExpressRoute connections from both on-premises datacenters to two Azure regions for the desired redundancy.
    • In this scenario, we can select East US 2 and Central US as the target regions.
  • Because of the requirement to communicate across regions through either of the ExpressRoute circuits, Global Reach must be configured on the circuits.
    • This also means that the circuits have to be ExpressRoute with the Premium add-on.
  • Finally, to address any possible severe outages across providers, the ExpressRoute circuits should be two different connectivity providers for each region.
  • The branch offices can continue to connect to the on-premises datacenters using site-to-site VPNs. You can configure additional VPNs for each branch office if or when they need the connectivity to Azure.

What should you use to provide hybrid connectivity to each datacenter?

Each datacenter should have an Azure ExpressRoute circuit with Global Reach to meet the design requirements.

What should you use to provide hybrid connectivity to each branch office?

Each branch office should have a site-to-site VPN defined to each Azure region.

How should the services be deployed to ensure highly available connectivity?

Each branch office should have an independent site-to-site VPN to each Azure region. Each ExpressRoute circuit should have Global Reach enabled for cross-region connectivity.

What should the IT and executive staff use to connect to Azure remotely?

The Azure regions should have a virtual network gateway with point-to-site connectivity configured for the IT and executive staff to connect remotely with their mobile devices.

Leave a Reply