Networking - VXLAN (Virtual Extensible LAN)

Virtual Extensible LAN (VXLAN) is a modern network virtualization technology designed to address the limitations of traditional VLANs in large-scale cloud and data center environments. VXLAN provides a scalable and flexible way to extend Layer 2 (L2) networks across Layer 3 (L3) boundaries, allowing virtual machines (VMs) and workloads to communicate as if they were on the same local network — even when they are physically located in different parts of a data center or across multiple data centers.

1. Background and Motivation

In traditional networking, VLANs (Virtual Local Area Networks) are used to segment networks logically within the same Layer 2 domain. However, VLANs have significant limitations:

• VLAN IDs are limited to 4096 (12-bit field in the VLAN header), which is insufficient for large-scale cloud and data center environments.

• VLANs cannot easily extend across multiple Layer 3 networks.

• Mobility of virtual machines across data centers becomes difficult due to broadcast domain boundaries.

As cloud computing, virtualization, and multi-tenant data centers grew, these limitations became critical. To overcome them, the VXLAN standard was developed by VMware, Cisco, Arista, and others, and later standardized by the IETF (RFC 7348).

2. Definition

VXLAN (Virtual Extensible LAN) is a Layer 2 overlay technology built on top of a Layer 3 network. It encapsulates Ethernet frames inside UDP packets, allowing Layer 2 networks to be extended over Layer 3 infrastructure.

In simple terms:

VXLAN creates a virtual tunnel between two or more network endpoints, allowing devices on separate physical networks to behave as though they’re connected to the same LAN.

3. How VXLAN Works

VXLAN operates by encapsulating Ethernet frames in UDP packets. This encapsulation allows L2 traffic to be transmitted over L3 networks, such as an IP network or the internet.

Key Elements of VXLAN Architecture

1. VTEP (VXLAN Tunnel Endpoint):

   • The most critical component of VXLAN.

   • VTEPs exist at the edge of the VXLAN network — usually implemented in virtual switches (like VMware vSwitch or Open vSwitch) or in physical switches.

   • A VTEP performs two main functions:

     • Encapsulation: Takes an Ethernet frame from a local virtual machine and encapsulates it inside a VXLAN header and UDP/IP packet.

     • Decapsulation: Receives VXLAN packets from the network, removes the VXLAN and UDP/IP headers, and delivers the original Ethernet frame to the destination VM.

2. VXLAN Network Identifier (VNI):

   • A 24-bit field in the VXLAN header that identifies the VXLAN segment.

   • Supports up to 16 million (2²⁴) logical networks, far exceeding the 4096 VLAN limit.

3. Underlay Network:

   • The physical IP network that carries VXLAN-encapsulated traffic.

   • This is a standard Layer 3 network that uses routing protocols like OSPF, BGP, or IS-IS to transport the packets.

4. Overlay Network:

   • The virtual network formed by VXLAN tunnels between VTEPs.

   • The overlay is where the logical Layer 2 communication happens, independent of the physical network topology.

4. VXLAN Encapsulation Process

When a packet is sent from one VM to another in a VXLAN network:

1. The source VTEP receives the Ethernet frame from the local VM.

2. It adds a VXLAN header, which includes the VNI (identifying the VXLAN segment).

3. This packet is then encapsulated inside a UDP header (usually using port 4789).

4. The IP header and Ethernet header of the underlay network are then added.

5. The encapsulated packet is routed through the underlay network to the destination VTEP.

6. The destination VTEP decapsulates the packet and delivers the original Ethernet frame to the target VM.

This process allows VMs on different physical networks to communicate as if they were on the same broadcast domain.

5. VXLAN Header Format

A VXLAN packet includes:

• Outer Ethernet header (used by the underlay network)

• Outer IP header (source/destination = VTEP IPs)

• UDP header (destination port 4789)

• VXLAN header (8 bytes, containing the 24-bit VNI)

• Inner Ethernet frame (original frame from the VM)

6. Control Plane: Flood and Learn vs. EVPN

VXLAN itself (RFC 7348) only defines the data plane — how encapsulation and decapsulation occur.
For control-plane learning (how VTEPs discover each other and know where to send traffic), two main approaches exist:

1. Flood and Learn (Multicast-based):

   • Early VXLAN implementations relied on multicast in the underlay network to discover remote VTEPs.

   • Broadcast, unknown unicast, and multicast (BUM) traffic is flooded to all VTEPs within a VXLAN segment.

2. EVPN (Ethernet VPN) Control Plane:

   • A modern and scalable approach using BGP EVPN to distribute MAC and IP address information among VTEPs.

   • EVPN eliminates the need for multicast and provides efficient, scalable, and loop-free VXLAN deployments.

7. Advantages of VXLAN

1. Scalability:

   • Supports up to 16 million VNIs, compared to only 4096 VLANs.

2. Layer 2 Extension over Layer 3:

   • Enables virtual machines and applications to move across different subnets and data centers seamlessly.

3. Multi-Tenancy:

   • Ideal for cloud environments with multiple customers or tenants who need isolated virtual networks.

4. Network Agility and Flexibility:

   • Simplifies network provisioning and allows for easier network reconfiguration without changing the underlying physical infrastructure.

5. Integration with SDN and NFV:

   • Works seamlessly with Software-Defined Networking (SDN) controllers and Network Function Virtualization (NFV) for automated provisioning and orchestration.

6. Fault Isolation and Security:

   • Each VXLAN segment (identified by a unique VNI) provides strong isolation between tenants.

8. Challenges of VXLAN

1. Increased Overhead:

   • VXLAN adds about 50 bytes of encapsulation overhead, which can lead to MTU (Maximum Transmission Unit) issues unless configured properly.

2. Complexity:

   • Managing and troubleshooting overlay networks can be more complex than traditional VLANs.

3. Hardware Support:

   • Older switches may not support VXLAN offloading, leading to higher CPU usage on servers.

4. Multicast Dependency (in older models):

   • Early VXLAN implementations relied on multicast for VTEP discovery, which adds complexity to network configuration.

9. VXLAN vs VLAN

10. Real-World Use Cases

• Cloud Service Providers: To isolate tenants in multi-tenant cloud environments.

• Data Centers: To create large, flexible Layer 2 overlays over Layer 3 infrastructure.

• Disaster Recovery: To extend Layer 2 domains across geographically separated sites.

• 5G and Edge Computing: For network slicing and flexible virtual network deployment.

11. Conclusion

VXLAN (Virtual Extensible LAN) is a cornerstone technology for modern data center and cloud network virtualization. By encapsulating Layer 2 traffic over Layer 3 infrastructure, it provides the scalability, flexibility, and tenant isolation required by large-scale, dynamic environments.

VXLAN effectively removes the limitations of traditional VLANs, supports millions of isolated networks, and enables seamless VM mobility across subnets and data centers. With EVPN integration and SDN orchestration, VXLAN has become the preferred standard for network overlays in modern cloud architectures.