EVPN VxLAN Multihoming HLD #1622

hasan-brcm · 2024-02-23T02:26:16Z

First draft of EVPN VxLAN Multihoming high level design.

philo-micas · 2024-03-06T02:13:08Z

doc/vxlan/EVPN/EVPN_VxLAN_Multihoming.md

+
+The following are functional requirements for EVPN VxLAN Multihoming:
+
+1. Support All-EVPN based active-active access redundancy with up to 4 VTEPs.


Is "up to 4 VTEPS" just software limitation, or ASIC hardware limitaion?

The software data-structures etc need to have some size, it is primarily for that. Not a hardware limitation.

philo-micas · 2024-03-06T02:14:13Z

doc/vxlan/EVPN/EVPN_VxLAN_Multihoming.md

+In the example output, L2 NHID group 536870913 has two member nexthops - 268435458 and 268435459 - which resolve to single path VTEP nexthops.
+
+
+#### 2.2.4.2 BUM traffic handling


Which component is responsible for installing tc rules to kernel?

Ideally it should be FRR (zebra.)

philo-micas · 2024-03-06T02:14:42Z

doc/vxlan/EVPN/EVPN_VxLAN_Multihoming.md

+
+# 3 Design
+
+## 3.1 Overview


suggest to add Architecture diagram for understanding software design.

philo-micas · 2024-03-06T02:15:29Z

doc/vxlan/EVPN/EVPN_VxLAN_Multihoming.md

+  - Known unicast traffic with Destination MACs pointing to the above bridgeport type will be load balanced among the remote VTEPs. 
+
+```
+typedef enum _sai_l2_ecmp_group_attr_t


it is noted that l2 ecmp group has almost the same attribute with sainexthopgroup.
why not to reuse sainexthopgroup object and add some attribute to identify l2 and l3 ecmp group?

@philo-micas , yes, it makes sense. SAI subsection is updated.

helloanandhi · 2024-03-19T15:30:35Z

doc/vxlan/EVPN/EVPN_VxLAN_Multihoming.md

+Achieve sub-second convergence in following failure scenarios:
+1. LAG link down
+2. LAG link up
+3. VTEP restart


How is sub-second expected for VTEP restart ? Is Fastboot being referred ?

Sub-second convergence is out of scope for this design. Removed this sec.

helloanandhi · 2024-03-19T16:02:19Z

doc/vxlan/EVPN/EVPN_VxLAN_Multihoming.md

+
+![](images/evpn_mh_unicast_forwarding.PNG)
+
+In the diagram above, H2 host mac address H2-MAC is advertised by Vtep-1 and/or Vtep-4 in a EVPN Type-2 route that also carries Ethernet-Segment identifier (ES-ID) of PortChannel1. On Vtep-5, the H2-MAC processing undergoes special handling since the Type-2 route has ES-ID. And H2-MAC is installed against L2 NHG corresponding to the ES-ID. Even if Type-2 route for H2-MAC is received only from either Vtep-1 or Vtep-4, the traffic towards H2 will still be load-balanced between Vtep-1 and Vtep-4. Later, if PortChannel1 interface goes down, on say Vtep-1, Vtep-1 will withdraw the Type-1 (AD-per-ES) route and this will result in Vtep-5 updating L2 NHG for the ES-ID. This re-balance of traffic will not wait for the individual MAC route updates to arrived and get processed. This update of L2 NHG on Multihomed ES link failures is referred as "Fast Convergence" in RFC 7432.


Wouldn't a MAC-to-ESI association in SAI be needed for mass withdrawl ?

ESI-based MACs will be installed with L2NHID. In case of remote ESI member change, L2NHID will be updated in the hw. There is no need for SAI to be aware of ESI or MAC-to-ESI association.

helloanandhi · 2024-03-19T16:04:44Z

doc/vxlan/EVPN/EVPN_VxLAN_Multihoming.md

+     * @default false
+     * @validonly SAI_BRIDGE_PORT_ATTR_TYPE == SAI_BRIDGE_PORT_TYPE_PORT
+     */
+    SAI_BRIDGE_PORT_ATTR_NON_DF,


SAI_BRIDGE_PORT_ATTR_NON_DF is for n/w facing ports or access facing ports ?

It is for access facing ports (ESI configured interfaces.)

How does SAI knows whether a PORT is part of Ethernet segment or not..? do we need one more attribute to define that. I think, SAI can't depend on (SAI_BRIDGE_PORT_ATTR_NON_DF) attribute, since this is "FALSE" even for both Ethernet segment and single homed switches.

and second doubt is, does this "default value of false" condition causes any Micro-loops in the network..?
Consider a case for PO2 configuration..

when an acess PORT with Ethernet segment enabled is added to Valn, SAI calls bridge port with (SAI_BRIDGE_PORT_ATTR_NON_DF== false), since still DF election is in progress.

above condition is smae on all 4 VTEPs. Now all the nodes which are part of PO2(VTEP-1,2,3,4) will start flooding all the BUM traffic.

after the DF election is completed, SAI will update the NON-DF nodes with (SAI_BRIDGE_PORT_ATTR_NON_DF= true)

till this time, traffic is flooding on all VTEPs. does this momentary flooded traffic will not cause transient loops in the network..?

adyeung · 2024-03-20T21:20:19Z

doc/vxlan/EVPN/EVPN_VxLAN_Multihoming.md

+### 1.1.3 Platform requirements
+Support EVPN Multihoming on platforms having VxLAN capabilities.
+
+### 1.1.4 Scalability Requirements


The solution is vendor agnostic, it's difficult to claim scaling performance and convergence in isolation of platform details and ASIC capabilities, please remove section 1.1.4, 1.1.5, and section 8

Agreed. Removed.

adyeung · 2024-03-20T21:51:47Z

doc/vxlan/EVPN/EVPN_VxLAN_Multihoming.md

+
+
+
+# 9 Unit Test


EVPN MH will require a separate testplan documentation, please move this to a test plan submission

Agreed. Removed from here for now.

adyeung · 2024-03-20T21:52:42Z

doc/vxlan/EVPN/EVPN_VxLAN_Multihoming.md

+
+
+
+# 8 Scalability


To be removed for the same reason stated in section 1

Agreed. Removed.

eddieruan-alibaba · 2024-03-25T21:04:41Z

doc/vxlan/EVPN/EVPN_VxLAN_Multihoming.md

+![image](https://github.com/hasan-brcm/SONiC/assets/56742004/4bd442a9-5275-4fb1-8ccf-a46083c57bf9)
+
+
+In the diagram above, H2 host mac address H2-MAC is advertised by Vtep-1 and/or Vtep-4 in a EVPN Type-2 route that also carries Ethernet-Segment identifier (ES-ID) of PortChannel1. On Vtep-5, the H2-MAC processing undergoes special handling since the Type-2 route has ES-ID. And H2-MAC is installed against L2 NHG corresponding to the ES-ID. Even if Type-2 route for H2-MAC is received only from either Vtep-1 or Vtep-4, the traffic towards H2 will still be load-balanced between Vtep-1 and Vtep-4. Later, if PortChannel1 interface goes down, on say Vtep-1, Vtep-1 will withdraw the Type-1 (AD-per-ES) route and this will result in Vtep-5 updating L2 NHG for the ES-ID. This re-balance of traffic will not wait for the individual MAC route updates to arrived and get processed. This update of L2 NHG on Multihomed ES link failures is referred as "Fast Convergence" in RFC 7432.


After Vtep-4 receives this update, it installs the MAC against Po1 but also advertises Type-2 route for the same MAC with Proxy=1, VTEP-4 receives MAC from H2 locally. In this case, would VTEP-4 update proxy bits to both remote VTEPs and VTEP1?

If the mac is learnt locally on Vtep-4, the type-2 route will be re-advertised with proxy=0 by vtep-4.

eddieruan-alibaba · 2024-03-25T21:08:14Z

doc/vxlan/EVPN/EVPN_VxLAN_Multihoming.md

+The MCLAG configuration will not be allowed if ESI is configured on one or more interface(s).
+
+### 2.2.8 ARP/ND suppression
+ARP/ND suppression will be supported in EVPN Multihoming scenarios. The VTEP will respond to the ARP/ND requests received on local access ports only for the ARP/ND installed against remote VTEPs. ARP/ND response will not be  generated by the VTEP if ARP/ND is installed on the local Multihomed ES, even though the ARP/ND learning happened on the remote VTEP multihoming a given ES.


Similar to my previous question, initially, VTEP1 learns ARP. But later, the ARP packets goes to VTEP4 due to some hash changing locally on the host. In this case, how would VTEP4 handle these ARP packets?

ARP rides on mac. If the mac is locally active on vtep-4, the corresponding type-2 routes will be advertised with proxy=0.

After VTEP1 learns ARPs, it would use BGP to send this type 2 route to VTEP4. How does VTEP4 install this ARP entry to kernel? since this ARP entry should not be timeout unless VTEP1 withdrew it.

Once VTEP4 learns the same ARP entry locally, would kernel be able to flip previous remote learnt ARP entry to local entry? If the local ARP time sout later, which module detects it and reinstalls remote learnt ARP?

adyeung · 2024-06-26T23:24:59Z

Converged HLD posted as #1702, closing this PR

hasan-brcm added 4 commits February 22, 2024 13:51

Create EVPN_VxLAN_Multihoming.md

9ac169c

Add files via upload

f1179b3

Update EVPN_VxLAN_Multihoming.md

b2e71e4

Update EVPN_VxLAN_Multihoming.md

8462c41

philo-micas reviewed Mar 6, 2024

View reviewed changes

helloanandhi reviewed Mar 19, 2024

View reviewed changes

adyeung reviewed Mar 20, 2024

View reviewed changes

Yuval-Mellanox mentioned this pull request Mar 21, 2024

EVPN-VxLAN Multihoming HLD #1638

Closed

hasan-brcm added 3 commits March 21, 2024 01:09

Update EVPN_VxLAN_Multihoming.md

f8fdcdc

Update EVPN_VxLAN_Multihoming.md

054c2fe

Update EVPN_VxLAN_Multihoming.md

009a236

eddieruan-alibaba reviewed Mar 25, 2024

View reviewed changes

Yuval-Mellanox requested a review from eddyk-nvidia March 26, 2024 11:46

adyeung mentioned this pull request Apr 23, 2024

EVPN VxLAN Multihoming #1662

Open

pbrisset mentioned this pull request May 21, 2024

Merged EVPN VxLAN MH HLD from Cisco and BCM #1702

Merged

adyeung closed this Jun 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EVPN VxLAN Multihoming HLD #1622

EVPN VxLAN Multihoming HLD #1622

hasan-brcm commented Feb 23, 2024

philo-micas Mar 6, 2024

hasan-brcm Mar 20, 2024

philo-micas Mar 6, 2024

hasan-brcm Mar 20, 2024

philo-micas Mar 6, 2024

philo-micas Mar 6, 2024

hasan-brcm Mar 21, 2024

helloanandhi Mar 19, 2024

hasan-brcm Mar 21, 2024

helloanandhi Mar 19, 2024

hasan-brcm Mar 21, 2024

helloanandhi Mar 19, 2024

hasan-brcm Mar 20, 2024

sathk-marvell Apr 9, 2024 •

edited

Loading

adyeung Mar 20, 2024

hasan-brcm Mar 21, 2024

adyeung Mar 20, 2024

hasan-brcm Mar 21, 2024

adyeung Mar 20, 2024

hasan-brcm Mar 21, 2024

eddieruan-alibaba Mar 25, 2024

hasan-brcm Mar 28, 2024 •

edited

Loading

eddieruan-alibaba Mar 25, 2024

hasan-brcm Mar 28, 2024

eddieruan-alibaba Mar 28, 2024

adyeung commented Jun 26, 2024


		The following are functional requirements for EVPN VxLAN Multihoming:

		1. Support All-EVPN based active-active access redundancy with up to 4 VTEPs.

		In the example output, L2 NHID group 536870913 has two member nexthops - 268435458 and 268435459 - which resolve to single path VTEP nexthops.


		#### 2.2.4.2 BUM traffic handling


		![](images/evpn_mh_unicast_forwarding.PNG)

		In the diagram above, H2 host mac address H2-MAC is advertised by Vtep-1 and/or Vtep-4 in a EVPN Type-2 route that also carries Ethernet-Segment identifier (ES-ID) of PortChannel1. On Vtep-5, the H2-MAC processing undergoes special handling since the Type-2 route has ES-ID. And H2-MAC is installed against L2 NHG corresponding to the ES-ID. Even if Type-2 route for H2-MAC is received only from either Vtep-1 or Vtep-4, the traffic towards H2 will still be load-balanced between Vtep-1 and Vtep-4. Later, if PortChannel1 interface goes down, on say Vtep-1, Vtep-1 will withdraw the Type-1 (AD-per-ES) route and this will result in Vtep-5 updating L2 NHG for the ES-ID. This re-balance of traffic will not wait for the individual MAC route updates to arrived and get processed. This update of L2 NHG on Multihomed ES link failures is referred as "Fast Convergence" in RFC 7432.

		![image](https://github.com/hasan-brcm/SONiC/assets/56742004/4bd442a9-5275-4fb1-8ccf-a46083c57bf9)


		In the diagram above, H2 host mac address H2-MAC is advertised by Vtep-1 and/or Vtep-4 in a EVPN Type-2 route that also carries Ethernet-Segment identifier (ES-ID) of PortChannel1. On Vtep-5, the H2-MAC processing undergoes special handling since the Type-2 route has ES-ID. And H2-MAC is installed against L2 NHG corresponding to the ES-ID. Even if Type-2 route for H2-MAC is received only from either Vtep-1 or Vtep-4, the traffic towards H2 will still be load-balanced between Vtep-1 and Vtep-4. Later, if PortChannel1 interface goes down, on say Vtep-1, Vtep-1 will withdraw the Type-1 (AD-per-ES) route and this will result in Vtep-5 updating L2 NHG for the ES-ID. This re-balance of traffic will not wait for the individual MAC route updates to arrived and get processed. This update of L2 NHG on Multihomed ES link failures is referred as "Fast Convergence" in RFC 7432.

EVPN VxLAN Multihoming HLD #1622

EVPN VxLAN Multihoming HLD #1622

Conversation

hasan-brcm commented Feb 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sathk-marvell Apr 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hasan-brcm Mar 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adyeung commented Jun 26, 2024

sathk-marvell Apr 9, 2024 •

edited

Loading

hasan-brcm Mar 28, 2024 •

edited

Loading