BESS WorkGroup

Internet Engineering Task Force (IETF)                        A. Sajassi
Internet-Draft
Request for Comments: 9784                                  P. Brissette
Intended status:
Category: Standards Track                                  Cisco Systems
Expires: 12 June 2025
ISSN: 2070-1721                                                R. Schell
                                                                 Verizon
                                                                J. Drake
                                                                 Juniper
                                                              J. Rabadan
                                                                   Nokia
                                                         9 December 2024
                                                                May 2025

                     EVPN Virtual Ethernet Segment
              draft-ietf-bess-evpn-virtual-eth-segment-19 Segments

Abstract

   Ethernet VPN (EVPN) and Provider Backbone Bridge EVPN (PBB-EVPN)
   introduce a comprehensive suite of solutions for delivering Ethernet
   services over MPLS/IP networks.  These solutions offer advanced
   multi-homing capabilities.  Specifically, they support Single-Active
   and All-
   Active All-Active redundancy modes for an Ethernet Segment (ES), which
   is defined as a collection of physical links connecting a multi-homed
   device or network to a set of Provider Edge (PE) devices.  This
   document extends the concept of an Ethernet Segment by allowing an ES
   to be associated with a set of Ethernet Virtual Circuits (EVCs, (EVCs), such
   as VLANs) VLANs, or other entities, including MPLS Label Switched Paths
   (LSPs) or Pseudowires pseudowires (PWs).  This extended concept is referred to as
   virtual Ethernet Segments (vES). (vESes).  This draft list document lists the
   requirements and specifies the necessary extensions to support vES in
   both EVPN and PBB-EVPN.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents an Internet Standards Track document.

   This document is a product of the Internet Engineering Task Force
   (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list  It represents the consensus of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid the IETF community.  It has
   received public review and has been approved for a maximum publication by the
   Internet Engineering Steering Group (IESG).  Further information on
   Internet Standards is available in Section 2 of six months RFC 7841.

   Information about the current status of this document, any errata,
   and how to provide feedback on it may be updated, replaced, or obsoleted by other documents obtained at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 12 June 2025.
   https://www.rfc-editor.org/info/rfc9784.

Copyright Notice

   Copyright (c) 2024 2025 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info)
   (https://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Revised BSD License text as described in Section 4.e of the
   Trust Legal Provisions and are provided without warranty as described
   in the Revised BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
     1.1.  Requirements Language . . . . . . . . . . . . . . . . . .   3
     1.2.  vESes in Access Ethernet Networks . . . . . . . . . . . .   3
     1.3.  vESes in Access MPLS Networks . . . . . . . . . . . . . .   5
   2.  Terminology . . . . . . . . . . . . . . . . . . . . . . . . .   6
   3.  Requirements  . . . . . . . . . . . . . . . . . . . . . . . .   7
     3.1.  Single-Homed and Multi-Homed vES  . . . . . . . . . . . .   8
     3.2.  Local Switching . . . . . . . . . . . . . . . . . . . . .   8
     3.3.  EVC Service Types . . . . . . . . . . . . . . . . . . . .   8
     3.4.  Designated Forwarder (DF) Election  . . . . . . . . . . .   9
     3.5.  EVC Monitoring  . . . . . . . . . . . . . . . . . . . . .   9
     3.6.  Failure and Recovery  . . . . . . . . . . . . . . . . . .  10
     3.7.  Fast Convergence  . . . . . . . . . . . . . . . . . . . .  10
   4.  Solution Overview . . . . . . . . . . . . . . . . . . . . . .  11
     4.1.  EVPN DF Election for vES  . . . . . . . . . . . . . . . .  11
     4.2.  Grouping and Route Coloring for vES . . . . . . . . . . .  13
       4.2.1.  EVPN Route Coloring for vES . . . . . . . . . . . . .  13
       4.2.2.  PBB-EVPN Route Coloring for vES . . . . . . . . . . .  14
   5.  Failure Handling and Recovery . . . . . . . . . . . . . . . .  14
     5.1.  EVC Failure Handling for Single-Active vES in EVPN  . . .  16
     5.2.  EVC Failure Handling for a Single-Active vES in PBB-EVPN  .  16
     5.3.  Port Failure Handling for Single-Active vESes in EVPN . .  17
     5.4.  Port Failure Handling for Single-Active vESes in PBB-EVPN  . . . . . . . . . . . . . . . . . . . . . . . .  18
     5.5.  Fast Convergence in (PBB-)EVPN  . . . . . . . . . . . . .  19 EVPN and PBB-EVPN
   6.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .  21
   7.  Security Considerations . . . . . . . . . . . . . . . . . . .  21
   8.
   7.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  21
   9.
   8.  References  . . . . . . . . . . . . . . . . . . . . . . . . .  21
     9.1.
     8.1.  Normative References  . . . . . . . . . . . . . . . . . .  21
     9.2.
     8.2.  Informative References  . . . . . . . . . . . . . . . . .  22
   Acknowledgements
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  23

1.  Introduction

   Ethernet VPN (EVPN, [RFC7432]) (EVPN) [RFC7432] and Provider Backbone Bridge EVPN (PBB-EVPN,
   [RFC7623])) (PBB-
   EVPN) [RFC7623] introduce a comprehensive suite of solutions for
   delivering Ethernet services over MPLS/IP networks.  These solutions
   offer advanced multi-homing capabilities.  Specifically, they support
   Single-Active and All-Active redundancy modes for an Ethernet Segment
   (ES).  As defined in [RFC7432], an Ethernet Segment (ES) ES represents a collection of
   Ethernet links that connect a customer site to one or more PEs Provider
   Edge (PE) devices.

   This document extends the concept of an Ethernet Segment by allowing
   an ES to be associated with a set of Ethernet Virtual Circuits (EVCs,
   such (EVCs)
   (such as VLANs) or other entities, including MPLS Label Switched
   Paths (LSPs) or Pseudowires pseudowires (PWs).  This extended concept is referred
   to as virtual Ethernet Segments (vES). (vESes).  This draft document lists the
   requirements and specifies the necessary extensions to support vES in
   both EVPN and PBB-EVPN.  The scope of this document includes PBB-EVPN
   [RFC7623], EVPN over MPLS [RFC7432], and EVPN over IP [RFC8365].
   However, [RFC8365];
   however, it excludes EVPN over SRv6 [RFC9252].

1.1.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in
   BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

1.2.  vESes in Access Ethernet Networks

   Some Service Providers service providers (SPs) seek to extend the concept of physical
   Ethernet links in an ES to encompass Ethernet Virtual Circuits
   (EVCs), EVCs, wherein multiple EVCs
   (such as VLANs) can be aggregated onto a single physical External Network-to-Network
   Network-Network Interface (ENNI).  An ES composed of a set of EVCs
   rather than physical links is referred to as a vES.  Figure 1
   illustrates two PE devices (PE1 and PE2), each with an ENNI
   aggregating several EVCs.  Some of these EVCs on a given ENNI can be
   associated with vESes.  For instance, the multi-homed vES depicted in
   Figure 1 consists of EVC4 on ENNI1 and EVC5 on ENNI2.

                     3rd Party

                     Third-Party
       +-----+       EAP
       | CE11|EVC1  +---------+
       +-----+   \  |         |       +---+
       Cust. A    \-0=========0--ENNI1|   |
       +-----+      |         |  ENNI1|   |   +-------+   +---+
       | CE12|EVC2--0=========0--ENNI1|PE1|---|       |   |   |
       +-----+      |         |  ENNI1|   |   |SP     |---|PE3|-
                    |       ==0--ENNI1|   |   |IP/MPLS|   |   | \  +---+
       +-----+      |      /  |       +---+   |Core   |   +---+  \-|   |
       | CE22|EVC3--0==== /   |               |Network|            |CE4|
       +-----+      |    X    |               |       |   +---+    |   |
       Cust. B      |   / \   |       +---+   |       |   |   |  /-|   |
       +-----+     -0===   ===0--ENNI2|   |   |       |---|PE4|-/  +---+
       | CE3 |EVC4/ |         |  ENNI2|PE2|---|       |   |   |
       |     |EVC5--0=========0--ENNI2|   |   +-------+   +---+
       +-----+      |         |       +---+
       Cust. C      +---------+   /\
              /\                  ||
              ||                  ENNI
              EVCs             Interface
       <--------802.1Q---------->  <---- EVPN Network -----> <-802.1Q->

      Figure 1: Dual-homed A Dual-Homed Device/Network (both (Both SA/AA) and SH on same the
                                 Same ENNI

   ENNIs are commonly used to reach remote customer sites via
   independent Ethernet access networks or third- party third-party Ethernet Access
   Providers (EAP). (EAPs).  ENNIs can aggregate traffic from many vESes (e.g.,
   hundreds to thousands), where each vES is represented by its
   associated EVC on that ENNI.  As a result, ENNIs and their associated
   EVCs are a key element elements of SP external boundaries that are carefully
   designed and closely monitored.  As a reminder, the ENNI is the
   demarcation between the SP (IP/MPLS Core Network) core network) and the third-party
   Ethernet Access Provider.

   To meet customers' Service Level Agreements (SLA), (SLAs), SPs build
   redundancy via multiple EVPN PEs and across multiple ENNIs (as shown
   in Figure 1) 1), where a given vES can be multi-homed to two or more
   EVPN PE devices (on two or more ENNIs) via their associated EVCs.
   Just like physical ESs in the solutions described in [RFC7432] and [RFC7623] solutions,
   [RFC7623], these vESes can be single-homed or multi-homed ESs ESs, and
   when multi-homed, then they can operate in either Single-Active or All-Active All-
   Active redundancy modes.  In a typical SP external-boundary scenario
   (e.g., with an EAP), an ENNI can be associated with several thousands
   of single-homed vESes, several hundreds of Single- Active vESes Single-Active vESes, and it may also be
   associated with
   tens or hundreds of All-Active vESes.  The specific figures (hundreds, thousands, etc.) used
   throughout this document reflect the relative quantities (hundreds,
   thousands, etc.) of various elements as understood at the time of
   writing.

1.3.  vESes in Access MPLS Networks

   Other Service Providers (SPs) SPs want to extend the concept of the physical links in an ES to
   individual Pseudowires (PWs) PWs or to MPLS
   Label Switched Paths (LSPs) LSPs in Access MPLS networks - networks, i.e., a vES
   consisting of a set of PWs or a set of LSPs.  Figure 2 illustrates
   this concept.

                    MPLS Aggregation
                    Network
      +-----+      +-----------------+
      | CE11|EVC1  |                 |
      +-----+   \ +AG1-+  PW1      +-+---+
      Cust. A    -0----|===========|     |
      +-----+     | ---+===========|     |   +-------+   +---+
      | CE12|EVC2-0/   |  PW2   /\ | PE1 +---+       |   |   |
      +-----+     ++---+      /=||=|     |   |       +---+PE3+-
                   |         //=||=|     |   |IP/MPLS|   |   | \  +---+
                   |        //  \/ +-+---+   |Core   |   +---+  \-+   |
      +-----+EVC3  |    PW3//  LSP1  |       |Network|            |CE4|
      | CE13|    \+AG2-+==//         |       |       |   +---+    |   |
      +-----+     0    |==/PW4  /\ +-+---+   |       |   |   |  /-+   |
                  0    |==PW5===||=|     |   |       +---+PE4+-/  +---+
      +-----+    /++---+==PW6===||=| PE2 +---+       |   |   |
      | CE14|EVC4  |            \/ |     |   +-------+   +---+
      +-----+      |           LSP2+-+---+
      Cust. C      +-----------------+
             /\
             ||
             EVCs
      <--802.1Q--> <-----MPLS Agg----> <--- EVPN Network ---> <-802.1Q->

          Figure 2: A Dual-Homed and Single-homed Single-Homed Network on MPLS
                            Aggregation networks Networks

   In certain scenarios, Service Providers SPs utilize MPLS Aggregation Networks that are
   managed by separate administrative entities or third-party
   organizations to gain access to their own IP/MPLS core network
   infrastructure.  This situation is depicted in Figure 2.

   In such scenarios, a vES is defined as a set of individual PWs when
   aggregation is not feasible.  If aggregation is possible, the vES can
   be associated with a group of PWs that share the same unidirectional
   LSP pair, where the LSP pair consists of the ingress and egress LSPs
   between the same endpoints.

   In the example of Figure 2, EVC3 is connected to a VPWS instance in AG2 that is
   connected to PE1 and PE2 via PW3 and PW5 PW5, respectively.  EVC4 is
   connected to another VPWS instance on AG2 that is connected to PE1
   and PE2 via PW4 and PW6, respectively.  Since the PWs for the two
   VPWS instances can be aggregated into the same LSP pair going to and
   coming from the MPLS network, a common vES can be defined for the
   four mentioned PWs.  In Figure 2, LSP1 and LSP2 represent the two LSP
   pairs between PE1 and AG2, AG2 and between PE2 and AG2, respectively.  The
   vES consists of these two LSP pairs (LSP1 and LSP2) LSP2), and each LSP
   pair has two PWs.  This vES will be shared by two separate EVPN
   instances (e.g., EVI-1 and EVI-2) in the EVPN network.  PW3 and PW4
   are associated with EVI-1 and EVI-2 respectively EVI-2, respectively, on PE1, and PW5
   and PW6 are associated with EVI-1 and EVI-2 respectively EVI-2, respectively, on PE2.

   In some cases, the aggregation of PWs that share the same LSP pair
   may not be possible.  For instance, if PW3 were terminated into a
   third PE, e.g. e.g., PE3, instead of PE1, the vES would need to be defined
   on a per individual PW on each PE.

   For MPLS/IP access networks where a vES represents a set of LSP pairs
   or a set of PWs, this document extends the Single-Active multi-homing
   procedures defined in [RFC7432] and [RFC7623] to accommodate vES.
   The extension of vES to support All-Active multi-homing in MPLS/IP
   access networks is beyond the scope of this document.

   This draft document defines the concept of a vES and specifies the
   additional extensions necessary to support a vES in accordance with
   [RFC7432] and [RFC7623].  Section 3 enumerates the set of
   requirements for a vES.  Section 4 details the extensions for a vES
   applicable to EVPN solutions, including those specified in [RFC7432]
   and [RFC7209].  These extensions are designed to meet the
   requirements listed in Section 3.  Section 4 also provides an
   overview of the solution, while Section 5 addresses failure handling,
   recovery, scalability, and fast convergence of [RFC7432] and
   [RFC7623] for vESes.

2.  Terminology

   AC:       Attachment Circuit

   B-MAC:    Backbone MAC Address

   CE:       Customer Edge Device

   C-MAC:    Customer/Client MAC Address

   DF:       Designated Forwarder

   ENNI:     External Network-Network Interface

   ES:       Ethernet Segment

   ESI:      Ethernet Segment Identifier

   Ethernet A-D:  Ethernet Auto-Discovery Route

   EVC:      Ethernet Virtual Circuit, Circuit [MEF63]

   EVI:      EVPN Instance

   EVPN:     Ethernet VPN

   I-SID:    Service Instance Identifier (24 bits and global within a
             PBB network network; see [RFC7080]) [RFC7080]).

   MAC:      Media Access Control

   PBB:      Provider Backbone Bridge

   PBB-EVPN:  Provider Backbone Bridge EVPN

   PE:       Provider Edge Device

   VPWS:     Virtual Pseudowire Private Wire Service

   Single-Active (SA) Redundancy Mode (SA): Mode:  When only a single PE, among a
             group of PEs attached to an Ethernet Segment, is allowed to
             forward traffic to/from that Ethernet Segment, then the Ethernet
             Segment is defined to be as operating in Single-
             Active Single-Active redundancy
             mode.

   All-Active (AA) Redundancy Mode (AA): Mode:  When all PEs attached to an
             Ethernet segment, Segment are allowed to forward traffic to/from
             that Ethernet Segment, then the Ethernet Segment is defined
             to be as
             operating in All-Active redundancy mode.

3.  Requirements

   This section describes the requirements specific to virtual Ethernet
   Segment (vES) vES for (PBB-)EVPN EVPN and
   PBB-EVPN solutions.  These requirements are in addition to the ones
   described in [RFC8214], [RFC7432], and [RFC7623].

3.1.  Single-Homed and Multi-Homed vES

   A PE device MUST support the following types of virtual Ethernet
   Segments (vES): vESes:

   (R1a)  The PE MUST handle single-homed vESes on a single physical
          port, such as a single ENNI.

   (R1b)  The PE MUST support a combination of single-homed vESes and
          Single-Active multi-homed vESes simultaneously on a single
          physical port, such as a single ENNI.  Throughout this
          document, Single-Active multi-homed vESes will be referred to
          as Single-Active vESes. "Single-Active vESes".

   (R1c)  The PE MAY support All-Active multi-homed vESes on a single
          physical port.  Throughout this document, All-Active multi-homed multi-
          homed vESes will be referred to as All-Active vESes. "All-Active vESes".

   (R1d)  The PE MAY support a combination of All-Active vESes along
          with other types of vESes on a single physical port.

   (R1e)  A Multi-Homed multi-homed vES, whether Single-Active or All-Active, can
          span across two or more ENNIs on any two or more PEs.

3.2.  Local Switching

   Many vESes of different types can be aggregated on a single physical
   port on a PE device and some of these vESes can belong to the same
   service instance (e.g., EVI).  This translates into the need for
   supporting local switching among the vESes for the same service
   instance on the same physical port (e.g., ENNI) of the PE.

   (R3a)  A PE device that supports the vES function MUST support local
          switching among different vESes associated with the same
          service instance on a single physical port.  For instance, in
          Figure 1, PE1 must support local switching between CE11 and
          CE12, which are mapped to two single-homed vESes on ENNI1.  In
          the case of Single-Active vESes, the local switching is
          performed among active EVCs associated with the same service
          instance on the same ENNI.

3.3.  EVC Service Types

   A physical port, such as an ENNI of a PE device, can aggregate
   numerous EVCs, each associated with a vES.  An EVC may carry one or
   more VLANs.  Typically, an EVC carries a single VLAN and is therefore
   associated with a single broadcast domain.  However, there are no
   restrictions preventing an EVC from carrying multiple VLANs.

   (R4a)  An EVC can be associated with a single broadcast domain, such
          as in a VLAN-based service or a VLAN bundle service.

   (R4b)  An EVC MAY be associated with several broadcast domains, such
          as in a VLAN-aware bundle service.

   Similarly, a PE can aggregate multiple LSPs and PWs.  In the case of
   individual PWs per vES, typically, a PW is typically associated with a single
   broadcast domain, although there are no restrictions preventing a PW
   from carrying multiple VLANs if the PW is configured in Raw mode.

   (R4c)  A PW can be associated with a single broadcast domain, such as
          in a VLAN-based service or a VLAN bundle service.

   (R4d) An  A PW MAY be associated with several broadcast domains, such as
          in a VLAN-aware bundle service.

3.4.  Designated Forwarder (DF) Election

   Section 8.5 of [RFC7432] specifies the default procedure for DF
   election in EVPN, which is also applied in [RFC7623] and [RFC8214].
   [RFC8584] elaborates on additional procedures for DF election in
   EVPN.  These DF election procedures are performed at the granularity
   of (ESI, Ethernet Tag).  In the context of a vES, the same EVPN
   default procedure for DF election is applicable, applicable but at the
   granularity of (vESI, Ethernet Tag).  In this context, the Ethernet
   Tag is represented by an I-SID in PBB-EVPN and by a VLAN ID (VID) in
   EVPN.  As described in [RFC7432], this default procedure for DF
   election at the granularity of (vESI, Ethernet Tag) is also known as
   "service carving."  The goal of service carving is to evenly
   distribute the DFs for different vESes among various PEs, thereby
   ensuring an even distribution of traffic across the PEs.  The
   following requirements are applicable to the DF election of vESes for
   (PBB-)EVPN.
   EVPN and PBB-EVPN.

   (R5a)  A PE that supports vES function, function MUST support a vES with m EVCs
          among n ENNIs belonging to p PEs in any arbitrary order; order, where
          n >= p >= m >=2.  For example, if there is a vES with 2 EVCs
          and there are 5 ENNIs on 5 PEs (PE1 through PE5), then vES can
          be dual homed dual-homed to PE2 and PE4 PE4, and the DF election must be
          performed between PE2 and PE4.

   (Rbc)

   (R5b)  Each vES MUST be identified by its own virtual ESI (vESI).

3.5.  EVC Monitoring

   To detect the failure of an individual EVC and subsequently perform
   DF election for its associated vES as a result of this failure, each
   EVC should be monitored independently.

   (R6a)  Each EVC SHOULD be independently monitored for its operational
          health.

   (R6b)  A failure in a single EVC, among many aggregated on a single
          physical port or ENNI, MUST trigger a DF election for its
          associated vES.

3.6.  Failure and Recovery

   (R7a)  Failure and failure recovery of an EVC for a Single-homed single-homed vES
          SHALL NOT impact any other EVCs within its service instance or
          any other service instances.  In other words, for PBB-EVPN, it
          SHALL NOT trigger any MAC flushing both within both its own I-SID as well as
          and other I-SIDs.

   (R7b)  In case of All-Active vES, failure and failure recovery of an
          EVC for that vES SHALL NOT impact any other EVCs within its
          service instance or any other service instances.  In other
          words, for PBB-
   EVPN, PBB-EVPN, it SHALL NOT trigger any MAC flushing both
          within both its own I-SID
   as well as and other I-SIDs.

   (R7c)  Failure and failure recovery of an EVC for a Single-Active vES
          SHALL impact only its own service instance.  In other words,
          for PBB-
   EVPN, PBB-EVPN, MAC flushing SHALL be limited to the associated
          I-SID only and SHALL NOT impact any other I-SIDs.

   (R7d)  Failure and failure recovery of an EVC for a Single-Active vES
          MUST only impact C-MACs associated with a multi-homed device/network device/
          network for that service instance.  In other words, MAC
          flushing MUST be limited to a single service instance (I-SID
          in the case of PBB-EVPN) and only C-MACs for a Single-Active
          multi-homed device/network.

3.7.  Fast Convergence

   Since many EVCs (and their associated vESes) are aggregated via a
   single physical port (e.g., ENNI), then the when there is a failure of that
   physical
   port port, it impacts many vESes and triggers equally triggers many ES
   route withdrawals.  Formulating, sending, receiving, and processing
   such large number numbers of BGP messages can introduce delay in DF election
   and convergence time.  As such, it is highly desirable to have a
   mass-withdraw mechanism similar to the one in [RFC7432] for
   withdrawing many Ethernet A-D per ES routes.

   (R8a)  There SHOULD be a mechanism equivalent to EVPN mass-withdraw mass withdraw
          such that upon an ENNI failure, only a single BGP message is needed
   to indicate to
          the remote PEs is needed to trigger DF election for all impacted
   vES
          vESes associated with that ENNI.

4.  Solution Overview

   The solutions described in [RFC7432] and [RFC7623] are leveraged
   as-is as
   is, with the modification that the ESI assignment is performed for an
   EVC or a group of EVCs or LSPs/PWs instead of a link or a group of
   physical links.  In other words, the ESI is associated with a virtual
   ES (vES), hereby vES
   (hereby referred to as vESI. the "vESI").

   In the EVPN solution, the overall procedures remain consistent, with
   the primary difference being the handling of physical port failures
   that can affect multiple vESes.  Sections 5.1 and 5.3 describe the
   procedures for managing physical port or link failures in the context
   of EVPN.  In a typical multi-homed setup, MAC addresses learned
   behind a vES are advertised using the ESI associated with the vES,
   referred to as vES
   (i.e., the vESI. vESI).  EVPN aliasing and mass-withdraw operations are
   conducted with respect to the vES identifier.  Specifically, the
   Ethernet Auto-Discovery (A-D) A-D routes for these operations are advertised using the
   vESI instead of the ESI.

   For the PBB-EVPN solution, the main change is with respect to the
   B-MAC address assignment assignment, which is performed in a similar way to what
   is described in
   section Section 7.2.1.1 of [RFC7623] [RFC7623], with the following
   refinements:

   *  One shared B-MAC address SHOULD be used per PE for the
      single-homed single-
      homed vESes.  In other words, a single B-MAC is shared for all
      single-homed vESes on that PE.

   *  One shared B-MAC address SHOULD be used per PE PE, per physical port
      (e.g., ENNI) for the Single-Active vESes.  In other words, a
      single B-MAC is shared for all Single-Active vESes that share the
      same ENNI.

   *  One shared B-MAC address MAY be used for all Single-Active vESes
      on that PE.

   *  One B-MAC address SHOULD be used per set of EVCs representing an
      All-Active vES.  In other words, a single B-MAC address is used
      per vES for All-Active scenarios.

   *  A single B-MAC address MAY also be used per vES vES, per PE for Single-
      Active
      Single-Active scenarios.

4.1.  EVPN DF Election for vES

   The procedure for service carving procedures for vESes is are almost the same as the
   ones
   procedures outlined in section Section 8.5 of [RFC7432] and [RFC8584] in [RFC8584],
   except for
   the fact that ES is replaced with vES.

   For the sake of clarity and completeness, the default DF election
   procedure of [RFC7432] is repeated below with the necessary changes:

   1.  When a PE discovers the vESI or is configured with the vESI
       associated with its attached vES, it advertises an Ethernet
       Segment route with the associated ES-Import extended community
       attribute.

   2.  The PE then starts a timer (default value = 3 seconds) to allow
       the reception of Ethernet Segment routes from other PE nodes
       connected to the same vES.  This timer value MUST be the same
       across all PEs connected to the same vES.

   3.  When the timer expires, each PE builds an ordered list of the IP
       addresses of all the PE nodes connected to the vES (including
       itself), in increasing numeric value.  Each IP address in this
       list is extracted from the "Originator Router's IP address" field
       of the advertised Ethernet Segment route.  Every PE is then given
       an ordinal indicating its position in the ordered list, starting
       with 0 as the ordinal for the PE with the numerically lowest IP
       address.  The ordinals are used to determine which PE node will
       be the DF for a given EVPN instance on the vES using the
       following rule: Assuming a redundancy group of N PE nodes, the PE
       with ordinal i is the DF for an EVPN instance with an associated
       Ethernet Tag value of V when (V mod N) = i.  It should be noted
       that using the "Originator Router's IP address" field in the
       Ethernet Segment route to get the PE IP address needed for the
       ordered
       list, list allows for a CE to be multi-homed across different ASes ASes,
       if such need ever arises.

   4.  The PE that is elected as a DF for a given EVPN instance will
       unblock traffic for that EVPN instance.  Note that the DF PE
       unblocks all traffic in both ingress and egress directions for
       Single-Active vES vESes and unblocks multi-destination in the egress
       direction for All-Active Multi-homed vES. multi-homed vESes.  All non-DF PEs block
       all traffic in both ingress and egress directions for Single-
       Active vES vESes and block multi-destination traffic in the egress
       direction for All-Active vES. vESes.

   In case of an EVC failure, the affected PE withdraws its
   corresponding Ethernet Segment route if there are no more EVCs
   associated to the vES in the PE.  This will re-trigger the DF
   Election
   election procedure on all the PEs in the Redundancy Group. redundancy group.  For PE
   node failure, or upon PE commissioning or decommissioning, the PEs
   re-trigger the DF Election election procedure across all affected vESes.  In
   case of a Single-Active, when a service moves from one PE in the
   Redundancy Group
   redundancy group to another PE because of DF re-election, the PE,
   which PE
   (which ends up being the elected DF for the service, service) MUST trigger a
   MAC address flush notification towards the associated vES if the
   multi-homing device is a bridge or the multi-homing network is an
   Ethernet bridged network.

   For LSP-based and PW-based vES, the non-DF PE SHOULD signal PW-status
   'standby' to the Aggregation PE (e.g., AG1 and AG2 in Figure 2), and
   a new DF PE MAY send an LDP a Label Distribution Protocol (LDP) MAC withdraw
   message as a MAC address flush notification.  It should be noted that
   the PW-status is signaled for the scenarios where there is a one-to-one one-to-
   one mapping between EVI (EVPN instance) and the PW.

4.2.  Grouping and Route Coloring for vES

   Physical ports (e.g. (e.g., ENNI) which that aggregate many EVCs are 'colored' to
   enable the grouping schemes described below.

   By default, the MAC address of the corresponding port (e.g. (e.g., ENNI) is
   used to represent the 'color' of the port, and the EVPN Router's MAC
   Extended Community defined in [RFC9135] is used to signal this color.

   The difference between coloring mechanism mechanisms for EVPN and PBB-EVPN is
   that for EVPN, the extended community is advertised with the Ethernet A-D per
   ES route whereas for PBB-EVPN, EVPN, whereas the extended community is advertised with
   the B-MAC route. route for PBB-EVPN.

   The subsequent sections detailing Grouping of Ethernet Auto-Discovery
   (A-D) A-D per ES and
   Grouping of B-MAC addresses will be essential for addressing port
   failure handling, as discussed in Sections
   Section 5.3, Section 5.4, and Section 5.5.

4.2.1.  EVPN Route Coloring for vES

   When a PE discovers the vESI or is configured with the vESI
   associated with its attached vES, an Ethernet-Segment Ethernet Segment route and
   Ethernet A-D per ES route are generated using the vESI identifier.

   These Ethernet-Segment Ethernet Segment and Ethernet A-D per ES routes specific to
   each vES are colored with an attribute representing their association
   to a physical port (e.g. (e.g., ENNI).

   The corresponding port 'color' is encoded in the EVPN Router's MAC
   Extended Community defined in [RFC9135] and advertised along with the
   Ethernet Segment and Ethernet A-D per ES routes for this vES.  The
   color (which is the MAC address of the port) MUST be unique.

   The PE also constructs a special Grouping Ethernet A-D per ES route
   which
   that represents all the vES vESes associated with the port (e.g. (e.g., ENNI).
   The corresponding port 'color' is encoded in the ESI field.  For this
   encoding, Type 3 ESI (Section 5 of [RFC7432]) is used with the MAC
   field set to the color (MAC address) of the port and the 3-octet
   local discriminator field set to 0xFFFFFF.

   The ESI label extended community (Section 7.5 of [RFC7432]) is not
   relevant to Grouping Ethernet A-D per ES route.  The label value is
   not used for encapsulating BUM (Broadcast, Unknown-unicast,
   Multicast) Broadcast, Unknown Unicast, and Multicast
   (BUM) packets for any split-horizon function.  The ESI label extended
   community MUST NOT be added to Grouping Ethernet A-D per ES route and
   MUST be ignored on receiving the PE.

   The Grouping Ethernet Auto-Discovery (A-D) A-D per ES route is advertised with a list of
   Route Targets corresponding to the affected service instances.  If
   the number of associated Route Targets exceeds the capacity of a
   single route, multiple Grouping Ethernet A-D per ES routes are
   advertised accordingly as specified in Section 8.2 of [RFC7432].

4.2.2.  PBB-EVPN Route Coloring for vES

   In PBB-EVPN, particularly when there are a large number numbers of service
   instances (i.e., I-SIDs) associated with each EVC, the PE device MAY
   assign a color attribute to each vES B-MAC route, indicating their
   association with a physical port (e.g., an ENNI).

   The corresponding port 'color' is encoded in the EVPN Router's MAC
   Extended Community defined in [RFC9135] and advertised along with the
   B-MAC for this vES in PBB-EVPN.

   The PE MAY then also construct a special Grouping B-MAC route which that
   represents all the vES vESes associated with the port (e.g. (e.g., ENNI).  The
   corresponding port 'color' is encoded directly into this special
   Grouping B-MAC route.

5.  Failure Handling and Recovery

   There are several failure scenarios to consider such as:

   A:  CE uplink port failure

   B:  Ethernet Access Network failure

   C:  PE access-facing port or link failure

   D:  PE node failure

   E:  PE isolation from IP/MPLS network

   The solutions specified in [RFC7432], [RFC7623], and [RFC8214]
   provide protection against failures as described in these respective
   references.  In the context of these solutions, the presence of vESes
   introduces an additional failure scenario beyond those already
   considered, specifically the failure of individual EVCs.  Addressing
   vES failure scenarios necessitates the independent monitoring of EVCs
   or PWs.  Upon detection of failure or service restoration,
   appropriate DF election and failure recovery mechanisms must be
   executed.

   [RFC7023] is used for monitoring EVCs EVCs, and upon failure detection of
   a given EVC, the DF election procedure per Section 4.1 is executed.
   For PBB-EVPN, some extensions are needed to handle the failure and
   recovery procedures of [RFC7623] to meet the above requirements.
   These extensions are described in the next section.

   [RFC4377] and [RFC6310] are used for monitoring the status of LSPs
   and/or PWs associated to vES.

                         B            D
                         ||           ||
                         \/           \/
                       +-----+
          +-----+      |     |       +---+
          | CE1 |EVC2--0=====0--ENNI1|   |   +-------+
          +-----+      |    =0--ENNI1|PE1|---|       |  +---+  +---+
          Cust. A      |   / |       |   |   |IP/MPLS|--|PE3|--|CE4|
          +-----+      |  /  |       +---+   |Network|  |   |  +---+
          |     |EVC2--0==   |               |       |  +---+
          | CE2 |      |     |       +---+   |       |
          |     |EVC3--0=====0--ENNI2|PE2|---|       |
          +-----+      |     |       |   |   +-------+
                       +-----+       +---+
                 /\                /\     /\
                 ||                ||     ||
                 A                 C      E

               Figure 4: 3: Failure Scenarios A,B,C,D A, B, C, D, and E

5.1.  EVC Failure Handling for Single-Active vES in EVPN

   In [RFC7432], when a DF PE connected to a Single-Active multi-homed
   Ethernet Segment loses connectivity to the segment, due to link or
   port failure, it signals to the remote PEs to invalidate all MAC
   addresses associated with that Ethernet Segment.  This is done by
   means of a mass-withdraw message, by withdrawing the Ethernet A-D per
   ES route.  It should be noted that for dual-homing use cases where
   there is only a single backup path, MAC invalidating can be avoided
   by the remote PEs as they can update their next hop associated with
   the affected MAC entries to the backup path per the procedure
   described in section Section 8.2 of [RFC7432].

   In case of an EVC failure which that impacts a single vES, this same EVPN
   procedure is used.  In this case, the mass-withdraw mass withdraw is conveyed by
   withdrawing the Ethernet A-D per vES route carrying the vESI
   representing the failed EVC.  The remote PEs upon  Upon receiving this
   message message, the remote
   PEs perform the same procedures outlined in section Section 8.2 of [RFC7432].

5.2.  EVC Failure Handling for a Single-Active vES in PBB-EVPN

   In [RFC7432] [RFC7432], when a PE connected to a Single-Active Ethernet Segment
   loses connectivity to the segment, due to link or port failure, it
   signals the remote PE to flush all C-MAC addresses associated with
   that Ethernet Segment.  This is done by updating the advertised a B-MAC
   route's MAC Mobility Extended community. Community.

   In case of an EVC failure that impacts a single vES, if the above
   PBB-EVPN procedure is used, it results in excessive C-MAC flushing
   because a single physical port can support a large number of EVCs
   (and their associated vESes) and thus vESes); therefore, updating the advertised
   B-MAC corresponding to the physical port, with MAC mobility Mobility Extended
   community,
   Community, will result in flushing C-MAC addresses not just for the
   impacted EVC but for all other EVCs on that port.

   To reduce the scope of C-MAC flushing to only the impacted service
   instances (the service instance(s) impacted by the EVC failure), the
   PBB-EVPN C-MAC flushing needs to be adapted on a per service instance per-service-instance
   basis (i.e., per I-SID).  [RFC9541] introduces a B-MAC/I-SID route
   where the existing PBB-EVPN B-MAC route is modified to carry an I-SID
   in the "Ethernet Tag ID" field instead of NULL value.  This field
   indicates to  To the
   receiving PE, to flush this field indicates flushing all C-MAC addresses
   associated with that I-SID for that B-MAC.  This C-MAC flushing
   mechanism per I-SID SHOULD be used in case of an EVC failure
   impacting a vES.  Since typically an EVC typically maps to a single broadcast
   domain and
   thus, thus a single service instance, the affected PE only needs
   to advertise a single B-MAC/I-SID route.  However, if the failed EVC
   carries multiple VLANs each with its own broadcast domain, then the
   affected PE needs to advertise multiple B-MAC/I-SID routes - -- one for
   each VLAN (broadcast domain) - -- i.e., one for each I-SID.  Each B-MAC/
   I-SID
   B-MAC/I-SID route basically instructs the remote PEs to perform
   flushing for C-MACs corresponding to the advertised B-MAC only for
   the advertised I-SID.

   The C-MAC flushing based on a B-MAC/I-SID route works fine when there
   are only a few VLANs (e.g., I-SIDs) per EVC.  However  However, if the number
   of I-SIDs associated with a failed EVC is large, then it is
   RECOMMENDED to assign a B-MAC per vES vES, and upon EVC failure, the
   affected PE simply withdraws this B-MAC message to other PEs.

5.3.  Port Failure Handling for Single-Active vESes in EVPN

   When many EVCs are aggregated via a single physical port on a PE,
   where each EVC corresponds to a vES, then the port failure impacts
   all the associated EVCs and their corresponding vESes.  If the number
   of EVCs corresponding to the Single-Active vESes for that physical
   port is in the thousands, then thousands of service instances are
   impacted.  Therefore, the propagation of failure in BGP needs to
   address all these impacted service instances.  In order to achieve
   this, the following extensions are added to the baseline EVPN
   mechanism:

   1.  The PE MAY color each Ethernet A-D per ES route for a given vES,
       as described in Section 4.2.1.  The PE SHOULD use the MAC
       physical port
       MAC by default.  The receiving PEs take note of this
       color and create a list of vESes for this color.

   2.  The PE MAY advertises advertise a special Grouping Ethernet A-D per ES route
       for that color, which represents all the vES vESes associated with
       the port.

   3.  Upon a port failure (e.g., an ENNI failure), the PE MAY send a
       mass-withdraw message by withdrawing the Grouping Ethernet A-D
       per ES route.

   4.  When this message is received, the remote PE MAY detect the
       special vES mass-withdraw message by identifying the Grouping
       Ethernet A-D per ES route.  The remote PEs MAY then access the
       list created in (1) of the vESes for the specified color, color and
       initiate
       locally initiate MAC address invalidating procedures for each of
       the vESes in the list.

   In scenarios where a logical ENNI is used used, the above procedure
   equally applies.  The logical ENNI is represented by a Grouping
   Ethernet A-D per ES where the Type 3 ESI and the 6 bytes used in the
   ENNI's ESI MAC address field is are used as a color for the vESes as
   described above and in Section 4.2.1.

5.4.  Port Failure Handling for Single-Active vESes in PBB-EVPN

   When many EVCs are aggregated via a single physical port on a PE,
   where each EVC corresponds to a vES, then the port failure impacts
   all the associated EVCs and their corresponding vESes.  If the number
   of EVCs corresponding to the Single-Active vESes for that physical
   port is in the thousands, then thousands of service instances
   (I-SIDs) are impacted.  In such failure scenarios, the following two
   MAC flushing mechanisms per [RFC7623] can be performed.

   1.  If the MAC address of the physical port is used for PBB
       encapsulation as B-MAC SA, then upon the port failure, the PE
       MUST use the EVPN MAC route withdrawal message to signal the
       flush.

   2.  If the PE PE's shared MAC address is used for PBB encapsulation as
       B-MAC SA, then upon the port failure, the PE MUST re-advertise
       this MAC route with the MAC Mobility Extended Community to signal
       the flush.

   The first method is recommended because it reduces the scope of
   flushing the most.

   As noted above, the advertisement of the extended community along
   with the B-MAC route for coloring purposes is optional and only
   recommended when there are many vESes per physical port and each vES
   is associated with a very large number of service instances (i.e., a
   large number of I-SIDs).

   If there are large number numbers of service instances (i.e., I-SIDs)
   associated with each EVC, and if there is a B-MAC assigned per vES as
   recommended in the above section, then in order to handle port
   failure efficiently, the following extensions are added to the
   baseline PBB-
   EVPN PBB-EVPN mechanism:

   1.  Each vES MAY be colored with a MAC address representing the
       physical port like the coloring mechanism for EVPN.  In other
       words, each B-MAC representing a vES is advertised with the
       'color' of the physical port per Section 4.2.2.  The receiving
       PEs take note of this color being advertised along with the B-MAC
       route
       route, and for each such color, they create a list of vESes
       associated with this color.

   2.  The PE MAY advertise a special Grouping B-MAC route for that
       color (consisting by default of a port MAC address), address by default), which
       represents all the vES vESes associated with the port.

   3.  Upon a port failure (e.g., ENNI failure), the PE MAY send a
       mass-withdraw mass-
       withdraw message by withdrawing the Grouping B-MAC route.

   4.  When this message is received, the remote PE MAY detect the
       special vES mass-withdraw message by identifying the Grouping
       B-MAC route.  The remote PEs MAY then access the list created in
       (1) above for the specified color, color and flush all C-MACs associated
       with the failed physical port.

5.5.  Fast Convergence in (PBB-)EVPN EVPN and PBB-EVPN

   As described above, when many EVCs are aggregated via a physical port
   on a PE, and where each EVC corresponds to a vES, then the port failure
   impacts all the associated EVCs and their corresponding vESes.  Two
   actions must be taken as the result of such a port failure:

   *  For EVPN EVPN, initiate the mass-withdraw procedure for all vESes
      associated with the failed port to invalidate MACs and for PBB-EVPN PBB-
      EVPN to flush all C-MACs associated with the failed port across
      all vESes and the impacted I-SIDs

   *  Use DF election for all impacted vESes associated with the failed
      port

   Section 5.3 already describes how to perform mass-withdraw a mass withdraw for all
   affected vESes and invalidating invalidate MACs using a single BGP withdrawal of
   the Grouping Ethernet A-D per ES route.  Section 5.4 describes how to
   only flush C-MAC address addresses associated with the failed physical port
   (e.g., optimum C-MAC flushing) as well as, optionally, the withdrawal
   of a Grouping B-MAC route.

   This section describes how to perform DF election in the most optimal
   way -
   way, e.g., to trigger by triggering DF election for all impacted vESes (which
   can be very large) among the participating PEs via a single BGP
   message as opposed to sending a large number of BGP messages (one per
   vES).  This section assumes that the MAC flushing mechanism described
   in Section 5.4 is used and route coloring is are used.

                     +-----+
          +----+     |     |       +---+
          | CE1|AC1--0=====0--ENNI1|   |  +-------+
          |    |AC2--0     |       |PE1|--|       |
          +----+     |\  ==0--ENNI2|   |  |       |
                     | \/  |       +---+  |       |
                     | /\  |              |IP/MPLS|
          +----+     |/  \ |       +---+  |Network|   +---+  +---+
          | CE2|AC4--0    =0--ENNI3|   |  |       |---|PE4|--|CE4|
          |    |AC4--0=====0--ENNI3|PE2|--|       |   +---+  +---+
          +----+     | ====0--ENNI3|   |  |       |
                     |/    |       +---+  |       |
                     0     |              |       |
          +----+    /|     |       +---+  |       |
          | CE3|AC5- |     |       |PE3|--|       |
          |    |AC6--0=====0--ENNI4|   |  +-------+
          +----+     |     |       +---+
                     +-----+

                Figure 5: 4: Fast Convergence Upon ENNI Failure

   As discussed in Section 4.2, it is highly desirable to have a mass mass-
   withdraw mechanism similar to the one in [RFC7432] . [RFC7432].  Although such an
   optimization is desirable, it is OPTIONAL.  If the optimization is
   implemented, the following describes the procedure: procedures are used:

   1.  When a vES is configured, the PE advertises the Ethernet Segment
       route for this vES with a color that corresponds to the
       associated physical port.

   2.  All receiving PEs within the redundancy group record this color
       and compile a list of vESes associated with it.

   3.  Additionally, the PE advertises a Grouping Ethernet A-D per ES
       for EVPN, and a Grouping B-MAC for PBB-EVPN, which corresponds to
       the color and vES grouping.

   4.  In the event of a port failure, such as an ENNI failure, the PE
       withdraws the previously advertised Grouping Ethernet A-D per ES
       or Grouping B-MAC associated with the failed port.  The PE should
       prioritize sending these Grouping route withdrawal messages over
       the withdrawal of individual vES routes affected by the failure.
       For instance, as depicted in Figure 5, 4, when the physical port
       associated with ENNI3 fails on PE2, it withdraws the previously
       advertised Grouping Ethernet A-D per ES route.  Upon receiving
       this withdrawal message, other multi-homing PEs (such as PE1 and
       PE3) recognize that the vESes associated with CE1 and CE3 are
       impacted, based on the associated color, and thus initiate the DF
       election procedure for these vESes.  Furthermore, remote PEs
       (such as PE4), upon receiving
       this withdrawal message, remote PEs (such as PE4) initiate the
       failover procedure for the vESes associated with CE1 and CE3, CE3 and
       switch to the other PE for each vES redundancy group.

   5.  On reception of Grouping Ethernet A-D per ES or Grouping B-MAC
       route withdrawal, other PEs in the redundancy group initiate DF
       election procedures across all their affected vESes.

   6.  The PE with the physical port failure (ENNI failure), failure) sends a vES
       route withdrawal for every impacted vES.  The other PEs upon  Upon receiving these
       messages, the other PEs clear up their BGP tables.  It should be
       noted that the vES route withdrawal messages are not used for
       executing DF election procedures by the receiving PEs when
       Grouping Ethernet A-D per ES or Grouping B-MAC withdrawal has
       been previously received.

6.  Acknowledgements

   The authors would like to thank Mei Zhang, Jose Liste, and
   Luc Andre Burdet for their reviews of this document and feedback.

7.  Security Considerations

   All the security considerations in [RFC7432] and [RFC7623] apply
   directly to this document because this document leverages the control
   and data plane procedures described in those documents.

   This document does not introduce any new security considerations
   beyond that of [RFC7432] and [RFC7623] because advertisements and the
   processing of Ethernet Segment route routes for vES in this document follows follow
   that of physical ES in those RFCs.

8.

7.  IANA Considerations

   This document requests has no actions from IANA.

9. IANA actions.

8.  References

9.1.

8.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC7432]  Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A.,
              Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based
              Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February
              2015, <https://www.rfc-editor.org/info/rfc7432>.

   [RFC7623]  Sajassi, A., Ed., Salam, S., Bitar, N., Isaac, A., and W.
              Henderickx, "Provider Backbone Bridging Combined with
              Ethernet VPN (PBB-EVPN)", RFC 7623, DOI 10.17487/RFC7623,
              September 2015, <https://www.rfc-editor.org/info/rfc7623>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/info/rfc8174>.

   [RFC8214]  Boutros, S., Sajassi, A., Salam, S., Drake, J., and J.
              Rabadan, "Virtual Private Wire Service Support in Ethernet
              VPN", RFC 8214, DOI 10.17487/RFC8214, August 2017,
              <https://www.rfc-editor.org/info/rfc8214>.

   [RFC8365]  Sajassi, A., Ed., Drake, J., Ed., Bitar, N., Shekhar, R.,
              Uttaro, J., and W. Henderickx, "A Network Virtualization
              Overlay Solution Using Ethernet VPN (EVPN)", RFC 8365,
              DOI 10.17487/RFC8365, March 2018,
              <https://www.rfc-editor.org/info/rfc8365>.

   [RFC9135]  Sajassi, A., Salam, S., Thoria, S., Drake, J., and J.
              Rabadan, "Integrated Routing and Bridging in Ethernet VPN
              (EVPN)", RFC 9135, DOI 10.17487/RFC9135, October 2021,
              <https://www.rfc-editor.org/info/rfc9135>.

   [RFC9541]  Rabadan, J., Ed., Sathappan, S., Nagaraj, K., Miyake, M.,
              and T. Matsuda, "Flush Mechanism for Customer MAC
              Addresses Based on Service Instance Identifier (I-SID) in
              Provider Backbone Bridging EVPN (PBB-EVPN)", RFC 9541,
              DOI 10.17487/RFC9541, March 2024,
              <https://www.rfc-editor.org/info/rfc9541>.

9.2.

8.2.  Informative References

   [MEF63]    Metro Ethernet Forum, MEF., "[MEF6.3]: Subscriber "Subscriber Ethernet Services
              Definitions", MEF Standard, MEF 6.3, November 2019.

   [RFC4377]  Nadeau, T., Morrow, M., Swallow, G., Allan, D., and S.
              Matsushima, "Operations and Management (OAM) Requirements
              for Multi-Protocol Label Switched (MPLS) Networks",
              RFC 4377, DOI 10.17487/RFC4377, February 2006,
              <https://www.rfc-editor.org/info/rfc4377>.

   [RFC6310]  Aissaoui, M., Busschbach, P., Martini, L., Morrow, M.,
              Nadeau, T., and Y. Stein, "Pseudowire (PW) Operations,
              Administration, and Maintenance (OAM) Message Mapping",
              RFC 6310, DOI 10.17487/RFC6310, July 2011,
              <https://www.rfc-editor.org/info/rfc6310>.

   [RFC7023]  Mohan, D., Ed., Bitar, N., Ed., Sajassi, A., Ed., DeLord,
              S., Niger, P., and R. Qiu, "MPLS and Ethernet Operations,
              Administration, and Maintenance (OAM) Interworking",
              RFC 7023, DOI 10.17487/RFC7023, October 2013,
              <https://www.rfc-editor.org/info/rfc7023>.

   [RFC7080]  Sajassi, A., Salam, S., Bitar, N., and F. Balus, "Virtual
              Private LAN Service (VPLS) Interoperability with Provider
              Backbone Bridges", RFC 7080, DOI 10.17487/RFC7080,
              December 2013, <https://www.rfc-editor.org/info/rfc7080>.

   [RFC7209]  Sajassi, A., Aggarwal, R., Uttaro, J., Bitar, N.,
              Henderickx, W., and A. Isaac, "Requirements for Ethernet
              VPN (EVPN)", RFC 7209, DOI 10.17487/RFC7209, May 2014,
              <https://www.rfc-editor.org/info/rfc7209>.

   [RFC8584]  Rabadan, J., Ed., Mohanty, S., Ed., Sajassi, A., Drake,
              J., Nagaraj, K., and S. Sathappan, "Framework for Ethernet
              VPN Designated Forwarder Election Extensibility",
              RFC 8584, DOI 10.17487/RFC8584, April 2019,
              <https://www.rfc-editor.org/info/rfc8584>.

   [RFC9252]  Dawra, G., Ed., Talaulikar, K., Ed., Raszuk, R., Decraene,
              B., Zhuang, S., and J. Rabadan, "BGP Overlay Services
              Based on Segment Routing over IPv6 (SRv6)", RFC 9252,
              DOI 10.17487/RFC9252, July 2022,
              <https://www.rfc-editor.org/info/rfc9252>.

Acknowledgements

   The authors would like to thank Mei Zhang, Jose Liste, and Luc André
   Burdet for their reviews of this document and their feedback.

Authors' Addresses

   Ali Sajassi
   Cisco Systems
   Email: sajassi@cisco.com

   Patrice Brissette
   Cisco Systems
   Email: pbrisset@cisco.com

   Rick Schell
   Verizon
   Email: richard.schell@verizon.com

   John E Drake
   Juniper
   Email: jdrake@juniper.net

   Jorge Rabadan
   Nokia
   Email: jorge.rabadan@nokia.com