Benchmarking Methodology (bmwg)
-------------------------------

 Charter
 Last Modified: 2011-02-14

 Current Status: Active Working Group

 Chair(s):
     Al Morton  <acmorton@att.com>

 Operations and Management Area Director(s):
     Dan Romascanu  <dromasca@avaya.com>
     Ronald Bonica  <rbonica@juniper.net>

 Operations and Management Area Advisor:
     Ronald Bonica  <rbonica@juniper.net>

 Mailing Lists: 
     General Discussion:bmwg@ietf.org
     To Subscribe:      bmwg-request@ietf.org
         In Body:       subscribe your_email_address
     Archive:           http://www.ietf.org/mail-archive/web/bmwg/index.html

Description of Working Group:

The Benchmarking Methodology Working Group (BMWG) will continue to 
produce a series of recommendations concerning the key performance 
characteristics of internetworking technologies, or benchmarks for 
network devices, systems, and services. Taking a view of networking 
divided into planes, the scope of work includes benchmarks for the 
management, control, and forwarding planes.

Each recommendation will describe the class of equipment, system, or
service being addressed; discuss the performance characteristics that
are pertinent to that class; clearly identify a set of metrics that aid
in the description of those characteristics; specify the methodologies
required to collect said metrics; and lastly, present the requirements
for the common, unambiguous reporting of benchmarking results.

The set of relevant benchmarks will be developed with input from the 
community of users (e.g, network operators and testing organizations) 
and from those affected by the benchmarks when they are published 
(networking and test equipment manufacturers). When possible, the 
benchmarks and other terminology will be developed jointly with 
organizations that are willing to share their expertise. Joint review 
requirements for a specific work area will be included in the detailed 
description of the task, as listed below.

To better distinguish the BMWG from other measurement initiatives in the
IETF, the scope of the BMWG is limited to the characterization of 
implementations of various internetworking technologies
using controlled stimuli in a laboratory environment. Said differently,
the BMWG does not attempt to produce benchmarks for live, operational
networks. Moreover, the benchmarks produced by this WG shall strive to
be vendor independent or otherwise have universal applicability to a
given technology class.

Because the demands of a particular technology may vary from deployment
to deployment, a specific non-goal of the Working Group is to define
acceptance criteria or performance requirements.

An ongoing task is to provide a forum for discussion regarding the
advancement of measurements designed to provide insight on the 
capabilities and operation of inter-networking technology 
implementations.

The BMWG will communicate with the operations community through 
organizations such as NANOG, RIPE, and APRICOT.

In addition to its current work plan, the BMWG is explicitly tasked to
develop benchmarks and methodologies for the following technologies:

* BGP Control-plane Convergence Methodology (Terminology is complete): 
With relevant performance characteristics identified, BMWG will prepare 
a Benchmarking Methodology Document with review from the Routing Area 
(e.g., the IDR working group and/or the RTG-DIR). The Benchmarking 
Methodology will be Last-Called in all the groups that previously 
provided input, including another round of network operator input during 
the last call. 

* SIP Networking Devices: Develop new terminology and methods to
characterize the key performance aspects of network devices using
SIP, including the signaling plane scale and service rates while
considering load conditions on both the signaling and media planes. This
work will be harmonized with related SIP performance metric definitions
prepared by the PMOL working group.

* Flow Export and Collection: Develop terminology and methods to
characterize network devices flow monitoring, export, and collection.
The goal is a methodology to assess the maximum IP flow rate that a
network device can sustain without losing any IP flow information or
compromising the accuracy of information exported on the IP flows,
and to asses the forwarding plane performance (if the forwarding 
function is present) in the presence of  Flow Monitoring. 

* Data Center Bridging Devices:
Some key concepts from BMWG's past work are not meaningful when testing
switches that implement new IEEE specifications in the area of data 
center bridging. For example, throughput as defined in RFC 1242 cannot 
be measured when testing devices that implement three new IEEE
specifications: priority-based flow control (802.1Qbb); priority groups
(802.1Qaz); and congestion notification (802.1Qau).
Since devices that implement these new congestion-management
specifications should never drop frames, and since the metric of
throughput distinguishes between non-zero and zero drop rates, no
throughput measurement is possible using the existing methodology.
The current emphasis is on the Priority Flow Control aspects of
Data Center Bridging, and the work will include an investigation
into whether TRILL RBridges require any specific treatment in the 
methodology. This work will update RFC 2544 and exchange periodic 
Liaisons with IEEE 802.1 DCB Task Group, especially at WG Last Call.

* Content Aware Devices: 
New classes of network devices that operate above the IP layer of the 
network stack require a new methodology to perform adequate 
benchmarking.  Existing BMWG RFCs (RFC2647 and RFC3511) provides useful 
measurement and performance statistics, though they may not reflect the 
actual performance of the device when deployed in production networks.  
Operating within the limitations of the charter, namely blackbox 
characterization in laboratory environments, the BMWG will develop a 
methodology that more closely relates the performance of these devices 
to performance in an operational setting. In order to confirm or 
identify key performance characteristics, BMWG will solicit input from 
operations groups such as NANOG, RIP and APRICOT.

* LDP Dataplane Convergence:
In order to identify key LDP convergence performance characteristics, 
BMWG will solicit input from operations groups such as NANOG, RIP and 
APRICOT. When relevant performance characteristics have been identified, 
BMWG will jointly prepare a Benchmarking Terminology Document with the 
Routing Area (e.g., the MPLS working group and or the RTG-DIR), which 
would define metrics relevant to LDP convergence. The Benchmark 
definition document would be Last-Called in all the working groups that 
produced it, and solicit operator input during the last call. The work 
will then continue in BMWG to define the test methodology, with input 
and review from the aforementioned parties.

 Goals and Milestones:

   Done         Expand the current Ethernet switch benchmarking methodology 
                draft to define the metrics and methodologies particular to the 
                general class of connectionless, LAN switches. 

   Done         Edit the LAN switch draft to reflect the input from BMWG. Issue 
                a new version of document for comment. If appropriate, 
                ascertain consensus on whether to recommend the draft for 
                consideration as an RFC. 

   Done         Take controversial components of multicast draft to mailing 
                list for discussion. Incorporate changes to draft and reissue 
                appropriately. 

   Done         Submit workplan for initiating work on Benchmarking Methodology 
                for LAN Switching Devices. 

   Done         Submit workplan for continuing work on the Terminology for 
                Cell/Call Benchmarking draft. 

   Done         Submit initial draft of Benchmarking Methodology for LAN 
                Switches. 

   Done         Submit Terminology for IP Multicast Benchmarking draft for AD 
                Review. 

   Done         Submit Benchmarking Terminology for Firewall Performance for AD 
                review 

   Done         Progress ATM benchmarking terminology draft to AD review. 

   Done         Submit Benchmarking Methodology for LAN Switching Devices draft 
                for AD review. 

   Done         Submit first draft of Firewall Benchmarking Methodology. 

   Done         First Draft of Terminology for FIB related Router Performance 
                Benchmarking. 

   Done         First Draft of Router Benchmarking Framework 

   Done         Progress Frame Relay benchmarking terminology draft to AD 
                review. 

   Done         Methodology for ATM Benchmarking for AD review. 

   Done         Terminology for ATM ABR Benchmarking for AD review. 

   Done         Terminology for FIB related Router Performance Benchmarking to 
                AD review. 

   Done         Firewall Benchmarking Methodology to AD Review 

   Done         First Draft of Methodology for FIB related Router Performance 
                Benchmarking. 

   Done         First draft Net Traffic Control Benchmarking Methodology. 

   Done         Methodology for IP Multicast Benchmarking to AD Review. 

   Done         Resource Reservation Benchmarking Terminology to AD Review 

   Done         First I-D on IPsec Device Benchmarking Terminology 

   Done         EGP Convergence Benchmarking Terminology to AD Review 

   Done         Resource Reservation Benchmarking Methodology to AD Review 

   Done         Net Traffic Control Benchmarking Terminology to AD Review 

   Done         IGP/Data-Plane Terminology I-D to AD Review 

   Done         IGP/Data-Plane Methodology and Considerations I-Ds to AD Review 

   Done         Hash and Stuffing I-D to AD Review 

   Done         IPv6 Benchmarking Methodology to AD Review 

   Done         IPsec Device Benchmarking Terminology to IESG Review 

   Done         IPsec Device Benchmarking Methodology to IESG Review 

   Done         Terminology For Protection Benchmarking to AD Review 

   Done         Methodology for MPLS Forwarding to AD Review 

   Done         Networking Device Reset Benchmark (Updates RFC 2544) to IESG 
                Review 

   Dec 2010       Methodology For Protection Benchmarking to IESG Review 

   Feb 2011       Methodology for Flow Export and Collection Benchmarking to IESG 
                Review 

   Jun 2011       Methodology for Data Center Bridging Benchmarking to IESG 
                Review 

   Jun 2011       Terminology for SIP Device Benchmarking to IESG Review 

   Jun 2011       Methodology for SIP Device Benchmarking to IESG Review 

   Jul 2011       Basic BGP Convergence Benchmarking Methodology to IESG Review 

   Dec 2011       Terminology for Content Aware Device Benchmarking to IESG 
                Review 

   Dec 2011       Methodology for Content Aware Device Benchmarking to IESG 

   Dec 2011       Terminology for LDP Convergence Benchmarking to IESG Review 

   Dec 2011       Methodology for LDP Convergence Benchmarking to IESG Review 


 Internet-Drafts:

Posted Revised         I-D Title   <Filename>
------ ------- --------------------------------------------
Jun 2003 Feb 2011   <draft-ietf-bmwg-igp-dataplane-conv-meth-23.txt>
                Benchmarking Methodology for Link-State IGP Data Plane Route 
                Convergence 

Jun 2003 Feb 2011   <draft-ietf-bmwg-igp-dataplane-conv-term-23.txt>
                Terminology for Benchmarking Link-State IGP Data Plane Route 
                Convergence 

Oct 2006 Jul 2010   <draft-ietf-bmwg-protection-term-09.txt>
                Benchmarking Terminology for Protection Performance 

Mar 2009 Mar 2011   <draft-ietf-bmwg-sip-bench-term-03.txt>
                Terminology for Benchmarking Session Initiation Protocol (SIP) 
                Networking Devices 

Mar 2009 Mar 2011   <draft-ietf-bmwg-sip-bench-meth-03.txt>
                Methodology for Benchmarking SIP Networking Devices 

Dec 2010 Jul 2011   <draft-ietf-bmwg-ipflow-meth-03.txt>
                IP Flow Information Accounting and Export Benchmarking 
                Methodology 

 Request For Comments:

  RFC   Stat Published     Title
------- -- ----------- ------------------------------------
RFC1242 I    Jul 1991    Benchmarking Terminology for Network Interconnection 
                       Devices 

RFC1944 I    May 1996    Benchmarking Methodology for Network Interconnect 
                       Devices 

RFC2285 I    Feb 1998    Benchmarking Terminology for LAN Switching Devices 

RFC2432 I    Oct 1998    Terminology for IP Multicast Benchmarking 

RFC2544 I    Mar 1999    Benchmarking Methodology for Network Interconnect 
                       Devices 

RFC2647 I    Aug 1999    Benchmarking Terminology for Firewall Performance 

RFC2761 I    Feb 2000    Terminology for ATM Benchmarking 

RFC2889 I    Aug 2000    Benchmarking Methodology for LAN Switching Devices 

RFC3116 I    Jun 2001    Methodology for ATM Benchmarking 

RFC3133 I    Jun 2001    Terminology for Frame Relay Benchmarking 

RFC3134 I    Jun 2001    Terminology for ATM ABR Benchmarking 

RFC3222 I    Dec 2001    Terminology for Forwarding Information Base (FIB) based 
                       Router Performance 

RFC3511 I    Apr 2003    Benchmarking Methodology for Firewall Performance 

RFC3918 I    Oct 2004    Methodology for IP Multicast Benchmarking 

RFC4098 I    Jun 2005    Terminology for Benchmarking BGP Device Convergence in 
                       the Control Plane 

RFC4063 I    Jun 2005    Considerations When Using Basic OSPF Convergence 
                       Benchmarks 

RFC4062 I    Jun 2005    OSPF Benchmarking Terminology and Concepts 

RFC4061 I    Jun 2005    Benchmarking Basic OSPF Single Router Control Plane 
                       Convergence 

RFC4689 I    Oct 2006    Terminology for Benchmarking Network-layer Traffic 
                       Control Mechanisms 

RFC4814 I    Mar 2007    Hash and Stuffing: Overlooked Factors in Network Device 
                       Benchmarking 

RFC4883 I    Jul 2007    Benchmarking Terminology for Resource Reservation 
                       Capable Routers 

RFC5180 I    May 2008    IPv6 Benchmarking Methodology for Network Interconnect 
                       Devices 

RFC5695 I    Nov 2009    MPLS Forwarding Benchmarking Methodology for IP Flows 

RFC6201 I    Mar 2011    Device Reset Characterization