Benchmarking Methodology (bmwg)
-------------------------------

 Charter
 Last Modified: 2006-12-18

 Current Status: Active Working Group

 Chair(s):
     Al Morton  <acmorton@att.com>

 Operations and Management Area Director(s):
     Dan Romascanu  <dromasca@avaya.com>
     David Kessens  <david.kessens@nokia.com>

 Operations and Management Area Advisor:
     David Kessens  <david.kessens@nokia.com>

 Mailing Lists: 
     General Discussion:bmwg@ietf.org
     To Subscribe:      bmwg-request@ietf.org
         In Body:       subscribe your_email_address
     Archive:           http://www.ietf.org/mail-archive/web/bmwg/index.html

Description of Working Group:

The major goal of the Benchmarking Methodology Working Group is to make
a series of recommendations concerning the measurement of the
performance characteristics of various internetworking technologies;
further, these recommendations may focus on the systems or services 
that are built from these technologies.

Each recommendation will describe the class of equipment, system, or
service being addressed; discuss the performance characteristics that
are pertinent to that class; clearly identify a set of metrics that aid
in the description of those characteristics; specify the methodologies
required to collect said metrics; and lastly, present the requirements
for the common, unambiguous reporting of benchmarking results.

To better distinguish the BMWG from other measurement initiatives in 
the IETF, the scope of the BMWG is limited to technology 
characterization using simulated stimuli in a laboratory environment. 
Said differently, the BMWG does not attempt to produce benchmarks for 
live, operational networks. Moreover, the benchmarks produced by this 
WG 
shall strive to be vendor independent or otherwise have universal 
applicability to a given technology class.

Because the demands of a particular technology may vary from
deployment to deployment, a specific non-goal of the Working Group is
to define acceptance criteria or performance requirements.

An ongoing task is to provide a forum for discussion regarding the
advancement of measurements designed to provide insight on the 
operation internetworking technologies.

 Goals and Milestones:

   Done         Expand the current Ethernet switch benchmarking methodology 
                draft to define the metrics and methodologies particular to the 
                general class of connectionless, LAN switches. 

   Done         Edit the LAN switch draft to reflect the input from BMWG. Issue 
                a new version of document for comment. If appropriate, 
                ascertain consensus on whether to recommend the draft for 
                consideration as an RFC. 

   Done         Take controversial components of multicast draft to mailing 
                list for discussion. Incorporate changes to draft and reissue 
                appropriately. 

   Done         Submit workplan for initiating work on Benchmarking Methodology 
                for LAN Switching Devices. 

   Done         Submit workplan for continuing work on the Terminology for 
                Cell/Call Benchmarking draft. 

   Done         Submit initial draft of Benchmarking Methodology for LAN 
                Switches. 

   Done         Submit Terminology for IP Multicast Benchmarking draft for AD 
                Review. 

   Done         Submit Benchmarking Terminology for Firewall Performance for AD 
                review 

   Done         Progress ATM benchmarking terminology draft to AD review. 

   Done         Submit Benchmarking Methodology for LAN Switching Devices draft 
                for AD review. 

   Done         Submit first draft of Firewall Benchmarking Methodology. 

   Done         First Draft of Terminology for FIB related Router Performance 
                Benchmarking. 

   Done         First Draft of Router Benchmarking Framework 

   Done         Progress Frame Relay benchmarking terminology draft to AD 
                review. 

   Done         Methodology for ATM Benchmarking for AD review. 

   Done         Terminology for ATM ABR Benchmarking for AD review. 

   Done         Terminology for FIB related Router Performance Benchmarking to 
                AD review. 

   Done         Firewall Benchmarking Methodology to AD Review 

   Done         First Draft of Methodology for FIB related Router Performance 
                Benchmarking. 

   Done         First draft Net Traffic Control Benchmarking Methodology. 

   Done         Methodology for IP Multicast Benchmarking to AD Review. 

   Done         Resource Reservation Benchmarking Terminology to AD Review 

   Done         First I-D on IPsec Device Benchmarking Terminology 

   Done         EGP Convergence Benchmarking Terminology to AD Review 

   Done         Resource Reservation Benchmarking Methodology to AD Review 

   Done         Net Traffic Control Benchmarking Terminology to AD Review 

   Done         IGP/Data-Plane Terminology I-D to AD Review 

   Done         IGP/Data-Plane Methodology and Considerations I-Ds to AD Review 

   Done         Hash and Stuffing I-D to AD Review 

   Dec 2006       IPsec Device Benchmarking Terminology to AD Review 

   Dec 2006       Router Accelerated Test Terminology I-D to AD Review 

   Dec 2006       Router Accelerated Test Method. and Considerations I-Ds to AD 
                Review 

   Dec 2006       IPsec Device Benchmarking Methodology to AD Review 

   Feb 2007       Terminology For Protection Benchmarking to AD Review 

   Mar 2007       IPv6 Benchmarking Methodology to AD Review 

   Jul 2007       Basic BGP Convergence Benchmarking Methodology to AD Review. 

   Jul 2007       Methodology For Protection Benchmarking to AD Review 

   Dec 2007       Net Traffic Control Benchmarking Methodology to AD Review. 

   Dec 2007       Router Accelerated Test Method for EBGP to AD Review 

   Dec 2007       Router Accelerated Test Method for Operational Security to AD 
                Review 


 Internet-Drafts:

Posted Revised         I-D Title   <Filename>
------ ------- --------------------------------------------
Jul 2001 Feb 2006   <draft-ietf-bmwg-benchres-term-07.txt>
                Benchmarking Terminology for Resource Reservation Capable 
                Routers 

Jun 2003 Jun 2006   <draft-ietf-bmwg-igp-dataplane-conv-meth-11.txt>
                Benchmarking Methodology for IGP Data Plane Route Convergence 

Jun 2003 Jun 2006   <draft-ietf-bmwg-igp-dataplane-conv-term-11.txt>
                Terminology for Benchmarking IGP Data Plane Route Convergence 

Jun 2003 Jun 2006   <draft-ietf-bmwg-igp-dataplane-conv-app-11.txt>
                Considerations for Benchmarking IGP Data Plane Route 
                Convergence 

Jun 2003 Oct 2006   <draft-ietf-bmwg-acc-bench-term-10.txt>
                Terminology for Accelerated Stress Benchmarking 

Jul 2004 Jan 2007   <draft-ietf-bmwg-hash-stuffing-08.txt>
                Hash and Stuffing: Overlooked Factors in Network Device 
                Benchmarking 

Jul 2004 Oct 2006   <draft-ietf-bmwg-acc-bench-meth-06.txt>
                Methodology Guidelines for Accelerated Stress Benchmarking 

Oct 2006 Oct 2006   <draft-ietf-bmwg-protection-term-00.txt>
                Benchmarking Terminology for Protection Performance 

Oct 2006 Oct 2006   <draft-ietf-bmwg-protection-meth-00.txt>
                Methodology for benchmarking MPLS Protection mechanisms 

Jan 2007 Jan 2007   <draft-ietf-bmwg-ipv6-meth-00.txt>
                IPv6 Benchmarking Methodology 

 Request For Comments:

  RFC   Stat Published     Title
------- -- ----------- ------------------------------------
RFC1242 I    Jul 1991    Benchmarking Terminology for Network Interconnection 
                       Devices 

RFC1944 I    May 1996    Benchmarking Methodology for Network Interconnect 
                       Devices 

RFC2285 I    Feb 1998    Benchmarking Terminology for LAN Switching Devices 

RFC2432 I    Oct 1998    Terminology for IP Multicast Benchmarking 

RFC2544 I    Mar 1999    Benchmarking Methodology for Network Interconnect 
                       Devices 

RFC2647 I    Aug 1999    Benchmarking Terminology for Firewall Performance 

RFC2761 I    Feb 2000    Terminology for ATM Benchmarking 

RFC2889 I    Aug 2000    Benchmarking Methodology for LAN Switching Devices 

RFC3116 I    Jun 2001    Methodology for ATM Benchmarking 

RFC3134 I    Jun 2001    Terminology for ATM ABR Benchmarking 

RFC3133 I    Jun 2001    Terminology for Frame Relay Benchmarking 

RFC3222 I    Dec 2001    Terminology for Forwarding Information Base (FIB) based 
                       Router Performance 

RFC3511 I    Apr 2003    Benchmarking Methodology for Firewall Performance 

RFC3918 I    Oct 2004    Methodology for IP Multicast Benchmarking 

RFC4098 I    Jun 2005    Terminology for Benchmarking BGP Device Convergence in 
                       the Control Plane 

RFC4063 I    Jun 2005    Considerations When Using Basic OSPF Convergence 
                       Benchmarks 

RFC4062 I    Jun 2005    OSPF Benchmarking Terminology and Concepts 

RFC4061 I    Jun 2005    Benchmarking Basic OSPF Single Router Control Plane 
                       Convergence 

RFC4689 I    Oct 2006    Terminology for Benchmarking Network-layer Traffic 
                       Control Mechanisms