Knowledge Search


×
 

[EX/QFX] Maximum number of ECMP groups supported by EX and QFX switches

  [KB34303] Show Article Properties


Summary:

This article discusses the table size for ECMP (Equal-Cost MultiPath) and how its related features affect the number of ECMP objects on EX and QFX product line.

Symptoms:

Except for EX4300, whose ECMP table size is 4K, the ECMP table size of EX and QFX product is 16K. An ECMP table is shared by all protocols (e.g. IPv4, IPv6, MPLS, VXLAN, etc). An ECMP object is created on ECMP tables for each ECMP set of active routes. Depending on the JUNOS version and protocols, Static or Dynamic ECMP entries are allocated for ECMP objects. The maximum number of ECMP objects is affected by how ECMP entries are allocated. Other than that, MBB (Make Before Break) and fragmentation also affect the maximum number of ECMP objects.

Solution:

There are several features and factors that affect the maximum number of ECMP object.

  1. Static ECMP Entries

    Before 14.1X53-D30 and 15.1R3, ECMP entries are allocated statically based on the value defined by command set chassis maximum-ecmp, whose default value is 16. If the default value is used, an ECMP object always allocates 16 entries in ECMP tables regardless the number of active routes in the ECMP set.

  2. Dynamic ECMP Entries

    Obviously, static ECMP entries is not an efficient way to create ECMP objects. In 14.1X53-D30 and 15.1R3, a feature called Dynamic ECMP Entries for IPv4, IPv6, and MPLS is added. Dynamic ECMP entries only allocates the number of entries needed in an ECMP object. For example, if there are 5 active routes in an ECMP set, only 5 entries are allocated for the ECMP object in the ECMP table. The feature significantly improves the maximum number of ECMP objects.

    Dynamic ECMP entries for VXLAN is added in the following software versions:

    14.1X53-D48, 17.1R3, 17.2R3, 17.4R2-S4, 17.4R3, 18.1R3-S1, 18.1R4, 18.2R2, 18.3R2, and 18.4R1.
  3. MBB (Make Before Break)

    Assume that a customer uses 16 routes for all ECMP objects. Theoretically, 1000 ECMP objects can be used. However, it is not wise to implement 1000 ECMP objects because of MBB. MBB is a feature always enable in JUNOS.

    Assume that there is a change in the routing table (e.g. routing protocol flap). The number of active routes in an ECMP object is reduced from 16 to 14. MBB creates a new ECMP object with 14 entries. Then, the old ECMP object is then deleted. If 1000 ECMP objects are created before the change in the routing table, MBB is unable to create an ECMP object, which result in a forwarding problem on the switch.

  4. Fragmentation

    An ECMP object requires all entries allocated in a contiguous block. After a large number of ECMP object changes, the ECMP table may be fragmented; so that, the largest contiguous block in the ECMP table may be smaller than the number of available ECMP entries. Fragmentation may cause a failure to create ECMP objects.

Below is the instruction on checking the number of ECMP entries in an ECMP table

  1. Enter start shell in CLI

    root@switch:RE> start shell
    root@switch:RE:0%
  2. Create a command.txt in /var/tmp

    root@switch:RE:0% echo 'set dcb bc "l3 multipath show"' > /var/tmp/command.txt
    root@switch:RE:0%
  3. Save the ECMP table on a text file

    root@switch:RE:0% cprod -A fpc0 /var/tmp/command.txt > /var/tmp/output.txt
    root@switch:RE:0%
  4. Count the number of ECMP entries

    root@switch:RE:0% more /var/tmp/output.txt | grep Interfaces | egrep -o '[0-9]{1,}' | wc -l
           6
    root@switch:RE:0% 
    

In this example, there are 6 ECMP entries in the ECMP table.

In summary, the ECMP table size is 16K for most product. It is not recommended to use up all 16K entries in ECMP table because of MBB and fragmentation. Depending on the frequency of changes in ECMP objects and the number of active routes in ECMP object, the utilization of ECMP table can reach 90%.

Related Links: