This article provides information to avoid double bandwidth issues for subscribers terminating on MX over an AE interface.  Strategies are provide for using the Shared-Bandwidth-Policer knob and Targeted-Distribution.

A standard policer restricts traffic at the configured rate per PFE. In the case of AE bundles spanning multiple FPCs, the overall rate for the subscriber could exceed the configured rate (depending on the number of PFEs involved). An example of this would be a policer with bandwidth-limit 40mbps and burst-size 40Kbytes configured on an AE interface that has member links ge-0/0/0 and ge-1/0/0. When the policer is applied to the AE interface, this will result in a total bandwidth of 80Mbps as policer is configured for two PFE's.

In the example below, AE has 4 member links configured over FPC11.  FPC11 is an MPC5 which has 1 PFE. The subscriber is configured with 100M policer.

FPC 11           REV 66   750-045715   CAFE6382          MPC5EQ-40G10G

jtac@ERX-MX960-II-RE0> show lacp interfaces

Aggregated interface: ae0
    LACP state:       Role   Exp   Def  Dist  Col  Syn  Aggr  Timeout  Activity
      xe-11/0/0      Actor    No    No   Yes  Yes  Yes   Yes     Fast    Active
      xe-11/0/0    Partner    No    No   Yes  Yes  Yes   Yes     Fast    Active
      xe-11/0/1      Actor    No    No   Yes  Yes  Yes   Yes     Fast    Active
      xe-11/0/1    Partner    No    No   Yes  Yes  Yes   Yes     Fast    Active
      xe-11/1/0      Actor    No    No   Yes  Yes  Yes   Yes     Fast    Active
      xe-11/1/0    Partner    No    No   Yes  Yes  Yes   Yes     Fast    Active
      xe-11/1/1      Actor    No    No   Yes  Yes  Yes   Yes     Fast    Active
      xe-11/1/1    Partner    No    No   Yes  Yes  Yes   Yes     Fast    Active
    LACP protocol:        Receive State  Transmit State          Mux State
      xe-11/0/0                 Current   Fast periodic Collecting distributing
      xe-11/0/1                 Current   Fast periodic Collecting distributing
      xe-11/1/0                 Current   Fast periodic Collecting distributing
      xe-11/1/1                 Current   Fast periodic Collecting distributing

jtac@ERX-MX960-II-RE0# show firewall family inet filter 100M   
interface-specific;
term 1 {
    then {
        policer 100M;
        accept;
    }
}
jtac@ERX-MX960-II-RE0# show firewall policer 100M               
filter-specific;
if-exceeding {
    bandwidth-limit 100m;
    burst-size-limit 8m;
}
then discard;

jtac@ERX-MX960-II-RE0# run show subscribers extensive
Type: VLAN
Logical System: default
Routing Instance: default
Interface: demux0.3221225476
Interface type: Dynamic
Underlying Interface: ae0
Dynamic Profile Name: DYN-VLAN-Q-IN-Q-AE-PROFILE
Dynamic Profile Version: 1
State: Active
Session ID: 5
PFE Flow ID: 21
Stacked VLAN Id: 0x8100.2
VLAN Id: 0x8100.200
Login Time: 2017-04-03 21:23:57 IST

Type: PPPoE
User Name: test@mx.com
IP Address: 10.200.0.4
IP Netmask: 255.255.255.255
Domain name server inet: 8.8.4.4
Domain name server inet6: 2001:4860:4860::8844
Logical System: default
Routing Instance: default
Interface: pp0.3221225477
Interface type: Dynamic
Underlying Interface: demux0.3221225476
Dynamic Profile Name: PPPOE-PROFILE
Dynamic Profile Version: 1
MAC Address: 00:16:01:00:00:01
State: Active
Radius Accounting ID: 6
Session ID: 6
PFE Flow ID: 22
Stacked VLAN Id: 2
VLAN Id: 200
Login Time: 2017-04-03 21:23:57 IST
IP Address Pool: ipv4-pool
IPv4 Input Filter Name: 100M-pp0.3221225477-in
IPv4 Output Filter Name: 100M-pp0.3221225477-out
Accounting interval: 1800
Dynamic configuration:
  junos-input-filter: 100M
  junos-output-filter: 100M

jtac@ERX-MX960-II-RE0# run show interfaces pp0.3221225477 extensive | match bps    
     Input  bytes  :          32693175289                    0 bps
     Output bytes  :           6120028012             99865464 bps  <-- 100Mbps

As seen above, the subscriber is getting the expected 100M bandwdith as configured. The egress policer is configured on a single PFE (MPC5).

Now, one more member link will be added to the AE interface. This link is configured on FPC8, another MPC5. The AE multiple PFEs contributing member links to the AE interface.

As illustrated below, the subscriber is getting double bandwidth due to policer programmed across each PFE.
 
FPC 8            REV 21   750-054563   CADH0955          MPC5E-40G10G
FPC 11           REV 66   750-045715   CAFE6382          MPC5EQ-40G10G

jtac@ERX-MX960-II-RE0> show lacp interfaces
Aggregated interface: ae0
    LACP state:       Role   Exp   Def  Dist  Col  Syn  Aggr  Timeout  Activity
      xe-11/0/0      Actor    No    No   Yes  Yes  Yes   Yes     Fast    Active
      xe-11/0/0    Partner    No    No   Yes  Yes  Yes   Yes     Fast    Active
      xe-11/0/1      Actor    No    No   Yes  Yes  Yes   Yes     Fast    Active
      xe-11/0/1    Partner    No    No   Yes  Yes  Yes   Yes     Fast    Active
      xe-11/1/0      Actor    No    No   Yes  Yes  Yes   Yes     Fast    Active
      xe-11/1/0    Partner    No    No   Yes  Yes  Yes   Yes     Fast    Active
      xe-11/1/1      Actor    No    No   Yes  Yes  Yes   Yes     Fast    Active
      xe-11/1/1    Partner    No    No   Yes  Yes  Yes   Yes     Fast    Active
      xe-8/0/6       Actor    No    No   Yes  Yes  Yes   Yes     Fast    Active
      xe-8/0/6     Partner    No    No   Yes  Yes  Yes   Yes     Fast    Active
    LACP protocol:        Receive State  Transmit State          Mux State
      xe-11/0/0                 Current   Fast periodic Collecting distributing
      xe-11/0/1                 Current   Fast periodic Collecting distributing
      xe-11/1/0                 Current   Fast periodic Collecting distributing
      xe-11/1/1                 Current   Fast periodic Collecting distributing
      xe-8/0/6                  Current   Fast periodic Collecting distributing

jtac@ERX-MX960-II-RE0# run show subscribers extensive                              
Type: VLAN
Logical System: default
Routing Instance: default
Interface: demux0.3221225478
Interface type: Dynamic
Underlying Interface: ae0
Dynamic Profile Name: DYN-VLAN-Q-IN-Q-AE-PROFILE
Dynamic Profile Version: 1
State: Active
Session ID: 7
PFE Flow ID: 24
Stacked VLAN Id: 0x8100.2
VLAN Id: 0x8100.200
Login Time: 2017-04-04 08:10:46 IST

Type: PPPoE
User Name: test@mx.com
IP Address: 10.200.0.5
IP Netmask: 255.255.255.255
Domain name server inet: 8.8.4.4
Domain name server inet6: 2001:4860:4860::8844
Logical System: default
Routing Instance: default
Interface: pp0.3221225479
Interface type: Dynamic
Underlying Interface: demux0.3221225478
Dynamic Profile Name: PPPOE-PROFILE
Dynamic Profile Version: 1
MAC Address: 00:16:01:00:00:01
State: Active
Radius Accounting ID: 8
Session ID: 8
PFE Flow ID: 26    <-- VBF FLOW ID
Stacked VLAN Id: 2
VLAN Id: 200
Login Time: 2017-04-04 08:10:46 IST
IP Address Pool: ipv4-pool
IPv4 Input Filter Name: 100M-pp0.3221225479-in
IPv4 Output Filter Name: 100M-pp0.3221225479-out
Accounting interval: 1800
Dynamic configuration:
  junos-input-filter: 100M
  junos-output-filter: 100M

jtac@ERX-MX960-II-RE0# run show interfaces pp0.3221225479 extensive | match bps    
     Input  bytes  :                  684                    0 bps
     Output bytes  :          11820983986            200031952 bps   <-- 200Mbps

Shared-Bandwidth-Policer (Targeted-Distribution is supported from Junos 16.1R4 and higher)

The purpose of the “Shared-bandwidth-policer” feature is to match the effective bandwidth/burst-size made available to an aggregate bundle by enhancing the implementation of the policer on the PFE. The implementation will statically carve up the bandwidth and burst size among the PFE-complex by a factor derived from number of member-links that belong to a given PFE-complex. The carve-up factor shown below.

Carve-up-factor= (Number of links on that PFE-complex)/(Total Number of links in the bundle)

Bandwidth-limit=(Total Bandwidth-limit)*(Carve-up-factor)
Burst-Size-limit=(Total burst-size-limit)* (Carve-up-factor)

policer 100M {
    filter-specific;
    shared-bandwidth-policer;  <-- Configured shared-bandwidth-policer under policer
    if-exceeding {
        bandwidth-limit 100m;
        burst-size-limit 8m;
    }
    then discard;

jtac@ERX-MX960-II-RE0# run show subscribers extensive
Type: VLAN
Logical System: default
Routing Instance: default
Interface: demux0.3221225480
Interface type: Dynamic
Underlying Interface: ae0
Dynamic Profile Name: DYN-VLAN-Q-IN-Q-AE-PROFILE
Dynamic Profile Version: 1
State: Active
Session ID: 9
PFE Flow ID: 27
Stacked VLAN Id: 0x8100.2
VLAN Id: 0x8100.200
Login Time: 2017-04-04 12:24:43 IST

Type: PPPoE
User Name: test@mx.com
IP Address: 10.200.0.6
IP Netmask: 255.255.255.255
Domain name server inet: 8.8.4.4
Domain name server inet6: 2001:4860:4860::8844
Logical System: default
Routing Instance: default
Interface: pp0.3221225481
Interface type: Dynamic
Underlying Interface: demux0.3221225480
Dynamic Profile Name: PPPOE-PROFILE
Dynamic Profile Version: 1
MAC Address: 00:16:01:00:00:01
State: Active
Radius Accounting ID: 10
Session ID: 10
PFE Flow ID: 29
Stacked VLAN Id: 2
VLAN Id: 200
Login Time: 2017-04-04 12:24:43 IST
IP Address Pool: ipv4-pool
IPv4 Input Filter Name: 100M-pp0.3221225481-in
IPv4 Output Filter Name: 100M-pp0.3221225481-out
Accounting interval: 1800
Dynamic configuration:
  junos-input-filter: 100M
  junos-output-filter: 100M


Carve-up factor programmed under FPC8

        Regular policer '100M'
        dfw:167772160 pfe_id:0
        carve-up factor:0.200 vcuf:0.200 <-- AE0 has only one member link from FPC8, so 1/5=0.2 Carve-up factor
        bandwidth:2500000 bytes/sec, burst size:1600000 bytes

Carve-up factor programmed under FPC11

        Regular policer '100M'
        dfw:167772160 pfe_id:0
        carve-up factor:0.800 vcuf:0.800 <-- AE0 has 4 links from FPC11, so 4/5=0.8 Carve-up factor
        bandwidth:10000000 bytes/sec, burst size:6400000 bytes
 

jtac@ERX-MX960-II-RE0# run show interfaces pp0.3221225481 extensive | match bps    
     Input  bytes  :                  684                    0 bps
     Output bytes  :           2256293826             80189776 bps   <-- ~100Mbps

Targeted-Distribution

Targeted distribution enables you to target the egress traffic for an IP or VLAN demux subscriber on a single member link using a single scheduler resource. To achieve load balancing over the member links, the system distributes the subscriber interfaces equally among the links.

Configuration

interfaces {
    ae0 {
        hierarchical-scheduler maximum-hierarchy-levels 2;
        interface-transmit-statistics;
        flexible-vlan-tagging;
        auto-configure {
            stacked-vlan-ranges {
                dynamic-profile DYN-VLAN-Q-IN-Q-AE-PROFILE {
                    accept pppoe;
                    ranges {
                        any,any;
                    }
                }
            }
            remove-when-no-subscribers;
        }
        mtu 9192;
        encapsulation flexible-ethernet-services;
        aggregated-ether-options {
            no-flow-control;
            minimum-links 1;
            link-speed 10g;
            lacp {
                active;
                periodic fast;
            }
        }
    }
}

jtac@ERX-MX960-II-RE0# show dynamic-profiles DYN-VLAN-Q-IN-Q-AE-PROFILE
routing-instances {
    "$junos-routing-instance" {
        interface "$junos-interface-name";
    }
}
interfaces {
    demux0 {
        unit "$junos-interface-unit" {
            actual-transit-statistics;
            demux-source [ inet inet6 ];
            no-traps;
            proxy-arp unrestricted;
            vlan-tags outer "$junos-stacked-vlan-id" inner "$junos-vlan-id";
            demux-options {
                underlying-interface "$junos-underlying-interface";
            }
            family inet {
                unnumbered-address "$junos-loopback-interface";
            }
            family inet6 {
                unnumbered-address "$junos-loopback-interface";
            }
            family pppoe {
                access-concentrator MX960;
                dynamic-profile PPPOE-PROFILE;
            }
        }
    }
}

jtac@ERX-MX960-II-RE0# show dynamic-profiles PPPOE-PROFILE
routing-instances {
    "$junos-routing-instance" {
        interface "$junos-interface-name";
    }
}
interfaces {
    pp0 {
        unit "$junos-interface-unit" {
            no-traps;
            ppp-options {
                pap;
            }
            pppoe-options {
                underlying-interface "$junos-underlying-interface";
                server;
            }
            targeted-distribution;   <-- Enable targeted distribution
            keepalives interval 20;
            family inet {
                filter {
                    input "$junos-input-filter";
                    output "$junos-output-filter";
                }
                unnumbered-address "$junos-loopback-interface";
            }
            family inet6 {
                unnumbered-address "$junos-loopback-interface";
            }
        }
    }
}
 

Verify

jtac@ERX-MX960-II-RE0# run show subscribers extensive
Type: VLAN
Logical System: default
Routing Instance: default
Interface: demux0.3221225482
Interface type: Dynamic
Underlying Interface: ae0
Dynamic Profile Name: DYN-VLAN-Q-IN-Q-AE-PROFILE
Dynamic Profile Version: 1
State: Active
Session ID: 11
PFE Flow ID: 30
Stacked VLAN Id: 0x8100.2
VLAN Id: 0x8100.200
Login Time: 2017-04-04 12:44:09 IST

Type: PPPoE
User Name: test@mx.com
IP Address: 10.200.0.7
IP Netmask: 255.255.255.255
Domain name server inet: 8.8.4.4
Domain name server inet6: 2001:4860:4860::8844
Logical System: default
Routing Instance: default
Interface: pp0.3221225483
Interface type: Dynamic
Underlying Interface: demux0.3221225482
Dynamic Profile Name: PPPOE-PROFILE
Dynamic Profile Version: 1
MAC Address: 00:16:01:00:00:01
State: Active
Radius Accounting ID: 12
Session ID: 12
PFE Flow ID: 33
Stacked VLAN Id: 2
VLAN Id: 200
Login Time: 2017-04-04 12:44:09 IST
IP Address Pool: ipv4-pool
IPv4 Input Filter Name: 100M-pp0.3221225483-in
IPv4 Output Filter Name: 100M-pp0.3221225483-out
Accounting interval: 1800
Dynamic configuration:
  junos-input-filter: 100M
  junos-output-filter: 100M

jtac@ERX-MX960-II-RE0# run show interfaces targeting
Aggregated interface: ae0
Targeting Type: Auto
Redundancy mode: Link Level Redundancy
Total number of distribution groups: 1
Total number of distributed interfaces: 1

Distribution Group name: default
Number of distributed interfaces: 1
Physical interface: xe-11/0/0, Link status: Up  
  Number of primary distributions: 1    
  Number of backup  distributions: 0    
Physical interface: xe-11/0/1, Link status: Up  
  Number of primary distributions: 0    
  Number of backup  distributions: 1    
Physical interface: xe-11/1/0, Link status: Up  
  Number of primary distributions: 0    
  Number of backup  distributions: 0    
Physical interface: xe-11/1/1, Link status: Up  
  Number of primary distributions: 0    
  Number of backup  distributions: 0    
Physical interface: xe-8/0/6, Link status: Up  
  Number of primary distributions: 0    
  Number of backup  distributions: 0    

jtac@ERX-MX960-II-RE0# run show interfaces pp0.3221225483 extensive | match bps
     Input  bytes  :                  190                    0 bps
     Output bytes  :             66540630            100378728 bps  <-- 100Mbps

jtac@ERX-MX960-II-RE0# run show interfaces xe-11/0/0 extensive | match bps
  Link-level type: Flexible-Ethernet, MTU: 9192, MRU: 9200, LAN-PHY mode, Speed: 10Gbps, BPDU Error: None, Loop Detect PDU Error: None, MAC-REWRITE Error: None, Loopback: None,
   Input  bytes  :              7879396                  992 bps
   Output bytes  :          66352704835             99880936 bps  <-- Traffic going out from one member link