[ACX] TCAM Management and Firewall Filter Scale Best Practices

The dynamic allocation of Ternary Content Addressable Memory (TCAM) space in ACX Series routers efficiently allocates the available TCAM resources for various filter applications. However, in some scaled setups, if filters that contain a large number of source-prefix-lists, destination-prefix-lists, or terms, the router may run out of TCAM space. When that condition occurs, the following error in acx_dfw tied to firewall rule creation will be seen:

Dec 10 07:20:31  test fpc0 ACX Error (dfw):acx_dfw_group_delete :Entry not found/-7:bcm_field_group_destroy(gid:0) called for  unit 0
Dec 10 07:20:31  test fpc0 ACX Error (dfw):acx_dfw_field_entry_create :failed to create entry in group 16 of unit 0
Dec 10 07:20:31  test fpc0 ACX Error (dfw):acx_dfw_field_entry_create :rv -14 error "No resources for operation"
Dec 10 07:20:31  test fpc0 ACX Error (dfw):acx_dfw_handle_shared_app_entry_add :Could not create field entry, unit 0, group 16
Dec 10 07:20:31  test fpc0 ACX Error (dfw):acx_dfw_rule_create :Failed in shared entry add  unit 0, entry-ret-1, group 16
Dec 10 07:20:31  test fpc0 ACX Error (dfw):acx_dfw_field_entry_destroy :failed to destroy entry -1 in unit 0
Dec 10 07:20:31  test fpc0 ACX Error (dfw):acx_dfw_field_entry_destroy :rv -7 error "Entry not found"
Dec 10 07:20:31  test fpc0 ACX Error (dfw):acx_dfw_filter_create_exp :[-1] from acx_dfw_rule_create_exp term(2)
Dec 10 07:20:31  test fpc0 ACX Error (dfw):acx_dfw_create_hw_instance :Status:-1 Could not program dfw(test-lo0.0-i) type(IFP_FOR_Lo0_FILTER)! [-1]
Dec 10 07:20:31  test fpc0 ACX Error (dfw):acx_dfw_bind_shim :[-1] Could not create dfw(test-lo0.0-i) type(IFP_FOR_Lo0_FILTER)
Dec 10 07:20:31  test fpc0 ACX Error (dfw):acx_dfw_bind :[1000] bind failed for filter test-lo0.0-i

A sample firewall filter and associated prefix-lists that is attached to lo0 which causes the problem is shown below:

set policy-options prefix-list source-prefix-test 192.168.1.0/24
set policy-options prefix-list source-prefix-test 192.168.2.0/24
set policy-options prefix-list source-prefix-test 192.168.3.0/24
set policy-options prefix-list source-prefix-test 192.168.4.0/24
set policy-options prefix-list source-prefix-test 192.168.5.0/24
set policy-options prefix-list source-prefix-test 192.168.6.0/24
set policy-options prefix-list source-prefix-test 192.168.7.0/24
set policy-options prefix-list source-prefix-test 192.168.8.0/24
set policy-options prefix-list source-prefix-test 192.168.9.0/24
set policy-options prefix-list source-prefix-test 192.168.10.0/24
set policy-options prefix-list dest-prefix-test 10.10.1.0/24
set policy-options prefix-list dest-prefix-test 10.10.2.0/24
set policy-options prefix-list dest-prefix-test 10.10.3.0/24
set policy-options prefix-list dest-prefix-test 10.10.4.0/24
set policy-options prefix-list dest-prefix-test 10.10.5.0/24
set policy-options prefix-list dest-multicast-test 224.0.0.1/32
set policy-options prefix-list dest-multicast-test 224.0.0.2/32
set policy-options prefix-list dest-multicast-test 224.0.0.5/32
set policy-options prefix-list dest-multicast-test 224.0.0.6/32

set firewall family inet filter test interface-specific
set firewall family inet filter test term 1 from source-prefix-list source-prefix-test
set firewall family inet filter test term 1 from destination-prefix-list dest-prefix-test
set firewall family inet filter test term 1 from destination-prefix-list dest-multicast-test
set firewall family inet filter test term 1 from protocol ospf
set firewall family inet filter test term 1 from protocol icmp
set firewall family inet filter test term 1 from protocol pim
set firewall family inet filter test term 1 then count count-1
set firewall family inet filter test term 1 then accept
set firewall family inet filter test term 2 from source-prefix-list source-prefix-test
set firewall family inet filter test term 2 from protocol tcp
set firewall family inet filter test term 2 from protocol udp
set firewall family inet filter test term 2 from destination-port ssh
set firewall family inet filter test term 2 from destination-port telnet
set firewall family inet filter test term 2 from destination-port bgp
set firewall family inet filter test term 2 from destination-port ldp
set firewall family inet filter test term 2 then count count-2
set firewall family inet filter test term 2 then accept
set firewall family inet filter test term last then accept

set interfaces lo0 unit 0 family inet filter input test
set interfaces lo0 unit 0 family inet address 192.124.193.182/32

To check the state of the TCAM memory in order to see if it is close to exhaustion or not, run ‘show pfe tcam usage all-tcam-stages detail’:

 labroot@ show pfe tcam usage all-tcam-stages detail
 
Slot 0
 
Tcam Resource Stage: Pre-Ingress
--------------------------------
Free [hw-grps: 3 out of 3]
No dynamic tcam usage
 
Tcam Resource Stage: Ingress
----------------------------
Free [hw-grps: 6 out of 8]
Group: 8, Mode: SINGLE, Hw grps used: 2, Tcam apps: 1
               Used  Allocated  Available     Errors
Tcam-Entries    662       1024        362          0
Counters        660       1024        364          0
Policers          0       1024       1024          0
 
App tcam usage:
----------------
App-Name              Entries Counters Policers Precedence  State
  Related-App-Name ..
-----------------------------------------------------------------
lo0-inet-fil              662      660        0          0     OK
 
Tcam Resource Stage: Egress
---------------------------
Free [hw-grps: 3 out of 3]
No dynamic tcam usage

In order to check the state of the TCAM, it is recommended to run the following CLI commands periodically as a health check of the system:

show pfe tcam usage all-tcam-stages
show pfe tcam usage all-tcam-stages detail
show pfe tcam usage tcam-stage <ingress|egress|pre-ingress>
show pfe tcam usage tcam-stage <ingress|egress|pre-ingress> app ?
show pfe tcam usage app <fw-ccc-in>
show pfe tcam errors
show pfe tcam errors <all-tcam-stages|tcam-stage|app> <detail>
show pfe tcam app ?

If TCAM memory space is close to exhaustion, the following Best Practices can be referenced to save TCAM space:

Avoid using long prefix-lists, if possible. Carefully audit the prefix-list against applied protocols and try to aggregate the prefix-list entries into as few entries as possible.
Keep in mind that all terms, counters, and policers will consume TCAM spaces. Consider using firewall counters only for trouble shooting purposes.
Avoid the use of a logical “or” for all filter terms. Each “or” matching term is going to multiply the TCAM space usage and create lots of unnecessary entries.
Avoid using “destination-prefix-list” for some protocols. For example, Junos routers are already expecting only Multicast and local addresses for OSPF. Adding the same addresses in the filter configuration would not add much value. In many deployments, source address based filters are sufficient to protect the control plane.

Knowledge Base