This article describes the best practices for ACX Series TCAM management and Firewall Filter scale.
The dynamic allocation of Ternary Content Addressable Memory (TCAM) space in ACX Series routers efficiently allocates the available TCAM resources for various filter applications. However, in some scaled setups, if filters that contain a large number of source-prefix-lists, destination-prefix-lists, or terms, the router may run out of TCAM space. When that condition occurs, the following error in acx_dfw tied to firewall rule creation will be seen:
Dec 10 07:20:31 test fpc0 ACX Error (dfw):acx_dfw_group_delete :Entry not found/-7:bcm_field_group_destroy(gid:0) called for unit 0
Dec 10 07:20:31 test fpc0 ACX Error (dfw):acx_dfw_field_entry_create :failed to create entry in group 16 of unit 0
Dec 10 07:20:31 test fpc0 ACX Error (dfw):acx_dfw_field_entry_create :rv -14 error "No resources for operation"
Dec 10 07:20:31 test fpc0 ACX Error (dfw):acx_dfw_handle_shared_app_entry_add :Could not create field entry, unit 0, group 16
Dec 10 07:20:31 test fpc0 ACX Error (dfw):acx_dfw_rule_create :Failed in shared entry add unit 0, entry-ret-1, group 16
Dec 10 07:20:31 test fpc0 ACX Error (dfw):acx_dfw_field_entry_destroy :failed to destroy entry -1 in unit 0
Dec 10 07:20:31 test fpc0 ACX Error (dfw):acx_dfw_field_entry_destroy :rv -7 error "Entry not found"
Dec 10 07:20:31 test fpc0 ACX Error (dfw):acx_dfw_filter_create_exp :[-1] from acx_dfw_rule_create_exp term(2)
Dec 10 07:20:31 test fpc0 ACX Error (dfw):acx_dfw_create_hw_instance :Status:-1 Could not program dfw(test-lo0.0-i) type(IFP_FOR_Lo0_FILTER)! [-1]
Dec 10 07:20:31 test fpc0 ACX Error (dfw):acx_dfw_bind_shim :[-1] Could not create dfw(test-lo0.0-i) type(IFP_FOR_Lo0_FILTER)
Dec 10 07:20:31 test fpc0 ACX Error (dfw):acx_dfw_bind :[1000] bind failed for filter test-lo0.0-i
A sample firewall filter and associated prefix-lists that is attached to lo0 which causes the problem is shown below:
set policy-options prefix-list source-prefix-test 192.168.1.0/24
set policy-options prefix-list source-prefix-test 192.168.2.0/24
set policy-options prefix-list source-prefix-test 192.168.3.0/24
set policy-options prefix-list source-prefix-test 192.168.4.0/24
set policy-options prefix-list source-prefix-test 192.168.5.0/24
set policy-options prefix-list source-prefix-test 192.168.6.0/24
set policy-options prefix-list source-prefix-test 192.168.7.0/24
set policy-options prefix-list source-prefix-test 192.168.8.0/24
set policy-options prefix-list source-prefix-test 192.168.9.0/24
set policy-options prefix-list source-prefix-test 192.168.10.0/24
set policy-options prefix-list dest-prefix-test 10.10.1.0/24
set policy-options prefix-list dest-prefix-test 10.10.2.0/24
set policy-options prefix-list dest-prefix-test 10.10.3.0/24
set policy-options prefix-list dest-prefix-test 10.10.4.0/24
set policy-options prefix-list dest-prefix-test 10.10.5.0/24
set policy-options prefix-list dest-multicast-test 224.0.0.1/32
set policy-options prefix-list dest-multicast-test 224.0.0.2/32
set policy-options prefix-list dest-multicast-test 224.0.0.5/32
set policy-options prefix-list dest-multicast-test 224.0.0.6/32
set firewall family inet filter test interface-specific
set firewall family inet filter test term 1 from source-prefix-list source-prefix-test
set firewall family inet filter test term 1 from destination-prefix-list dest-prefix-test
set firewall family inet filter test term 1 from destination-prefix-list dest-multicast-test
set firewall family inet filter test term 1 from protocol ospf
set firewall family inet filter test term 1 from protocol icmp
set firewall family inet filter test term 1 from protocol pim
set firewall family inet filter test term 1 then count count-1
set firewall family inet filter test term 1 then accept
set firewall family inet filter test term 2 from source-prefix-list source-prefix-test
set firewall family inet filter test term 2 from protocol tcp
set firewall family inet filter test term 2 from protocol udp
set firewall family inet filter test term 2 from destination-port ssh
set firewall family inet filter test term 2 from destination-port telnet
set firewall family inet filter test term 2 from destination-port bgp
set firewall family inet filter test term 2 from destination-port ldp
set firewall family inet filter test term 2 then count count-2
set firewall family inet filter test term 2 then accept
set firewall family inet filter test term last then accept
set interfaces lo0 unit 0 family inet filter input test
set interfaces lo0 unit 0 family inet address 192.124.193.182/32
To check the state of the TCAM memory in order to see if it is close to exhaustion or not, run ‘show pfe tcam usage all-tcam-stages detail’
:
labroot@ show pfe tcam usage all-tcam-stages detail
Slot 0
Tcam Resource Stage: Pre-Ingress
--------------------------------
Free [hw-grps: 3 out of 3]
No dynamic tcam usage
Tcam Resource Stage: Ingress
----------------------------
Free [hw-grps: 6 out of 8]
Group: 8, Mode: SINGLE, Hw grps used: 2, Tcam apps: 1
Used Allocated Available Errors
Tcam-Entries 662 1024 362 0
Counters 660 1024 364 0
Policers 0 1024 1024 0
App tcam usage:
----------------
App-Name Entries Counters Policers Precedence State
Related-App-Name ..
-----------------------------------------------------------------
lo0-inet-fil 662 660 0 0 OK
Tcam Resource Stage: Egress
---------------------------
Free [hw-grps: 3 out of 3]
No dynamic tcam usage
In order to check the state of the TCAM, it is recommended to run the following CLI commands periodically as a health check of the system:
show pfe tcam usage all-tcam-stages
show pfe tcam usage all-tcam-stages detail
show pfe tcam usage tcam-stage <ingress|egress|pre-ingress>
show pfe tcam usage tcam-stage <ingress|egress|pre-ingress> app ?
show pfe tcam usage app <fw-ccc-in>
show pfe tcam errors
show pfe tcam errors <all-tcam-stages|tcam-stage|app> <detail>
show pfe tcam app ?
If TCAM memory space is close to exhaustion, the following Best Practices can be referenced to save TCAM space:
-
Avoid using long prefix-lists, if possible. Carefully audit the prefix-list against applied protocols and try to aggregate the prefix-list entries into as few entries as possible.
-
Keep in mind that all terms, counters, and policers will consume TCAM spaces. Consider using firewall counters only for trouble shooting purposes.
-
Avoid the use of a logical “or” for all filter terms. Each “or” matching term is going to multiply the TCAM space usage and create lots of unnecessary entries.
-
Avoid using “destination-prefix-list” for some protocols. For example, Junos routers are already expecting only Multicast and local addresses for OSPF. Adding the same addresses in the filter configuration would not add much value. In many deployments, source address based filters are sufficient to protect the control plane.