Search our Knowledge Base sites to find answers to your questions.
Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articlesSRX Getting Started -- Troubleshooting Traffic Flows and Session Establishment
This article addresses troubleshooting traffic flows and session establishment on all SRX devices. It also addresses packet capture and analysis on SRX Branch devices.
This article contains the following sections for troubleshooting traffic through SRX devices. Click the section title below to jump directly to that section:
For detailed information regarding the packet flow and session management, refer to the technical documentation: Flow-Based Processing Feature Guide for Security Devices.
The trace configuration is:192.168.224.30 (Server in Trust zone) ---------SRX-------- 1.1.1.100 (client in Untrust zone)
SRX interfaces:
ge-0/0/0.0: IP Address 192.168.224.3/24 (Trust)
fe-0/0/7.0: IP Address 1.1.1.50/24 (Untrust)
Static NAT on SRX for server:
1.1.1.30 (public IP address for server) -> 192.168.224.30 (internal Server IP address)
Source NAT on SRX for incoming clients: (for this scenario because 192.168.244.30 network does not have knowledge of 1.1.1.0/24 network)
1.1.1.0/24 -> 192.168.224.3
Note that the packet filters are configured with the addresses that the packets will arrive at the SRX as, not as the translated IP Addresses in NAT. Within each packet-filter the arguments serve to match as a logical AND (e.g. for MatchTraffic statement the traffic will match source IP’s within 1.1.1.100/32 AND destination IP 1.1.1.30/32.) Adding additional Traffic Filters will function as a logical OR between the filters so each filter will operate independently.set security flow traceoptions file DebugTraffic
set security flow traceoptions flag basic-datapath
# define filter to capture traffic from client to Server's public IP address 1.1.1.30
set security flow traceoptions packet-filter MatchTraffic source-prefix 1.1.1.100/32 destination-prefix 1.1.1.30/32
(Note: You may also specify a subnet for the prefix, 1.1.1.0/24, instead of the client's /32 IP address)
#define filter to capture traffic from Server in Trust to Client's NAT'd address
set security flow traceoptions packet-filter MatchTrafficReverse source-prefix 192.168.224.30/32 destination-prefix 192.168.224.3/32
##IPID of the packet shows its 885, which is very useful for comparing packets across multiple capture points (e.g. viewing packet on originating machine, on FW, and on destination to make sure the packet arrivesAug 12 16:22:22 16:22:21.1100315:CID-0:RT:<1.1.1.100/57650->1.1.1.30/3389;6> matched filter MatchTraffic:
)
## This packet arrived on port fe-0/0/7.0Aug 12 16:22:22 16:22:21.1100315:CID-0:RT:packet [48] ipid = 885, @422ec91e
Aug 12 16:22:22 16:22:21.1100315:CID-0:RT:---- flow_process_pkt: (thd 1): flow_ctxt type 13, common flag 0x0, mbuf 0x422ec780
Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: flow process pak fast ifl 69 in_ifp fe-0/0/7.0
## Platform knows the 5 tuple of the packet and does a flow lookupAug 12 16:22:22 16:22:21.1100315:CID-0:RT: fe-0/0/7.0:1.1.1.100/57650->1.1.1.30/3389, tcp, flag 2 syn
## No existing session found in the table lookup, a policy lookup must be performedAug 12 16:22:22 16:22:21.1100315:CID-0:RT: find flow: table 0x4dd31658, hash 32955(0xffff), sa 1.1.1.100, da 1.1.1.30, sp 57650, dp 3389, proto 6, tok 448
## First, check to see if destination address should be NAT’d (static or destination NAT)Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: no session found, start first path. in_tunnel - 0, from_cp_flag - 0
Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: flow_first_create_session
## The destination should be NAT’d from 1.1.1.30 to 192.168.224.30Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: flow_first_in_dst_nat: in <fe-0/0/7.0>, out <N/A> dst_adr 1.1.1.30, sp 57650, dp 3389
Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: chose interface fe-0/0/7.0 as incoming nat if.
## Do a route lookup to determine what the destination interface and zone should be so the SRX can do a policy lookup.Aug 12 16:22:22 16:22:21.1100315:CID-0:RT:flow_first_rule_dst_xlate: packet 1.1.1.100->1.1.1.30 nsp2 0.0.0.0->192.168.224.30.
## The route lookup that points out the interface ge-0/0/0.0 with a next hop which is the destination on the attached 192.168.224.0/24 network.Aug 12 16:22:22 16:22:21.1100315:CID-0:RT:flow_first_routing: call flow_route_lookup(): src_ip 1.1.1.100, x_dst_ip 192.168.224.30, in ifp fe-0/0/7.0, out ifp N/A sp 57650, dp 3389, ip_proto 6, tos 0
Aug 12 16:22:22 16:22:21.1100315:CID-0:RT:Doing DESTINATION addr route-lookup
## Now a policy lookup is performed that determines that the destination zone on ge-0/0/0.0 is trust, and the packet arrived on fe-0/0/7.0 which is untrust.Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: routed (x_dst_ip 192.168.224.30) from untrust (fe-0/0/7.0 in 0) to ge-0/0/0.0, Next-hop: 192.168.224.30
## Policy lookup shows that the SRX does not find a match and the traffic is dropped due to no matching policy.Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: policy search from zone untrust-> zone trust
Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: app 0, timeout 1800s, curr ageout 20s
Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: packet dropped, denied by policy
Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: packet dropped, policy deny.
Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: flow find session returns error.
Aug 12 16:22:22 16:22:21.1100315:CID-0:RT: ----- flow_process_pkt rc 0x7 (fp rc -1)
## Packet has the IPID 5015. The IP ID field is very useful when correlating this debug trace to packets in a simultaneous PCAP packet capture. Comparing the debug trace and PCAP packet captures on a source or destination machine help to identify packets that did or did not arrive.Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:<1.1.1.100/51303->1.1.1.30/3389;6> matched filter MatchTraffic:
## Packet Arrived on the fe-0/0/7.0 interfaceAug 4 16:21:02 16:21:00.1872004:CID-0:RT:packet [48] ipid = 5015, @423d7e9eAug 4 16:21:02 16:21:00.1872004:CID-0:RT:---- flow_process_pkt: (thd 1): flow_ctxt type 13, common flag 0x0, mbuf 0x423d7d00
## Deeper look at the packet shows that it is the syn packet of the connection, indicating the first packet of the flowAug 4 16:21:02 16:21:00.1872004:CID-0:RT: flow process pak fast ifl 72 in_ifp fe-0/0/7.0
## Packet Lookup with the information above, where sa=source address, da=destination address, sp=source port, dp=destination port, proto=protocolAug 4 16:21:02 16:21:00.1872004:CID-0:RT: fe-0/0/7.0:1.1.1.100/51303->1.1.1.30/3389, tcp, flag 2 syn
## First packet in flow, so no session is foundAug 4 16:21:02 16:21:00.1872004:CID-0:RT: find flow: table 0x5258d7b0, hash 17008(0xffff), sa 1.1.1.100, da 1.1.1.30, sp 51303, dp 3389, proto 6, tok 448
## Create new sessionAug 4 16:21:02 16:21:00.1872004:CID-0:RT: no session found, start first path. in_tunnel - 0, from_cp_flag – 0
## Before a policy lookup, a check must be performed for destination NAT; in this case it is static NAT, which NAT’s the 1.1.1.30 address to 192.168.224.30. This lookup must be done BEFORE route lookup, since if there is a destination NAT change, then the SRX needs to do a route lookup on the translated destination address.Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: flow_first_create_session
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: flow_first_in_dst_nat: in <fe-0/0/7.0>, out <N/A> dst_adr 1.1.1.30, sp 51303, dp 3389
## Begin Routing LookupAug 4 16:21:02 16:21:00.1872004:CID-0:RT: chose interface fe-0/0/7.0 as incoming nat if.
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:flow_first_rule_dst_xlate: packet 1.1.1.100->1.1.1.30 nsp2 0.0.0.0->192.168.224.30.
## Perform Route LookupAug 4 16:21:02 16:21:00.1872004:CID-0:RT:flow_first_routing: call flow_route_lookup(): src_ip 1.1.1.100, x_dst_ip 192.168.224.30, in ifp fe-0/0/7.0, out ifp N/A sp 51303, dp 3389, ip_proto 6, tos 0
## Performing the route lookup on destination IP address 192.168.224.30 shows that it should go out interface ge-0/0/0.0, since ge-0/0/0.0 IP address is 192.168.224.3/24, the network is directly attached, and therefore the next-hop is the destination itself 192.168.224.30. If the destination was not directly attached, then our Next-Hop would be the Next-Hop of the router.Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:Doing DESTINATION addr route-lookup
## Since the ingress interface is fe-0/0/7.0 which is zone Untrust, a route lookup determines that the outgoing interface is going to be ge-0/0/0.0, which is zone Trust. Therefore a policy lookup can be performed.Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: routed (x_dst_ip 192.168.224.30) from untrust (fe-0/0/7.0 in 0) to ge-0/0/0.0, Next-hop: 192.168.224.30
## The session timeout is set to 1800 seconds, which is based on the standard TCP application timeout of 30 minutes. (Note that the SRX does not use Ticks like ScreenOS; it uses seconds.) Also the curr ageout of 20 seconds is the initial TCP timeout, if the session isn’t established within 20 seconds, then the session will be dropped.Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: policy search from zone untrust-> zone trust
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: policy has timeout 900
## Here the output shows the SRX NAT pool for source translation is used.Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: app 0, timeout 1800s, curr ageout 20s
## DIP id 2/0 is used, where the term DIP is a legacy term from ScreenOS, but it shows that the SRX is translating the source from 1.1.1.100 (source port 51303) to 192.168.224.3 (source port 48810). In this example, translation is needed because the destination host 192.168.224.30 does not know about our 1.1.1.0/24 network.Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:flow_first_src_xlate: src nat 0.0.0.0(51303) to 192.168.224.30(3389) returns status 1, rule/pool id 1/2.
## The SRX selects ge-0/0/0.0 as our outgoing interfaceAug 4 16:21:02 16:21:00.1872004:CID-0:RT: dip id = 2/0, 1.1.1.100/51303->192.168.224.3/48810
## Message references other service processing not applicable for this ruleAug 4 16:21:02 16:21:00.1872004:CID-0:RT: choose interface ge-0/0/0.0 as outgoing phy if
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:is_loop_pak: No loop: on ifp: ge-0/0/0.0, addr: 192.168.224.30, rtt_idx:0
## The traffic is matched to Policy 9Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:policy is NULL (wx/pim scenario)
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:sm_flow_interest_check: app_id 0, policy 9, app_svc_en 0, flags 0x2. not interested
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:sm_flow_interest_check: app_id 1, policy 9, app_svc_en 0, flags 0x2. not interested
## TIP: The policy 9 content can be checked by looking through the policy list with the command “show security policies” or with the command “show security policies | find "Index\: 9"” which uses the regular expression to search for Index: 9 in the policy and will show the first instance. Visually make sure that the traffic is matching the correct rule.
root@SRX210# run show security policies | find "Index\: 9"
Policy: Allow-RDP-Server, State: enabled, Index: 9, Sequence number: 1
Source addresses: 1.1.1.0/24
Destination addresses: 192.168.224.30
Applications: RDP
Action: permit, log
## Packet is sent out interface ge-0/0/0.0Aug 4 16:21:02 16:21:00.1872004:CID-0:RT:flow_first_service_lookup(): natp(0x51ee4680): app_id, 0(0).
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: service lookup identified service 0.
Aug 4 16:21:02 16:21:00.1872004:CID-0:RT: flow_first_final_check: in <fe-0/0/7.0>, out <ge-0/0/0.0>
## Do a hash lookup on the session to find it in the tableAug 14 19:11:40 19:11:40.192283:CID-0:RT:<192.168.224.30/3389->192.168.224.3/48810;6> matched filter MatchTrafficReverse:Aug 14 19:11:40 19:11:40.192283:CID-0:RT:packet [40] ipid = 8174, @42308b9e
Aug 14 19:11:40 19:11:40.192283:CID-0:RT:---- flow_process_pkt: (thd 1): flow_ctxt type 13, common flag 0x0, mbuf 0x42308a00
Aug 14 19:11:40 19:11:40.192283:CID-0:RT: flow process pak fast ifl 67 in_ifp ge-0/0/0.0
Aug 14 19:11:40 19:11:40.192283:CID-0:RT: ge-0/0/0.0:192.168.224.30/3389->192.168.224.3/48810, tcp, flag 10
## Since this is return traffic, it is found it in the session table, as id 12535Aug 14 19:11:40 19:11:40.192283:CID-0:RT: find flow: table 0x4dd31658, hash 44831(0xffff), sa 192.168.224.30, da 192.168.224.3, sp 3389, dp 48810, proto 6, tok 384
Aug 14 19:11:40 19:11:40.192283:CID-0:RT: flow got session.
## Translate the packet to the appropriate translation for outbound flowAug 14 19:11:40 19:11:40.192283:CID-0:RT: flow session id 12535Aug 14 19:11:40 19:11:40.192283:CID-0:RT: tcp seq check.
## Place packet in buffer to be sent outAug 14 19:11:40 19:11:40.192283:CID-0:RT: post addr xlation: 1.1.1.30->1.1.1.100.
Aug 14 19:11:40 19:11:40.192283:CID-0:RT:mbuf 0x42308a00, exit nh 0x80010
Aug 14 19:11:40 19:11:40.192283:CID-0:RT: ----- flow_process_pkt rc 0x0 (fp rc 0)
datapath-debug
method. For datapath-debug
instructions, refer to KB21563- How to capture packets on High-End SRX devices.monitor traffic interface
' command. This is supported on both SRX Branch and High-end SRX devices. For more information, see KB15779 - SRX Getting Started - Troubleshooting Commands.root@SRX210# set forwarding-options packet-capture file filename pcap files 10 size 10000
root@SRX210# set forwarding-options packet-capture maximum-capture-size 1500
root@SRX210# set firewall filter PCAP term capture from source-address 192.168.224.0/24
root@SRX210# set firewall filter PCAP term capture from destination-address 192.168.224.3/32
root@SRX210# set firewall filter PCAP term capture from protocol ospf
root@SRX210# set firewall filter PCAP term capture from protocol esp
root@SRX210# set firewall filter PCAP term capture then sample
root@SRX210# set firewall filter PCAP term capture then accept
root@SRX210# set firewall filter PCAP term allow-all-else then accept
root@SRX210# set interfaces fe-0/0/7 unit 0 family inet filter input PCAP
root@SRX210# commit and-quit
First, configure the packet capture file information, including the file name, size, copies, and the amount of traffic to capture per packet. The minimum amount of traffic that must be captured will be 68 Bytes (Layer 2-4) but a maximum of 1500 Bytes can be captured.
Next, a packet capture filter will be applied to select what packets should be captured in this configuration. Technically the SRX does not need to filter the traffic, however it is highly recommended to filter only the desired traffic, which is what is shown in this example. This is done by configuring the firewall filter which defines what to match. You can select the source-address, destination-address, source-port, destination-port, and protocol. The following packet capture is going to collect traffic that arrives on the interface with a source address in the prefix 192.168.224.0/24 AND destination-address 192.168.224.3 AND protocol OSPF AND protocol ESP. Also the action allow-all-else is used to make sure that the SRX does not drop any other traffic, but do not sample it either. It is important to remember that the packet capture will match the traffic as it leaves or arrives on the interface based on the Layer 3 or Layer 4 properties in the packet itself. So you need to take this into account when NAT is being performed on the packet, and match the traffic as is.
Lastly, apply the filter to the interface to define that it should capture the traffic. Then commit the configuration. In this example, only the traffic arriving on the input direction is captured, but the outbound direction can be set as well in another statement.
Disabling Packet Captures: To stop the packet capture, either deactivate or delete the configuration “interfaces fe-0/0/7 unit 0 family inet filter input PCAP” and optionally, remove the configuration “firewall filter PCAP” and “forwarding-options packet-capture” as shown below:
For additional information on the Packet Capture feature, refer to http://www.juniper.net/techpubs/en_US/junos11.4/information-products/topic-collections/security/software-all/monitoring-and-troubleshooting/index.html?config-pcap-chapter.html.root@SRX210# delete interfaces fe-0/0/7 unit 0 family inet filter input PCAP
(deactivates packet capture)
root@SRX210# delete firewall filter PCAP
(optional: delete filter if not using it again)
root@SRX210# delete forward-options packet-capture
root@SRX210> start shell
root@SRX210% cd /var/tmp
Either login to the SRX with the root account, or use the “start shell” command with a super-user class account permission.
Change directories to the /var/tmp directory
Issue the tcpdump command with the –r option to read the pcap formatted file (in this case the name pcap is used but that name is arbitrary). The –v option gives additional verboseness to the output and shows some additional info, and finally pcap.fe-0.0.0 is the name of the capture. There are other options that can be explored by viewing the tcpdump manual page at http://www.tcpdump.org/tcpdump_man.html.
root@SRX210% tcpdump –r –v pcap.fe-0.0.7
21:05:19.974301 In IP (tos 0xc0, ttl 64, id 3464, offset 0, flags [none], proto: ESP (50), length: 120) 192.168.224.1 > 192.168.224.3: ESP(spi=493530462,seq=0x324)
21:05:24.979969 In IP (tos 0xc0, ttl 1, id 31089, offset 0, flags [none], proto: OSPF (89), length: 72) 192.168.224.6 > ospf-all.mcast.net: OSPFv2, Hello, length 52
Router-ID 192.168.225.1, Backbone Area, Authentication Type: none (0)
Options [External]
Hello Timer 10s, Dead Timer 40s, Mask 255.255.255.0, Priority 1
Designated Router 192.168.224.3, Backup Designated Router 192.168.224.1
Neighbor List:
Juniper2350
192.168.230.1
21:05:24.980113 In IP (tos 0xc0, ttl 64, id 3500, offset 0, flags [none], proto: ESP (50), length: 120) 192.168.224.1 > 192.168.224.3: ESP(spi=493530462,seq=0x325)
21:05:26.977403 In IP (tos 0x0, ttl 64, id 3513, offset 0, flags [none], proto: ESP (50), length: 104) 192.168.224.1 > 192.168.224.3: ESP(spi=493530462,seq=0x326)
21:05:27.985879 In IP (tos 0xc0, ttl 64, id 3515, offset 0, flags [none], proto: ESP (50), length: 136) 192.168.224.1 > 192.168.224.3: ESP(spi=493530462,seq=0x327)
21:05:28.976092 In IP (tos 0xc0, ttl 1, id 3521, offset 0, flags [none], proto: OSPF (89), length: 72) 192.168.224.1 > ospf-all.mcast.net: OSPFv2, Hello, length 52
Router-ID 192.168.230.1, Backbone Area, Authentication Type: none (0)
Options [External]
Hello Timer 10s, Dead Timer 40s, Mask 255.255.255.0, Priority 1
Designated Router 192.168.224.3, Backup Designated Router 192.168.224.1
Neighbor List:
Juniper2350
192.168.225.1
Getting Up and Running with Junos
Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search