Support Support Downloads Knowledge Base Case Manager My Juniper Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

Troubleshoot MSRPC problems on firewalls running ScreenOS

0

0

Article ID: KB11951 KB Last Updated: 20 Mar 2019Version: 6.0
Summary:

This document provides basic guidelines to troubleshoot MSRPC related problems.  Each problem is considered on a case by case basis, depending on the customer network design and other various factors. In this document, the troubleshooting steps for a specific customer example are explained, but it can be considered as a basic guideline on how to troubleshoot MSRPC related problems. When in the real-time debugging, depending on the amount of traffic, a lot of output for "debug rpc map" is displayed.  It is always good to collect the buffer information and analyze the problem offline.

Symptoms:
  • Outlook client can send and receive emails, but has problem accessing the directory services for looking up user details in the address book and other name services connected with the email server.

  • Users lost the ability to search address book entries, and the only way to get back into service is to restart the Outlook client.

This problem can occurred in a scenario where the environment is hosted by multiple active directory domains and the user details can be part of any of the Active Directory servers in the domain. The mail server IP may be different from the Active Directory server.

Solution:
Following are the steps to troubleshoot such problem.  Because the Outlook client deals with MSRPC, this document will also gives insight on how to debug MSRPC-related problems.

To start , debug the problem in the flow level to understand if such packets are getting dropped or not.

Set the appropriate filters with source and destination IP and port (if you know the port #).  Generally for directory services, Microsoft uses either port 1026 or 1025:
set ff src-ip a.b.c.d dst-ip a.b.c.d
set ff dst-port 1026
Then run the commands to start the debug flow and view the packets captured:
clear db
undebug all
debug flow basic
<have Outlook client attempt to search address book entries that fail>
undebug all
get db stream
 
 
In our customer scenario, the debug showed that the firewall was dropping these packets with the 'packet dropped, first pak not sync' error:
****** 7669619.0: <Untrust/ethernet0/1> packet received [248]******
  ipid = 617(0269), @2e708910
  packet passed sanity check.
  ethernet0/1:10.10.16.22/5337->172.16.65.16/1026,6<Root>
  no session found
  flow_first_sanity_check: in <ethernet0/1>, out <N/A>
  packet dropped, first pak not sync
With this message, we understand that the packet came into the firewall and there was no existing session on the firewall.  Then because syn-check is enabled of the firewall (set flow tcp-syn-check), the packet is dropped because it is a non-syn packet.

Since there was no session on the firewall, it could be because the session died on the firewall when the timeout was reached or the session was closed abruptly for other reasons.

To figure this out, run the command "get session | i 1026" or "get session with the specific IP or port filter as below,  and check the time out value:
FW1(M)-> get session src-ip 10.10.16.22 dst-ip 172.16.65.16
alloc 3996/max 256064, alloc failed 0, mcast alloc 0, di alloc failed 0
total reserved 0, free sessions in shared pool 252068
Total 1 sessions according filtering criteria.
id 254152/s**,vsys 0,flag 08000040/0000/0001,policy 48,time 3, dip 0 module 0
 if 5(nspflag 801801):10.10.16.22/5353->172.16.65.16/1026,6,00000c07ac07,sess token 6,vlan 0,tun 0,vsd 0,route 5,wsf 0
 if 0(nspflag 801800):10.10.16.22/5353<-172.16.65.16/1026,6,00000c07ac07,sess token 4,vlan 0,tun 0,vsd 0,route 52,wsf 0
Total 1 sessions shown


FW1(M)-> get session id 254152
id 254152(0003e0c8), flag 08000040/0000/0001, vsys id 0(Root)
policy id 48, application id 0, dip id 0, state 0
current timeout 20, max timeout 60 (second)
status normal, start time 7669707, duration 0
session id mask 0, app value 0
From the above output, note that the session timeout is set to 60 seconds max.  Therefore a session is cleared from the firewall when there is no traffic for more than 60 seconds. It is common in an Outlook client scenario to want a greater timeout because the client queries one user in the address book and may not do it again for other users after approximately 5 minutes.

But according to the config, the policy used on the session has a service "MSRPC-ANY" with a timeout of 240 minutes, but still the timeout on the session is 60 seconds.

To figure why the value of 60 seconds is set on the session, further debugging is needed using MSRPC debug commands, as this is related to MSRPC traffic from Microsoft.


MSRPC Troubleshooting
====================

MSRPC OVERVIEW

RPC is a protocol that allows a program running on one machine to call procedures in a program running on another machine.
  • There are two flavors of RPC: ONC/RPC and DCE/RPC. MSRPC is Microsoft’s implementation of DCE/RPC (Sun RPC conforms to ONC/RPC).

  • Sun RPC ALG and MSRPC ALG have similar requirements and design considerations. They both use a port-mapping channel (TCP/UDP 111 for Sun RPC and TCP/UDP 135 for MSRPC) to find the dynamic ports for services.

  • Sun RPC services are identified by 32-bit program numbers. MSRPC services are identified by 16-octet interface UUID’s. 
The ALG (Application Layer gateway) on the firewall for both requires service-based firewall policy enforcement. RPC dynamic port negotiation is different from the case of FTP and H.323 in the sense that the control channel (portmapper) and data channel (specific RPC services) are completely independent; they don’t have a parent-child relationship. For this reason, the gate mechanism used by FTP and H.323 does not match the RPC requirement.  ScreenOS needs to use a service-port mapping table and a software implemented RPC rule table. It also needs a UUID2OID table and a RPC service timeout table.


MSRPC MAPPING TABLE

To start with the troubleshooting, check whether or not the MSRPC mapping table is built on the firewall:
FW1(M)-> get service map-ms-rpc

MS RPC service mapping - hashed by (vsys, ip, port, prot)
#    Bkt   Vsys    IP              Port    Prot   UUID
0    21    Root    10.10.101.57    21278   TCP    a4f1db00-ca47-1067-b31f-00dd010662da
1    27    Root    172.16.65.13    1026    TCP    e3514235-4b06-11d1-ab04-00c04fc2dcd2
2    27    Root    172.16.65.13    1026    TCP    12345678-1234-abcd-ef00-01234567cffb
3    27    Root    172.16.65.13    1026    TCP    f5cc5a18-4264-101a-8c59-08002b2f8426
4    29    Root    172.16.65.14    1026    TCP    e3514235-4b06-11d1-ab04-00c04fc2dcd2
5    29    Root    172.16.65.14    1026    TCP    12345678-1234-abcd-ef00-01234567cffb
6    29    Root    172.16.65.15    1026    TCP    12345678-1234-abcd-ef00-01234567cffb
7    29    Root    172.16.65.15    1026    TCP    e3514235-4b06-11d1-ab04-00c04fc2dcd2
8    29    Root    172.16.65.14    1026    TCP    f5cc5a18-4264-101a-8c59-08002b2f8426
9    29    Root    172.16.65.15    1026    TCP    f5cc5a18-4264-101a-8c59-08002b2f8426

 
The above table shows the mapping of the MSRPC table; it contains the destination IP / port number and the UUID associated with that request.
The Bkt field contains the internal naming mechanism used to identify the UUID in 'debug flow'.
The firewall will dynamically create this table based on the MSRPC traffic traversing the firewall and by looking at the UUID in the packet.

Note:  There could multiple duplicate destination ports and multiple duplicate source IP's in the table, but they will have different UUID's mapped to it.
If you find the same UUID, then you will see a different IP or destination port mapped to it.
This is normal behavior in the MSRPC environment.

Note: To clear the MSRPC mapping table manually you can execute "clear service rpc-map"


MSRPC UUID

After retrieving the MRRPC Mapping Table, then one needs to find out which UUID associated with the traffic coming into the firewall and creating the session with the 60 seconds timeout.  This is done with an additional MSRPC debug.

Run the debug flow basic with appropriate flow filters along with "debug rpc map":
set ff src-ip a.b.c.d dst-ip a.b.c.d
set ff dst-port 1026


clear db
undebug all
debug flow basic

debug rpc map (This debug will tell us what UUID is matching in the flow)
<have Outlook client attempt to search address book entries that fail>
undebug all
get db stream
The data captured will show the following.

Keep in mind that IP 10.10.16.22 is a client.  IP 172.16.65.16 is the server.
Also, the customer had more than one server and the IPs were 172.16.65.13, 172.16.65.14, 172.16.65.15, and 172.16.65.16.
****** 7769171.0: <Untrust/ethernet0/1> packet received [248]******
  ipid = 19349(4b95), @2e7ed910
  packet passed sanity check.
  ethernet0/1:10.10.16.22/6981->172.16.65.16/1026,6<Root>
  no session found
  flow_first_sanity_check: in <ethernet0/1>, out <N/A>
  chose interface ethernet0/1 as incoming nat if.
  flow_first_routing: in <ethernet0/1>, out <N/A>
  search route to (ethernet0/1, 10.10.16.22->172.16.65.16) in vr trust-vr for vsd-0/flag-0/ifp-null
  [ Dest] 52.route 172.16.65.16->172.27.253.49, to ethernet0/0
  routed (x_dst_ip 172.16.65.16) from ethernet0/1 (ethernet0/1 in 0) to ethernet0/0
  policy search from zone 1-> zone 2
 policy_flow_search  policy search nat_crt from zone 1-> zone 2
## 2008-07-16 16:47:29 : rpc_map_search: search for (0, 172.16.65.16, 1026, 6)
## 2008-07-16 16:47:29 : rpc_map_search: found MS RPC uuid: 12345678-1234-abcd-ef00-01234567cffb
## 2008-07-16 16:47:29 : msrpc_uuid2oid_search: search for uuid 12345678-1234-abcd-ef00-01234567cffb
## 2008-07-16 16:47:29 : msrpc_uuid2oid_search: bucket 91 uuid: 12345678-1234-abcd-ef00-01234567cffb
## 2008-07-16 16:47:29 : msrpc_uuid2oid_search: found match at bucket 91, msrpc_id 0x8000002b
## 2008-07-16 16:47:29 : rpc_map_add_svc_to_list: svc 0x8000002b added to index 1
## 2008-07-16 16:47:29 : rpc_map_search: found MS RPC uuid: f5cc5a18-4264-101a-8c59-08002b2f8426
## 2008-07-16 16:47:29 : msrpc_uuid2oid_search: search for uuid f5cc5a18-4264-101a-8c59-08002b2f8426
## 2008-07-16 16:47:29 : msrpc_uuid2oid_search: bucket 244 uuid: f5cc5a18-4264-101a-8c59-08002b2f8426
## 2008-07-16 16:47:29 : msrpc_uuid2oid_search: found match at bucket 244, msrpc_id 0x80000007
## 2008-07-16 16:47:29 : rpc_map_add_svc_to_list: svc 0x80000007 added to index 0
## 2008-07-16 16:47:29 : rpc_map_search: found MS RPC uuid: e3514235-4b06-11d1-ab04-00c04fc2dcd2
## 2008-07-16 16:47:29 : msrpc_uuid2oid_search: search for uuid e3514235-4b06-11d1-ab04-00c04fc2dcd2
## 2008-07-16 16:47:29 : msrpc_uuid2oid_search: bucket 12 uuid: e3514235-4b06-11d1-ab04-00c04fc2dcd2
## 2008-07-16 16:47:29 : msrpc_uuid2oid_search: found match at bucket 12, msrpc_id 0x80000002
## 2008-07-16 16:47:29 : rpc_map_add_svc_to_list: svc 0x80000002 added to index 2
  RPC Mapping Table search returned 3 matched service(s) for (vsys Root, ip 172.16.65.16, port 1026, proto 6)
  first RPC service matched: uid -2147483605(0x-2147483605)
  SW RPC Rule Table match: uid -2147483605(0x8000002b), polid id 48, index 1
  Permitted by policy 48
  No src xlate   choose interface ethernet0/0 as outgoing phy if
  check nsrp pak fwd: in_tun=0xffffffff, VSD 0 for out ifp ethernet0/0
  vsd 0 is active
  no loop on ifp ethernet0/0.
  session application type 0, name None, nas_id 0, timeout 60sec
  service lookup identified service 0.
  flow_first_final_check: in <ethernet0/1>, out <ethernet0/0>
  existing vector list 123-368d5a4.
  Session (id:250697) created for first pak 123
  flow_first_install_session======>
  route to 172.27.253.49
  arp entry found for 172.27.253.49
  ifp2 ethernet0/0, out_ifp ethernet0/0, flag 00800800, tunnel ffffffff, rc 1
  outgoing wing prepared, ready
  handle cleartext reverse route
  search route to (ethernet0/0, 172.16.65.16->10.10.16.22) in vr trust-vr for vsd-0/flag-3000/ifp-ethernet0/1
  [ Dest] 5.route 10.10.16.22->172.27.253.1, to ethernet0/1
  route to 172.27.253.1
  arp entry found for 172.27.253.1
  ifp2 ethernet0/1, out_ifp ethernet0/1, flag 00800801, tunnel ffffffff, rc 1
  nsrp msg sent.
  flow got session.
  flow session id 250697
  vsd 0 is active
  tcp seq check.
  post addr xlation: 10.10.16.22->172.16.65.16.
 flow_send_vector_, vid = 0, is_layer2_if=0
From the above output note that the packet came in and the MS-RPC mapping table was searched.  More than one rule matching the Destination IP and Destination Port of 1026 was found.  Remember that  "1026" is a common port for MS Directory Services and any UUID can match this port.

At this point, which of the three UUIDs that were matched is the one we are interested in?
Look closely at the output from the debugs:
## 2008-07-16 16:47:29 : rpc_map_search: found MS RPC uuid: 12345678-1234-abcd-ef00-01234567cffb
## 2008-07-16 16:47:29 : msrpc_uuid2oid_search: search for uuid 12345678-1234-abcd-ef00-01234567cffb
## 2008-07-16 16:47:29 : msrpc_uuid2oid_search: bucket 91 uuid: 12345678-1234-abcd-ef00-01234567cffb
## 2008-07-16 16:47:29 : msrpc_uuid2oid_search: found match at bucket 91, msrpc_id 0x8000002b
## 2008-07-16 16:47:29 : rpc_map_add_svc_to_list: svc 0x8000002b added to index 1
After the UUID is found (in the first line), there is an index number set by the flow:  "0x8000002b".   This number is a pre-defined number for each UUID on the firewall .
So to match the right UUID , look further in the debugs to see which index is matching.  In the 3 lines below, you can see that 0x8000002b was used with policy id 48:

     RPC Mapping Table search returned 3 matched service(s) for (vsys Root, ip 172.16.65.16, port 1026, proto 6)
  first RPC service matched: uid -2147483605(0x-2147483605)
  SW RPC Rule Table match: uid -2147483605(0x8000002b), polid id 48, index 1


The debugs above show that  0x8000002b maps to UUID: 12345678-1234-abcd-ef00-01234567cffb.


MSRPC SERVICE

Now, the service that maps to that UUID needs to be determined.

Following is some of the pre-defined MS-RPC services with the UUID mapping (which were retrieved from the Functional Spec and RFCs):
MS-RPC-EPM              e1af8308-5d1f-11c9-91a4-08002b14a0fa
MS-AD-BR                ecec0d70-a603-11d0-96b1-00a0c91ece30
                        16e0cf3a-a604-11d0-96b1-00a0c91ece30
MS-AD-DRSUAPI            e3514235-4b06-11d1-ab04-00c04fc2dcd2
MS-AD-DSROLE            1cbcad78-df0b-4934-b558-87839ea501c9
MS-AD-DSSETUP            3919286a-b10c-11d0-9ba8-00c04fd92ef5
MS-DTC                    906b0ce0-c70b-1067-b317-00dd010662da
MS-EXCHANGE-DATABASE    1a190310-bb9c-11cd-90f8-00aa00466520
MS-EXCHANGE-DIRECTORY    f5cc5a18-4264-101a-8c59-08002b2f8426                  
                        f5cc5a7c-4264-101a-8c59-08002b2f8426
                        f5cc59b4-4264-101a-8c59-08002b2f8426
MS-EXCHANGE-INFO-STORE    0e4a0156-dd5d-11d2-8c2f-00c04fb6bcde
                        1453c42c-0fa6-11d2-a910-00c04f990f3b          
                        10f24e8e-0fa6-11d2-a910-00c04f990f3b                
                        1544f5e0-613c-11d1-93df-00c04fd7bd09
MS-EXCHANGE-MTA            9e8ee830-4459-11ce-979b-00aa005ffebe   
                        38a94e72-a9bc-11d2-8faf-00c04fa378ff
MS-EXCHANGE-STORE    99e66040-b032-11d0-97a4-00c04fd6551d
                        89742ace-a9ed-11cf-9c0c-08002be7ae86
                        a4f1db00-ca47-1067-b31e-00dd010662da
                        a4f1db00-ca47-1067-b31f-00dd010662da
MS-EXCHANGE-SYSATD    67df7c70-0f04-11ce-b13f-00aa003bac6c
                        f930c514-1215-11d3-99a5-00a0c9b61b04
                        83d72bf0-0d89-11ce-b13f-00aa003bac6c
                        469d6ec0-0d87-11ce-b13f-00aa003bac6c
                        06ed1d30-d3d3-11cd-b80e-00aa004b9c30
MS-FRS                    f5cc59b4-4264-101a-8c59-08002b2f8426
                        d049b186-814f-11d1-9a3c-00c04fc9b232 
                        a00c021c-2be2-11d2-b678-0000f87a8f8e
MS-IIS-COM            70b51430-b6ca-11d0-b9b9-00a0c922e750
                        a9e69612-b80d-11d0-b9b9-00a0c922e750
MS-IIS-IMAP4            2465e9e0-a873-11d0-930b-00a0c90ab17c
MS-IIS-INETINFO      82ad4280-036b-11cf-972c-00aa006887b0
MS-IIS-NNTP            4f82f460-0e21-11cf-909e-00805f48a135
MS-IIS-POP3            1be617c0-31a5-11cf-a7d8-00805f48a135
MS-IIS-SMTP            8cfb5d70-31a4-11cf-a7d8-00805f48a135
MS-ISMSERV            68dcd486-669e-11d1-ab0c-00c04fc2dcd2
                        130ceefb-e466-11d1-b78b-00c04fa32883
MS-MESSENGER            17fdd703-1827-4e34-79d4-24a55c53bb37
                        5a7b91f8-ff00-11d0-a9b2-00c04fb6e6fc
MS-MQQM                    fdb3a030-065f-11d1-bb9b-00a024ea5525
                        76d12b80-3467-11d3-91ff-0090272f9ea3
                        1088a980-eae5-11d0-8d9b-00a02453c337
                       5b5b3580-b0e0-11d1-b92d-0060081e87f0
                       41208ee0-e970-11d1-9b9e-00e02c064c39
MS-NETLOGON           12345678-1234-abcd-ef00-01234567cffb
MS-SCHEDULER           1ff70682-0a51-30e8-076d-740be8cee98b
                       378e52b0-c0a9-11cf-822d-00aa0051e40f
                       0a74ef1c-41a4-4e06-83ae-dc74fb1cdd53
MS-WIN-DNS           50abc2a4-574d-40b3-9d66-ee4fd5fba076
MS-WINS                   45f52c28-7f9f-101a-b52b-08002b2efabe
                       811109bf-a4e1-11d1-ab54-00a0c91e9b45

 
From the above pre-defined UUID table, the UUID "12345678-1234-abcd-ef00-01234567cffb" is part of the "MS-NETLOGON" service

Check the service timeout for this service on the firewall.
get service ms-netlogon         
Name:       MS-NETLOGON
Category:   other          ID:  0   Flag:  Pre-defined

Transport    UUID                                 Timeout(min) Application
MS-RPC       12345678-1234-abcd-ef00-01234567cffb           1     
  
By default ,the timeout for this service is 1 minute / 60 seconds and that is the reason why the session was getting installed with 60 seconds timeout.

Changing the timeout value for this service will resolve the problem.



MSRPC TIMEOUT  NOTES

There are scenarios where the service "MS-RPC-ANY" is configured for a higher timeout value, and MS-RPC-ANY is suppose to cover all the MS-RPC services including the MS-NETLOGON, but still the session is installed with a 60 seconds timeout and the "MS-RPC-ANY" timeout is ignored.

MS-RPC-ANY will allow any MS-RPC service or UUID to allow as long as the action is permit. The pre-defined timeout for MS-RPC-ANY is 1 minute / 60 seconds
get service ms-rpc-any
Name:       MS-RPC-ANY
Category:   other          ID:  0   Flag:  Pre-defined

Transport    UUID                                 Timeout(min) Application
MS-RPC       ANY                               1       
If the user modified the value for MS-RPC-ANY and if the policy is matched where the MS-RPC-ANY is defined, then any session getting installed thru that policy will have the timeout defined for the MS-RPC-ANY, which is a higher timeout value.

In the above scenario, this would have helped to resolve the problem, but this can cause side effects on the session table based on the number of packets traversing the firewall for MSRPC. For example if the MS-RPC-ANY has a higher timeout value and there are lot of packets hitting the firewall for MS-RPC, the session table is open with the higher timeout value, and if the server or client is not closing the session properly be sending a RST or FIN, then these sessions are sitting idle on the firewall and can fill up the session table.

If the same scenario is defined as a group service in the policy like below:
set policy id 48 from "Untrust" to "Trust"  "Any" "Any" "MS-RPC-ANY" permit log
set policy id 48
set service "MS-RPC-EPM"
set service "MS-AD"
set service "MS-EXCHANGE"
exit
This is considered as a policy with a group of services or a multi-cell policy.  When in this condition, the session timeout lookup logic is different (see logic steps below) and that is why the "MS-RPC-ANY" timeout does not kick in, if it is part of the multi-cell policy.
 
  1. If the policy is using a specified service entry other than “Any”, ScreenOS will get the timeout value from the service entry directly. If the policy is using a service group or is a multi-cell service policy, go to (c).

  2. If the policy is using the “Any” service, if the timeout value of the “Any” service is configured by user, ScreenOS will get the value directly. Otherwise go to (c).

  3. To find the service timeout according to the dst-port in our service timeout table. If the port is not overlapped, ScreenOS will get the value directly. If the port is overlapped, go to (d).

  4. If the logic is going to (d), it means that the policy is using a service group or is using service “Any”. If the policy is using service “Any”, ScreenOS will go to (e). Otherwise, ScreenOS will try to find the matched service in the group referenced by the matched policy according to the destination port.
     
    - Go through all the service entries on the group, if ScreenOS gets the matched one, ScreenOS will get the service timeout value.

    - Go through each sub group and match the all the service entries in the sub group, if ScreenOS gets the matched one, ScreenOS will get the service timeout  value.

  5. If ScreenOS still gets a zero value for service timeout (the default timeout value of a user-defined service is zero), ScreenOS will get the system default service timeout value according to the protocol.
Modification History:

2019-03-20: Updated 'get session filters'

Related Links

Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search