Support Support Downloads Knowledge Base Case Manager My Juniper Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

Troubleshooting ISG/IDP policy push failures

0

0

Article ID: KB10738 KB Last Updated: 06 Oct 2020Version: 11.0
Summary:

A firewall policy with IDP rules can fail to be pushed to ISG/IDP for different reasons. ISG has two parts of the policy: firewall and IDP. When NSM compiles the policy, it generates the CLI for the firewall policy and compiles/pushes the IDP policy to the device.

This article details how to troubleshoot these failures.

 

Symptoms:

Symptoms and Errors:

  • Policy installation on the device can fail due to the The file might be invalid -1 error.

  • Compilation errors on NSM.

  • Failure to install a policy on the device.

  • Security module (SM) is in the hung state; the management module (MM) cannot push the policy to it.

  • NSM compiles the IDP policy for a specific detector version, but the policy push failed.

  • A signature fails to compile in the security module; phase1 of the policy push fails and the new policy is not installed.

  • At the time of policy push, the security module can coredump.

  • Insufficient memory for the security module.

  • Policy push succeeds with warnings.

 

Solution:

The policy push might fail for two reasons:

  • Firewall policy push failure

    Failure when generating or pushing CLI for the firewall policy. This might occur due to NSM generating the wrong or unsupported CLI.

    Solution:
    • Check the gproDDM.log file for the exception error.

    • Run the summarize config command and look for unusual CLI.
  • IDP policy push failure
    IDP policy push can fail due to:
     
    • Compilation errors on NSM

      or

    • Failure to install the policy on the device. For additional troubleshooting tips, refer to the section below.

Policy compilation in NSM

Policy compilation in NSM involves the creation of the following files under /usr/netscreen/DevSvr/var/idp/:
 
  • 1_x.cmpin (before the policy compiler)

  • 1_x.cmpout (after the policy compiler)

  • 1_x.cmpout.gz (compressed with gzip)

  • 1_x.cmpout.gz.v (compressed with gzip + header)

    Where 1 is the domain ID, and x is the device ID.

These files are overwritten with every policy push in NSM 2007.x or later. Policy compilation can fail due to various reasons:

  • IDP compiler invocation failure:

    This problem happens when the gmp package is not installed and the compilation fails. When NSM compiles the IDP policy, it requires the gmp package for the compilation to work. To verify if the gmp package is installed, see KB9098 - [Archive] NSM 2006.1 ISG IDP Invocation Compiler error.

  • Missing libraries in the installed OS version. To check for missing libraries, see KB8761 - [Archive] IDP Policy compilation fails on RedHat ES3 Update 8.

  • Missing signatures in the signature group or wrong signature format. These failures can be seen in the UI.

  • If the policy is still failing, place devSvrDirectiveHandler in debug mode (in DevSvr.cfg), compile the policy manually via the CLI, and check for errors in gproDDM.log. To compile the policy manually, run the following on the NSM server:
    #export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/lib:/usr/lib:/usr/netscreen/DevSvr/lib/gcc-3.4.3/linux_86/lib/

    #/usr/netscreen/DevSvr/utils/policy_compiler -i /usr/netscreen/DevSvr/var/idp/1.x_0.cmpin -o /usr/netscreen/DevSvr/var/idp/1.x_0.cmpout -b /usr/netscreen/DevSvr/var/idp/dfa_cache -r 5.4IDP

Troubleshooting steps involved in understanding the IDP policy compilation failures on NSM
When an IDP policy is pushed to firewall devices (ISG-IDP devices), the IDP policy is compiled on the NSM itself and NSM pushes the compiled policy to the firewall. In the case of standalone IDP devices, the sensor device itself takes care of compiling the policy.

If the job manager displays any IDP policy compilation failure errors, then the following steps can be performed to understand the reason for failure:
 
  1. Log in to the NSM server (device server if the GUI and DEV are installed on separate servers) via SSH and become the root user.

  2. Change to the location /usr/netscreen/DevSvr/var/errorLog.

  3. Under this directory, run the following command to list the recent IDP policy compilation commands printed in the log file:

         grep cmp deviceDamon.0

  4. The above command should result in the compilation command internally run by the NSM server. An example output is shown below:

         deviceDaemon.5:[10/29/2008 09:31:00.732] [Notice] [13-nsIdpCompilerComm.c:319]
         Invoking: /usr/netscreen/DevSvr/utils/policy_compiler -i /usr/netscreen/DevSvr/var/idp/4.11_13.cmpin 
         -o /usr/netscreen/DevSvr/var/idp/4.11_13.cmpout -b /usr/netscreen/DevSvr/var/idp/dfa-cache -r 6.0

  5. The same command can be run with slight modification manually on the command line to verify if any missing libraries might be causing the compilation failure. Modification needed in the command is just the output filename.

         /usr/netscreen/DevSvr/utils/policy_compiler -i /usr/netscreen/DevSvr/var/idp/4.11_13.cmpin 
         -o /usr/netscreen/DevSvr/var/idp/4.11_13.cmpout_new -b /usr/netscreen/DevSvr/var/idp/dfa-cache -r 6.0

  6. Result of this command will give clues to the problem in compilation which might not be clearly printed on the Job Manager window of the NSM GUI client.

Policy installation on device can fail due to error “The file might be invalid -1”

This is a generic error indicating that the policy was compiled successfully in NSM but failed to get installed on the device. There might be several reasons for this failure. To understand this, we need to know how IDP policy is installed on the device

 

Note: Before continuing, make sure the latest Detector Engine is loaded on the device.
To update the Detector version on the firewall, see KB9773 - How to update the detector version on IDP.
If you are having problems in updating the detector on ISG, see KB10515 - Error loading a new IDP Detector Engine on ISG-IDP.
Also see KB9769 - IDP Detector Engine FAQ.

IDP policy installation workflow on firewall

NSM compiles the policy successfully and sends the compiled policy (firewall and IPD) to the firewall Mgt board. The Mgt board extracts the firewall policy and IDP policy in two parts.
IDP policy loading happens in two phases:

Phase I
  • Management module (MM) sends this policy to all the security modules (SM) installed in the device.
  • Each SM receives the compile message from the MM, decompresses and parses the policy, and loads it into memory. If this is successful, all SMs send a successful message to the MM.

Phase II
  • Only after the SMs return a success message, the MM sends an install message to all the SMs to install the new policy.
  • Each SM installs the policy and saves a copy in RAM of the MM.
  • The new policy is copied to the /idp/bin folder in the SM.
  • To check if the policy is installed successfully, verify the timestamp of policy.gz.v in /idp/bin:
       #exec sm # ksh “ls –l /idp/bin/policy.gz.v”
    where ‘#’ is number of SM.
  • After the policy is installed in the SM, the MM also keeps the last ISG-IDP policy on flash.

Case 1: SM is in a hung state
If an SM is in a hung state, MM cannot push the policy to it. Check the status of the SMs:
#get chassis – Shows you the number of modules installed on the device
#get sm status

Make sure the SM CPUs are enabled and available.
Case 2:  Specific detector version
NSM compiles the IDP policy for a specific detector version. If the detector version on the device is different, then policy push will fail.
To check the detector version on the device:
#get system
IDP files version:
detector.so 3.1.99116


This indicates the device has the 3.1.99116 detector version.
To check the detector on NSM:
Edit the device, Security > IDP SM Settings > IDP Detector version.

Make sure you have the same detector version in NSM and on the device.

To update the detector version on the firewall, see KB9773 - How to update the detector version on IDP.
If you are having problems in updating the detector on ISG, see KB10515 - Error loading a new IDP Detector Engine on ISG-IDP.
Also see the Detector Engine FAQ - KB9769 - IDP Detector Engine FAQ.


Case 3: Signature fails to compile 
When a signature fails to compile in the security module, phase1 of the policy push fails and the new policy is not installed. In order to find if a bad signature is causing the issue, run the following debugs on one security module:
# exec sm # ksh “scio const set d_log 1”
# exec sm # ksh “scio const set sc_debug_level 3”
# exec sm # ksh “scio const set d_subs 1”

Re-push the policy from NSM and check the output of the debug logs with the command:
#exec sm # ksh “sloginfo”
Look for error messages in sloginfo similar to the following:
sc_pmanager_policy_compile: before parser sc_malloc_comsumed = 204964688
sc_ids_parse_context: bad context string ldap-modifydn-request
sc_ids_parse_attack_fields: sc_ids_parse_context() failed
sc_ids_parse_attack: sc_ids_parse_attack_fields(LDAP:SUN-REQ-FS) failed
sc_ids_parse_context: bad context string ldap-modifydn-request
sc_ids_parse_attack_fields: sc_ids_parse_context() failed
sc_ids_parse_data: sc_ids_parse_load_attacks() failed
sc_ids_parser: sc_ids_parse_data() failed
sc_parser_compile: parser for ids failed
sc_pmanager_policy_compile: sc_parser_compile() failed
In the above error message it is clear that the compilation was failing on signature “LDAP:SUN-REQ-FS”. Remove this signature from the policy and update the device to see if this resolves the problem.

Once the policy push succeeds, turn off the debugs with the following command:
#exec sm # ksh “scio const set d_log 0”
#exec sm # ksh “scio const set sc_debug_level 1”
#exec sm # ksh “scio const set d_subs 0”
Case 4: Memory required 
The security module needs sufficient memory to compile the policy. Usually with a big policy, SM needs 600 MB of memory to compile the policy. An insufficient memory situation can occur when SM is busy processing traffic, and this can cause a policy push failure.
 
  • Checking the sloginfo (exec sm # ksh “sloginfo”) file should show malloc failure or not enough memory to allocate error messages.

  • Memory status can also be checked with the pidin info cmd.
    #exec sm # ksh "pidin info
    CPU:PPC Release:6.3.0 FreeMem:292Mb/1536Mb BootTime:Mar 06 10:30:50 UTC 2007


    The above output shows that the SM only has a free memory of 290M.
  • Check the memory used by the current policy:
    #exec sm # ksh “scio policy list s0.

    If the policy uses more than 150 – 200 MB of RAM, it is a huge policy. We can unload the last policy from RAM to free up the memory.

    #exec sm # ksh "scio const -s s0:flow set sc_flow_reset_on_policy 1"
    #exec sm # ksh “scio policy unload s0”
    When the policy is unloaded in the SM, the existing sessions are put to cut-through (i.e., the sessions will be only going through firewall inspection). However, new sessions will be inspected once the new policy is pushed.
Case 5: Security module coredumps 
At the time of policy push, if the security module coredumps due to an error, then the SM process will restart, causing a policy failure. Check for coredumps in the /idp/log directory:

      # exec sm # ksh “ls –l /idp/log

     Check for the files: Engine.core, engine.1.core, and pcid.core.

If a coredump is seen, collect the coredumps, idpinfo, and tech-support from the firewall and open a TAC case.

To collect coredumps and idpinfo:
 
  • exec sm <#> “ls –l /idp/log”

    Do this once for each module to check for core on every module and also display the timestamp.

    exec sm <#> “ls –l /idp/bin”

  • TFTP the core files from SM to a TFTP server.

    # exec sm <#> save tftp <tftp-ip> <filename> from /idp/log/engine.core

    If there are multiple core files, get all the corefiles from the SMs, for example: engine.core, engine.1.core, etc. If there are hundreds of files, get only a few.

  • TFTP the idp.info file. This has the memory address that will give an indication of what might have caused the crash.

    # exec sm <#> save tftp <tftp-ip> <filename> from /idp/log/idp.info

  • TFTP the binary files of the process that generated the core. For example, if the core is generated by pcid, you need to provide the pcid binary.

    # exec sm # save tftp <tftp-ip> <filename> from /idp/bin/pcid

  • Log the output of the following get commands:

    exec sm <#> ksh “sloginfo

    get tech-support
    get sm status
    get sm pkt
Case 6: Policy push succeeds with warnings
When a policy is pushed from NSM to ISG with IDP modules, a warning is issued by NSM stating that some attacks/groups cannot be updated to the device.
 
The following attacks/groups can not be updated (see "Reason Code" column below):

IDP Attack/Group Name                  Attack Type         In Rules      Reason Code
-------------------------------------------------------------------------------------------------
 SIP: Ethereal SIP Decoder Exploit     predef signature       I-1             6

 DNS: Non-RFC1035 Type Used            predef signature       I-1             6

 Reason Codes:
(6)      This attack is not supported by the current detector on the device. A detector update may be required

IDP policy update will only select those attack versions that are valid for the latest detector on the device. None of the IDP or ISG-IDP detectors support all the attack objects. If the signature is obsolete and is not applicable for the current version of the detector, the signature will not be pushed. Similarly, if the signature is applicable only to a higher version of the detector, it will not be pushed to devices running a lower version of the detector.

Edit the signature in NSM to find the supported platforms (detector version) for a signature. Following are the detector versions supported on different OS:
ScreenOS 6.3.x, 6.2.x 3.5.xxxxxx
ScreenOS 6.0.x, 6.1.x 3.4.xxxxxx
ScreenOS 5.4
ScreenOS 5.0
3.1.xxxxxx
IDP 5.1.x 5.1.xxxxx
IDP 5.0.x 5.0.xxxxx
IDP 4.1.x 4.1.xxxxx
IDP 4.0.x 4.0.xxxxx

Only the signatures applicable for the specific detector version on the device can be pushed.

 

Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search