Support Support Downloads Knowledge Base Case Manager My Juniper Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[SRX] Overview of common problems when using ThreatSTOP with SRX

0

0

Article ID: KB25813 KB Last Updated: 19 Nov 2020Version: 2.0
Summary:

This article provides an overview of common problems that are encountered when an SRX device is used with the ThreatSTOP IP Reputation service.

 

Symptoms:
  • Overview of common problems when SRX is used with the ThreatSTOP IP Reputation service

  • Debugging issues with ThreatSTOP

 

Solution:

The problems encountered tend to be due to the fact that ThreatSTOP will add many thousands of additional lines to a configuration (in the Address book version at worst case it is just over 60,000, about half that in the prefix-list one) and that the configuration is updated regularly (by default every 2 hours) from a cron job. The result of this is that commit times for any change can take a while – particularly when the SRXes are clustered – and there can be problems with the file system because the configs are so big. ThreatSTOP also exercises the syslog functions of the SRX and there is a known issue there – see FTP section below. As a general rule, none of the issues are ThreatSTOP-specific but we do tend to stress the SRX so that problems occur.

Methods to Apply Blocklists

ThreatSTOP offers two different ways to apply its blocklists to traffic passing through Juniper SRX devices.

  1. The first method uses SRX's security zones, address books, and security policies for enforcement.

  2. The second method uses firewall filters and prefix lists (a.k.a. ACLs).

On branch SRX devices (SRX100-SRX650), the firewall filter method is recommended, as this permits many more blocks. It is also generally faster and has lower impact on the CPU; however, it is also less intuitive to set up particularly if other firewall filters are active on the device.

On high-end SRX devices (SRX1400 through SRX5800), either method seems to work equally well although the filter code does provide a shorter commit. On the other hand, the address book method fits in better with most recommendations for how to manage firewalls.

The address book code is installed in the /root/ts-juniper directory; the filter code is installed in /root/ts-jfilter.

How does it work?

ThreatSTOP’s service works by downloading the latest list of untrusted IPs periodically from our servers using DNS and applying them as targets to rules. ThreatSTOP does not mandate how these targets are used in the rules although we recommend that they be used to deny traffic and log the connection attempt. Most users implement them this way. There is also a list of trusted (allowed) IPs that is downloaded and which is expected to be used to explicitly allow traffic. Normally the allow rule will be set to hit ahead of the deny rule. The firewall also periodically uploads its firewall logs to ThreatSTOP so that we can analyze the logs for blocked traffic and produce a report.

During installation ThreatSTOP puts a little shell script on the SRX that is called from cron to periodically (by default every 2 hours) update the address-books or prefix lists. The script first uses our patent pending DNS method to retrieve the data and store it in temporary files. It then creates a commit op script that will apply the changes and then finally it calls the “cli” program to enter configuration and commit the changes. The ThreatSTOP scripts all have to run as root in order to have the correct permissions to work.

In both variants, the addresses are broken down into groups of 1,000 and you will see up to 30 address sets/prefix lists that start with ThreatSTOP-block in the configuration.

Basic Debugging of Possible ThreatSTOP Problems

  1. No processes available / FTP Log upload failures

There is an open case with JTAC which has been partially resolved where either the log rotate/upload process or the syslog process encounters a null pointer and then bad things tend to happen. This has nothing to do with ThreatSTOP per se except that we tend to write more to log files and upload them more often. If the problem is not corrected, then eventually the SRX runs out of processes and a ps -ax command displays a large number of lines like the following:

4027  ??  S      0:00.00 /usr/sbin/eventd -N -r -s -A  
4028  ??  S      0:00.02 sh -c /usr/libexec/transfer-file /var/log/upload.BjUc  
4029  ??  S      0:00.03 /bin/sh /usr/libexec/transfer-file /var/log/upload.Bj  
4032  ??  S      0:00.05 fetch -L -o ftp://ftp@172.21.203.36/logs/beavis_64.87
  1. SRX overloaded/unstable

If the problem is suspected to be due to the size of the list, then the best thing to do is reduce it. The best way to do this is to get the customer to login to their ThreatSTOP account and uncheck most of the blocklists they have selected on the account – just leave the BASIC list checked. Then once that has been saved, wait about 15 minutes and force an update by running the line in the root cron tab (crontab –l to show it). This will work to reduce the size of the configuration and thus the commit time but it may not actually reduce the space used in the file system (see below).

Note: Deleting the prefix-lists or address book entries directly on the SRX is not a good idea as this will likely create problems in the config because of dependencies AND even if you sort those out, unless you stop the cron job you’ll see them come right back next time it runs. Also if you are running the filter/prefix-list variant and delete all ThreatSTOP prefix-lists, you will need to reinstall ThreatSTOP to get the script working again once you’ve fixed whatever other problem you have. For this variant, you should always have the ‘-00’ block and allow ThreatSTOP prefix lists present with at least one entry in each.

If the problem seems to be related to clashes of commits, then you may wish to edit the crontab to reduce the frequency and/or specify the time(s) precisely.

The cron tasks are usually set up like the following:

22 */2 * * * /bin/sh /root/ts-path/tsscriptname.sh

Replacing the */2 with, say, 1,8,18 would get you updates at 1:22am, 8:22am and 6:22pm. You can also comment the line out temporarily to ensure no conflict with a particular set of commits when doing major system changes.

If, for some reason, you have a failed commit then you will probably need to remove the ThreatSTOP op script from the config before you do anything else (delete system scripts commit file …) as without that no subsequent commits will be successful. Failed commits are usually due to the SRX running out of flash space (see below).

  1. ThreatSTOP service not working

In the event that the ThreatSTOP lists do not appear to be being updated, then normally you should ask the customer to contact ThreatSTOP support at this point. However the instructions below will help resolve a number of simple configuration issues.

You should check the SRX can ping www.threatstop.com and do DNS requests from our DNS servers (currently 64.87.26.147 and 24.249.204.58). A good way to test DNS is to do
dig dns.threatstop.local @64.87.26.147

If it times out (but ping works) then you may need to do the following in the SRX config :

set security alg dns disable

(Note it may also be a good idea to do the same thing for ftp so that logs can be uploaded)

If you get a response to the dig that is REFUSED or similar then probably the SRX’s external IP address is not in our database correctly. You can do the following from the SRX’s CLI to verify the SRX’s actual external IP address and whether it is in our database or not:

% fetch -qo - http://www.threatstop.com/cgi-bin/validip.pl
Your IP address: 79.84.187.19
Address is in the list of authorized hosts

If the response is in the negative then the address should be corrected in the device’s configuration at https://www.threatstop.com/

  1. File System Issues

The ThreatSTOP scripts are installed in directories below /root/ which is normally on the /cf partition. The scripts are tiny and do not take up meaningful space. However the scripts create a number of intermediate files in /tmp/ (normally on the /mfs partition) which can get to be quite large and they put op scripts in /var/db/scripts/commit/ (normally also on the /cf partition) which can also be large.

On branch SRX devices with limited flash space we have encountered two places where the sizes of the files we create (or cause to be created) are a problem. The first and simplest is that if the /cf partition is quite full then it may not be possible to copy the entire opscript to /var/db/scripts/commit/ . In that case the subsequent attempt to commit the op script will fail - generally with an error about syntax - and the SRX will need to be cleaned up before ThreatSTOP can run again successfully. We do check for this problem but there could be a race condition so that it happens anyway. As well as deleting unnecessary files, you must also delete the commit op script from the configuration (delete system scripts commit file …) as without that no subsequent commits will be successful.

The second place where we have hit problems are the files in /var/run/db/ which is also mounted on /mfs/ It seems that the various juniper.* files grow large but never reduce in size even when the configuration is small. In particular we have seen a problem when we have more than about 25,000 prefix-list addresses/networks where the size of these files grows rapidly to over 30MB each e.g. below

% ls -l /var/run/db/
total 198916
-rw-r--r-- 1 root config 3145728 May 29 15:57 chassisd.dynamic.db
-rw-r----- 1 root config 31457280 Jul 18 08:58 juniper.data
-rw-r----- 1 root config 31457280 Jul 18 08:58 juniper.db
-rw-r----- 1 root config 2621440 Jun 17 23:51 juniper.dyn
-rw-r----- 1 root config 31457280 Jul 18 08:58 juniper.save
drwxr-xr-x 2 root wheel 512 May 29 15:55 private
-rw-r----- 1 root config 13631488 May 29 15:55 schema.db

This is a problem because the partition is only 170 MB in size and, particularly in cluster mode, it seems that a number of temporary copies of these files are created during commit. The result is that the commit fails during a ThreatSTOP update with an error message about lack of space.

When there is no problem the mfs partition looks something like below but the partition can sometimes become more than 100% used which is definitely a problem!

% df /var/run/db/
Filesystem 512-blocks Used Avail Capacity Mounted on
/dev/md1 343720 152752 163472 48% /mfs

The workaround for this issue is to delete juniper.save and juniper.data, reduce the size of the configuration, do a commit (ignore the warnings about files missing) and then then reboot the SRX. In clustered environments you will need to reboot the primary, wait until it is back up and in the cluster as secondary and then reboot the secondary. Once you are back up and stable you should probably reduce the size of the ThreatSTOP bock lists so that this doesn’t happen again (see above). We have hard coded the maximum prefix list count in 3 digit SRXes to be 25,000 as opposed to 30,000 for 4 digit ones but it is possible that some complex configurations may require this limit to be reduced further. Lower maximum sizes can also be put in the scripts but the customer should contact ThreatSTOP support to work this through.

 

Modification History:

2020-11-19: Removed reference to forums.juniper.net

 

Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search