Support Support Downloads Knowledge Base Juniper Support Portal Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[Archive] [Contrail] Keystone-signing directories fill up /tmp folder and disrupt services

0

0

Article ID: KB34167 KB Last Updated: 26 May 2020Version: 2.0
Summary:

Due to a known software defect in python-keystonemiddleware package which is used by Contrail 3.x/4.x releases, i.e., Ubuntu Bug 1533724, hundreds of thousands of Keystone-signing directories fill up the /tmp directories overtime. Since there are so many of these directories created, the operating system is unable to write or access /tmp. A few contrail services require writing and reading the /tmp directory during startup. This causes some Contrail services to fail when trying to come up after hypervisor restarts. This issue is mainly reported from pre-3.2.7.0 releases but can be seen in pre-4.0.3.0 releases as well.

This article describes Juniper's proactive effort to address this keystone issue from the Contrail side which still uses pre-4.4.1-0ubuntu1~cloud0 Openstack version. Refer to the Juniper fix in Juniper Openstack Bug 1722787.

Symptoms:

Over 144k keystone-signing- folders are found in /tmp which drastically slows down system performance. When /tmp is filled up with too many keystone-signing-XXXX directories, the OS is unable to write or read to /tmp, hence affecting many Contrail services that require access to /tmp during startup.

root@control-002:/tmp# ls -al | grep keystone-sign | wc -l
144200
more info, the folders are owned by non-keystone services, mainly glance for us:
drwx------ 2 glance glance 4096 Jul 9 2015 keystone-signing-zZocUc
drwx------ 2 glance glance 4096 Jul 20 07:55 keystone-signing-ZZOibD
drwx------ 2 designate designate 4096 May 26 2015 keystone-signing-ZZoKgT
drwx------ 2 glance glance 4096 Jul 14 2015 keystone-signing-zzOmtb
drwx------ 2 glance glance 4096 Jul 12 2015 keystone-signing-zzOubp
drwx------ 2 glance glance 4096 Jul 15 17:22 keystone-signing-zzpD6x
drwx------ 2 designate designate 4096 Jun 9 2015 keystone-signing-ZzPeNQ
drwx------ 2 glance glance 4096 Jul 2 2015 keystone-signing-ZZPJ4H
drwx------ 2 glance glance 4096 Jul 9 2015 keystone-signing-zZPnd0
drwx------ 2 designate designate 4096 May 20 2015 keystone-signing-ZZQK3i
drwx------ 2 glance glance 4096 Jun 30 2015 keystone-signing-ZZQmEI
Cause:

According to Ubuntu Bug 1533724, the ​python-keystonemiddleware package has a software defect when a signing_dir value is not provided in the keystone_authtoken section for a given service, the keystonemiddleware will create a secure temporary directory upon startup. For users not using the 'PKI' or 'PKIZ' token providers, this is unnecessary and causes /tmp to fill up.

Solution:

For Contrail versions using older Openstack releases, i.e. pre-4.4.1-0ubuntu1~cloud0, this issue is addressed via Juniper Openstack Bug 1722787. Juniper explicitly ​configures the default signing_dir for keystone as /var/lib/contrail/keystone-signing.

Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search