Support Support Downloads Knowledge Base Case Manager My Juniper Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[CSO] Errorlog collection script - getlogs is not working

0

0

Article ID: KB35042 KB Last Updated: 12 Nov 2019Version: 3.0
Summary:

This article explains an issue with CSO 4.1.1 where the getlogs script is not working and the workaround that can be used.

Symptoms:

Output of getlogs script:

# ./getlogs.sh 
Log collection for csp-central-msvm2 - X.X.X.X
[+                                                                                                                                                                                                         ] 0% csp.admin-portal-ui-4262910192-0qs6lError from server (NotFound): the server could not find the requested resource ( pods/log csp.admin-portal-ui-4262910192-0qs6l)
[++                                                                                                                                                                                                        ] 0% csp.admin-portal-ui-4262910192-7p2lcError from server (NotFound): the server could not find the requested resource ( pods/log csp.admin-portal-ui-4262910192-7p2lc)
[+++                                                                                                                                                                                          
] 1% csp.admin-portal-ui-4262910192-cb1d6Error from server (NotFound): the server could not find the requested resource ( pods/log csp.admin-portal-ui-4262910192-cb1d6)
[++++                                                                                                                                                                                                      ] 1% csp.csp-ams-2930958688-lnj23Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-ams-2930958688-lnj23)
[+++++                                                                                                                                                                                                     ] 2% csp.csp-ams-2930958688-rggcsError from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-ams-2930958688-rggcs)
[++++++                                                                                                                                                                                       
] 2% csp.csp-ams-2930958688-sm9mcError from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-ams-2930958688-sm9mc)
[+++++++                                                                                                                                                                                                   ] 3% csp.csp-cslm-signature-ims-central-3520248340-7qlvmError from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-cent
ral-3520248340-7qlvm)
Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-3520248340-7qlvm)
Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-3520248340-7qlvm)
[++++++++                                                                                                                                                                                     
] 3% csp.csp-cslm-signature-ims-central-3520248340-h0x8nError from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-cent
ral-3520248340-h0x8n)
Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-3520248340-h0x8n)
Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-3520248340-h0x8n)
[+++++++++                                                                                                                                                                                    
] 4% csp.csp-cslm-signature-ims-central-3520248340-rkspzError from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-cent
ral-3520248340-rkspz)
Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-3520248340-rkspz)
Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-3520248340-rkspz)
[++++++++++                                                                                                                                                                                   
] 4% csp.csp-cslm-signature-ims-central-core-1107922945-8gg43Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims
-central-core-1107922945-8gg43)
Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-core-1107922945-8gg43)
Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-core-1107922945-8gg43)
[+++++++++++                                                                                                                                                                                  
] 5% csp.csp-cslm-signature-ims-central-core-1107922945-jr899Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims
-central-core-1107922945-jr899)
Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-core-1107922945-jr899)
Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-core-1107922945-jr899)
[++++++++++++                                                                                                                                                                                 
] 5% csp.csp-cslm-signature-ims-central-core-1107922945-tkppwError from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims
-central-core-1107922945-tkppw)
Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-core-1107922945-tkppw)
Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-core-1107922945-tkppw)
Cause:

This issue is due to kubectl logs being blocked in CSO 4.1.1 (refer to ​https://contrail-jws.atlassian.net/browse/CXU-36343 for complete details).

Commit was done with the following changes to control the kubelet log size:

The verbosity level for the log was reduced from 6 to 2. This prevents kubelet from aggressively writing logs, and the log file size will then be in control. To see this change in the customer setup, locate the file /etc/default/kubelet in MSVM. Then change the following lines:

Before: ---enable-debugging-handlers=true \

Change To:  ---enable-debugging-handlers=false \

Before: ---v=6 \

Change To: ---v=2 \

Restart kubelet to have these changes take effect: service kubelet restart

However, after this change, kubectl logs <pod> will not be present. But docker logs and Kibana will continue to work.

Solution:

There are two workarounds, and any one of them can be used:

Workaround 1 - Enable the debug in kubelet log size temporarily.

This workaround can be used to collect logs and should be reverted in order to use disk space properly.

Note: Be aware that the kubectl logs that are consuming most of the disk space are reported in ​https://contrail-jws.atlassian.net/browse/CXU-36343 after enabling kubectl logs.

​vi /etc/default/kubelet file in all msvms,
4.1.1 FRS: ---enable-debugging-handlers=false \
Change To: ---enable-debugging-handlers=true \

4.1.1 FRS: ---v=2 \
Change To: ---v=6 \

service kubelet restart

Workaround 2 - Use the attached script which uses docker instead of k8.

Download the archive that contains getdockerlogs.sh and getlogs.sh scripts from here. Then extract the zip files into your workstation. It contains two scripts: getdockerlogs.sh and getlogs.sh.

Below are the instructions:

  1. Log in to the installer VM.
  2. Copy the getlogs.sh script and place it in the current Contrail_Service_Orchestration_4.1.1 folder. (In order to avoid any DOS format bash error, execute the command: sed -i -e 's/\r$//' getlogs.sh)
  3. Copy the getdockerlogs.sh script into /home/Contrail_Service_Orchestration_4.1.1/deployments/central/file_root/kubeminion/files.
  4. Run getlogs.sh. It will give the tgz file as ($ _ms_logs.tgz).

Snippet:

root@ip-X.X.X.X:/home/ubuntu/Contrail_Service_Orchestration_4.1.1# bash getlogs.sh
Running docker logs in kubeminion VMs...
Pushing docker logs to installer VM...
Compressing the Logs...
***** All the logs are collected under 09-20-19-10-38-50_ms_logs.tgz******
root@ip-100-74-0-52:/home/ubuntu/Contrail_Service_Orchestration_4.1.1# ls -la
total 204808
drwxr-xr-x 33 root   root        4096 Sep 20 10:39 .
drwxr-xr-x  6 ubuntu ubuntu      4096 Aug 15 13:44 ..
-rw-r--r--  1 root   root   209290644 Sep 20 10:39 09-20-19-10-38-50_ms_logs.tgz  <- LOG FILE
drwxr-xr-x  3 root   root        4096 Aug 14 11:31 apache_deb_packages

Note: It is recommended to delete the log files after usage to avoid getting the disk full.

Modification History:

2019-10-14: Removed internal and added docker workaround

2019-11-12: Added comment about DOS format bash precaution in Step 2 > Workaround 2 in Solution

Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search