Knowledge Search


×
 

[CSO] Errorlog collection script - getlogs is not working

  [KB35042] Show Article Properties


Summary:

This article explains an issue with CSO 4.1.1 where the getlogs script is not working and the workaround that can be used.

Symptoms:

Output of getlogs script:

# ./getlogs.sh 
Log collection for csp-central-msvm2 - X.X.X.X
[+                                                                                                                                                                                                         ] 0% csp.admin-portal-ui-4262910192-0qs6lError from server (NotFound): the server could not find the requested resource ( pods/log csp.admin-portal-ui-4262910192-0qs6l)
[++                                                                                                                                                                                                        ] 0% csp.admin-portal-ui-4262910192-7p2lcError from server (NotFound): the server could not find the requested resource ( pods/log csp.admin-portal-ui-4262910192-7p2lc)
[+++                                                                                                                                                                                          
] 1% csp.admin-portal-ui-4262910192-cb1d6Error from server (NotFound): the server could not find the requested resource ( pods/log csp.admin-portal-ui-4262910192-cb1d6)
[++++                                                                                                                                                                                                      ] 1% csp.csp-ams-2930958688-lnj23Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-ams-2930958688-lnj23)
[+++++                                                                                                                                                                                                     ] 2% csp.csp-ams-2930958688-rggcsError from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-ams-2930958688-rggcs)
[++++++                                                                                                                                                                                       
] 2% csp.csp-ams-2930958688-sm9mcError from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-ams-2930958688-sm9mc)
[+++++++                                                                                                                                                                                                   ] 3% csp.csp-cslm-signature-ims-central-3520248340-7qlvmError from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-cent
ral-3520248340-7qlvm)
Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-3520248340-7qlvm)
Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-3520248340-7qlvm)
[++++++++                                                                                                                                                                                     
] 3% csp.csp-cslm-signature-ims-central-3520248340-h0x8nError from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-cent
ral-3520248340-h0x8n)
Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-3520248340-h0x8n)
Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-3520248340-h0x8n)
[+++++++++                                                                                                                                                                                    
] 4% csp.csp-cslm-signature-ims-central-3520248340-rkspzError from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-cent
ral-3520248340-rkspz)
Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-3520248340-rkspz)
Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-3520248340-rkspz)
[++++++++++                                                                                                                                                                                   
] 4% csp.csp-cslm-signature-ims-central-core-1107922945-8gg43Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims
-central-core-1107922945-8gg43)
Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-core-1107922945-8gg43)
Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-core-1107922945-8gg43)
[+++++++++++                                                                                                                                                                                  
] 5% csp.csp-cslm-signature-ims-central-core-1107922945-jr899Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims
-central-core-1107922945-jr899)
Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-core-1107922945-jr899)
Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-core-1107922945-jr899)
[++++++++++++                                                                                                                                                                                 
] 5% csp.csp-cslm-signature-ims-central-core-1107922945-tkppwError from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims
-central-core-1107922945-tkppw)
Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-core-1107922945-tkppw)
Error from server (NotFound): the server could not find the requested resource ( pods/log csp.csp-cslm-signature-ims-central-core-1107922945-tkppw)
Cause:

This issue is due to kubectl logs being blocked in CSO 4.1.1 (refer to ​https://contrail-jws.atlassian.net/browse/CXU-36343 for complete details).

Commit was done with the following changes to control the kubelet log size:

The verbosity level for the log was reduced from 6 to 2. This prevents kubelet from aggressively writing logs, and the log file size will then be in control. To see this change in the customer setup, locate the file /etc/default/kubelet in MSVM. Then change the following lines:

Before: ---enable-debugging-handlers=true \

Change To:  ---enable-debugging-handlers=false \

Before: ---v=6 \

Change To: ---v=2 \

Restart kubelet to have these changes take effect: service kubelet restart

However, after this change, kubectl logs <pod> will not be present. But docker logs and Kibana will continue to work.

Solution:

There are two workarounds, and any one of them can be used:

Workaround 1 - Enable the debug in kubelet log size temporarily.

This workaround can be used to collect logs and should be reverted in order to use disk space properly.

Note: Be aware that the kubectl logs that are consuming most of the disk space are reported in ​https://contrail-jws.atlassian.net/browse/CXU-36343 after enabling kubectl logs.

​vi /etc/default/kubelet file in all msvms,
4.1.1 FRS: ---enable-debugging-handlers=false \
Change To: ---enable-debugging-handlers=true \

4.1.1 FRS: ---v=2 \
Change To: ---v=6 \

service kubelet restart

Workaround 2 - Use the attached script which uses docker instead of k8.

Download the archive that contains getdockerlogs.sh and getlogs.sh scripts from here. Then extract the zip files into your workstation. It contains two scripts: getdockerlogs.sh and getlogs.sh.

Below are the instructions:

  1. Log in to the installer VM.
  2. Copy the getlogs.sh script and place it in the current Contrail_Service_Orchestration_4.1.1 folder. (In order to avoid any DOS format bash error, execute the command: sed -i -e 's/\r$//' getlogs.sh)
  3. Copy the getdockerlogs.sh script into /home/Contrail_Service_Orchestration_4.1.1/deployments/central/file_root/kubeminion/files.
  4. Run getlogs.sh. It will give the tgz file as ($ _ms_logs.tgz).

Snippet:

root@ip-X.X.X.X:/home/ubuntu/Contrail_Service_Orchestration_4.1.1# bash getlogs.sh
Running docker logs in kubeminion VMs...
Pushing docker logs to installer VM...
Compressing the Logs...
***** All the logs are collected under 09-20-19-10-38-50_ms_logs.tgz******
root@ip-100-74-0-52:/home/ubuntu/Contrail_Service_Orchestration_4.1.1# ls -la
total 204808
drwxr-xr-x 33 root   root        4096 Sep 20 10:39 .
drwxr-xr-x  6 ubuntu ubuntu      4096 Aug 15 13:44 ..
-rw-r--r--  1 root   root   209290644 Sep 20 10:39 09-20-19-10-38-50_ms_logs.tgz  <- LOG FILE
drwxr-xr-x  3 root   root        4096 Aug 14 11:31 apache_deb_packages

Note: It is recommended to delete the log files after usage to avoid getting the disk full.

Modification History:

2019-10-14: Removed internal and added docker workaround

2019-11-12: Added comment about DOS format bash precaution in Step 2 > Workaround 2 in Solution

Related Links: