Support Support Downloads Knowledge Base Case Manager My Juniper Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[Contrail] Contrail log collection in Release 5.0+

0

0

Article ID: KB34063 KB Last Updated: 06 Apr 2019Version: 1.0
Summary:

This article explains the log collection techniques for contrail releases post 5.0

Solution:

Application and systems logs can help you understand what is happening inside your cluster. The easiest and most embraced logging method for containerized applications is to write to the standard output and standard error streams. The logs are particularly useful for debugging problems and monitoring cluster activity. 

Kubectl output Format

Output format Description
-o=custom-columns=<spec> Print a table using a comma separated list of custom columns
-o=custom-columns-file=<filename> Print a table using the custom columns template in the <filename> file
-o=json Output a JSON formatted API object
-o=jsonpath=<template> Print the fields defined in a jsonpath expression
-o=jsonpath-file=<filename> Print the fields defined by the jsonpath expression in the <filename> file
-o=name Print only the resource name and nothing else
-o=wide Output in the plain-text format with any additional information, and for pods, the node name is included
-o=yaml Output a YAML formatted API object
 

Examples of –output-format usage

kubectl get pod -o json -n contrail |jq -r .items[].status.podIP
 
172.17.0.1
172.17.0.1
172.17.0.1
172.17.0.1
172.17.0.1
172.17.0.1
172.17.0.1
172.17.0.1
172.17.0.1
172.17.0.1
172.17.0.1
172.17.0.1
192.168.172.135
192.168.172.134
 
kubectl get pods --all-namespaces -o=jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "} '
 
contrail-analytics-8lsbx:       docker.io/opencontrailnightly/contrail-analytics-api:latest, docker.io/opencontrailnightly/contrail-analytics-collector:latest, docker.io/opencontrailnightly/contrail-analytics-alarm-gen:latest, docker.io/opencontrailnightly/contrail-analytics-query-engine:latest, docker.io/opencontrailnightly/contrail-nodemgr:latest,
contrail-analytics-zookeeper-s4xdj:     docker.io/opencontrailnightly/contrail-external-zookeeper:latest,
contrail-analyticsdb-nodemgr-9vqbr:     docker.io/opencontrailnightly/contrail-nodemgr:latest,
contrail-analyticsdb-xg76d:     docker.io/opencontrailnightly/contrail-external-cassandra:latest,
contrail-config-g9fn7:  docker.io/opencontrailnightly/contrail-controller-config-api:latest, docker.io/opencontrailnightly/contrail-controller-config-devicemgr:latest, docker.io/opencontrailnightly/contrail-controller-config-schema:latest, docker.io/opencontrailnightly/contrail-controller-config-svcmonitor:latest, docker.io/opencontrailnightly/contrail-nodemgr:latest,
contrail-config-zookeeper-swmrt:        docker.io/opencontrailnightly/contrail-external-zookeeper:latest,
contrail-configdb-cnwzb:        docker.io/opencontrailnightly/contrail-external-cassandra:latest,
contrail-control-56p7r: docker.io/opencontrailnightly/contrail-controller-control-control:latest, docker.io/opencontrailnightly/contrail-controller-control-dns:latest, docker.io/opencontrailnightly/contrail-controller-control-named:latest, docker.io/opencontrailnightly/contrail-nodemgr:latest,
contrail-kafka-f4xkb:   docker.io/opencontrailnightly/contrail-external-kafka:latest,
contrail-redis-pggmc:   redis:4.0.2,
contrail-vrouter-agent-kernel-f92x5:    docker.io/opencontrailnightly/contrail-vrouter-agent:latest, docker.io/opencontrailnightly/contrail-nodemgr:latest,
contrail-webui-hjgzk:   docker.io/opencontrailnightly/contrail-controller-webui-job:latest, docker.io/opencontrailnightly/contrail-controller-webui-web:latest,
ingress-68bdd5bf66-n9n7h:       quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0,
ingress-error-pages-f57f86986-fb47t:    gcr.io/google_containers/defaultbackend:1.0,

 
kubectl get pod -o=custom-columns=Namespace:.metadata.namespace,PODS:.metadata.name,CONTAINERS:.spec.containers[*].name -n contrail
 
Namespace   PODS                                  CONTAINERS
contrail    contrail-analytics-8lsbx              contrail-analytics-api,contrail-collector,contrail-alarm-gen,contrail-query-engine,contrail-analytics-nodemgr
contrail    contrail-analytics-zookeeper-s4xdj    zookeeper
contrail    contrail-analyticsdb-nodemgr-9vqbr    contrail-analyticsdb-nodemgr
contrail    contrail-analyticsdb-xg76d            contrail-analyticsdb
contrail    contrail-config-g9fn7                 contrail-config-api,contrail-devicemgr,contrail-schema-transformer,contrail-svcmonitor,contrail-config-nodemgr
contrail    contrail-config-zookeeper-swmrt       zookeeper
contrail    contrail-configdb-cnwzb               contrail-configdb
contrail    contrail-control-56p7r                contrail-control,contrail-dns,contrail-named,contrail-control-nodemgr
contrail    contrail-kafka-f4xkb                  kafka
contrail    contrail-redis-pggmc                  redis
contrail    contrail-vrouter-agent-kernel-f92x5   contrail-vrouter-agent,contrail-vrouter-nodemgr
contrail    contrail-webui-hjgzk                  contrail-webui-middleware,contrail-webui
contrail    ingress-68bdd5bf66-n9n7h              ingress
contrail    ingress-error-pages-f57f86986-fb47t   ingress-error-pages

 
kubectl get pod -n contrail -o=custom-columns-file=template
 
NAME                                  Containers
contrail-analytics-8lsbx              contrail-analytics-api,contrail-collector,contrail-alarm-gen,contrail-query-engine,contrail-analytics-nodemgr
contrail-analytics-zookeeper-s4xdj    zookeeper
contrail-analyticsdb-nodemgr-9vqbr    contrail-analyticsdb-nodemgr
contrail-analyticsdb-xg76d            contrail-analyticsdb
contrail-config-g9fn7                 contrail-config-api,contrail-devicemgr,contrail-schema-transformer,contrail-svcmonitor,contrail-config-nodemgr
contrail-config-zookeeper-swmrt       zookeeper
contrail-configdb-cnwzb               contrail-configdb
contrail-control-56p7r                contrail-control,contrail-dns,contrail-named,contrail-control-nodemgr
contrail-kafka-f4xkb                  kafka
contrail-redis-pggmc                  redis
contrail-vrouter-agent-kernel-f92x5   contrail-vrouter-agent,contrail-vrouter-nodemgr
contrail-webui-hjgzk                  contrail-webui-middleware,contrail-webui
ingress-68bdd5bf66-n9n7h              ingress
ingress-error-pages-f57f86986-fb47t   ingress-error-pages
root@ubuntu1604:~#
root@ubuntu1604:~#
root@ubuntu1604:~# cat template
NAME          Containers
metadata.name spec.containers[*].name

 
kubectl get pods --sort-by=.metadata.name -n contrail
 
NAME                                  READY     STATUS    RESTARTS   AGE
contrail-analytics-8lsbx              5/5       Running   0          37d
contrail-analytics-zookeeper-s4xdj    1/1       Running   0          37d
contrail-analyticsdb-nodemgr-9vqbr    1/1       Running   0          37d
contrail-analyticsdb-xg76d            1/1       Running   0          37d
contrail-config-g9fn7                 5/5       Running   0          37d
contrail-config-zookeeper-swmrt       1/1       Running   0          37d
contrail-configdb-cnwzb               1/1       Running   0          37d
contrail-control-56p7r                4/4       Running   0          37d
contrail-kafka-f4xkb                  1/1       Running   0          37d
contrail-redis-pggmc                  1/1       Running   0          37d
contrail-vrouter-agent-kernel-f92x5   2/2       Running   1          37d
contrail-webui-hjgzk                  2/2       Running   0          37d
ingress-68bdd5bf66-n9n7h              1/1       Running   0          37d
ingress-error-pages-f57f86986-fb47t   1/1       Running   0          37d

Display Kubernetes logs

Everything contained in Kubernetes is represented by a RESTful resource. We refer to these resources as Kubernetes objects. Each Kubernetes object exists at a unique HTTP path.

The kubectl command makes HTTP requests to these URLs to access the Kubernetes objects that reside at these paths.

kubectl also makes a number of commands available for debugging your containers. You can use the following to see the logs for a running container:
 
$ kubectl logs <pod-name>

If you have multiple containers in your pod you can choose the container to view using the -c flag.

Kubernetes provides two commands for debugging running containers. The kubectl logs command downloads the current logs from the running instance:

$ kubectl logs <pod-name>

Adding the -f flag will cause you to continuously stream logs.

The kubectl logs command always tries to get logs from the currently running container. Adding the --previous flag will get logs from a previous instance of the container. This is useful, for example, if your containers are continuously restarting due to a problem at container startup.

root@ubuntu1604:~# kub logs contrail-vrouter-agent-kernel-f92x5 -c contrail-vrouter-agent -n contrail --previous
INFO: agent started in kernel mode
Device "vhost0" does not exist.
cat: /sys/class/net/vhost0/address: No such file or directory
INFO: creating vhost0 for nic mode: nic: ens4, mac=52:54:00:4b:b8:76
INFO: there is no ifcfg-ens4 a
nd ifcfg-vhost0, so initialize vhost0 manually
INFO: Changing physical interface to vhost in ip table
RTNETLINK answers: File exists
ERROR: failed to add address fe80::5054:ff:fe4b:b876/64 to vhost0.
INFO: set default gateway
ERROR: failed to add default gateway 10.168.6.1
INFO: set mtu
FATAL: failed to init vhost0
RTNETLINK answers: File exists

Usage:

kubectl logs [-f] [-p] (POD | TYPE/NAME) [-c CONTAINER] [options]
 

Examples:

  # Return snapshot logs from the redis pod with only one container
     kubectl logs contrail-redis-pggmc -n contrail 
 
  # Return snapshot logs for the pods defined by label app=nginx
     kubectl logs -lapp=nginx
  
  # Display only the most recent 20 lines of output in pod nginx
     kubectl logs --tail=20 nginx
 
  # Show all logs from pod nginx written in the last hour
     kubectl logs --since=1h nginx
  
  # Return snapshot logs from container nginx-1 of a deployment named nginx
     kubectl logs deployment/nginx -c nginx-1
 
 
Examples of kubectl logs with explanation 
 
# dump pod logs
kubectl logs my-pod                                
 
# dump pod logs for a previous instantiation of a container
kubectl logs my-pod --previous                     
 
# dump pod container logs (stdout, multi-container case)
kubectl logs my-pod -c my-container                
 
# dump pod container logs (stdout, multi-container case) for a previous instantiation of a container
kubectl logs my-pod -c my-container --previous     
 
# stream pod logs
kubectl logs -f my-pod                             
 
# stream pod container logs (stdout, multi-container case)
kubectl logs -f my-pod -c my-container             
 
# Run pod as interactive shell
kubectl run -i --tty busybox --image=busybox -- sh 
 
# Attach to Running Container
kubectl attach my-pod -i                           
 
# Run command in existing pod (1 container case)
kubectl exec my-pod -- ls /         
 
# Run command in existing pod (multi-container case)
kubectl exec my-pod -c my-container -- ls /       
 

Kubectl output verbosity and debugging

Kubectl verbosity is controlled with the -v or --v flags followed by an integer representing the log level.
 
Verbosity Description
--v=0 Generally useful for this to ALWAYS be visible to an operator.
--v=1 A reasonable default log level if you don’t want verbosity.
--v=2 Useful steady state information about the service and important log messages that may correlate to significant changes in the system. This is the recommended default log level for most systems.
--v=3 Extended information about changes.
--v=4 Debug level verbosity.
--v=6 Display requested resources.
--v=7 Display HTTP request headers.
--v=8 Display HTTP request contents.
--v=9 Display HTTP request contents without truncation of contents.
 
 
Log and .conf files
 
Config Folder - /etc/kubernetes/
Certificate Files - /etc/kubernetes/pki/
Kubernetes Working Dir - /var/lib/kubelet/
Log Files - /var/log/pods
log in master node - /var/log/kube-apiserver.log, kube-scheduler.log, kube-controller-manager.log
log in worker node - /var/log/kubelet.log, kubelet-proxy.log
Docker Working Dir - /var/lib/docker, /var/log/containers

 root@ubuntu1604:~# kubectl exec -it contrail-vrouter-agent-kernel-f92x5 -c contrail-vrouter-agent -n contrail -- tail -f /var/log/contrail/contrail-vrouter-agent.log
2018-07-27 Fri 01:15:01:401.876 UTC  ubuntu1604 [Thread 139678473840384, Pid 166]: SANDESH: Send FAILED: 1532654101401758  [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.168.6.9 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1706
2018-07-27 Fri 01:15:01:402.003 UTC  ubuntu1604 [Thread 139678473840384, Pid 166]: SANDESH: Send FAILED: 1532654101401832  [SYS_NOTICE]: NodeStatusUVE: data= [ name = ubuntu1604 process_status= [  [ [  module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [  [ [  type = XMPP name = control-node:10.168.6.9 server_addrs= [  [ (*_iter6) = 10.168.6.9:5269,  ] ] status = Down description = Idle ],  [  type = Collector name =  server_addrs= [  [ (*_iter6) = 172.17.0.1:8086,  ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ],  ] ] description = XMPP:control-node:10.168.6.9, Collector connection down, No Configuration for self Number of connections:2, Expected: 3 ],  ] ] ]

**** SNIP ****
 
Dump lots of relevant information about the cluster for debugging and diagnosis 
 
kubectl cluster-info dump > file
 

Copying Files to and from Containers

At times you may need to copy files from a remote container to a local machine for more in-depth exploration. For example, you can use a tool like Wireshark to visualize tcpdump packet captures.  Suppose you had a file called /captures/capture3.txt inside a container in your Pod. You could securely copy that file to your local machine by running:

$ kubectl cp <pod-name>:/captures/capture3.txt ./capture3.txt

Other times you may need to copy files from your local machine into a container. Let’s say you want to copy $HOME/config.txt to a remote container. In this case, you can run:

$ kubectl cp $HOME/config.txt <pod-name>:/config.txt
  
List pods Sorted by Restart Count

kubectl get pods --sort-by='.status.containerStatuses[0].restartCount'
  
kubectl top pod contrail-vrouter-agent-kernel-f92x5 --containers -n contrail

 
 Accessing Docker container processes and logs
 
root@ubuntu1604:~# docker ps -a|grep vrouter
d348c988cdce        opencontrailnightly/contrail-vrouter-agent@sha256:ede65841bf277479b982310aa6423e9a987d7a1e001e97ea753fb549e5597ce4                       "/entrypoint.sh /u..."   3 months ago        Up 3 months                                     k8s_contrail-vrouter-agent_contrail-vrouter-agent-kernel-f92x5_contrail_722cdf8d-9138-11e8-976a-525400db35a1_1
adf22040454d        opencontrailnightly/contrail-nodemgr@sha256:d09d687e0d3b91b53679ebd20441414eb3ac07b0555a0036939f60153d7cb2ba                             "/entrypoint.sh /b..."   3 months ago        Up 3 months                                     k8s_contrail-vrouter-nodemgr_contrail-vrouter-agent-kernel-f92x5_contrail_722cdf8d-9138-11e8-976a-525400db35a1_0
fef46544c6cc        opencontrailnightly/contrail-vrouter-agent@sha256:ede65841bf277479b982310aa6423e9a987d7a1e001e97ea753fb549e5597ce4                       "/entrypoint.sh /u..."   3 months ago        Exited (1) 3 months ago                         k8s_contrail-vrouter-agent_contrail-vrouter-agent-kernel-f92x5_contrail_722cdf8d-9138-11e8-976a-525400db35a1_0
a94337399549        opencontrailnightly/contrail-vrouter-kernel-build-init@sha256:52582975874f16841056cf7915303dd132bcd9f14da90dc1a780d9d37fd3a692           "/entrypoint.sh"         3 months ago        Exited (0) 3 months ago                         k8s_contrail-vrouter-init-kernel_contrail-vrouter-agent-kernel-f92x5_contrail_722cdf8d-9138-11e8-976a-525400db35a1_0
df5e0ff4cb9b        opencontrailnightly/contrail-node-init@sha256:e9b8dda63953e94e2a51fe058a9fd0d508c257132ba4fa6737b72532a6898d4e                           "/entrypoint.sh"         3 months ago        Exited (0) 3 months ago                         k8s_contrail-node-init_contrail-vrouter-agent-kernel-f92x5_contrail_722cdf8d-9138-11e8-976a-525400db35a1_0
b6b250499f6c        quay.io/stackanetes/kubernetes-entrypoint@sha256:cd8758c539501cd3d4c2b3029d2e774ce1f2260584c0c8d3d8d8c473ba4285bb                        "kubernetes-entryp..."   3 months ago        Exited (0) 3 months ago                         k8s_init_contrail-vrouter-agent-kernel-f92x5_contrail_722cdf8d-9138-11e8-976a-525400db35a1_0
72fddded7461        gcr.io/google_containers/pause-amd64:3.0                                                                                                 "/pause"                 3 months ago        Up 3 months                                     k8s_POD_contrail-vrouter-agent-kernel-f92x5_contrail_722cdf8d-9138-11e8-976a-525400db35a1_0
root@ubuntu1604:~# docker exec -it d348c988cdce bash
(vrouter-agent)[root@ubuntu1604 /]$ ps -efw
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Jul27 ?        00:00:00 /bin/bash /entrypoint.sh /usr/bin/contrail-vrouter-agent
root       166     1  5 Jul27 ?        5-00:53:55 /usr/bin/contrail-vrouter-agent
root       636     0  4 02:43 ?        00:00:00 bash
root       657   636  0 02:43 ?        00:00:00 ps -efw
(vrouter-agent)[root@ubuntu1604 /]$
 
 
root@ubuntu1604:~# docker ps|grep vrouter
d348c988cdce        opencontrailnightly/contrail-vrouter-agent@sha256:ede65841bf277479b982310aa6423e9a987d7a1e001e97ea753fb549e5597ce4                       "/entrypoint.sh /u..."   5 weeks ago         Up 5 weeks                              k8s_contrail-vrouter-agent_contrail-vrouter-agent-kernel-f92x5_contrail_722cdf8d-9138-11e8-976a-525400db35a1_1
adf22040454d        opencontrailnightly/contrail-nodemgr@sha256:d09d687e0d3b91b53679ebd20441414eb3ac07b0555a0036939f60153d7cb2ba                             "/entrypoint.sh /b..."   5 weeks ago         Up 5 weeks                              k8s_contrail-vrouter-nodemgr_contrail-vrouter-agent-kernel-f92x5_contrail_722cdf8d-9138-11e8-976a-525400db35a1_0
72fddded7461        gcr.io/google_containers/pause-amd64:3.0                                                                                                 "/pause"                 5 weeks ago         Up 5 weeks                              k8s_POD_contrail-vrouter-agent-kernel-f92x5_contrail_722cdf8d-9138-11e8-976a-525400db35a1_0

 
 
To see container startup logs, do the following:
 
root@ubuntu1604:~# docker logs --details d348c988cdce
 INFO: agent started in kernel mode
 INFO: vhost0 is already up
 Device "crypt0" does not exist.
 Successfully added ipvlan interface crypt0
 Device "decrypt0" does not exist.
 Successfully added tunnel interface decrypt0 and the required iptables rules
 INFO: Physical interface: ens4, mac=52:54:00:4b:b8:76, pci=0000:00:04.0
 INFO: vhost0 cidr 10.168.6.9/24, gateway 10.168.6.1
 INFO: Preparing /etc/contrail/contrail-vrouter-agent.conf
 INFO: /etc/contrail/contrail-vrouter-agent.conf
 [CONTROL-NODE]
 servers=10.168.6.9:5269
<SNIP>
 
root@ubuntu1604:~# docker logs --tail 10 d348c988cdce
 
[CRYPT]
crypt_interface=crypt0
 
[SESSION]
slo_destination = collector
sample_destination = collector
 
INFO: Preparing /etc/contrail/contrail-lbaas-auth.conf
Successfully added tunnel interface decrypt0 to vrouter
root@ubuntu1604:~#
 
 
root@ubuntu1604:~# docker exec -it d348c988cdce bash
(vrouter-agent)[root@ubuntu1604 /]$
(vrouter-agent)[root@ubuntu1604 /]$
(vrouter-agent)[root@ubuntu1604 /]$
(vrouter-agent)[root@ubuntu1604 /]$ pwd

root@ubuntu1604 /]$ tailf /var/log/contrail/contrail-vrouter-agent.log
2018-07-27 Fri 01:15:01:401.876 UTC  ubuntu1604 [Thread 139678473840384, Pid 166]: SANDESH: Send FAILED: 1532654101401758  [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.168.6.9 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1706

<SNIP>
Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search