Support Support Downloads Knowledge Base Case Manager My Juniper Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[Contrail] SSH/TCP traffic fails in-between virtual machines hosted on different compute nodes due to checksum errors

0

0

Article ID: KB30500 KB Last Updated: 29 Jul 2015Version: 1.0
Summary:

SSH/TCP traffic fails in-between virtual machines hosted on different compute nodes due to checksum errors.

Symptoms:

SSH/TCP traffic fails in-between:

  • Virtual machines (VMs) on different compute nodes
  • Compute nodes and the MX Gateway router
Cause:

The following is a simple use case to demonstrate the issue:

Compute: Host1 ( vm1, vm2 in network A)
Compute: Host2 ( vm3, vm4 in network A)

You might observe the following:

vm1 to vm2 - SSH works
vm1 to vm3 - SSH fails
vm2 to vm4 - SSH fails
vm3 to vm4 - SSH works

The SSH traffic from vm1 to vm2 and vm3 to vm4 passes through as the traffic gets handled at the vRouter within the compute node, and won't pass through the physical NIC of the compute node as the VM pair is hosted on the same compute node.

However, SSH traffic from vm1 to vm3 and vm2 to vm4 has to pass through the physical NIC of the compute nodes. In the case of SSH traffic passing through the physical node, cases are observed in which the traffic drops.

This is mostly observed (but not limited to) the following scenarios:

  • Nested VMs
  • VMs running in ESXi Hypervisor
  • NIC cards that do not support checksum and generate an incorrect checksum (e.g., Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe NIC cards).
Solution:

Check if there are any incorrect checksum errors on the host and the virtual machine using the following command:

tcpdump -i <interface> -v -nn | grep -i incorrect

If you observe any errors as the output of the above command, you can turn off the tx-checksumming parameter. In Contrail, the vrouter makes use of the NIC cards to perform checksum offload for the inner packet (after the header has been added). Some NIC cards do not perform this operation correctly. In such cases, it is advisable to make a note of the driver and the hardware being used before turning off the tx-checksumming parameter. 

To get around this problem, it is advised to turn off the tx-checksumming parameter on the compute node data interface using the ethtool utility from the CLI.

By default, the ethtool utility is not installed on the servers. It must be installed on the compute nodes. 

If the OS is CentOS, it can be installed using the following command:

yum install ethtool

If the OS is Ubuntu, it can be installed using the following command:

sudo apt-get install ethtool

Once the ethtool utility is installed, perform the following steps to disable tx-checksumming:

  1. Determine the interface used for data traffic by mapping the MAC address of vhost0 to the physical interface's MAC address on the compute node.

  2. Enter the following command: ethtool -K eth1 tx off (assuming that eth1 is the data interface)

  3. Repeat step 1 and step 2 on all compute nodes. 

You can then check the status of tx-checksumming using the ethtool -k <interface_name> command. The output indicates whether tx-checksumming is on or off.

Disabling tx-checksumming should allow the SSH/TCP traffic (in the failed use cases) to be allowed without any problems.

Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search