Support Support Downloads Knowledge Base Juniper Support Portal Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[NorthStar] Nested VM interoperability of multi-port equipped ESXi server with L3 switches

0

0

Article ID: KB35091 KB Last Updated: 27 Sep 2019Version: 1.0
Summary:

This article describes a possible scenario with a design issue in a nested VM (Virtual Machine).

NorthStar Controller is supported to work in a single VM by nested virtualization using VMware ESXi. Under a particular network setting between ESXi host server and L3 node device with bridging ports, an unexpected issue may occur. Any broadcast/unknown-unicast/multicast packets from Junos VM in NorthStar Controller are never able to communicate out of ESXi nodes.

Note: This situation can happen in general nested VMs other than Juniper NorthStar product.
Symptoms:

In the example setup below, the Linux OS which accommodates NorthStar Controller (NorthStar APP and Junos VM) is located in ESXi host server. The ESXi host server has 2 NIC ports; and they are connected to L3 switch for the sake of redundancy.

The circled numbers in the figure correspond to the following lines to explain the problematic scenario:

[1]

Junos VM transmits ARP Request by broadcast to find the gateway IP address.

[2]

At the port external0 (Linux bridge), ARP request is received from "port no 2" side.

MAC Address of Junos VM 52:54:00:11:11:00 is learnt in external0.

[root@linux ~]#  brctl showmacs external0   
port no    mac addr        is local?        ageing timer
  2     52:54:00:11:11:00       no                 0.00


Linux port eth0 is bound to esternal0 bridge.
Since this ARP packet destines broadcast, it goes through vSwitch and being sent out from physical NIC port2 of ESXi host server.
Here is the frame captured at external0 at this time. Size is 42 Bytes.

16:08:19.935832 52:54:00:11:11:00 -> ff:ff:ff:ff:ff:ff ARP 42 Who has 10.0.0.254?  Tell 10.0.0.1

0000  ff ff ff ff ff ff 52 54 00 11 11 00 08 06 00 01   ......RT. Q.....
0010  08 00 06 04 00 01 52 54 00 11 11 00 0a 63 01 e6   ......RT. Q..c..
0020  00 00 00 00 00 00 0a 63 01 e4                     .......c..

[3]

When ARP request is sent from Physical NIC Port2, padding data 0000... is added because of filling out to meet the minimum length of Ethernet frame, 64 bytes.

Note: Captured packet shows 60 bytes because the 4 last bytes of FCS(Frame Check Sequence) is omitted.

[4]

ARP request is received by VLAN L3 interface(vlan.x, irb.x, etc.) in Switch. Now, the packet itself is broadcast. Hence, it is flooded to the same L2 segment.

[5]

As a result of [4], ARP request packet is transmitted towards another physical NIC port 1 of ESXi host server. By this process, ARP request packet arrives at ESXi host and goes back through vSwitch.

[6]

The flooded ARP request is received at "port no 1" of external0, hence MAC 52:54:00:11:11:00 moves to "port no 1" as below.

[root@linux ~]#  brctl showmacs external0   
port no    mac addr        is local?        ageing timer
  1     52:54:00:11:11:00       no                 0.50

Also, the ARP request packet was added padding data in [3], then it is seen as 60 bytes in capture hex data at external0.

16:08:19.935895 52:54:00:11:11:00 -> ff:ff:ff:ff:ff:ff ARP 60 Who has 10.0.0.254?  Tell 10.0.0.1

0000  ff ff ff ff ff ff 52 54 00 11 11 00 08 06 00 01   ......RT. Q.....
0010  08 00 06 04 00 01 52 54 00 11 11 00 0a 63 01 e6   ......RT. Q..c..
0020  00 00 00 00 00 00 0a 63 01 e4 00 00 00 00 00 00   .......c........  <-- 18 bytes of "0" are padded.
0030  00 00 00 00 00 00 00 00 00 00 00 00               ............

This broadcast packet is forwarded to Junos VM. In/Out both packets can be collected in Junos VM.

03:17:47.589523 Out 52:54:00:11:11:00 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: arp who-has 10.0.0.254  tell 10.0.0.1
03:17:47.589745  In 52:54:00:11:11:00 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.0.0.254  tell 10.0.0.1

[7]

During [5] is processed in the switch, ARP reply is sent by unicast from L3 switch vlan interface.
Generally, this happens later than [5] because generating ARP reply packet takes more time than broadcast packet forwarding.

[8]

ARP reply by [7] reaches to external0.

16:08:19.937041 00:33:33:33:33:33 -> 52:54:00:11:11:00 ARP 60 10.0.0.254 is at 00:33:33:33:33:33

0000  52 54 00 11 11 00 00 33 33 33 33 33 08 06 00 01   RT. Q.$.........
0010  08 00 06 04 00 02 00 33 33 33 33 33 0a 63 01 e4   ......$......c..
0020  52 54 00 11 11 00 0a 63 01 e6 00 00 00 00 00 00   RT. Q..c........
0030  00 00 00 00 00 00 00 00 00 00 00 00               ............

However, the second ARP request by [6] has already reached to external0 and Junos VM's MAC 52:54:00:11:11:00 was learnt as "port no 1" side. Therefore, the ARP reply is returned back to vSwitch direction.
As a result, ARP resolution in Junos VM never completes since ARP reply is unable to reach to Junos VM. In fact, external0 keeps learning "port no 1" for Junos VM's MAC most of its working. Momentarily, index 2 is seen 1 out of 10000 collecting 'brctl showmacs <>'.

[root@linux ~]#  cat showmac.log | grep -C3  -E "^  2"
  1    52:54:00:11:11:00    no           0.19
  1    52:54:00:11:11:00    no           0.19
  1    52:54:00:11:11:00    no           0.19
  2    52:54:00:11:11:00    no           0.00  <-- This moment
  1    52:54:00:11:11:00    no           0.00
  1    52:54:00:11:11:00    no           0.00
  1    52:54:00:11:11:00    no           0.00
--
--
  1    52:54:00:11:11:00    no           0.49
  1    52:54:00:11:11:00    no           0.50
  1    52:54:00:11:11:00    no           0.50
  2    52:54:00:11:11:00    no           0.00  <-- This moment
  1    52:54:00:11:11:00    no           0.00
  1    52:54:00:11:11:00    no           0.00
  1    52:54:00:11:11:00    no           0.00
--


 
Solution:

If Junos VM does not need to send BUM packet in the customer's design, configuring static ARP entry can be a solution or workaround.

In NorthStar Controller, Junos VM uses PCEP and BGP. However, it depends on the customer's requirement. (OSPF/ISIS are options)

Junos VM

set interfaces em1 unit 0 family inet address 10.0.0.1 arp 10.0.0.254 mac 00:33:33:33:33:33

As another workaround, setting aging timer to zero in Linux can solve the situation. However, you must confirm the Linux OS usage with the customer.

Linux OS

brctl setaging external0 0

The root cause is that ESXi host server has two available interfaces with remote switch. It creates a type of L2 loop condition among the physical switch, vSwitch, and bridge ports. Network design may be needed to consider changing that Aggregation/Teaming, or other interface control feature is to be used.

Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search