Support Support Downloads Knowledge Base Juniper Support Portal Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[QFX] Fabric-Controller Load-Balance behavior on Qfabric-DG

0

0

Article ID: KB32735 KB Last Updated: 30 Jun 2018Version: 1.0
Summary:

This article explains the scenario and behavior for Fabric-Controller when one of the Director-device is rebooted. Fabric-Controllers are the Route-Reflectors in Qfabric devices, which has internal IBGP peering with all the other VMs and Node-Groups in a Qfabric System.

Consider this scenario:

DG1 is primary and DG0 is backup for director-group cluster. FC-1 is running on DG-1 and FC-0 is running on DG-0 respectively.

[root@dg1 ~]# lsvm
NODE    ACTIVE  TAG                             UUID
dg0     1       _DCF_default___RR-INE-0_RE0_    0bbf7b2a-5859-11e6-bad6-f3a9e63c1a6a 
dg1     1       _DCF_default___RR-INE-1_RE0_    0ce190ec-5859-11e6-9773-a72e24c42352  
Symptoms:

All tests are done on version 14.1D17.1

1-Scenario-1 is in expected condition.

DG1 (primary) is rebooted, FC-1 is shifted/respawned on DG0 (Expected behaviour).

Before reboot Both FC-0 and FC-1 has BGP session established to all node-groups and between RRs too:

root@FC-0> show bgp summary fabric | grep "est|idl|act|con" 
Unconfigured peers: 6
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
128.0.128.4             100      32433      32033       0       6 1w3d 4:13:54 Establ
128.0.128.8             100     119040     119045       0       0 5w2d 13:10:00 Establ
128.0.130.0             100       1768       1731       0       2    13:09:10 Establ
128.0.130.6             100     119657     119339       0       0 5w2d 13:09:58 Establ
128.0.130.8             100     119660     119340       0       0 5w2d 13:09:58 Establ
128.0.130.10            100      90385      89199       0       0 4w0d 8:43:35 Establ
128.0.130.18            100      90625      89472       0       0 4w0d 10:28:19 Establ
root@FC-1> show bgp summary fabric | no-more | grep "Est|Act|conn|idl"  
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
128.0.128.4             100      32428      32009       0       6 1w3d 4:05:09 Establ
128.0.128.6             100     119045     119033       0       1 5w2d 13:01:20 Establ
128.0.130.0             100       1763       1729       0       2    13:06:17 Establ
128.0.130.6             100     119667     119181       0       0 5w2d 13:07:57 Establ
128.0.130.8             100     119670     119181       0       0 5w2d 13:07:57 Establ
128.0.130.10            100      90379      89182       0       0 4w0d 8:34:50 Establ
128.0.130.18            100      90619      89461       0       0 4w0d 10:19:34 Establ

During DG1 reboot:

FC-1 VM shuts down too and respawned on DG0. Now both FC VMs are on DG0:

[root@dg0 ~]# lsvm
NODE    ACTIVE  TAG                             UUID
dg0     1       _DCF_default___NW-INE-0_RE1_    0a7b58ec-5859-11e6-aa3b-03617c167afc
dg0     1       _DCF_default___RR-INE-0_RE0_    0bbf7b2a-5859-11e6-bad6-f3a9e63c1a6a
dg0     1       _TAG_DCF_ROOT_RE0_              c5816470-5858-11e6-a26a-ef094445ab07
dg1     1       _DCF_default___RR-INE-1_RE0_    0ce190ec-5859-11e6-9773-a72e24c42352  ?
dg1     1       _TAG_DRE_                       c9fe7f38-5858-11e6-8ed1-470321454c61
dg1     1       _DCF_default___NW-INE-0_RE0_    09474094-5859-11e6-b704-573a3d7bca43
dg1     1       _TAG_DCF_ROOT_RE1_              c58669e8-5858-11e6-a653-3792a1592d6a
[root@dg0 ~]# lsvm
NODE    ACTIVE  TAG                             UUID
dg0     1       _DCF_default___NW-INE-0_RE1_    0a7b58ec-5859-11e6-aa3b-03617c167afc
dg0     1       _DCF_default___RR-INE-0_RE0_    0bbf7b2a-5859-11e6-bad6-f3a9e63c1a6a
dg0     1       _TAG_DCF_ROOT_RE0_              c5816470-5858-11e6-a26a-ef094445ab07
---     0       _DCF_default___RR-INE-1_RE0_    0ce190ec-5859-11e6-9773-a72e24c42352  ? 
---     0       _TAG_DRE_                       c9fe7f38-5858-11e6-8ed1-470321454c61
---     0       _DCF_default___NW-INE-0_RE0_    09474094-5859-11e6-b704-573a3d7bca43
---     0       _TAG_DCF_ROOT_RE1_              c58669e8-5858-11e6-a653-3792a1592d6a
[root@dg0 ~]# date
Fri Jan 27 18:20:13 UTC 2017
[root@dg0 ~]# lsvm
NODE    ACTIVE  TAG                             UUID
dg0     1       _DCF_default___NW-INE-0_RE1_    0a7b58ec-5859-11e6-aa3b-03617c167afc
dg0     1       _DCF_default___RR-INE-0_RE0_    0bbf7b2a-5859-11e6-bad6-f3a9e63c1a6a
dg0     1       _TAG_DCF_ROOT_RE0_              c5816470-5858-11e6-a26a-ef094445ab07
dg0     1       _DCF_default___RR-INE-1_RE0_    0ce190ec-5859-11e6-9773-a72e24c42352  ? 
dg0     1       _TAG_DRE_                       c9fe7f38-5858-11e6-8ed1-470321454c61
dg0     1       _DCF_default___NW-INE-0_RE0_    09474094-5859-11e6-b704-573a3d7bca43
---     0       _TAG_DCF_ROOT_RE1_              c58669e8-5858-11e6-a653-3792a1592d6a

As FC-1 VM shuts down, FC-0 lost BGP to peer RR FC-1:

128.0.128.4             100         13         18       0       7        3:52 Establ
128.0.128.8             100     119210     119215       0       1        4:19 Connect  <<<<<
128.0.130.0             100       1931       1920       0       2    14:22:23 Establ
128.0.130.6             100     119819     119527       0       0 5w2d 14:23:11 Establ
128.0.130.8             100     119822     119527       0       0 5w2d 14:23:11 Establ

As FC-1 VM respawned on DG0 now, FC-1 established BGP session to FC-0:

---(refreshed at 2017-01-27 18:26:52 UTC)---
Unconfigured peers: 4
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
128.0.128.4             100         14         20       0       7        4:08 Establ
128.0.128.8             100          4         17       0       1           1 Establ  <<<<<
128.0.130.0             100       1932       1921       0       2    14:22:39 Establ
128.0.130.6             100     119820     119528       0       0 5w2d 14:23:27 Establ
128.0.130.8             100     119822     119529       0       0 5w2d 14:23:27 Establ

Now both FC-0 and FC-1 are residing on DG0 and have all BGP sessions established:

NODE    ACTIVE  TAG                             UUID
dg0     1       _DCF_default___NW-INE-0_RE1_    0a7b58ec-5859-11e6-aa3b-03617c167afc
dg0     1       _DCF_default___RR-INE-0_RE0_    0bbf7b2a-5859-11e6-bad6-f3a9e63c1a6a  <<<<
dg0     1       _TAG_DCF_ROOT_RE0_              c5816470-5858-11e6-a26a-ef094445ab07
dg0     1       _DCF_default___RR-INE-1_RE0_    0ce190ec-5859-11e6-9773-a72e24c42352  <<<<
dg0     1       _TAG_DRE_                       c9fe7f38-5858-11e6-8ed1-470321454c61
dg0     1       _DCF_default___NW-INE-0_RE0_    09474094-5859-11e6-b704-573a3d7bca43
---     0       _TAG_DCF_ROOT_RE1_              c58669e8-5858-11e6-a653-3792a1592d6a

Post DG1 reboot:

To load balance VMs, FC-0 and NNG RE0 VMs are released by DG-0(current primary) and spawned on DG-1(current Backup)

If you notice VMs that are released by DG-0, NNG RE VM is the newly spawned one whereas FC VM is the native one (FC with longest uptime) but not the newly spawned one:

Fri Jan 27 18:27:05 UTC 2017
[root@dg0 ~]# lsvm
NODE    ACTIVE  TAG                             UUID
dg0     1       _DCF_default___NW-INE-0_RE1_    0a7b58ec-5859-11e6-aa3b-03617c167afc
dg0     1       _DCF_default___RR-INE-0_RE0_    0bbf7b2a-5859-11e6-bad6-f3a9e63c1a6a
dg0     1       _TAG_DCF_ROOT_RE0_              c5816470-5858-11e6-a26a-ef094445ab07
dg0     1       _DCF_default___RR-INE-1_RE0_    0ce190ec-5859-11e6-9773-a72e24c42352
dg0     1       _TAG_DRE_                       c9fe7f38-5858-11e6-8ed1-470321454c61
dg0     1       _DCF_default___NW-INE-0_RE0_    09474094-5859-11e6-b704-573a3d7bca43
---     0       _TAG_DCF_ROOT_RE1_              c58669e8-5858-11e6-a653-3792a1592d6a
[root@dg0 ~]# lsvm
NODE    ACTIVE  TAG                             UUID
dg0     1       _DCF_default___NW-INE-0_RE1_    0a7b58ec-5859-11e6-aa3b-03617c167afc
---     0       _DCF_default___RR-INE-0_RE0_    0bbf7b2a-5859-11e6-bad6-f3a9e63c1a6a
dg0     1       _TAG_DCF_ROOT_RE0_              c5816470-5858-11e6-a26a-ef094445ab07
dg0     1       _DCF_default___RR-INE-1_RE0_    0ce190ec-5859-11e6-9773-a72e24c42352
dg0     1       _TAG_DRE_                       c9fe7f38-5858-11e6-8ed1-470321454c61
---     0       _DCF_default___NW-INE-0_RE0_    09474094-5859-11e6-b704-573a3d7bca43
dg1     1       _TAG_DCF_ROOT_RE1_              c58669e8-5858-11e6-a653-3792a1592d6a

As a result FC-0 shuts down and BGP peering between FC-0 and FC-1 went down:

FC-1:
---(refreshed at 2017-01-27 18:30:06 UTC)---
Unconfigured peers: 4
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
128.0.128.4             100         26         17       0       0        5:45 Establ
128.0.128.6             100         30         15       0       1        2:44 Connect
128.0.130.0             100         24         19       0       0        5:47 Establ
128.0.130.6             100         17         17       0       0        5:46 Establ
128.0.130.8             100         17         17       0       0        5:46 Establ

Unconfigured peers: 4
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
128.0.128.4             100         26         18       0       0        5:47 Establ
128.0.128.6             100          5          6       0       1           2 Establ
128.0.130.0             100         24         20       0       0        5:49 Establ
128.0.130.6             100         18         18       0       0        5:48 Establ
128.0.130.8             100         17         18       0       0        5:48 Establ

4.	Final State:

Only FC VMs UUID getting swapped between DGs prior and post reboot of the actual primary DG (DG-1) whereas NNG VMs UUID remained on actual DG:

[root@dg0 ~]# lsvm
NODE    ACTIVE  TAG                             UUID
dg0     1       _DCF_default___NW-INE-0_RE1_    0a7b58ec-5859-11e6-aa3b-03617c167afc
dg1     1       _DCF_default___RR-INE-0_RE0_    0bbf7b2a-5859-11e6-bad6-f3a9e63c1a6a  <<
dg0     1       _TAG_DCF_ROOT_RE0_              c5816470-5858-11e6-a26a-ef094445ab07
dg0     1       _DCF_default___RR-INE-1_RE0_    0ce190ec-5859-11e6-9773-a72e24c42352  <<
dg0     1       _TAG_DRE_                       c9fe7f38-5858-11e6-8ed1-470321454c61
dg1     1       _DCF_default___NW-INE-0_RE0_    09474094-5859-11e6-b704-573a3d7bca43
dg1     1       _TAG_DCF_ROOT_RE1_              c58669e8-5858-11e6-a653-3792a1592d6a

root@FC-1> show bgp summary fabric | match "est|act|idl|con" | refresh 2    
Jan 27 19:11:04
---(refreshed at 2017-01-27 19:11:04 UTC)---
Unconfigured peers: 4
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
128.0.128.4             100        117        113       0       0       46:42 Establ
128.0.128.6             100        113        107       0       1       40:57 Establ
128.0.130.0             100        114        117       0       0       46:44 Establ
128.0.130.6             100        107        120       0       0       46:43 Establ
128.0.130.8             100        107        120       0       0       46:43 Establ

root@FC-0> show bgp summary fabric | match "est|act|ild|con"    
Unconfigured peers: 4
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
128.0.128.4             100        104        100       0       0       41:22 Establ
128.0.128.8             100        111        110       0       0       41:24 Establ
128.0.130.0             100        103        104       0       0       41:23 Establ
128.0.130.6             100         95        107       0       0       41:22 Establ
128.0.130.8             100         96        106       0       0       41:22 Establ

Both FCs show change in uptime during recent VM spawned time:

root@FC-0> show system uptime 
Current time: 2017-01-27 19:21:50 UTC
System booted: 2017-01-27 18:28:45 UTC (00:53:05 ago)
Protocols started: 2017-01-27 18:29:44 UTC (00:52:06 ago)
Last configured: 2017-01-27 18:29:49 UTC (00:52:01 ago) by root
 7:21PM  up 53 mins, 1 user, load averages: 0.00, 0.00, 0.00

root@FC-1> show system uptime 
Jan 27 19:22:06
Current time: 2017-01-27 19:22:06 UTC
System booted: 2017-01-27 18:22:14 UTC (00:59:52 ago)
Protocols started: 2017-01-27 18:23:13 UTC (00:58:53 ago)
Last configured: 2017-01-27 18:23:16 UTC (00:58:50 ago) by root
 7:22PM  up 1 hr, 1 user, load averages: 0.00, 0.01, 0.00

This can cause disruption in traffic at times.

Cause:

The reason behind this is that the existing logic of selection and construction of a list of only lesser priority VMs to powerdown and activate on peer DG.

Solution:
This will be fixed in version 14.1X53-D43 and higher.
Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search