After rebooting a QFX5000 device, it is unable to send BGP notification messages to the peer device.
Topology:
QFX5100-r019 (xe-0/1/2, 192.168.20.1/30) --- (xe-0/0/0:0, 192.168.20.2/30 ) QFX5100-r030
Setting eBGP and iBGP between two QFX5100s, eBGP uses physical interfaces and iBGP uses logical interfaces (Loopback interface).
Both lo0 routes will be exported to each other via eBGP.
root@QFX5100-r019# show protocols bgp | display set
set protocols bgp traceoptions file bgp.txt
set protocols bgp traceoptions flag all
set protocols bgp group ibgp type internal
set protocols bgp group ibgp local-address 192.168.10.11
set protocols bgp group ibgp family evpn signaling
set protocols bgp group ibgp multipath
set protocols bgp group ibgp neighbor 192.168.10.1 peer-as 64512
set protocols bgp group ebgp type external
set protocols bgp group ebgp family inet unicast
set protocols bgp group ebgp export export-bgp
set protocols bgp group ebgp local-as 65101
set protocols bgp group ebgp multipath multiple-as
set protocols bgp group ebgp neighbor 192.168.20.2 peer-as 65201
root@QFX5100-r019# show interfaces lo0 | display set
set interfaces lo0 unit 0 family inet address 192.168.10.11/32
[master:0][edit]
root@QFX5100-r019# show policy-options | display set
set policy-options policy-statement export-bgp term 1 from protocol direct
set policy-options policy-statement export-bgp term 1 from protocol static
set policy-options policy-statement export-bgp term 1 from protocol local
set policy-options policy-statement export-bgp term 1 from interface lo0.0
set policy-options policy-statement export-bgp term 1 then accept
**************
root@QFX5100-r030# show protocols bgp | display set
set protocols bgp traceoptions file bgp.txt
set protocols bgp traceoptions flag all
set protocols bgp group ibgp type internal
set protocols bgp group ibgp local-address 192.168.10.1
set protocols bgp group ibgp family evpn signaling
set protocols bgp group ibgp multipath
set protocols bgp group ibgp neighbor 192.168.10.11 peer-as 64512
set protocols bgp group ebgp type external
set protocols bgp group ebgp family inet unicast
set protocols bgp group ebgp export export-bgp
set protocols bgp group ebgp local-as 65201
set protocols bgp group ebgp multipath multiple-as
set protocols bgp group ebgp neighbor 192.168.20.1 peer-as 65101
root@QFX5100-r030# show interfaces lo0 | display set
set interfaces lo0 unit 0 family inet address 192.168.10.1/32
[master:0][edit]
root@QFX5100-r030# show policy-options | display set
set policy-options policy-statement export-bgp term 1 from protocol direct
set policy-options policy-statement export-bgp term 1 from protocol local
set policy-options policy-statement export-bgp term 1 from interface lo0.0
set policy-options policy-statement export-bgp term 1 then accept
When rebooting r019 via the command 'request system reboot', then checking the BGP status on r030, IBGP (192.168.10.11) will not change to down immediately.
root@QFX5100-r030# run show bgp summary
Groups: 2 Peers: 2 Down peers: 1
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
0 0 0 0 0 0
bgp.evpn.0
0 0 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
192.168.10.11 64512 6 6 0 0 2:57 Establ
bgp.evpn.0: 0/0/0/0
192.168.20.1 65101 18 17 0 2 1:07 Idle
[master:0][edit]
root@QFX5100-r030# run show bgp summary
Groups: 2 Peers: 2 Down peers: 2
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
0 0 0 0 0 0
bgp.evpn.0
0 0 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
192.168.10.11 64512 0 0 0 1 49 Connect
192.168.20.1 65101 18 17 0 2 2:14 Idle
[master:0][edit]
root@QFX5100-r030# run show bgp neighbor
Peer: 192.168.10.11+179 AS 64512 Local: 192.168.10.1 AS 64512
Group: ibgp Routing-Instance: master
Forwarding routing-instance: master
Type: Internal State: Connect Flags: <>
Last State: Active Last Event: ConnectRetry
Last Error: Hold Timer Expired Error
Options: <Preference LocalAddress AddressFamily PeerAS Multipath Rib-group Refresh>
Address families configured: evpn
Local Address: 192.168.10.1 Holdtime: 90 Preference: 170
Number of flaps: 1
Last flap event: HoldTime <-- it is not RecvNotify
Error: 'Hold Timer Expired Error' Sent: 1 Recv: 0
Trace options: all
Trace file: /var/log/bgp.txt size 131072 files 10
Peer: 192.168.20.1 AS 65101 Local: 192.168.20.2 AS 65201
Group: ebgp Routing-Instance: master
Forwarding routing-instance: master
Type: External State: Idle Flags: <PeerInterfaceError>
Last State: Established Last Event: Stop
Last Error: Cease
Export: [ export-bgp ]
Options: <Preference AddressFamily PeerAS Multipath LocalAS Refresh>
Options: <MultipathAs>
Address families configured: inet-unicast
Holdtime: 90 Preference: 170 Local AS: 65201 Local System AS: 64512
Number of flaps: 2
Last flap event: Stop <-- it is not RecvNotify
Error: 'Cease' Sent: 2 Recv: 0
Trace options: all
Trace file: /var/log/bgp.txt size 131072 files 10
The same symptom is also seen on QFX110 and QFX5200.
This is a limitation because there is no hypervisor on QFX5000 PFE, so PFE will go down immediately.
From the following messages, the physical interface is down before sending notifications. Thus, notification messages will not be sent to the peer.
Mar 1 06:23:16 QFX5100-r019 fpc0 fxpc_check_signal: Going to shutdown network ports
Mar 1 06:23:16 QFX5100-r019 fpc0 ifp xe-0/1/2 ifd_mdown
Mar 1 06:23:21 QFX5100-r019 rpd[2053]: bgp_peer_delete:9410: NOTIFICATION sent to 192.168.10.1 (Internal AS 64512): code 6 (Cease) subcode 3 (Peer Unconfigured), Reason: Peer Deletion
Mar 1 06:23:21 QFX5100-r019 rpd[2053]: bgp_peer_delete:9410: NOTIFICATION sent to 192.168.20.2 (External AS 65201): code 6 (Cease) subcode 3 (Peer Unconfigured), Reason: Peer Deletion
The Bidirectional Forwarding Detection (BFD) protocol is a simple hello mechanism that detects failures in a network。
Please add BFD, which will let all BGP sessions go down immediately when the peer device reboots.