[EX/QFX] BGP session taking too long to go down

Summary:

In normal scenarios, when a Border Gateway Protocol (BGP) peer goes down, a router only waits for the hold timer to expire to declare the neighborship as down (90 seconds default). But in some cases, a router will retain this dead adjacency up for a longer time even if the BGP timers have expired.

This article explains why a router would behave in this manner and lists two methods to change this behavior.

Symptoms:

Dead adjacency is retained for a longer time even if the BGP timers have expired.

Cause:

By default, when BGP and other routing protocols are configured on Juniper devices, graceful restart helper mode gets automatically enabled. If a restarting router (a peer device configured with graceful restart) goes down, helper routers assist their BGP neighbor by keeping its information under stale status for an extra period.

Terminology:

Graceful restart router: Device configured with graceful restart protocol
Graceful restart helper: Device configured with helper mode

Topology Example

Scenario: Juniper_host is learning 100.100.100.0/24 from BGP peer 10.19.19.2:

Juniper_host> show route protocol bgp
  100.100.100.0/24   *[BGP/170] 00:03:39, localpref 100
                   AS path: 30079 I, validation-state: unverified
              > to 10.19.19.2 via irb.100

When you check the output of show bgp neighbor on the Juniper side, it is confirmed that graceful restart is enabled on the peer switch, as indicated:

Juniper_host>  show bgp neighbor 10.19.19.2
Peer: 10.19.19.2+59683 AS 30079 Local: 10.19.19.1+179 AS 30080
  Type: External    State: Established    Flags: <Sync>
  Last State: OpenConfirm   Last Event: RecvKeepAlive
  Last Error: None
  Options: <Preference HoldTime PeerAS LocalAS Refresh>
  Holdtime: 90 Preference: 170 Local AS: 30080 Local System AS: 0
  Number of flaps: 0
  Peer ID: 10.19.19.2      Local ID: 10.19.19.1        Active Holdtime: 90
  Keepalive Interval: 30         Group index: 0    Peer index: 0
  BFD: disabled, down
  Local Interface: irb.100
  NLRI for restart configured on peer: inet-unicast
  NLRI advertised by peer: inet-unicast
  NLRI for this session: inet-unicast
  Peer supports Refresh capability (2)
  Stale routes from peer are kept for: 300
  Restart time requested by this peer: 120
   NLRI that peer supports restart for: inet-unicast
   NLRI peer can save forwarding state: inet-unicast
   NLRI that peer saved forwarding for: inet-unicast
  NLRI that restart is negotiated for: inet-unicast
  NLRI of received end-of-rib markers: inet-unicast
  NLRI of all end-of-rib markers sent: inet-unicast
  Peer supports 4 byte AS extension (peer-as 30079)
   (...)

Since graceful restart helper mode is automatically enabled on Juniper_host when BGP is configured, if peer "10.19.19.2" goes down, Juniper_host will keep the session up and mark the corresponding received routes as "stale" till the graceful restart timers expire:

Jul 19 15:21:47.923161 bgp_io_mgmt_cb:1785: NOTIFICATION sent to 10.19.19.2 (External AS 30079): code 4 (Hold Timer Expired Error), Reason: holdtime 
expired for 10.19.19.2 (External AS 30079), socket buffer sndcc: 57 rcvcc: 0 TCP state: 4, snd_una: 3170400882 snd_nxt: 3170400939 snd_wnd: 16925 
rcv_nxt: 1738124642 rcv_adv: 1738141026, hold timer 90s, hold timer remain 0s, last sent 22s, TCP port (local 179, remote 49936)
Jul 19 15:21:47.923291 bgp_peer_close_and_restart: peer 10.19.19.2 (External AS 30079), state is 7 (Established) event HoldTime, flags=0x8010000
Jul 19 15:21:47.923533 BGP peer 10.19.19.2 (External AS 30079) rt terminate: enqueue in close list, start close job (flags Closing GRHelperMark)
Jul 19 15:21:47.923660 bgp_mark_route_stale 10.19.19.2 (External AS 30079): peer flags 0x134218752, peer's rst flags 0x4000, rt 100.100.100.0/24
Jul 19 15:21:47.923685 bgp_rti_terminate 10.19.19.2 (External AS 30079): rt 100.100.100.0/24, marked stale

Juniper_host> show route protocol bgp
  100.100.100.0/24   *[BGP/170] 00:03:39, localpref 100
                   AS path: 30079 I, validation-state: unverified
              > to 10.19.19.2 via irb.100

Juniper_host>  show bgp neighbor 10.19.19.2
Peer: 10.19.19.2+179 AS 30079  Local: 10.19.19.1 AS 30080
  Group: external              Routing-Instance: master
  Forwarding routing-instance: master
  Type: External    State: Connect        Flags: <TryConnect>
  Last State: Active        Last Event: ConnectRetry
  Last Error: Hold Timer Expired Error
  Options: <Preference HoldTime PeerAS LocalAS Refresh>
  Holdtime: 90 Preference: 170 Local AS: 30080 Local System AS: 0
  Number of flaps: 3
  Last flap event: HoldTime
  Error: 'Hold Timer Expired Error' Sent: 3 Recv: 0
  Error: 'Cease' Sent: 0 Recv: 2
  NLRI we are holding stale routes for: inet-unicast
 Time until stale routes are deleted or become long-lived stale: 00:00:34
  Time until end-of-rib is assumed for stale routes: 00:03:34
  Table inet.0
  RIB State: BGP restart is complete
  Send state: not advertising
    Active prefixes:              1
    Received prefixes:            1
    Accepted prefixes:            1
    Suppressed due to damping:    0
    Stale prefixes:                          1

Solution:

There are two solutions to modify this behavior:

Solution 1

Disable graceful restart on the peer device if it is not needed.

Example: delete routing-options graceful-restart

Results

BGP neighborship information on Juniper_host will refresh showing graceful restart and its corresponding timers disabled on the peer side:

Juniper_host>  show bgp neighbor 10.19.19.2
Peer: 10.19.19.2+60933 AS 30079 Local: 10.19.19.1+179 AS 30080
  Type: External    State: Established    Flags: <Sync>
  Last State: OpenConfirm   Last Event: RecvKeepAlive
  Last Error: None
  Options: <Preference HoldTime PeerAS LocalAS Refresh>
  Holdtime: 3 Preference: 170 Local AS: 30080 Local System AS: 0
  Number of flaps: 1
  Peer ID: 10.19.19.2      Local ID: 10.19.19.1        Active Holdtime: 3
  Keepalive Interval: 1          Group index: 0    Peer index: 0
  BFD: disabled, down
  Local Interface: irb.100
  NLRI for restart configured on peer: inet-unicast
  NLRI advertised by peer: inet-unicast
  NLRI for this session: inet-unicast
  Peer supports Refresh capability (2)
  Stale routes from peer are kept for: 300
  Peer does not support Restarter functionality
  NLRI that restart is negotiated for: inet-unicast
  NLRI of received end-of-rib markers: inet-unicast
  NLRI of all end-of-rib markers sent: inet-unicast
  Peer supports 4 byte AS extension (peer-as 30079)
  Peer does not support Addpath

Solution 2

Disable graceful restart helper mode on Juniper's side by using graceful-restart disable.

Example: set protocols bgp group external graceful-restart disable

Results

BGP neighborship information on Juniper_host will refresh showing helper mode locally disabled:

Juniper_host>  show bgp neighbor 10.19.19.2
Peer: 10.19.19.2+179 AS 30079  Local: 10.19.19.1+49401 AS 30080
  Group: external              Routing-Instance: master
  Forwarding routing-instance: master
  Type: External    State: Established    Flags: <Sync>
  Last State: OpenConfirm   Last Event: RecvKeepAlive
  Last Error: None
  Options: <Preference PeerAS LocalAS Refresh>
  Options: <GracefulRestartHelperDisabled LLGRHelperDisabled>
  Holdtime: 90 Preference: 170 Local AS: 30080 Local System AS: 0
  Number of flaps: 0
  Peer ID: 10.19.19.2      Local ID: 10.19.19.1        Active Holdtime: 90
  Keepalive Interval: 30         Group index: 0    Peer index: 0    SNMP index: 0
  I/O Session Thread: bgpio-0 State: Enabled
  BFD: disabled, down
  Local Interface: irb.100
  (...)

Note: In cases where BFD and graceful restart for BGP are both configured on the network, failures may cause other unexpected network issues. Though disabling helper mode can stop these behaviors, it is recommended not to have both these features enabled at the same time on your devices. See Example: Configuring BFD for BGP.

Knowledge Base