EVPN/VXLAN: MAC entry incorrectly programmed in PFE, leading to some traffic blackhole

An incorrect PE router is attached to an ESI when the router receives two copies of the same AD/ESI route (for example, one through eBGP and another one received from an iBGP neighbor). This causes a partial traffic black hole and stale MAC entries. You can confirm the issue by checking the members of the ESI: user@router> show evpn instance extensive ... Number of ethernet segments: 5 ESI: 00:13:78:00:00:00:00:00:00:01 Status: Resolved Number of remote PEs connected: 3 Remote PE MAC label Aliasing label Mode 87.233.39.102 0 0 all-active 87.233.39.1 200 0 all-active <<<< this PE is not part of the ESI 87.233.39.101 200 0 all-active 

Error messages might be seen if flapping the AE interface hosted on MPC-3D-16XGE card

In a scaled setup, triggering a flap of the aggregate ethernet interfaces using the commands "set interfaces ae disable" and "set interfaces ae enable" could result in the error messages "mqchip_disable_ostream() MQCHIP(2) timed out waiting for phy_stream 1025 queue empty". 

DCD Micro BFD seems to be failing in dcd_commit_check log file even when BFD is not configured

These logs are debugging logs and doesn't indicate an error condition. When Micro BFD is configured they give debugging info regarding the local address configuration and commit checks. The issue here is that these logs are being populated even if Micro BFD is not configured. This PR adds a check in the Junos code, to make sure that these logs are not populated unless Micro BFD is configured on the AE interface. 

The "LIBJNX_REPLICATE_RCP_ERROR" error message might be seen when the backup RE is not present in the dual RE Scenario

The message "LIBJNX_REPLICATE_RCP_ERROR" is repeated multiple times in syslog files in the master RE, when the backup is not reachable. Though the message is marked as ERROR in the syslog, The user need not to take any action on this ERROR and this will not have any impact on the system and can ignore it. 

With Resource Certification (RPKI) enabled, RPD successive crashes during route validation DB processing

In JUNOS with Resource Certification (RPKI) enabled for BGP Route Origin validation, in some scenarios successive RPD crashes generated with route validation DB processing enabled due to buffering issues in string, generating the coredumps due to invalid pointer. 

Non-optimal route to source might be selected for NG-MVPN with unicast-umh-election enabled

The configuration knob unicast-umh-election for NG-MVPN (next-generation multicast virtual private network) might not work as expected in special cases. This knob is to use the unicast route preference for UMH (upstream multicast hop) selection. However the non-optimal route might be selected if the routes have the same IP address value in route-import community. 

PTX1K:flabel Mem alloc failure followed by FPC core

Over a period of time, network events such as route flaps and MBB scenario cause the PFE heap memory to get fragmented. This change improves memory management and reduces the chance of memory fragmentation. 

Interface down due to "PFE Marked Disabled" on PECHIP causing traffic loss

Consider Ingress FPC to be up, fully Init and stable and follow below flow: We restart the egress FPC. It will come up and do its Block driver Init and program the PFE states. During fpc and ASIC initialization Init all global ASIC tables and during initialization we bring up paths for special path IFD's like RHI. After this FPC will send to RE (chasisd) FPC ONLINE message. As soon as this message is sent chaassid it will send IFD ATTACH to all cards. This will start the traffic in case of AE on Ingress card (The root problem). As soon as Ingress FPC start traffic a request will be generated that will traverse fabric path and reach FI which is the first block on egress. Since IFD Init is not complete on egress card this request will lookup in a queue mapping table, at this moment this table will point to an uninitialized value causing the traffic drops. 

Momentary dip in traffic when a GRES is performed.

On GRES the implicit filters set by DFWD are cleared by DCD. Hence we are seeing a momentary dip in traffic. 

The rpd process might crash when there is high priority route/nexthop involved along with high number of ACKs requested by rpd

The rpd process might crash and restart when there is high priority route/nexthop involved along with high number of ACKs requested by rpd. The core files could be seen by executing CLI command "show system core-dumps". The routing protocols are impacted and traffic disruption will be seen due to loss of routing information. 

fpc temperature mismatch for mpc6/8/9 on mx2k platform

The temperature mismatch between "show chassis fpc" & "show chassis fpc detail" for mpc6/8/9 on mx2k platform 

The fxp0 interface does not accept IP address with "master-only" applied

On MX Series, if GRES is not configured, while "master-only" IP is configured on fxp0.0, the IP address might not be applied to the interface after reboot. 

Error message "RE does not have MAC map for mac type 7" might be seen on Summit platforms

Packet drop might be seen on the logical tunnel interfaces lt-x/2/x or lt-x/3/x

On MX platform with MPC5/MPC6/MPC2E-NG/MPC3E-NG, if the 3rd or 4th LT (Logical Tunnel) interface is configured, e.g. lt-x/2/x or lt-x/3/x, the queuing logic of those LT interfaces will not work properly and therefore packet drop might be seen on them. 

Bogus DDOS counter values and syslog messages could be seen after clearing DDOS statistics for a specific protocol on QFX10000 series switches

On the QFX10000 line of switches, in a DDoS scenario, incorrect DDoS counter values and syslog messages might be seen after a specific protocol statistics is manually cleared. 

The process rpd may crash after configuration change in an L2VPN scenario

In an L2VPN scenario, rpd may crash if an interface (that is already in downstate) is added to any operating site of an L2VPN instance. 

Traffic drop might be seen after GRES if uRPF is configured

On all Junos platforms with Graceful Routing Engine Switchover (GRES) and Non Stop Routing (NSR) enabled, if unicast Reverse Path Forwarding (uRPF) is configured under the interface, traffic drop might be seen after GRES due to this issue. 

The "show chassis fpc" might show "Bad Voltage" for FPC powered off by configuration or CLI command after the command "show chassis environment fpc" is executed

When a FPC (or an incompatible one) is powered off by configuration or CLI command and the command "show chassis environment fpc" is issued, the status of the FPC will change to "---Bad Voltage---" under "show chassis fpc". 

FPC core might be observed after GRES switchover

In the dual Routing Engine (RE) platform with telemetry sensor configured. After graceful routing engine switchover (GRES) switchover, flexible PIC concentrator (FPC) core might be observed in the master RE. This issue might impact the device traffic. 

Executing "show route prefix proto ip detail" during route churm in a route scale scenario may lead to FPC crash.

Executing "show route prefix proto ip detail" during route churm in a route scale scenario may lead to FPC crash. 

Configuration commit might be delayed by 30 seconds.

In Junos releases that support ephemeral configuration databases, the configuration commit time might be delayed by ~30 seconds as "Routing protocols process" (rpd) validates the new configuration. If the synchronized commit is used, the delay time is therefore ~1 min. 

The bbe-smgd process might crash during the authentication phase for L2BSA subscriber

In L2BSA (Layer 2 Bit Stream Access) subscriber scenario, if there is a misconfiguration on Radius profile for L2BSA subscriber (for example, the routing-instance returned from Radius is not configured as VPLS) or authentication part is missing in the physical interface configuration, the bbe-smgd process might crash during the L2BSA subscribers login. 

The authd process might not be started after executing RE switchover on backup RE without GRES enabled

In a dual Routing Engine (RE) system with the enhanced subscriber management feature enabled, if Graceful Routing Engine Switchover (GRES) is not configured, the authd process might not be started after executing RE switchover on backup RE. 

The 'Normal discards' pfe statistics traffic counter might increase at a higher rate when Inline-Jflow or sFlow is enabled.

The 'Normal discards' pfe statistics traffic counter might increase at a higher rate when Inline-Jflow or sFlow is enabled. This has no operational impact and is a purely cosmetic issue. 

About 10min traffic loss is caused by BGP flap during MX ISSU

While performing ISSU in a dual REs or MXVC deployment, the system may clear TCP connections causing BGP peerings to flap. 

IPsec-VPN IKE security-associations might get stuck in "Not Matured" state

In IPsec-VPN scenario, some special peers (e.g. Huawei enodeB) might start new IPsec-VPN IKE (Internet Key Exchange) session without clearing the old session upon detecting session failure, which results in the old IKE session stuck in "Not Matured" state. There is no impact to service but these sessions might consume too many memory resources. 

Traffic might drop on new added interfaces on MX after ISSU

On MX Series platforms, after a unified in-service software upgrade (ISSU), traffic might drop on newly added interfaces because of unified ISSU hardware synchronize phase issue. 

The Routing Engine might crash after non-GRES switchover

When LAG-enhanced is disabled, one child next hop is created for each member link of a LAG interface. During the Non-GRES switchover, the kernel memory might be exhausted, which leads to the creation failure of the child next hop, hence the Routing Engine crash happens. This crash can be avoided by enabling LAG-enhanced. 

The rpd might crash when executing RE switchover under BGP environment and route churn occurs

On MX/PTX/QFX platforms and in BGP environment, if GRES (graceful Routing Engine switchover) and NSR (Nonstop active routing) are configured, rpd might crash when doing RE switchover. The reason is that severe route churn events result in multiple cycles of addition/deletion of routes/labels. And due to a software bug introduced in 17.2R1, the labels freed up in the process may not get reallocated when executing RE switchover. As a result the rpd might crash.

Traffic might lose for the CoS-based forwarding services if evpn is configured

When EVPN is configured with class-of-service-based forwarding (CBF), traffic might be lost for the CBF services. 

Address pool does not correctly cycle to the beginning of the pool when linked-pool-aggregation parameter is defined

In subscriber scenario, when "linked-pool-aggregation" is enabled on MX platform, the address pool might not cycle to the beginning of the pool and it might report "Out of Addresses" even though not all addresses are in use. 

bbe-smgd cores continiously while deleting multicast group node from the tree.

In case of Centralized IGMP Configuration, bbe-smgd daemon may restart on an event of last subscriber of a multicast group is leaving the group. it is not able to delete this multicast group node from the tree, in this case on daemon restart, in INIT phase, bbe-smgd will again try to delete the multicast group node and its associated multicast group service and restart again, due to this bbe-smgd will never complete the INIT phase and restart continuously in INIT phase only. 

The dcd process might go down when 'vlan-id none' is configured for interface

If 'vlan-id none' is configured for interface (for example 'set interfaces unit vlan-id none'), the dcd process will go down after committing this configuration. The fix is to report check error when committing this configuration so as to avoid the dcd crash. 

"PE Chip:pe0[0]: IPW: oversize_drop error" causes Major error on FPC

PTX routers with FPC3s that encounter oversize_drop errors will trigger an FPC Major alarm with each error seen. The alarm should clear on its own. The severity of this alarm is being reduced as there is no lasting impact to the FPC. 

The 802.1P rewrite may not work on inner VLAN

If a logical interface (IFL) is configured with 802.1P rewrite-rules (for both outer and inner VLAN) and fixed classification, after deactivating Class of Service (CoS) on any other IFL, the packets sent from this IFL may still have the original 802.1P bit set in the inner VLAN without being rewritten. 

SFB and PDM/PSU related info is missing in jnxBoxAnatomy MIB on high end MX routers (MX2010/2020).

SFB and PDM/PSU related info is missing in jnxBoxAnatomy MIB on high end MX routers (MX2010/2020) 

The 'accept-remote-source' knob configured on the core interface might cause traffic outage

In MVPN environment, the 'accept-remote-source' knob is designed to configure under a vrf instance where the traffic is coming. If it is configured on the core interface where traffic is not expected, some unusual things would happen and it might cause traffic outage or failure. 

EVPN active/active multihomed PE occasionally prefers to route to a directly connected prefix using LSPs towards the multihomed peer instead of going directly out the IRB interface (which is up).

EVPN active/active multihomed PE occasionally prefers to route to a directly connected prefix using LSPs towards the multihomed peer instead of going directly out the IRB 

Packets might be dropped on data plane in the inline Jflow scenario

On MX series with MPC, in the inline Jflow scenario, due to a software defect, the data structure associated with inline Jflow feature may not be initialized correctly. This leads to not being able to forward traffic correctly on the affected MPC. 

The fxpc might crash after an interface is changed on ACX5000 routers.

On ACX5000 platforms, if a GE (Gigabit Ethernet) interface is replaced with an XE (10-Gigabit Ethernet) interface or vice versa, or configuration is changed after the interface change, the fxpc might crash. 

Traffic might be dropped on third-generation FPCs on PTX.

On PTX with third-generation FPCs, if optics not certified by Juniper Networks (NON-JNPR) are used and there is specific traffic pattern with congestion, traffic might be dropped. 

The subscribers with service session might encounter connect failure if file-based shmlog with filtering enabled is configured

On MX-series platforms running the enhanced subscriber management feature, such as MX5/10/40/80/104, the subscriber service session could not be established once file-based shmlog with filtering enabled is configured. 

Remove the chassisd alarms for fpcs exceeding 90 percent of power budget and exeeding 100 percent of power budget

Starting in Junos OS Release with this change, PTX Series Routers do not raise a chassis alarm in the following events; instead, it registers a system log. 

MSQQ error logs and potential MPC traffic impact when IFD link goes down

When an IFD interface link goes down or flaps, MSQQ errors might be seen in the log and may impact traffic flow, or in some cases trigger a restart of the FPC. Logs may include some or all of the logs listed: Jan 24 17:26:11 Lab-R-1 fpc0 mqss_wo_coreif_conn_credits_wait_for_init_value: Timeout occured while waiting for available credits value to become initial value - conn_num 7, credits 43, init_credits 3 Jan 24 17:26:11 Lab-R-1 fpc0 mqss_stream_out_disable: Waiting for available credits value to become initial value for WO connection failed - status 29, conn_num 7 Jan 24 17:26:11 Lab-R-1 fpc0 mqss_ifd_link_up_down_handler: Disabling PHY stream for egress side failed - status 29, instance 0, phy_stream 1111 Jan 24 17:26:11 Lab-R-1 fpc0 pfe_ifd_link_updown: Handling IFD link DOWN failed - status 29, ifd xe-0/1/1 Jan 24 17:26:11 Lab-R-1 fpc0 MQSS(0): WANIO_CR: Parity Protect: Multiple Errors 0x1 Jan 24 17:26:17 Lab-R-1 chassisd[13412]: ASIC Error detected errorno 0x002207d1 Restart action performed 

Some error messages could be seen when running 'show interface extensive' command from CLI or Junos Space

On SRX5000-series platforms, some uspipc failed messages might be seen while running 'show interface extensive' command from CLI or Junos Space. The impact is that some interfaces' counters cannot be collected from some PFEs. It has no other service or traffic impact. 

2019-01 Security Bulletin: Junos OS: OpenSSL Security Advisories [16 Apr 2018] and [12 June 2018]

The OpenSSL project has published security advisories for vulnerabilities resolved in the OpenSSL library on April 16, 2018, and June 12, 2018. See https://kb.juniper.net/JSA10919 for details. 

Daemon dfwd might crash with DFWD_TRASHED_RED_ZONE log messages

In certain scenario with OTN options configuration, memory corruption might occur in dfwd (the firewall daemon) due to large IFL (logical interface) ifstate messages. This can lead to DFWD_TRASHED_RED_ZONE messages reported in dfwd log and occasionally dfwd crashes. 

CoA updates subscriber with original dynamic-profile if radius has returned different dynamic-profile name

When radius sends CoA (Change of Authorization) for the subscriber after radius has returned different dynamic-profile name in access-accept, the subscriber will be updated with original dynamic-profile. The issue is because the new dynamic-profile name which sent by the radius is not saved in the subscriber's table, hence when the CoA message arrives, the old dynamic-profile name is used. The issue results in CoA updates subscriber with unexpected values (The old dynamic-profile instead of the new dynamic-profile is used). 

Some subscribers fail to get SRL service as provided in Radius accept message even though the Radius messages can be sent and received

In Dual-stack PPP/PPPoE-based subscriber scenario, when V4+V6 service is installed with family v4, if some daemon (such as dfwd) fails to add family inet6 IFF during instantiation of the family inet6 portion of some services (such as SRL service), thus family activation for family inet6 would fail. But only the family inet6 portion of service should be removed. The family inet and L2 services such as CoS should be unchanged, but it does not. So some subscribers cannot get some services (such as SRL service) even though the Radius messages can be sent and received. It is a timing specific issue. 

The L2circuit might stop forwarding traffic when one core interface flapping happens

On ACX-series platforms with L2circuit scenario, if the L2cicuit peer is connected via RSVP (with link protection) through two interfaces (e.g. master and backup interface, AE0 and AE1), and after both interfaces flapped, the L2circuit link is brought down and back to up. And at this point, if one core interface (such as AE0 or AE1) flaps alone, the L2circuit link still stays up because another interface is up, but the traffic to CE would be dropped due to one internal table in kernel is not be programmed correctly. 

The value of 'predefined-variable-defaults routing-instances' overrides the RADIUS-supplied VSA (26-1 Virtual-Router)

If the default value for the $junos-routing-instance predefined variable is configured (i.e. 'dynamic-profiles <> predefined-variable-defaults routing-instances <>'), the subsciber will come up in the configured default routing-instance even if RADIUS has already supplied the VSA of '26-1 Virtual-Router'. 

RADIUS accounting statistics are not cleared after subscriber logout

On MX platform, if static demux interface over underlying is configured, after subscriber logout, the accounting statistics are not cleared. 

LTS Subscriber stats reporting to RADIUS

LTS subscriber stats are reported to RADIUS. 

The RAA message may consist of additional AVP "Destination-Host" even it is not configured for Gx-Plus session

On MX platforms for subscriber management scenario, when Gx-Plus is used, the Reauthorization Answer (RAA) message may have additional Attribute-Value Pair (AVP) "Destination-Host" even it is not configured. 

PRPD app tcan be configured to have pruge timeout of never

It is possible to configure the purge timeout of programmable RPD clients to never. This will mean that the routes added by PRPD clients will not be deleted when client disconnects. They will stay until routing daemon restarts or it is deleted by the client that added the route. This can be configured using following CLI command. Note the programmable API for setting purge timeout does not support this feature yet. set routing-options programmable-rpd purge-timeout never 

The RA packets may be sent out without using the configured virtual gateway address

In an EVPN scenario, even if an IPv6 "virtual-gateway-address" is configured on "IRB" interface, the router advertisement (RA) packets may be sent out with the physical interface/link-local IPv6 address instead of configured virtual-gateway-address. 

BFD sessions might flap consistently

On QFX10000/PTX Series platforms, the BFD sessions flapping might be seen during the device init itself. 

Vmcore may be seen on the 18.1R3 release.

Due to an API introduced in 18.1R3, a kernel core dump may occur when a configuration change is done. This results from invalid pointer access by the API. 

The LACP might be detached state when deleting native-vlan-id on AE interface with flexible-vlan-tagging configured

If an aggregated Ethernet interface is configured with LACP, "flexible-vlan-tagging" and "native-vlan-id", then after deleting the "native-vlan-id option", the LACP state will be detached state. 

MX might drop packets larger than tunnel interface MTU as Tail Drops in Egress queue

On MX-Series platforms with MPC7/8/9, when the packets which exceed the MTU and whose DF-bit is set go into a tunnel (such as GRE, LT), they might be dropped in the tunnel egress queue. 

Default route configured gets deleted during ZTP

During Zero Touch Provisioning (ZTP) process, default route is being cleaned up by code. Due to this if a static default route is configured in the initial configuration (configuration file downloaded from the file server for ZTP), the route will fail to work. This might lead to ZTP failure or device access issue after ZTP. 

Penultimate-hop router does not install BGP LU label causing traffic blackhole.

On the penultimate-hop router in BGP LU (labeled unicast) scenario using PHP (penultimate-hop popping), when a link flap causes the next-hop of a label received from the egress router to change, once the link comes back, the penultimate-hop router might fail to install the clone route (S=0) entry for that label and result in traffic blackhole. 

The bbe-smgd might not respond the NS message for the SLAAC client on dynamic VLAN

On all MX-Series with dynamic VLAN and NDRA deployment for the subscriber access, the SLAAC (Stateless Address Autoconfiguration) client might not get the gateway MAC address from the BNG (Broadband Network Gateways), as a result, the SLAAC client will not get a link-local prefix and the IPv6 global connectivity will not be established. 

Fabric drops might be seen if using a newer generation of MPC with SFB2

On MX2020, MX2010 and MX2008 platforms with SFB2 cards installed, if a newer generation of MPC (e.g. MPC type 3, 4, 5, 6, 7, 8 or 9) is installed into a slot which had MPC 3D 16x10GE, MPC type 1 or MPC type 2 previously installed, the available fabric bandwidth to the new MPC card would be rate-limited due to residual programming on the fabric planes. Traffic impact is observed during peak utilization. 

FPC might crash when BGP multipath is configured with protection

When running with Border Gateway Protocol (BGP) multipath with protection configured, it is possible to encounter a situation where nexthops references are not properly decremented, thus causing the system to hold onto nexthops when they should be freed. This leads to a memory hog situation which eventually results in a Flexible PIC Concentrator (FPC) crash. 

Forwarding issue on mixed link-speed AE interface after FPC reloads

On PTX series, in mixed link-speed AE interface scenario, if the FPC which hosts one of the member links of the AE reloads and after the FPC comes back, the member link of the AE on the FPC recovers to be up and joins the AE bundle, but the member link is not able to forward traffic, which results in uneven load-balance or black-hole of the AE interface traffic. 

The rpd might crash continuously when IPv6 prefix with IPv4 next-hop exists in BGP multipath scenario.

In BGP scenario with multipath enabled, if applying import/export policy of IPv6 routes with a IPv4 next-hop to a BGP neighbor, the rpd might crash continuously. 

The rpd crash might be seen if a BGP unresolved route is withdrawn

If an import policy is applied to a BGP neighbor and the policy has indirect IPv4 next-hop for IPv4 and IPv6 routes (IPv6 routes resolved over IPv4), when BGP unresolved route is withdrawn, rpd crash might be seen. 

Certain builds of Junos OS do not allow you to upgrade or commit configuration changes when the SI service interface is used.

Certain builds of JUNOS do not allow you to upgrade or commit config changes when the SI service interface is used. 17.2R3, 17.4R2, 18.1R3, 18.2R2, 18.3.R1, 16.1R8, 16.2R3 are known to encounter this issue. 

If FPGA on the new master CB has a specific hardware failure, the chassid might keep crashing after GRES switchover

On MX/EX/SRX platforms, after GRES switchover, if a chassis has bent-pin or failed Field Programmable Gate Array (FPGA) on the new CB has a specific hardware failure and fails to detect FPC presence properly, the chassisd might keep crashing. 

The best and the second-best routes might have the same weight value if BGP PIC is enabled.

In BGP PIC (Prefix Independent Convergence) scenario, next-hop of unequal BGP multipath routes might have the same weight (0x1), resulting in unexpected load balance for traffic. 

"MIC Error code: 0x1b0002" alarm might not be cleared for MIC on MPC6 when the voltage has returned to normal

The voltage high alarm might not be cleared when voltage level comes back to normal for MIC on MPC6. 

The transportd might consume 100% CPU for a prolonged period

There might be memory leak on tarnsportd when bulk SNMP polling are on large-scale IFLs and large number of traps are created due to interface flapping etc. The memory leak could cause the transportd consuming high CPU for a prolonged period. 

RVT interface might get flapping

RVT (redundant-virtual-tunnel) interface might get flapping due to a config change made on chassis FPC which is not part of RVT. It might cause traffic loss. 

The UHP behavior is not supported for LDP to SR stitching scenario

In LDP to SR (Segment-Routing) stitching scenario, when connecting with a different vendor`s router who advertises P-Flag (no-PHP Flag, requires UHP behavior) in their Prefix-SIDs, the SR traffic destined to such neighbors could get dropped because the UHP (Ultimate-Hop-Popping) behavior is not supported in some versions of Junos platforms. 

Traffic impact might be seen if auto-bandwidth is configured for RSVP LSPs

With auto-bandwidth configured for Resource Reservation Protocol (RSVP) Label Switched Path (LSP), when timeout occurs during LSP statistics query, large bandwidth might be wrongly reserved for the LSP. If there is no sufficient resources (e.g. bandwidth, alternative path) in the network, other LSPs might be torn down, or might not go up. 

The ISIS-SR route sent by the mapping server might be broken for ECMP

On MX/PTX series, in ISIS-SR (segment routing) scenario, the ECMP (equal-cost multi-path) route sent by ISIS-SR mapping server stitched from LDP might not be pushed labels on one of ECMP paths, which results in traffic blackhole on the next-hop with no label.