Support Support Downloads Knowledge Base Apex Support Portal Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[PTX] Error message - t6e_dfe_tuning_state:et-6/0/0 - Failed to dfe tuning count

0

0

Article ID: KB37421 KB Last Updated: 16 Sep 2021Version: 1.0
Summary:

On PTX platforms, a port may not come up (physical link is down) after an interface flap, reboot of the line card, or transport gear restart on the remote end.

This article discusses a scenario where port et-1/0/8 did not come up after FPC8 reboot and how to resolve the issue.
Symptoms:

Topology:

PTX5000-A : et-1/0/8 < Transport device > PTX500o-B: et-8/1/0 is down.

Description

Model: ptx5000
Junos: 17.4X6.15
FPC1: FPC-P2
xcvr: QSFP-100G-CWDM4

Physical interface: et-1/0/8, Enabled, Physical link is Down
  Interface index: 282, SNMP ifIndex: 586, Generation: 285
  Link-level type: Ethernet, MTU: 1514, Speed: 100Gbps, BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Enabled
  Device flags   : Present Running Down
  Interface flags: Hardware-Down SNMP-Traps Internal: 0x4000
  Link flags     : None
  CoS queues     : 8 supported, 8 maximum usable queues
  Hold-times     : Up 0 ms, Down 0 ms
  Damping        : half-life: 0 sec, max-suppress: 0 sec, reuse: 0, suppress: 0, state: unsuppressed
  Current address: 38:4f:49:7c:d9:59, Hardware address: 38:4f:49:7c:d9:59
  Last flapped   : 2020-11-30 20:36:59 UTC (14:50:16 ago)
  Statistics last cleared: 2020-11-30 22:00:42 UTC (13:26:33 ago)
  Traffic statistics:
   Input  bytes  :       11121699880758                    0 bps
   Output bytes  :       11121720347341                    0 bps
   Input  packets:           2446480842                    0 pps
   Output packets:           2446539907                    0 pps
   IPv6 transit statistics:
    Input  bytes  :                   0
    Output bytes  :                   0
    Input  packets:                   0
    Output packets:                   0
  Input errors:
    Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0, Resource errors: 0
  Output errors:
    Carrier transitions: 0, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0
  Egress queues: 8 supported, 5 in use
  Queue counters:       Queued packets  Transmitted packets      Dropped packets
    0                                0                    0                    0
    1                                0                    0                    0
    2                                0                    0                    0
    3                                0                    0                    0
    5                                0                    0                    0
  Queue number:         Mapped forwarding classes
    0                   FC-BEST-EFFORT, FC-STREAMING
    1                   FC-LOW-PRIORITY-DATA
    2                   FC-LOW-LATENCY-DATA, FC-SIGNALING, FC-REAL-TIME
    3                   FC-NETWORK-CONTROL
    5                   FC-TELEPHONY
  Active alarms  : LINK
  Active defects : LINK, LOCAL-FAULT
  PCS statistics                      Seconds
    Bit errors                             0
    Errored blocks                         0
  Ethernet FEC Mode  :                  FEC91
  Ethernet FEC statistics              Errors
    FEC Corrected Errors                    0
    FEC Uncorrected Errors                  0
    FEC Corrected Errors Rate               0
    FEC Uncorrected Errors Rate             0
  MAC statistics:                      Receive         Transmit
    Total octets                             0                0
    Total packets                            0                0
    Unicast packets                          0                0
    Broadcast packets                        0                0
    Multicast packets                        0                0
    CRC/Align errors                         0                0
    FIFO errors                              0                0
    MAC control frames                       0                0
    MAC pause frames                         0                0
    Oversized frames                         0
    Jabber frames                            0
    Fragment frames                          0
    VLAN tagged frames                       0
    Code violations                          0
  Filter statistics:
    Input packet count                       0
    Input packet rejects                     0
    Input DA rejects                         0
    Input SA rejects                         0
    Output packet count                                       0
    Output packet pad count                                   0
    Output packet error count                                 0
    CAM destination filters: 0, CAM source filters: 0
  Packet Forwarding Engine configuration:
    Destination slot: 0 (0x00)
  CoS information:
    Direction : Output
    CoS transmit queue               Bandwidth               Buffer Priority   Limit
                              %            bps     %           usec
    0 FC-BEST-EFFORT, FC-STREAMING 95 95000000000 95              0      low    none
    3 FC-NETWORK-CONTROL      5     5000000000     5              0      low    none
  Link Degrade :
    Link Monitoring                   :  Disable
  Interface transmit statistics: Disabled

Log messages

Dec  1 05:11:38  PTX5000-A : %PFE-3: fpc1 t6e_dfe_tuning_state:et-1/0/8 - Failed to dfe tuning count 10 
Dec  1 05:30:29  PTX5000-A : %PFE-3: fpc1 t6e_dfe_tuning_state:et-1/0/8 - Failed to dfe tuning count 10 
Dec  1 05:49:21  PTX5000-A : %PFE-3: fpc1 t6e_dfe_tuning_state:et-1/0/8 - Failed to dfe tuning count 10 
Dec  1 06:08:13  PTX5000-A : %PFE-3: fpc1 t6e_dfe_tuning_state:et-1/0/8 - Failed to dfe tuning count 10 
Dec  1 06:27:07  PTX5000-A : %PFE-3: fpc1 t6e_dfe_tuning_state:et-1/0/8 - Failed to dfe tuning count 10 
Dec  1 06:46:01  PTX5000-A : %PFE-3: fpc1 t6e_dfe_tuning_state:et-1/0/8 - Failed to dfe tuning count 10 
Dec  1 07:04:52  PTX5000-A : %PFE-3: fpc1 t6e_dfe_tuning_state:et-1/0/8 - Failed to dfe tuning count 10 
Dec  1 07:23:46  PTX5000-A : %PFE-3: fpc1 t6e_dfe_tuning_state:et-1/0/8 - Failed to dfe tuning count 10 
Dec  1 07:42:38  PTX5000-A : %PFE-3: fpc1 t6e_dfe_tuning_state:et-1/0/8 - Failed to dfe tuning count 10 
Dec  1 08:01:31  PTX5000-A : %PFE-3: fpc1 t6e_dfe_tuning_state:et-1/0/8 - Failed to dfe tuning count 10 
Cause:

On the PTX5000/PTX3000 platforms with t6e-pic installed (e.g. P3-10-U-QSFP28, P3-15-U-QSFP28, P3-24-U-QSFP28), the interface may fail to perform DFE tuning after link flaps on those PICs. Because of this, the interface may be stuck in down status. The issue is that under some instances, t6e tuning code can bypass retimer false pmd-lock check. This could happen due to one of the following:

  • Ingress signal is lost or degraded during final stages of the tuning.
  • Repetitive requests to start tuning after Avago serdes failed to respond to previous stop tuning request.

By enabling and adding debug logs, it revealed the existing serdes recovery mechanism (serdes shutdown + serdes re-init) in the PIC driver was not getting executed due to not meeting the required conditions even after a long wait time (approx. 1-2 hours of wait).

Solution:

The issue is fixed in Junos 17.4X6.20 and later releases.

The following workaround can be used if you are running a release earlier than 17.4X6.20 and upgrade is not possible. 
Please contact your JTAC Representative to assist with the restoration steps.

Restoration steps:

  1. Keep the hard loop for 15 minutes check if link comes UP.

  2. Collect the script output, session logs, /var/log/messages. Check if any serdes lane is busy. 

  3. If busy, issue the following commands:

    vty fpc<fpc_num>
    set parser security 10
    test t6e-pic select <t6e_pic_number>
    test t6e-pic serdes-repair <port_num> <chan_num> 2
  4. Wait for at least 15 minutes to allow DFE tuning to complete and see if the link comes UP. If not, try step 2 again.

  5. If recovery does not help, contact your JTAC Representative to assist with manual recovery.

Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search