Support Support Downloads Knowledge Base Case Manager My Juniper Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[MX] Syslog message: EA. *I2C Failed device: HMC:U65010:MEM EA0 HMC0, group 0x45 address 0x10

0

0

Article ID: KB35256 KB Last Updated: 07 Dec 2019Version: 2.0
Summary:

This article explains what must be done when the "​I2C Failed device: HMC:U65010:MEM EA0 HMC0, group 0x45 address 0x10" log message is observed on MX Series routers, and why the error is output.

Note: This is a troubleshooting article for a PFE ASIC Syslog Event. To view other documented syslog events related to XMCHIP, XLCHIP, MQCHIP, LUCHIP, EACHIP, and PECHIP, see KB31893 - Master Index of Articles for Troubleshooting PFE ASIC Syslog Events.

Symptoms:

When an "​I2C Failed device: HMC:U65010:MEM EA0 HMC0, group 0x45 address 0x10" event occurs, a message similar to the following is reported:

Sep 28 00:16:37.407 2019  MX_RE0 : %PFE-3: fpc2 I2C Failed device: HMC:U65011:MEM EA0 HMC1, group 0x45 address 0x11 
Sep 28 00:16:37.407 2019  MX_RE0 : %PFE-3: fpc2 cmtfpc_hmc_iofunc HMC 1 0x45/0x11 write reg offset 0x2b0000 data 0x3f failed 
Sep 28 00:16:37.407 2019  MX_RE0 : %PFE-3: fpc2 hmc_reg_wr HMC 63 pio error 1 
Sep 28 00:16:37.407 2019  MX_RE0 : %PFE-3: fpc2 mpcs_i2c_smem_read: MPCS(0) ctlr 0 smem_idx 0 group 5 addr 0x11 prio 0 flags 0x6 failed status 0xa 
Sep 28 00:16:37.407 2019  MX_RE0 : %PFE-3: fpc2 I2C Failed device: HMC:U65011:MEM EA0 HMC1, group 0x45 address 0x11 
Sep 28 00:16:37.408 2019  MX_RE0 : %PFE-3: fpc2 cmtfpc_hmc_iofunc HMC 1 0x45/0x11 write reg offset 0x2b0000 data 0x3f failed 
Sep 28 00:16:37.408 2019  MX_RE0 : %PFE-3: fpc2 hmc_reg_wr HMC 63 pio error 2 
Sep 28 00:16:37.408 2019  MX_RE0 : %PFE-3: fpc2 mpcs_i2c_smem_read: MPCS(0) ctlr 0 smem_idx 1 group 5 addr 0x11 prio 0 flags 0x4 failed status 0xa 
Sep 28 00:16:37.408 2019  MX_RE0 : %PFE-3: fpc2 I2C Failed device: HMC:U65011:MEM EA0 HMC1, group 0x45 address 0x11 
Sep 28 00:16:37.408 2019  MX_RE0 : %PFE-3: fpc2 cmtfpc_hmc_iofunc HMC 1 0x45/0x11 write reg offset 0x2b0000 data 0x3f failed 
Sep 28 00:16:37.408 2019  MX_RE0 : %PFE-3: fpc2 hmc_reg_wr HMC 63 pio error 3 
Sep 28 00:16:37.408 2019  MX_RE0 : %PFE-3: fpc2 hmc_eri_config_access, HMC 63, eri 10,  ext_data 0 write error 
Sep 28 00:16:37.408 2019  MX_RE0 : %PFE-3: fpc2 micron_hmc_get_temp_info, eri 10 error 
Sep 28 00:16:37.408 2019  MX_RE0 : %PFE-3: fpc2 hmc_misc_support vector NULL or failure return from vector 
Sep 28 00:16:37.408 2019  MX_RE0 : %PFE-3: fpc2 cmtfpc_hmc_get_temp hmc get temp failed for asic 63, status 1 
Sep 28 00:16:47.411 2019  MX_RE0 : %PFE-3: fpc2 mpcs_i2c_smem_read: MPCS(0) ctlr 0 smem_idx 0 group 5 addr 0x10 prio 0 flags 0x6 failed status 0xa 
Sep 28 00:16:47.425 2019  MX_RE0 : %PFE-3: fpc2 I2C Failed device: HMC:U65010:MEM EA0 HMC0, group 0x45 address 0x10 
Sep 28 00:16:47.426 2019  MX_RE0 : %PFE-3: fpc2 cmtfpc_hmc_iofunc HMC 0 0x45/0x10 write reg offset 0x2b0000 data 0xe00d3e18 failed 
Sep 28 00:16:47.428 2019  MX_RE0 : %PFE-3: fpc2 hmc_reg_wr HMC 62 pio error 1 
Sep 28 00:16:47.429 2019  MX_RE0 : %PFE-3: fpc2 mpcs_i2c_smem_read: MPCS(0) ctlr 0 smem_idx 1 group 5 addr 0x10 prio 0 flags 0x4 failed status 0xa 
Sep 28 00:16:47.431 2019  MX_RE0 : %PFE-3: fpc2 I2C Failed device: HMC:U65010:MEM EA0 HMC0, group 0x45 address 0x10 
Sep 28 00:16:47.431 2019  MX_RE0 : %PFE-3: fpc2 cmtfpc_hmc_iofunc HMC 0 0x45/0x10 write reg offset 0x2b0000 data 0xe00d3e18 failed 
Sep 28 00:16:47.434 2019  MX_RE0 : %PFE-3: fpc2 hmc_reg_wr HMC 62 pio error 2 
Sep 28 00:16:47.438 2019  MX_RE0 : %PFE-3: fpc2 mpcs_i2c_smem_read: MPCS(0) ctlr 0 smem_idx 0 group 5 addr 0x10 prio 0 flags 0x6 failed status 0xa 
Sep 28 00:16:47.443 2019  MX_RE0 : %PFE-3: fpc2 I2C Failed device: HMC:U65010:MEM EA0 HMC0, group 0x45 address 0x10 
Sep 28 00:16:47.446 2019  MX_RE0 : %PFE-3: fpc2 cmtfpc_hmc_iofunc HMC 0 0x45/0x10 write reg offset 0x2b0000 data 0xe00d3e18 failed 
Sep 28 00:16:47.448 2019  MX_RE0 : %PFE-3: fpc2 hmc_reg_wr HMC 62 pio error 3 
Sep 28 00:16:47.452 2019  MX_RE0 : %PFE-3: fpc2 hmc_eri_config_access, HMC 62, eri 10,  ext_data 0 write error 
Sep 28 00:16:47.452 2019  MX_RE0 : %PFE-3: fpc2 micron_hmc_get_temp_info, eri 10 error 
Sep 28 00:16:47.452 2019  MX_RE0 : %PFE-3: fpc2 hmc_misc_support vector NULL or failure return from vector 
Sep 28 00:16:47.452 2019  MX_RE0 : %PFE-3: fpc2 mpcs_i2c_smem_read: MPCS(0) ctlr 0 smem_idx 1 group 5 addr 0x10 prio 0 flags 0x4 failed status 0xa 
Sep 28 00:16:47.452 2019  MX_RE0 : %PFE-3: fpc2 I2C Failed device: HMC:U65010:MEM EA0 HMC0, group 0x45 address 0x10 
Sep 28 00:16:47.452 2019  MX_RE0 : %PFE-3: fpc2 cmtfpc_hmc_iofunc HMC 0 0x45/0x10 write reg offset 0x2b0000 data 0xe00d3e18 failed 
Sep 28 00:16:47.453 2019  MX_RE0 : %PFE-3: fpc2 hmc_reg_wr HMC 62 pio error 1 
Sep 28 00:16:47.453 2019  MX_RE0 : %PFE-3: fpc2 mpcs_i2c_smem_read: MPCS(0) ctlr 0 smem_idx 0 group 5 addr 0x10 prio 0 flags 0x6 failed status 0xa 
Sep 28 00:16:47.453 2019  MX_RE0 : %PFE-3: fpc2 I2C Failed device: HMC:U65010:MEM EA0 HMC0, group 0x45 address 0x10 
Sep 28 00:16:47.453 2019  MX_RE0 : %PFE-3: fpc2 cmtfpc_hmc_iofunc HMC 0 0x45/0x10 write reg offset 0x2b0000 data 0xe00d3e18 failed 
Sep 28 00:16:47.453 2019  MX_RE0 : %PFE-3: fpc2 hmc_reg_wr HMC 62 pio error 2 
Sep 28 00:16:47.453 2019  MX_RE0 : %PFE-3: fpc2 mpcs_i2c_smem_read: MPCS(0) ctlr 0 smem_idx 1 group 5 addr 0x10 prio 0 flags 0x4 failed status 0xa 
Sep 28 00:16:47.453 2019  MX_RE0 : %PFE-3: fpc2 I2C Failed device: HMC:U65010:MEM EA0 HMC0, group 0x45 address 0x10 
Sep 28 00:16:47.453 2019  MX_RE0 : %PFE-3: fpc2 cmtfpc_hmc_iofunc HMC 0 0x45/0x10 write reg offset 0x2b0000 data 0xe00d3e18 failed 
Sep 28 00:16:47.453 2019  MX_RE0 : %PFE-3: fpc2 hmc_reg_wr HMC 62 pio error 3 
Sep 28 00:16:47.453 2019  MX_RE0 : %PFE-3: fpc2 hmc_eri_config_access, HMC 62, eri 10,  ext_data 0 write error 
Sep 28 00:16:47.453 2019  MX_RE0 : %PFE-3: fpc2 micron_hmc_get_temp_info, eri 10 error 
Sep 28 00:16:47.453 2019  MX_RE0 : %PFE-3: fpc2 hmc_misc_support vector NULL or failure return from vector 
Sep 28 00:16:47.453 2019  MX_RE0 : %PFE-3: fpc2 mpcs_i2c_smem_read: MPCS(0) ctlr 0 smem_idx 0 group 5 addr 0x10 prio 0 flags 0x6 failed status 0xa 
Sep 28 00:16:47.453 2019  MX_RE0 : %PFE-3: fpc2 I2C Failed device: HMC:U65010:MEM EA0 HMC0, group 0x45 address 0x10 
Sep 28 00:16:47.453 2019  MX_RE0 : %PFE-3: fpc2 cmtfpc_hmc_iofunc HMC 0 0x45/0x10 write reg offset 0x2b0000 data 0xe00d3e18 failed 
Sep 28 00:16:47.453 2019  MX_RE0 : %PFE-3: fpc2 hmc_reg_wr HMC 62 pio error 1 
Sep 28 00:16:47.453 2019  MX_RE0 : %PFE-3: fpc2 mpcs_i2c_smem_read: MPCS(0) ctlr 0 smem_idx 1 group 5 addr 0x10 prio 0 flags 0x4 failed status 0xa 
Sep 28 00:16:47.454 2019  MX_RE0 : %PFE-3: fpc2 I2C Failed device: HMC:U65010:MEM EA0 HMC0, group 0x45 address 0x10 
Sep 28 00:16:47.454 2019  MX_RE0 : %PFE-3: fpc2 cmtfpc_hmc_iofunc HMC 0 0x45/0x10 write reg offset 0x2b0000 data 0xe00d3e18 failed 
Sep 28 00:16:47.454 2019  MX_RE0 : %PFE-3: fpc2 hmc_reg_wr HMC 62 pio error 2 
Sep 28 00:16:47.454 2019  MX_RE0 : %PFE-3: fpc2 mpcs_i2c_smem_read: MPCS(0) ctlr 0 smem_idx 0 group 5 addr 0x10 prio 0 flags 0x6 failed status 0xa 


Indications:

  • Traffic impact or PFE packet forwarding is permanently compromised if errors are seen repeatedly.

  • A major alarm will also be raised.

Cause:

The above message indicates that the system failed while interacting with the Hybrid Memory Cube (HMC) via the i2c bus from which it cannot recover without MPC reset. It is related to an HMC memory read error issue, with the HMC error being caused by a Bit flip in the HMC's SRAM. The error occurs when the affected memory is accessed.

This is a transient hardware issue on HMC.

Solution:

Perform the following steps to determine the cause and resolve the problem (if any).

  1. Collect the following show command outputs to investigate further.​​

Capture the output to a file (in case you need to open a Technical Service Request). To do this, configure each SSH client/terminal emulator to log your session.

show log messages
show log chassisd
start shell network pfe <fpc#>
show nvram
show syslog messages
  1. Analyze the show command output.

In the show log messages file, review the events that occurred at or just before the appearance of the "I2C Failed device: HMC:U65010:MEM EA0 HMC0, group 0x45 address 0x10" message.

  1. After confirming the error message in the above file, restart the FPC during a maintenance window to fix the issue. Restarting the FPC is seen to resolve the issue in most cases.

request chassis fpc slot <slot number> restart
  1. If the issue is seen repeatedly even after restarting the FPC, open a Technical Service Request for further investigation and assistance. Traffic or permanent packet forwarding on the Packet Forwarding Engine (PFE) may be impacted if the errors are seen repeatedly and may require replacement of the FPC.

Tip: When looking at an event in the logs, it is important to focus on the first error message in a collection of syslog messages. The first error message is usually the cause of all follow-on error messages. The subsequent collateral damage error messages can be ignored usually.

Modification History:
2019-12-07: Minor non-technical edit.
Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search