1 Reply Latest reply: Nov 15, 2017 8:55 AM by Karlo Bobiles RSS

    VIRL 1.3 Metadata CRC error detected xfs_repair

    sleepwalkerx97

      Hello,

       

      I thought I would share this issue and how I was able to resolved it for my coworker.

       

      I got a phone call today from a co-worker about having issues with VIRL 1.3 (Aug 2017 Build). He ran across an issue after deploying VIRL 1.3 ova (1 nic or 5 nic ova on ESXi 6.5 update1) After deploying the OVA he would configure CPU and Memory needed on the virtual machine. VIRL would successfully boot into CLI allowing him to run virl_setup. However, after configuring VIRL no matter if you rebooted or shutdown the VIRL virtual machine. You receive the following erros on next reboot. (I was able to run xfs_repair but would only temporary fix the issue and would error next reboot or shutdown)

      IMG_0240.jpg

       

      My co-worker downloaded ova again (Both 1nic or 5nic) and re-deployed the ova's about 3 or 4 times before involving me. After doing some troubleshooting with both ESXi and VIRL I was able to find the issue. It was due to a firmware issue with Intel 600p NVMe SSD that was installed on ESXi host. 600p was used as VMFS Datastore hosting VMDK for VIRL 1.3 virtual machine. I decided to SSH into ESXi host and run voma on the intel 600p. ESXi reported about 1150 errors on the device and was unable to fix the issues.

       

      What I found really funny was datastore had the following virtual machine running with no problems

      VIRL 1.2

      Window 10 1709 64-bit

       

      So I did some research and came across a bug reported with XFS and NVMe drive related to firmware currently loaded on 600p. Intel had released new firmware to resolve this issue. https://bugzilla.redhat.com/show_bug.cgi?id=1402533

       

      Resolved the issue by doing the following:

      1. Download Intel SSD firmware update tool (In this case I selected to download the Bootable ISO) Download Intel® SSD Firmware Update Tool

      2. Mounted the ISO, Restarted ESXi host, and booted off the ISO.

      3. Upgraded the firmware to v121c on the SSD

      4. Restarted the host back into ESXi

      5. Downloaded intel-nvme vib for ESXi  and extracted the ZIP. (ESXi Intel Drive)

      6. Uploaded "VMW-ESX-6.5.0-intel-nvme-1.2.1.15-offline_bundle-5330543.zip" to the VMFS datastore.

      7.SSH into ESXi host

      8.Ran the following command

           esxcli software vib install -d /vmfs/volumes/nameofdatastore/VMW-ESX-6.5.0-intel-nvme-1.2.1.15-offline_bundle-5330543.zip

      9. Verified driver was successfully installed from vib.

      10. Reboot ESXi host.

      11. Ran another "voma" check and came back with 0 errors

      12. Created new Virl 1.3 Virtual machine using OVA.

       

      Hopefully this helps somebody else who might come across the same issue. If you are having the same issue but with another brand NVMe SSD I would encourage you to update the firmware if you are having the same issue.