Post

Fixing Intel i226-V Hard Freezes and Link Drops on N-Series Mini PCs

Fixing Intel i226-V Hard Freezes and Link Drops on N-Series Mini PCs

If you run a small Intel N100, N150, N305 or N355 mini PC as a NAS, home server or router, there is a good chance it uses an onboard Intel i226-V or i225-V 2.5 GbE controller. There is also a good chance it will freeze on you under network load.

Mine did. My ZimaBlade 2 hard froze roughly every one to two days while ingesting backups, and each time it needed a physical power cycle. After a fair amount of digging I found that two separate problems were in play, and I have written everything up in a repo at github.com/karubits/intel-nic-i226-issues. This post is the readable version.

The symptom

  • The whole system hard freezes under sustained network traffic. No response on the network or the console.
  • There is no kernel panic, no oops, no thermal or MCE message. The journal just stops mid line with no shutdown sequence. That is the signature of a hard PCIe lock rather than a software crash.
  • Recovery requires pulling the power. A warm reboot is not possible because the OS is completely hung.
  • A milder version of the same fault shows up as the NIC link dropping for about a minute and then recovering while the host stays up.

For a headless always-on device this is about as bad as it gets: total loss of service until somebody is physically at the box.

Who is affected

Any board with an onboard Intel i226-V (8086:125C) or i225-V (8086:15F3) driven by the Linux igc driver. That covers ZimaBlade and ZimaBoard, Protectli, Odroid H4, Minisforum, Topton and CWWK, Beelink and many more, plus add-in 2.5 GbE cards. This is not specific to any one vendor.

The tell-tale sign is this line on every boot:

1
igc 0000:0X:00.0: can't disable ASPM; OS doesn't have ASPM control

It means the BIOS enabled ASPM but did not hand control over to the operating system, so the driver cannot turn it off.

Ruling out the usual suspects

Freezes like this are easy to blame on the wrong thing. In every case I looked at, the following were fine:

CheckedFinding
TemperatureNot thermal. The SoC sits far below its throttle limit
MemoryNot OOM. Plenty of RAM free and swap unused at freeze time
StorageNot disk. Filesystems healthy, no I/O errors
Kernel errorsNo panic, oops, hung task, soft lockup or MCE in the logs
RAM / EDACA boot time EDAC igen6 ... IBECC MEMORY ERROR ADDR 0x7fffffffe0 is a known benign false positive of the igen6_edac probe on Alder Lake-N

What points at the NIC is the per boot ASPM warning, the tight correlation with sustained throughput, and most convincingly the fact that disabling ASPM makes the freeze go away completely.

Root cause

PCIe Active State Power Management (ASPM) on the i225 and i226. The BIOS enables it but does not grant the OS control, so igc cannot disable it. Under load the NIC’s link power management state machine (L1 and the L1 substates) wedges, which either drops the link or hard locks the PCIe path and takes the whole system with it. This is a widely documented failure mode across the whole i225 and i226 family.

Reproducing it

  1. Run the board on a modern Linux kernel (6.x, igc driver) with stock NIC firmware and default BIOS settings.
  2. Push sustained traffic through the port. A multi gigabyte transfer, a backup job or an iperf3 soak all work. Light traffic will not trigger it quickly.
  3. Within hours to days the system hard freezes and needs a power cycle. The journal shows an abrupt end of log with no shutdown, and previous boots show the same unclean pattern.

The link drop variant reproduces more often and is the lighter signal of the same instability.

Fix 1: disable ASPM

This is the one that stops the freezes. There are two ways to do it.

Kernel command line (works on any board):

1
pcie_aspm=off intel_idle.max_cstate=1
  • pcie_aspm=off disables PCIe ASPM globally. This is the direct fix. Afterwards the can't disable ASPM warning is gone and the link reports ASPM as disabled.
  • intel_idle.max_cstate=1 caps deep CPU C-states. It is belt and braces against N-series idle related hangs.

On Debian, add these to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub, run update-grub and reboot.

BIOS (cleaner, if your vendor exposes it): disable ASPM and L1 Substates for the PCIe root ports, or set DMI/PEG ASPM to Disabled. This avoids the small system wide idle power cost of pcie_aspm=off.

A couple of notes:

  • Energy Efficient Ethernet (EEE) is already disabled by the igc driver by default, so EEE is not the freeze trigger. Old firmware does have a separate EEE link flap bug, which is covered by Fix 2.
  • A targeted upstream igc fix that only disables ASPM L1.2 landed in newer kernels around 6.18. On older kernels the blanket pcie_aspm=off is the reliable stand in.

Fix 2: update the NIC firmware

Stock units often ship with old NVM firmware. My ZimaBlade 2 came with i226-V 2.17 from 2023. The latest public versions are:

ControllerLatest NVM
i226-V2.32
i225-V1.89

What the firmware update fixes:

  • The device sometimes not being enumerated after a power cycle or warm reset (fixed in i226-V NVM 2.22 and 2.23).
  • Link flaps with Energy Efficient Ethernet enabled (fixed in i226-V NVM 2.22 to 2.25).
  • Community reports confirm that going from 2.17 to 2.32 resolves load related link drops and enumeration issues on N100 and N150 boxes.

What the firmware update does not fix:

  • The ASPM induced hang. Even on the latest NVM, ASPM still has to be disabled in the BIOS or on the kernel command line. Firmware and ASPM are two separate problems, so do Fix 1 regardless.

Updating is where it gets awkward. Vendors rarely publish an NVM image or a sanctioned update path for these onboard NICs, so people rely on community firmware images and Intel’s nvmupdate64e tool at their own risk. The repo includes an auto detecting helper script that removes most of the footguns: it picks the correct 1 MB or 2 MB image, sets iomem=relaxed, and flashes in a safe order. See firmware/README.md in the repo.

Flashing NIC firmware can permanently brick the controller. Read the warnings in the firmware README first, and only flash with physical access to the box and reliable power.

What to ask your hardware vendor

None of these workarounds should be necessary. If you own one of these boards, these are reasonable things to ask for:

  1. A BIOS option to disable PCIe ASPM and L1 Substates, or to grant the OS ASPM control, for the i225/i226 root ports. This is the proper fix.
  2. The latest NVM shipped on new units, and a sanctioned firmware image plus update procedure for existing owners, with the correct image for the board’s flash size.
  3. Documentation of the recommended interim Linux settings so users do not have to reverse engineer them.

My unit: ZimaBlade 2

For completeness, here is the hardware that started all of this. Since applying the ASPM fix the box has had zero further hard freezes.

I reported the findings to IceWhale and asked for a BIOS toggle to disable ASPM and L1 Substates, plus a sanctioned i226-V 2.32 NVM image and update path. Their response, in short: they are unable to provide an NVM firmware update for the i226 NIC at this time, and they suggested that a kernel update may help with the ASPM errors and improve stability. They did thank me for the detailed investigation and asked to be kept posted on anything new.

A newer kernel is a fair pointer, since the upstream igc change that disables ASPM L1.2 landed around 6.18, but on Debian 13 with kernel 6.12 the pcie_aspm=off workaround remains the practical fix for now, and there is still no official firmware path for existing owners.

  
DeviceZimaBlade 2
SoCIntel N150 (Alder Lake-N / Twin Lake), 4 cores
RAM16 GB
NIC2x Intel i226-V, 8086:125C rev 04, driver igc
NIC NVM2.17 (ethtool 2017:888d, eTrack 80000303, PHY FW 4C07_888D, EFI OROM 0.1.4, PBA G23456-000, 2 MB flash)
OSDebian 13 (Trixie), kernel 6.12.90
TriggerSustained inbound network plus disk I/O as a backup and file server target

Before the fix it froze every one to two days under backup load. After adding pcie_aspm=off intel_idle.max_cstate=1 to the kernel command line it has run the exact same workload with no freezes and no more trips to pull the power cable.

Evidence

The ASPM warning on both ports, every boot:

1
2
igc 0000:01:00.0: can't disable ASPM; OS doesn't have ASPM control
igc 0000:02:00.0: can't disable ASPM; OS doesn't have ASPM control

Link capability showing ASPM L1 and L1 Substates advertised, but the OS lacking control:

1
2
LnkCap:   Speed 5GT/s, Width x1, ASPM L1, Exit Latency L1 <4us
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+

NIC firmware inventory from nvmupdate64e -i:

1
2
3
Device: 125C   Subvendor: 8086   Subdevice: 0000   Revision: 4
ETrackId: 80000303   NVM Version: 2.23(2.17)   PBA: G23456-000
EFI: 0.1.4   checksum: Valid

And lspci:

1
2
01:00.0 Ethernet controller: Intel Corporation Ethernet Controller I226-V [8086:125c] (rev 04)
02:00.0 Ethernet controller: Intel Corporation Ethernet Controller I226-V [8086:125c] (rev 04)

The freeze signature in the journal is an abrupt end mid line on a routine log entry, with no Reached target Shutdown, no reboot: and no panic, repeated across multiple boots. A hard lock, not a clean reboot or a software crash.

Wrapping up

If your N-series mini PC with an i226-V or i225-V freezes under load, do both fixes: disable ASPM (this stops the freezes) and update the NIC firmware (this fixes the enumeration and EEE bugs). Everything, including the firmware helper, is in the intel-nic-i226-issues repo. If it helps you, or you have data from another board, please open an issue there.

This post is licensed under CC BY 4.0 by the author.