ARM64 hypervisor bug traced to NX bit enabling instruction cache incoherence

The bug emerged during development of a bare-metal hypervisor for ARM64, where enabling the CTR_EL0 intercept, a mechanism to trap reads of the cache type register, triggered random system lockups. The initial assumption was that the hypervisor’s emulation of the MRS instruction was corrupting registers, but verification of the exception handler and stack allocation ruled this out. Debug prints confirmed that the handler was not modifying any registers on real hardware, despite functioning correctly in QEMU.

The root cause was traced to instruction cache incoherence, a known limitation of ARM architectures where modifications to executable data do not automatically propagate to instruction fetches. The hypervisor sorted an array of executable machine instructions at runtime, which worked in QEMU but failed on physical hardware due to stale cached instructions. Moving the sorting to the build phase resolved this issue, but the system still failed to boot, indicating additional layers of complexity.

ARM’s fragmented hardware ecosystem further complicated debugging. Unlike x86, where Intel and AMD dominate, ARM implementations vary widely across vendors, leading to subtle bugs or vendor-specific behaviors. The developer added exhaustive exception handlers to capture any misbehavior, but no exceptions were triggered. This suggested the issue might stem from hardware quirks or undocumented deviations from the ARM specification, rather than a straightforward software bug.

The incident highlights the unintended consequences of enabling the NX bit, which is primarily associated with security but can also disrupt low-level hardware interactions. Engineers working on ARM64 hypervisors or bare-metal code must account for instruction cache incoherence when modifying executable data, as ARM does not guarantee automatic coherence between data and instruction caches. This bug also serves as a reminder that hardware assumptions valid in emulation may not hold on physical devices.