Pruning LLMs Like a Physicist: Block Removal as an Ising Optimization Problem

Pruning large language models by removing transformer blocks, known as depth pruning, offers predictable inference speedups and memory savings, and is compatible with other optimization techniques. The challenge lies in selecting which blocks to remove, as choices interact, making it a combinatorial problem. This problem can be modeled using spin systems, similar to the physics of Ising optimization. The CBO method, which searches the coupled configuration space, has shown superior performance in identifying optimal block removal configurations, even in hybrid models with unevenly distributed redundancy, outperforming methods like block influence.

This report uniquely frames LLM block removal as an Ising optimization problem, unlike prior approaches that treated it as a ranking problem, and introduces a CBO method that outperforms block influence.