Yingru Li Argues the Policy Gradient Is a Covariance and the Fisher a Variance

Yingru Li argues in Part 1 of a three-part theory series that the expected policy gradient is a covariance and the Fisher a variance, capping the gradient for any reward and explaining why post-training updates stay sparse. Li cites measurements that the top 1% of coordinates by Fisher hold 47.40% of the top 1% by gradient magnitude.

• Li cites Mukherjee et al. 2025 measuring that RL fine-tuning changes only 5 to 30 percent of parameters across seven algorithms.
• Li credits Marek and Ryabinin 2026 for score centering, which needs the sampler's full next-token distribution at every prefix.
• Li notes a baseline depending only on the prompt leaves the covariance unchanged, shifting only the spread of the signal.