J-lens default targeting final layer inherits dominant language direction in DeepSeek-V3

A J-lens translates hidden states into words by asking how a nudge at an earlier layer changes the representation at a target layer, then reading through the unembedding. On DeepSeek-V3, the choice of target layer fundamentally changes what the lens shows because the final transformer block pushes down Chinese-token probabilities during English text, creating a large residual-stream shift that barely affects predictions but dominates the Jacobian.

This amplified direction becomes the top singular direction at every earlier layer when differentiating through the final block, causing roughly half of mid-depth readouts to show majority Chinese tokens on English inputs. Targeting the penultimate layer excludes that block and restores readouts to 3% Chinese tokens, matching the logit lens baseline.

The distortion cascades into evaluation metrics: on order-of-operations prompts, the final-layer target lifts every number token by about one standard deviation, making J-lens appear better than logit lens on that test. The same target choice also flips the sign of excess kurtosis, a statistic used in Anthropic's original paper to characterize representation geometry.

The author checked 78 publicly released J-lenses and found 80% target the final layer by default, while the library default and most practitioners follow this pattern. Gurnee et al. notably used penultimate targeting for their Sonnet experiments, suggesting awareness of the issue that did not propagate to the default configuration.

The broader lesson is that any strong downstream specialization, whether language calibration, token suppression, or other output-layer behavior, can dominate a J-lens and make earlier layers appear to carry that specialization. Engineers should test whether changing the target layer removes dominant directions before interpreting what a lens reveals about model internals.