- Score 84
- Official
BOS Summary: The First Token Matters
arXiv:2611.23508v1 Announce Type: new Abstract: Large vision-language models (LVLMs) and language models (LLMs) often exhibit attention sinks, whereby particular positions, such as the beginning-of-sequence (BOS) token, receive a disproportionate share of attention. Under standard causal masking, however, the BOS query attends only to itself, so the representation that later tokens repeatedly acce
So what
What this event means by reading role—not a longer recap.
- BuilderA parameter-free prefill tweak that relaxes causal masking for the BOS row alone could be cheap to test in existing inference stacks.
- ResearcherMechanistic analysis links making BOS input-specific to weakened sink dominance and redistributed attention toward content tokens.
Score dimensions
Higher total means read first. Each bar is one factor we use to rank the system pool. How we score