Lend Me Your Eyes

Instruction-Aware Text Embeddings via Attention Relay

Yiyuan Luo, Vaggos Chatziafratis
University of California, Santa Cruz

Embedding models usually have to learn to follow instructions, through extra contrastive training on instruction data. Instruction-tuned LLMs, on the other hand, already know how. We show that an embedder can borrow this ability from an LLM, with no training at all. To make this transfer, we pass the attention weights an LLM produces to the embedder's own attention. We call it Attention Relay.

Question

Read by

An LLM reads a customer message, then a question. Each token is shaded by how much the model attends to it as it is about to answer.

What changes

Take a news article about a farmer in western France. As released, bge-large spreads its attention over the whole article, most of all on “the” and full stops, and places the article among technology stories from around the world, whatever we want to know about it. Putting the question in its input barely moves where it looks in the article. With Attention Relay, it looks at the words the question asks about, and its closest articles follow the question: asked where the news happened, all five are from France; asked for the topic, they stay technology.

Question

Embedding

New York Times articles from InBedder's evaluation set, lowercased as the dataset stores them, and the dataset's labels. Each token is shaded by how much bge-large-en-v1.5's [CLS] token attends to it, averaged over its layers and heads. With Attention Relay, Qwen3-1.7B's reading weights are multiplied into that attention.

If we shuffle the same weights over the article's words, the place disappears again for three of these four articles. What matters is which words the LLM reads.

How it works

Read. The LLM reads the text first and the question after it. As it is about to answer, its attention gives each token of the text a weight. The text comes first, so the LLM computes it the same way under every question. The question changes only these weights.
Align. These weights are a distribution over the text itself, so unlike the LLM's hidden states, they can leave the LLM. The embedding model splits the text into different tokens, so we carry the weights over through the characters.
Relay. The embedding model already pools its text with weights of its own. We multiply them by the LLM's weights and renormalize. When one token pools the text, every attention head at every layer keeps how much it attends to the text, but spreads it the way the LLM reads.

These eyes come from instruction tuning. Switch the figure at the top to the base checkpoint: before instruction tuning, the LLM's reading barely changes with the question.

Citation

@article{luo2026lend,
  title   = {Lend Me Your Eyes: Instruction-Aware Text Embeddings via Attention Relay},
  author  = {Luo, Yiyuan and Chatziafratis, Vaggos},
  journal = {arXiv preprint},
  year    = {2026}
}