Choosing an embedding layer without labels

Which layer to use

Each case below is backed by the tests further down. The stability pick is the label-free rule we tested: the layer whose k-means groups come out most alike when the grouping is repeated.

• No labels, an ordinary model: the second-to-last layer. It costs nothing, and no label-free rule we tested beat it.
• A few labels, even one per class: choose the layer with them. In our first test that matched the stability pick.
• No labels, and a model whose layers change sharply partway through (in the main post, Gemma 4), or a model you have not checked: run the stability pick as a check on the default. It costs one pass over a sample of the collection and k-means at every layer. On Gemma 4 the default missed by several points and the pick did not; we noticed this after the test, on two models of one family.
• The task is not about topic (sentiment, ratings, length, other surface properties): use labels. The stability pick follows the strongest structure in the collection, usually its topics, and drifts when the task is something else.

The question

Every layer of a language model holds a vector for each token. To turn a whole text into one vector, a common recipe takes one layer and averages its token vectors. Which layer to take is a real choice: middle and late layers often give better embeddings than the last one (Skean et al. 2025). Finding the best layer normally needs labelled examples of the task. The question here is whether a signal computed from unlabelled texts alone can find a layer that is nearly as good.

The rule we tested

The idea comes from the main post: some layers sort texts into clear groups, others do not. For each layer we take the averaged vectors of a sample of the collection and split them into 512 groups with k-means, a standard method that groups points by distance. k-means starts from random points, so we run it twice and measure how much the two groupings agree. We call that agreement the layer's repeatability. The stability pick is the layer where it is highest, averaged over ten pairs of runs. Before grouping, each layer's vectors are reduced to their 64 main directions, each rescaled to the same spread; the few coordinates with huge values are left in. No labels are used.

To score a choice we need labels, but only for checking. Topic search asks, for each text, how many of its 10 nearest neighbours share its label. A choice's shortfall is how many points of topic search it loses against the best layer of that model on that collection. Lower is better, and 0 means it found the best layer.

The first test, and the gap it left

A first pre-registered test, on 11 combinations of 7 models and two topic collections (DBpedia article summaries and 20 Newsgroups posts), went well: the pick fell well under a point short on average and beat the last layer in 10 of 11. But one comparison was only made afterwards: simply taking the second-to-last layer did almost as well, and there was no interval on the difference. Two other label-free rules from recent work, based on how many dimensions a layer's data effectively uses (its intrinsic dimension), had not been compared at all (Wang et al. 2026; Tang & Yang 2026).

The second test pre-registered · fresh models

So we registered a second test before extracting anything: three models never used to develop the rule (Llama-3.2-1B, SmolLM2-360M, Qwen3.5-4B-Base) on the same two topic collections, plus nine models on Yelp reviews, where the label is the star rating, not a topic. We fixed five predictions in advance.

The figure shows one model, chosen in advance, layer by layer. Repeatability (top) rises with search quality (third panel) and peaks at the same layer, which here is also the second-to-last. The intrinsic-dimension rules land elsewhere: the first peak of intrinsic dimension falls in the middle, where search is worse.

Our registration said what a failed P2 would mean, and we hold to it: for most models, take the second-to-last layer. It is free, and the stability pick did not beat it.

Where the pick earns its cost noticed after the test

Looking pair by pair, the second-to-last layer fails badly in only a few places, and they share one family: Gemma 4. In the main post, Gemma 4 is the family whose geometry turns sharply partway through, so its best layer for search can sit far from the end. There, the stability pick still finds it.

Read the orange rows: on three of the four Gemma pairs the second-to-last layer loses several points, and the pick loses far less. On the other models the two often pick the same layer, and otherwise trade small differences. We found this after seeing the results, it rests on two models of one family, and it is not a tested claim.

How far to trust these results

• Is it just picking a late layer? Mostly it does pick late layers. But on the topic collections, repeatability still tracks search quality after layer depth is accounted for, in five of the six new pairs. On star ratings that link disappears. post hoc
• Is the pick stable? Only when averaged. A single pair of k-means runs agrees with the ten-pair pick in between a fifth and four fifths of its runs, depending on the pair, because the top few layers are nearly tied. The cost of that noise in points is small, but "the layer" is really "one of the last few". post hoc
• Is the best layer a fair reference? It is chosen on the same labels it is scored on, which flatters it slightly. This affects every rule the same way, so comparisons between rules are fair; the absolute shortfalls are a little generous.
• Did it beat the intrinsic-dimension rules on their terms? No. Those rules were built for other inputs (the last-token state of a generated answer; a layer for routing attention). We applied them to averaged text vectors, so "beats" means "beats our adaptation".
• How much independent evidence is there? Six topic pairs from three models. Two pairs from one model are not independent, so intervals that treat them as independent are optimistic.

Where it fails

Every test above uses topic collections, where texts fall into clear subject groups. We also registered tests where that is not the job, and the pick did worse in each. When nearby texts should share something other than a topic, such as being of similar length or having many digits or capital letters, it falls several points short of the best layer. On restaurant reviews scored by star rating it falls about two points short on average, as we predicted, though it still beats the last layer on most models. On single sentences, both short customer-service questions sorted by intent and sentence pairs rated for similar meaning, it beat the last layer in only about half the models.

It is also less reliable when the model differs from the ones it was developed on. On encoders, models that read the whole text at once rather than left to right, it met its goal in only one of four tests. On chat-tuned versions of the models, and on compressed versions of one chat model that store their weights in fewer bits, it missed its goal by modest margins. And it is sensitive to the few coordinates with huge values that most language models carry: removing them moves its result noticeably on about half the collections.

All of these fit one reading. Repeatability follows the strongest structure in the collection, which is usually its topics. When the task asks for something else, or the model's layers are shaped differently, that structure stops pointing at the best layer.

Open questions and possible improvements

Why repeatability tracks search quality at all is unknown. Three explanations we registered failed: it is not just how well separated a layer's groups are, not how many dimensions the layer uses, and not how settled the layers are. What remains is a description: across the layers of a trained model, repeatability and search quality rise and fall together. The link is learned. In Pythia models before training it runs the other way, and after training it is positive in all eight runs we checked.

None of the ideas below has been tested. Each follows from a failure above.

• Check the geometry first. The pick earned its cost only where the layers turn sharply, and the main post measures that turn directly. A combined rule would take the second-to-last layer unless a sharp turn shows up in the last part of the model, and only then run the pick. This needs a registered test on models outside Gemma 4 whose layers turn.
• Match the grouping to the task. The pick always splits texts into 512 groups, which suits fine topic structure. A task with a few coarse classes, such as star ratings, might need far fewer groups, or groups steered by a handful of labels.
• Pick a range, not a layer. The last few layers are often nearly tied, so averaging the vectors of the top few layers may be steadier than choosing one.
• Handle huge coordinates explicitly. Rescaling or removing them before grouping, decided in advance, could remove one source of drift.

Prior work

Taking the second-to-last layer is an old habit. bert-as-service pooled it by default, reasoning that the last layer is too close to the pre-training objective (Xiao 2018), and Stable Diffusion 2 conditions on the penultimate layer of its text encoder (Stability AI 2022). Intermediate layers often make better embeddings than the last (Skean et al. 2025). Label-free layer choice has been proposed with intrinsic dimension (Wang et al. 2026; Tang & Yang 2026; the TwoNN estimator is from Facco et al. 2017). Cluster stability is a classical tool for choosing a clustering (Ben-Hur et al. 2002; von Luxburg 2010). We did not find it used to choose a layer. So what is new here is not the default but the test of it: a registered comparison of a label-free rule against the second-to-last layer, and a map of where each one fails.

Pre-registrations: PREREG_layer_selection.md (first test) and PREREG_stability_v2.md (second test), in the repository's docs/preregistrations/. Every number behind this page is in the claims ledger with its source file.

References