From what I can gather it depends, but could be a simple Softmax-based layer[1] or just argmax[2].

There was also a recent post[3] about a model where they used a cross-attention layer to let the expert selection be more context aware.

[1]: https://arxiv.org/abs/1701.06538

[2]: https://arxiv.org/abs/2208.02813

[3]: https://news.ycombinator.com/item?id=40675577

Usually there’s a small neural network that makes the choice for each token in an LLM.

I don't know the details, but there are a variety of routing mechanisms that have been tried. One goal is to load balance tokens among the experts so that each expert's parameters are equally utilized, which it seems must sometimes conflict with wanting to route to an expert based on the token itself.