I wonder if there is a principle here we can apply to neural nets like dividing up the layers for different tasks (no not MoE but something more fundamental).
I wonder if there is a principle here we can apply to neural nets like dividing up the layers for different tasks (no not MoE but something more fundamental).
There is some research on layer duplication on existing pre-trained models. This paper also references layer skipping: https://aclanthology.org/2025.findings-emnlp.967.pdf
nice