That's not how MoE works, you need to train the FFN directly or else the FFN gate would have no clue how to activate the expert.