HF says activation function is "SiTU-GLU", but I can't find any info on that?
Anyone know / is this a typo?
Even the python code for inference seems to use normal activations.
HF says activation function is "SiTU-GLU", but I can't find any info on that?
Anyone know / is this a typo?
Even the python code for inference seems to use normal activations.
It stands for Sigmoid Tanh Unit Gated Linear Unit. Check the tech report/paper (page 7): https://github.com/MoonshotAI/Kimi-K3/blob/main/k3_tech_repo...
Thanks! I had a clanker graph it with sliders, its a very interesting function to play with! seems like it has the negative 'hump' typical of a GELU coupled with the saturation of a sigmoid.
Its interesting that saturation is ideal at this scale, i thought we were all in on self-activating functions (i.e. swish) but in fairness all those papers lacked ablations at scale nor discussion of stability.