In reality it’s probably not a RELU modern LLMs use GeLU or something more advanced.