If you treat it as vectors instead of words, it might be differentiable.