I can't speak for others, but I downvote to recommend other readers that the post is not worth reading. Not because I disagree with it, but because it's off topic in a way or another.

If TFA is about a tool, I tend to downvote comments that don't talk about the merits of tool but rather about the hosting website, the language it's written in, whether or not it "smells AI", English mistakes in the readme, and so on.

On the other side, if I reply to a comment I always upvote it, even if my reply is to refute it. In fact if I felt the need to add anything to it, it was by definition worth dealing with it!