Do LLMs really care?