If the literal meaning doesn't make sense, derive the meaning from the way it is used.

"To be honest" typically means "Here is an opinion that I'm embarrassed to share, and would rather lie about"

They're not lying about everything else, they're lying about that one thing, every other time.

e.g. "I tell people my favorite movie is 'The Godfather', but, to be honest, it's actually Ratatouille"