You might be interested in `git notes`. It does what it sounds like you are missing.