You could try to work around that by first grading all anonymized proposals, then grading all potential speakers without knowing their proposal. In the third round you deanonymize and look at the weighted average of the two grades. You probably still need some judgment calls because the combination of speaker and topic can be important. But the score would give you a good base to work of.

Maybe you could make it even more impartial by allowing conditional scores in the first two rounds. Like "Jim is a 6, but a 8 if his talk is about molecular biology" or "this Lessons Learnt talk is a 5, but if it's by X, Y or Z it's a 9"