Hi all, I'm Nick, Product Manager for Kaggle Benchmarks and one of the co-organizers and judges for this AGI hackathon.
First off, I want to set some context on the AGI hackathon. This was co-organized by Kaggle and Google DeepMind, and we had ~20 judges from both organizations. The hackathon concluded on Apr 16 and we had initially anticipated a judging period of 1.5 months (till May 31). However, we ended up extending the judging period by another 1.5 months (to Jul 13) because we wanted to do right by participants.
Second, I want to emphasize and unequivocally clarify that every single winning submission went through at least 2 human judges, and in some cases, up to 3-4 human judges. These judges reviewed and scored the submissions independently based on the rubric we highlighted on the hackathon page.
Thirdly, I acknowledge that there is always an element of human subjectivity to reviewing qualitative submissions in hackathons. As best we can, we have put in place processes that ensure rigorous human review against objective standards and to reduce the possibility of bias by having multiple independent judges. We understand there may be valid disagreement over outcomes, but hopefully the above context clarifies this was not carelessly outsourced to LLM judges.
Thanks, Nick
Why not address the objective evidence the OP provided? To an impartial observer, it seemed quite overwhelming.
I don't want to comment on and single out any one participant, but please check out the rubric on how we did the grading: https://www.kaggle.com/competitions/kaggle-measuring-agi/ove...
Writeup quality is 20% of the total grade and there are other factors like dataset quality and results, which hold a much larger weight to the scores.
This has been made public to participants since day 1 of the launch and we've adhered very closely to this rubric:)
The data set quality and much of the methodology is just as abysmal as the writeup... as pointed out in my second post: https://www.kaggle.com/competitions/kaggle-measuring-agi/dis...
So it must have been the "results" that moved the needle :D
> Second, I want to emphasize and unequivocally clarify that every single winning submission went through at least 2 human judges, and in some cases, up to 3-4 human judges. These judges reviewed and scored the submissions independently based on the rubric we highlighted on the hackathon page.
How did you verify this? The results seem to indicate otherwise.