I took my 6 and 8 year old to SFMOMA and they loved it, to the point that they’ve asked to go (and have gone) to several more “boring” art museums since. We had a talk about ground rules (quiet voices, hands to self, no running, no exceptions) beforehand, and the mood of the place helped enforce those rules. A big, crowded space can be powerful in its quietness.
A lot of the weird, experimental, and experiential pieces seemed to scratch the novelty itch that they might otherwise get by running around or touching stuff. We were all ready to leave at the same time … or actually, I wanted to leave before they were ready, so it wasn’t like they got bored quickly. They are not uniquely quiet or well behaved kids, either—quite chaotic a lot of the time, really. I think a lot of people don’t give kids a chance to experience these kinds of places because they assume the kids won’t do well, which is too bad.
I took my 6 and 9 year olds to SFMOMA and they played along for about 20 minutes and then started rolling all over the furniture and complaining about being bored, despite my best efforts to engage them in discussions about the art pieces. I got them to settle for a while by playing pencil-and-paper games with them, but then I couldn't look at the art either.
A more extensive talk about ground rules wouldn't have helped. Kids aren't all the same, and most art museums aren't really designed to meet their needs.
(By comparison, they would be happy to spend all day every day at the Exploratorium, and the hardest part there is occasionally pulling one away from some exhibit so that the next kid can get a turn.)