At least in arts, there is a true "creative" element complicated to replicate at least in the current LLMs. True artistic invention involves a "spirit of rule-breaking" that pushes culture forward. If AI is only being trained on past data, it then struggles to make the non-linear, revolutionary creative jumps we've seen throughout history.

For instance, the transition of music from the 1950s to the Beatles, then to Nirvana, hip-hop, and beyond. Before Beatles, there wasn't such thing as Beatles. Before Nirvana, there wasn't anything that sounded like that. Of course you can point out underground bands doing something like that around that era, but stray way back in time to like 1940s and chances of hearing something like Nirvana are near zero.

True art is not easy to make. It is, rather, difficult. If you lower the difficulty to make art, then it is not already a piece of art. It is just another product down the line of plenty of copy cats. Art involves risk taking that AIs I don't think are capable of producing. Art is a medium to express human emotions, and the inherent concept here is that it will end up being consumed by another human being, and it has value because you know that another human being was on the other side of the table making it. Even if you do something you consider art for yourself with an AI, the human being on the other side was, well... you. You prompted it and you guided it. I don't believe people in general would be contempt with an endless stream of autonomous AI generated stuff (if they invent such loop) because it would not only be generic, repetitive and predictable at some point, but also the fact that you know its AI it destroys the whole vibe. Same as if you get an email and you don't know if it was an AI or human behind it. If it was a human being, great, you feel a connection there. If it was AI, well... that sucks to no end, and I guess that is for a very good reason.

Although I'm using art here as the argument for creativity, I believe there may be something analogous to scientific leaps and discoveries. To my understanding, people who seem to be fairly successful in uncover the truths of the universe via science also happen to be pretty good when it comes to some artistic endeavors, even if its only about consuming and appreciating art and not necessarily creating it. So, there might be some link in creativity there. I'm open to be wrong, of course. Creativity is about risk taking and pushing the edges, not re-iterating upon what's been already there.