There are two problems with this:
(1) It doesn't get any more difficult to fix your accent. But most people won't, because there's virtually no benefit to doing it.
This is related to
(2) Once you learn to speak a language, you're not at any risk of people thinking that you can't speak it, even if you speak with a strong accent.
It does actually get much more difficult to fix your accent as you improve in a language. You have to significantly regress, slowing down your speech and taking pains to say everything correctly. You can lock in a good accent early on with much less effort.
There's really no risk in having too good of an accent early on. People will assume you're more advanced than you are, but once you tell them you're learning, they'll simply be impressed by your lack of an accent. There are worse things that could happen.