You would still get a notification inside Telegram that someone else logged in (plus it sends the login code to your session first before you can fallback to SMS) and you can log them out. The "attacker" can't delete your account or log out your sessions because he can't use certain features on a fresh login, there is a downtime. But yeah, he can read all your chats. Same should be true for any app that uses phone number login. That's why there is a password feature.
You can delete an account if you have the phone number but not the password, it's on a 7-day timer before the account gets deleted. It can be canceled by a logged-in instance.
That still means you have to check it every 7 days.
You still get notified of a login and multiple times for the deletion request. Opening a chat app once in a week is not something rare and by that time you already changed your phone number inside Telegram.