The main difficulty is that most people do not like running a business, especially a small business. For most of these businesses, you are buying yourself a job. Let's say you take out $1.5MM loan to open a shop, and you net $70,000 per year on a good year after expenses and debt service. After a few years, you have $1MM left to pay and there's no way you could sell the business for that. You have a personal guarantee on the loan.
Your vendors are always coming up with new ways to tack on extra charges. You have to deal with training, HR, bookkeeping, payroll, handyman tasks, cleaning, working shifts when your employees flake out, annoying customers, dangerous people, destructive customers, employee drama, the list goes on. If that is not what you enjoy, you will have a lot of your life doing things you do not enjoy. Sipping tasty coffee and chatting with your happy customers is a small part of the whole.