I'm not an expert in network security, so I can only give you an idea. The correct way to do things depends on so many factors, but the TLDR n probably use multiple ways to identify malicious traffic, and then block it as early as possible. Dropping all traffic from abusive IP addresses with netfilter is where you start because it's easy/simple. Then you can move on to grouping by ASN, and dropping closer to a load balancer.
The problem (as far as I know) that cloudflare actually provides, is being able to identify abusive networks, and making them prove they're not bots before they reach your server. If you can't even identify the abusive traffic, you don't have many options other than cloudflare.
I'm not an expert in network security, so I can only give you an idea. The correct way to do things depends on so many factors, but the TLDR n probably use multiple ways to identify malicious traffic, and then block it as early as possible. Dropping all traffic from abusive IP addresses with netfilter is where you start because it's easy/simple. Then you can move on to grouping by ASN, and dropping closer to a load balancer.
The problem (as far as I know) that cloudflare actually provides, is being able to identify abusive networks, and making them prove they're not bots before they reach your server. If you can't even identify the abusive traffic, you don't have many options other than cloudflare.