Something that seems obvious but not always implied by people's comments is that people are rarely trying to match an entire document with a regular expression so it doesn't really matter that "HTML is not a regular language".
If I am trying to e.g. count div tags with a regex like "<div" or whatever, then clearly this would work in 99.9% of cases and probably achieve what the poster is looking for.
As soon as you also add character classes to ignore various parts of the document that you are not interested in like "<div[^>]*>" or whatever it is, then it is eminently useful even if the bit we are ignoring is not fully regular.
One lovely thing about regex is how fast it is. I was asked to parse a massive CAN Bus log file for how many times some event had logged. This was the early 2000s and the file was 6GB, which was pretty big. I tried .Net's string.StartsWith or something and that took ages to run through the file. I did the same thing with a regex and it finished in like 5 seconds (HDD, not SSD!). I don't know how the magic works but it is very impressive.
RegExps is the way to tokenize, so it's not surpassing you can look for individual tokens using them.
It's parsing that's hard , for example when it needs to match up braces, or start and end tags, even if either is easily matched by a RegExp.
And you still need to be careful if the source you're looking in has any way to escape text or have different meanings for the same text. In source code, you should recognize comments and strings (and RegExp literals) so you don't match inside those. In HTML, you should recognize CDATA sections, including script elements. If they contain `<div`, it's not a tag.
That's is, your 99.9% is probably too damn high.
Are we also counting divs inside comments or literals within scripts?
There is a reason this advice is default. The chances an edge case exist are probably a lot higher than anyone is prepared to accept. Even in the "simple" cases.
Regex can also be horribly slow - it depends on the particular regex you are using.
> Regex can also be horribly slow - it depends on the particular regex you are using.
And the alternative approach we are comparing to.