You have basically two options: treat HTML as a string or parse it then process it with higher level DOM features.
The problem with the second approach is that HTML may look like an XML dialect but it is actually immensely quirky and tolerant. Moreover the modern web page is crazy bloated, so mass processing pages might be surprisingly demanding. And in the end you still need to do custom code to grab the data you’re after.
On the other hand string searching is as lightweight as it gets and you typically don’t really need to care about document structure as a scraper anyways.
Hey, you guys got any cool tips for website scraping?
Consider free API first if possible.
what do you want to scrape.
Beautiful Soup (python library, bs4) is also fren
They’re gonna tell not to parse HTML with regular expressions. Heed this warning, and do it anyways.
Thanks for your reply. What are your arguments in favour of parsing HTML with regex instead of using another method?
You have basically two options: treat HTML as a string or parse it then process it with higher level DOM features.
The problem with the second approach is that HTML may look like an XML dialect but it is actually immensely quirky and tolerant. Moreover the modern web page is crazy bloated, so mass processing pages might be surprisingly demanding. And in the end you still need to do custom code to grab the data you’re after.
On the other hand string searching is as lightweight as it gets and you typically don’t really need to care about document structure as a scraper anyways.
it’s quick, it’s easy and it’s free
Are you a LLM?
Selenium is your fren