When scraping headings, questions and sentences, especially > 100, the result contain unrelated topic + low quality and it is time consuming to clean/filtering.
Use AI to show only high quality and highly related headings, questions and sentences using scoring system, below certain number or low, medium and high parameters, it should hide low quality and unrelated results.
We talking ai writer and article creator?
Then yes, there is already a content filter being applied automatically to remove spam.
However, the more results you look back eg 100+ the lower the quality of said content actually becomes. Its a problem with search results.
As for filtering 300 results, I think maybe forcing the content to have the ‘keyword’ appear in it works the best.
If you can attach example of some non relevant content it can help me figure out some solutions.
–
Dynamic scraper, that has no filtering outside regex.