Try to thing about this assignment in terms of the software engineering involved. IE. an interface between components that can change where you would like to automatically detect this and perhaps loads up an alternate scrapper component as a result, perhaps a simpler one. IE. you have a tailored scrapper but you never know when or where the content publishers will mess with the interface so you ripple back to a more basic scrapper at each stage being able to test that what you scrap is content and not noise…

Aaron.