I need a function to plug into a bigger piece of software that I wrote, that scrapes Google.
It should be pretty simple for an experienced PHP programmer. I initially wrote a function that scrapes Scroogle, but they ban IPs after 100 queries, and I’d like to cut to the source since the function has to be rewritten anyway.
– Create a PHP function that receives $search_term, scrapes Google, and returns the resulting hostnames as an array. If there are zero results, the function should return FALSE.
– The function must be usable for at least 5,000 queries per day. Something more like 20,000 would be ideal, of course.
– This is not a requirement, but in order to be able to do 5,000 queries per day, it may need to support proxies. See the links below.
– This is the function that I wrote, that you will be replacing. It’s perfectly functional, but isn’t usable in the long-term due to Scroogle’s IP banning.
– This page outlines what you’re up against, and offers some solutions.
– Another example PHP snippet.
– Yes another example of working PHP code that scrapes Google.
– It has been suggested (but I have Not confirmed) that it’s easier to scrape Google while passing accessibility flags: http://labs.google.com/accessible/
Initial bounty offer: $250. Let me know if you want to bid higher.
Contact me if you’re interested. Good luck!