Webbots, Spiders, and Screen Scrapers: A Guide to Developing Internet Agents with PHP/CURL

by Michael Schrenk

There's a wealth of data online, but sorting and gathering it by hand can be tedious and time consuming. Rather than click through page after endless page, why not let bots do the work for you?

Webbots, Spiders, and Screen Scrapers will show you how to create simple programs with PHP/CURL to mine, parse, and archive online data to help you make informed decisions. Michael Schrenk, a highly regarded webbot developer, teaches you how to develop fault-tolerant designs, how best to launch and schedule the work of your bots, and how to create Internet agents that:


Send email or SMS notifications to alert you to new information quickly Search different data sources and combine the results on one page, making the data easier to interpret and analyze Automate purchases, auction bids, and other online activities to save time Sample projects for automating tasks like price monitoring and news aggregation will show you how to put the concepts you learn into practice.

This second edition of Webbots, Spiders, and Screen Scrapers includes tricks for dealing with sites that are resistant to crawling and scraping, writing stealthy webbots that mimic human search behavior, and using regular expressions to harvest specific data. As you discover the possibilities of web scraping, you'll see how webbots can save you precious time and give you much greater control over the data available on the Web.

Genres: Programming, Technical, Computers, Technology, Computer Science, Software, Coding

392 pages, Paperback

First published March 30, 2007

Reviews

[Oskar Kosciolek · Rating 2 out of 5]

When I approached this book I had quite high expectations, because I had read books from this publishing house before. And... I am a bit disappointed. On the one hand, a book gives an inexperienced reader a lot of ideas, but it presents and implements them not through "teaching" and "explaining" things, but like it was an advertisement for the author's written libraries. When started reading, I was hoping to learn something about CURL and here I was very disappointed because there was not much about it throughout the book - actually I could read Appendix A and finish reading it there. Nevertheless, I will write again: the book gives many examples, ideas and advices for beginners. Finally, I know that I shouldn't expect much from a book from a few years ago (I read the 2012 version), but a lot of things presented in the book are no longer up to date, so if you want to learn something, it's worth looking for newer books.

[Trekka12 · Rating 5 out of 5]

A Great Resource for Web Developers to learn how to build PHP-based webbots and Web scrapers. Amazing reference litterature for this. A definite Must-read and Bible to any good web developers library and arsenal. Great segway for getting one step closer to creating more automation to save time and money for businesses as well.

[Greg · Rating 5 out of 5]

Great book!