HTTP Programming Recipes for C# Bots

by Jeff Heaton

The Hypertext Transfer Protocol (HTTP) allows information to be exchanged between a web server and a web browser. C# allows you to program HTTP directly. HTTP programming allows you to create programs that access the web much like a human user would. These programs, which are called bots, can collect information or automate common web programming tasks. This book presents a collection of very reusable recipes for C# bot programming. This book covers many topics related to C# HTTP programming. Both secure and insecure HTTP communications are covered, as well as HTTP authentication. Learn to interact with HTTP forms and support both HTTP POST and HTTP GET requests. Collect data from a wide array of HTML constructs, such as tables, and lists. Learn about advanced topics that complicate the life of a bot, such as AJAX and Javascript. Also learn about the ethical use of bots, and when bots should not be used. This book also introduces the Heaton Research Spider. The Heaton Research Spider is an open source spider framework. Using the Heaton Research Spider you can create spiders that will crawl a web site, much like a real spider crawls the web. The Heaton Research Spider is available in both Java and Microsoft Dot Net form.

618 pages, Paperback

First published April 3, 2007

Reviews

[Reza Taheri · Rating 5 out of 5]

This is the most instructive resource on writing http bots I've seen, reading the first two chapters is enough to give you a complete knowledge of how your browser communicates with the web, continue reading, and you'll learn how to program bots to do the same in a way you want.
i started this book from zero background on cookies, get and post forms, bots, c#, etc... and after a month or so i could automate my university enrolling so i dont even have to bother getting up early in the morning to do that, highly recommended.