require 'httpclient' require 'uri' require 'net/smtp' # Pick your user agent targetUserAgent = 'Mozilla/4.0 (compatible; MSIE 6.0; Windows+NT 5.1)' #targetUserAgent = 'Googlebot/2.1+(+http://www.googlebot.com/bot.html)' #targetUserAgent = 'Mozilla/5.0+(compatible;+Yahoo!+Slurp;+http://help.yahoo.com/help/us/ysearch/slurp)' #targetUserAgent = 'msnbot/1.0+(+http://search.msn.com/msnbot.htm)' # Pick your URL targetURL = 'http://sevenwires.blogspot.com/' begin client = HTTPClient.new(nil, targetUserAgent, "") response = client.head(URI.parse(targetURL)) # Print the entire header message print "\n===== HEADER =====\n" print response.dump # Print some summary print "===== SUMMARY =====\n" print "URL = #{targetURL}\n" print "UserAgent = #{targetUserAgent}\n" print "Response Code = #{response.header.response_status_code} \n" rescue print "Failed to connect to the page\n" end
Saturday, September 05, 2009
[code] User Agent / Browser Simulator in Ruby
Totally forgot that I wrote this early last year -- Test / simulate header response, redirect, and other behavior with user agent for various browsers, crawlers, and robots.
Subscribe to:
Posts (Atom)