Saturday, September 05, 2009

[code] User Agent / Browser Simulator in Ruby

Totally forgot that I wrote this early last year -- Test / simulate header response, redirect, and other behavior with user agent for various browsers, crawlers, and robots.

require 'httpclient'
require 'uri'
require 'net/smtp'

# Pick your user agent
targetUserAgent = 'Mozilla/4.0 (compatible; MSIE 6.0; Windows+NT 5.1)'
#targetUserAgent = 'Googlebot/2.1+(+http://www.googlebot.com/bot.html)'
#targetUserAgent = 'Mozilla/5.0+(compatible;+Yahoo!+Slurp;+http://help.yahoo.com/help/us/ysearch/slurp)'
#targetUserAgent = 'msnbot/1.0+(+http://search.msn.com/msnbot.htm)'

# Pick your URL
targetURL = 'http://sevenwires.blogspot.com/'

begin
  client = HTTPClient.new(nil, targetUserAgent, "")
  response = client.head(URI.parse(targetURL))
  
  # Print the entire header message
  print "\n===== HEADER =====\n"
  print response.dump 
  
  # Print some summary
  print "===== SUMMARY =====\n"
  print "URL = #{targetURL}\n"
  print "UserAgent = #{targetUserAgent}\n"
  print "Response Code = #{response.header.response_status_code} \n"

rescue 
  print "Failed to connect to the page\n"

end