Saturday, September 05, 2009

[code] User Agent / Browser Simulator in Ruby

Totally forgot that I wrote this early last year -- Test / simulate header response, redirect, and other behavior with user agent for various browsers, crawlers, and robots.

require 'httpclient'
require 'uri'
require 'net/smtp'

# Pick your user agent
targetUserAgent = 'Mozilla/4.0 (compatible; MSIE 6.0; Windows+NT 5.1)'
#targetUserAgent = 'Googlebot/2.1+(+'
#targetUserAgent = 'Mozilla/5.0+(compatible;+Yahoo!+Slurp;+'
#targetUserAgent = 'msnbot/1.0+(+'

# Pick your URL
targetURL = ''

  client =, targetUserAgent, "")
  response = client.head(URI.parse(targetURL))
  # Print the entire header message
  print "\n===== HEADER =====\n"
  print response.dump 
  # Print some summary
  print "===== SUMMARY =====\n"
  print "URL = #{targetURL}\n"
  print "UserAgent = #{targetUserAgent}\n"
  print "Response Code = #{response.header.response_status_code} \n"

  print "Failed to connect to the page\n"