Tag Archive: search engines

Jan 22 2009

whitehouse.gov/robots.txt

Jason Kottke pointed out that the whitehouse.gov robots.txt file was changed almost immediately after the inauguration … It went from this … User-agent: * Disallow: /cgi-bin Disallow: /search Disallow: /query.html Disallow: /omb/search Disallow: /omb/query.html Disallow: /expectmore/search Disallow: /expectmore/query.html Disallow: /results/search Disallow: /results/query.html Disallow: /earmarks/search Disallow: /earmarks/query.html Disallow: /help Disallow: /360pics/text Disallow: /911/911day/text Disallow: /911/heroes/text … …

Continue reading »

css.php