What's New   Popular   Top Rated   Search Search :   All CategoriesAjax ASP ASP DOT NET C and CPP CFML CGI and Perl Flash Java JavaScript PHP Python Remotely Hosted Ruby on Rails Tools and Utilities Web Hosting XML Home | How to Add Link |  My Account | Sponsor Listing |  Login | Register  | Getting Rated Sponsers Buy & Sell Text Link   Find Any Hosting   Offshore Software Development Categories ASP Scripts ASP.NET Scripts Ajax Scripts C And C++ Scripts CFML Scripts CGI & Perl Scripts Flash Scripts Java JavaScript PHP Scripts Python Scripts Remotely Hosted Ruby on Rails Tools and Utilities Web Hosting XML Scripts    Home  » PHP » Scripts and Programs » Searching   robots.txt class Hits: 595 Visit robots.txt class Homepage Download Now Review Reffer it to friends Report This Link Description: Give this class the contents of a robots.txt file or the URL to it and it will parse it for you. As an option you can give it the name or your robot/spider/crawler and it will return only the section that applies to your case. Works through proxies with the help of Snoopy (see script for more details). Rating Graph 0.0% 0.0% 0.0% 0.0% 0.0%  1   2   3   4   5  Resource Details: Version : 1 Price : GPL Date Added : Sun Nov 20, 2005 Author : Toon Goedhart Rate:   0.0(out of 5)  Vote:   0  Votes - Select -Excellent!Very GoodGoodFairPoor Bookmark Resource Other Listing by This User: Logging to text file   Web AdvanceScripts.com   Home | Feedback | Advertise with us (c) Copyright 2006 AdvanceScripts™. All rights reserved. No part of this website may be reproduced, copied and/or distributed in any medium without express written permission Trademarks are property of their respective owners Site Developed and Hosted by Ethic Solutions Hosting Resources  Free Tutorials    Flash Tutorials  Webmaster dicussion forum   Web Directory    phpLDHacks    Bollywood Discussion     96 users online