TalkDev.com Forums - View Single Post

paul.e · 07-29-2008, 02:06 PM

You can put the directory/folder of the files you don't want to be crawled by google in robots.txt file. Here's how to use robots txt file - How do I use a robots.txt file to control access to my site?

07-29-2008, 02:06 PM	#3 (permalink)
paul.e Status: Junior Member Join Date: Jul 2008 Posts: 3 Trader Rating: (0) TD$: 30.00 Donate	Re: prevent spider grabbing URLs.... You can put the directory/folder of the files you don't want to be crawled by google in robots.txt file. Here's how to use robots txt file - How do I use a robots.txt file to control access to my site?