wierd question about web design.<< lock it. throw it away.>>
Moderator: Moderators
-
shamanistcrage
- Data [Authenticated]
- Posts: 224
- Joined: Fri Jun 15, 2007 6:13 am
- Location: Loveland, Co GO C.S.U. RAMS
wierd question about web design.<< lock it. throw it away.>>
ok. so would a website have a /robots.txt page for any reason, like searching, or something. because societyoftheancients.com used to have one. i cant get to it now, but it shows up in web archive.
Last edited by shamanistcrage on Mon Jun 18, 2007 4:29 am, edited 1 time in total.
Re: wierd question about web design. (involving the arg, not OT)
robots.txt files are used to tell search engine spiders what pages they should store in their database, and which to ignore. It's very likely that until they wanted the site to go live they had a robots.txt file that told all the search engines to ignore all of the pages.
-
shamanistcrage
- Data [Authenticated]
- Posts: 224
- Joined: Fri Jun 15, 2007 6:13 am
- Location: Loveland, Co GO C.S.U. RAMS
Re: wierd question about web design. (involving the arg, not OT)
which is what i figured, and why i posted this question instead of "ZMOG I FOUND TEH ROBOTZ"