Although the RFC (for example, http://www.robotstxt.org/wc/norobots-rfc.html) doesn't explicitly require a leading slash (/) before the page name, I have found that, as of late Oct, 2005, many 'bots, including Googlebots, have started requiring them.
For example, before the change,
would prevent well-behaved 'bots from trying to index tiki-pagehistory.php. However, after the change, I had to have:
in robots.txt, or else all my page history would be indexed! I verified this using my server log, and also by doing google searches against my site for phrases that only appeared in page history. I have every reason to believe this is a problem for all other TikiWiki-based sites.
Others have noticed this. There is discussion in the forums at:
I have many years of web dev experience, and two plus years experience with PHP and TikiWiki, but almost no CVS experience. I'm happy to learn CVS and implement this solution, but I am hoping first for some feedback from the community re have I overlooked any reason not to make these changes. Thanks.
Assign this back to me and I'll start working on the changes (except for the 3rd change which a *.tw.o admin will need to do).
Show.tiki.org snapshot creation is in progress... Please monitor http:///snapshots/ for progress. Note that if you get a popup asking for a username/password, please just enter "show" and "show".
Password reset was successful
Password reset failed
Show.tiki.org instance destruction is in progress... Please wait...
The public/private keys configured to connect to show2.tikiwiki.org were not accepted. Please make sure you are using RSA keys. Thanks.
Unable to connect to show2.tikiwiki.org. Please let us know of the problem so that we can do something about it. Thanks.
Show.tiki.org is currently under maintenance. Sorry for the inconvenience.
Unable to get information from show2.tikiwiki.org. Please let us know of the problem so that we can do something about it. Thanks.
Show.tiki.org is in the progress of creating the new instance. Please continue waiting for a minute or two. If this continues on for more than 10 minutes, please let us know of the problem so that we can do something about it. Thanks.
To help developers solve the bug, we kindly request that you demonstrate your bug on a show2.tikiwiki.org instance. To start, simply select a version and click on "Create show2.tikiwiki.org instance". Once the instance is ready (in a minute or two), as indicated in the status window below, you can then access that instance, login (the initial admin username/password is "admin") and configure the Tiki to demonstrate your bug. Priority will be given to bugs that have been demonstrated on show2.tikiwiki.org.
The URL for the show2.tikiwiki.org instance that demonstrates this bug is at: http://. Note that if you get a popup asking for a username/password, please just enter "show" and "show". This is different from the initial login and password for a new Tiki which is "admin" and "admin".
For the install log, see http:///info.txt
Note that if you see PHP errors or a Tiki claiming to be missing third party software, the instance creation is probably not finished. Please wait a couple minutes and reload.
Snapshots are database dumps of the configuration that developers can download for debugging. Once you have reproduced your bug on the show2.tikiwiki.org instance, create a snapshot that can then be downloaded by developers for further investigation.
Snapshots can be accessed at: http:///snapshots/. Note that if you get a popup asking for a username/password, please just enter "show" and "show".Create new snapshot