Robots.txt Disallow entries now must have leading slash
- Status
- Closed
- Subject
- Robots.txt Disallow entries now must have leading slash
- Version
- 1.8.x
1.9.x
2.x - Category
- Error
- Feature
- All / Undefined
Installer (profiles, upgrades and server-related issues)
Search engine optimization (SEO) - Resolution status
- Fixed or Solved
- Submitted by
- John Hadjisky
- Lastmod by
- Marc Laporte
- Rating
- Description
Although the RFC (for example, http://www.robotstxt.org/wc/norobots-rfc.html) doesn't explicitly require a leading slash (/) before the page name, I have found that, as of late Oct, 2005, many 'bots, including Googlebots, have started requiring them.
For example, before the change,
Disallow: tiki-pagehistory.php
would prevent well-behaved 'bots from trying to index tiki-pagehistory.php. However, after the change, I had to have:
Disallow: /tiki-pagehistory.php
in robots.txt, or else all my page history would be indexed! I verified this using my server log, and also by doing google searches against my site for phrases that only appeared in page history. I have every reason to believe this is a problem for all other TikiWiki-based sites.
Others have noticed this. There is discussion in the forums at:
- Googlebot ignoring robots.txt — includes a graph of my bandwidth usage before and after.
- Adding <meta name="robots" content="noindex,nofollow"> to History pages
- Yahoo search indexes Print pages instead of Read pages
- Solution
Putting a leading slash before all page references in robots.txt solved the problem. See http://ihuck.com/robots.txt (a TikiWiki site), compare to e.g. http://dupli.tikiwiki.org/robots.txt
- Change the robots.txt in the CVS so that there is a leading slash before all page references (there is already a leading slash before all vdir references)
- Post an article on tw.o urging existing users to change their robots.txt
- Change the robots.txt on all sites *.tikiwiki.org that use the standard robots.txt, for example, dupli.tw.o, doc.tw.o, probably others.
I have many years of web dev experience, and two plus years experience with PHP and TikiWiki, but almost no CVS experience. I'm happy to learn CVS and implement this solution, but I am hoping first for some feedback from the community re have I overlooked any reason not to make these changes. Thanks.Assign this back to me and I'll start working on the changes (except for the 3rd change which a *.tw.o admin will need to do).
- Importance
- 5
- Priority
- 25
- Demonstrate Bug (Tiki 19+)
-
This bug has been demonstrated on show2.tiki.org
Please demonstrate your bug on show2.tiki.org
Version: Create show2.tiki.org instanceAbout show2.tiki.orgTo help developers solve the bug, we kindly request that you demonstrate your bug on a show2.tiki.org instance. To start, simply select a version and click on "Create show2.tiki.org instance". Once the instance is ready (in a minute or two), as indicated in the status window below, you can then access that instance, login (the initial admin username/password is "admin") and configure the Tiki to demonstrate your bug. Priority will be given to bugs that have been demonstrated on show2.tiki.org.
- Ticket ID
- 442
- Created
- Wednesday 21 December, 2005 00:51:59 GMT-0000
by Unknown - LastModif
- Tuesday 14 April, 2009 22:56:22 GMT-0000
;){kind=link}
;){kind=link}
- Wikiplugin img, the plugin is not able to display an image from anywhere on the internet
- Server check is raising false alarms about Tiki requirements
- File Galleries, Upload; have to try several times before successfully upload an image on dev.t.o
- Structures, Wiki pages; It should not be possible to create a page through structure with an "illegal" name
- Daily build zip file for Tiki26 is not available (404)
;){kind=link}
;){kind=link}