Loading...
 

Robots.txt Disallow entries now must have leading slash

Status
Closed
Subject
Robots.txt Disallow entries now must have leading slash
Version
1.8.x
1.9.x
2.x
Category
  • Error
Feature
All / Undefined
Installer (profiles, upgrades and server-related issues)
Search engine optimization (SEO)
Resolution status
Fixed
Submitted by
John Hadjisky
Lastmod by
Marc Laporte
Rating
(0)
Description

Although the RFC (for example, http://www.robotstxt.org/wc/norobots-rfc.html) doesn't explicitly require a leading slash (/) before the page name, I have found that, as of late Oct, 2005, many 'bots, including Googlebots, have started requiring them.

For example, before the change,

Disallow: tiki-pagehistory.php

would prevent well-behaved 'bots from trying to index tiki-pagehistory.php. However, after the change, I had to have:

Disallow: /tiki-pagehistory.php

in robots.txt, or else all my page history would be indexed! I verified this using my server log, and also by doing google searches against my site for phrases that only appeared in page history. I have every reason to believe this is a problem for all other TikiWiki-based sites.

Others have noticed this. There is discussion in the forums at:

Solution

Putting a leading slash before all page references in robots.txt solved the problem. See http://ihuck.com/robots.txt (a TikiWiki site), compare to e.g. http://dupli.tikiwiki.org/robots.txt

  1. Change the robots.txt in the CVS so that there is a leading slash before all page references (there is already a leading slash before all vdir references)
  2. Post an article on tw.o urging existing users to change their robots.txt
  3. Change the robots.txt on all sites *.tikiwiki.org that use the standard robots.txt, for example, dupli.tw.o, doc.tw.o, probably others.


I have many years of web dev experience, and two plus years experience with PHP and TikiWiki, but almost no CVS experience. I'm happy to learn CVS and implement this solution, but I am hoping first for some feedback from the community re have I overlooked any reason not to make these changes. Thanks.

Assign this back to me and I'll start working on the changes (except for the 3rd change which a *.tw.o admin will need to do).

Importance
5
Demonstrate Bug
Please demonstrate your bug on show2.tikiwiki.org

 About show2.tikiwiki.org

To help developers solve the bug, we kindly request that you demonstrate your bug on a show2.tikiwiki.org instance. To start, simply select a version and click on "Create show2.tikiwiki.org instance". Once the instance is ready (in a minute or two), as indicated in the status window below, you can then access that instance, login (the initial admin username/password is "admin") and configure the Tiki to demonstrate your bug. Priority will be given to bugs that have been demonstrated on show2.tikiwiki.org.

Version: Create show2.tikiwiki.org instance
Ticket ID
442
Created
Wednesday 21 December, 2005 00:51:59 GMT-0000
LastModif
Tuesday 14 April, 2009 22:56:22 GMT-0000

Attachments

 filenamecreatedhitscommentversionfiletype 
robotsneedroot.png 21 Dec 05 13:28 GMT-00004411


Keywords

The following is a list of keywords that should serve as hubs for navigation within the Tiki development and should correspond to documentation keywords.

Each feature in Tiki has a wiki page which regroups all the bugs, requests for enhancements, etc. It is somewhat a form of wiki-based project management. You can also express your interest in a feature by adding it to your profile. You can also try out the Dynamic filter.

Accessibility (WAI & 508)
Accounting
Administration
Ajax
Articles & Submissions
Backlinks
Banner
Batch
BigBlueButton audio/video/chat/screensharing
Blog
Bookmark
Browser Compatibility
Calendar
Category
Chat
Comment
Communication Center
Consistency
Contacts Address book
Contact us
Content template
Contribution
Cookie
Copyright
Credits
Custom Home (and Group Home Page)
Database MySQL - MyISAM
Database MySQL - InnoDB
Date and Time
Debugger Console
Directory (of hyperlinks)
Documentation link from Tiki to doc.tiki.org (Help System)
Docs
DogFood
Draw
Dynamic Content
Preferences
Dynamic Variable
External Authentication
FAQ
Featured links
Feeds (RSS)
File Gallery
Forum
Friendship Network (Community)
Gantt
Group
Groupmail
Help
History
Hotword
HTML Page
i18n (Multilingual, l10n, Babelfish)
Image Gallery
Import-Export
Install
Integrator
Interoperability
Inter-User Messages
InterTiki
jQuery
Kaltura video management
Karma
Live Support
Logs (system & action)
Lost edit protection
Mail-in
Map
Menu
Meta Tag
Missing features
Visual Mapping
Mobile
Mods
Modules
MultiTiki
MyTiki
Newsletter
Notepad
OS independence (Non-Linux, Windows/IIS, Mac, BSD)
Organic Groups (Self-managed Teams)
Packages
Payment
PDF
Performance Speed / Load / Compression / Cache
Permission
Poll
Profiles
Quiz
Rating
Realname
Report
Revision Approval
Scheduler
Score
Search engine optimization (SEO)
Search
Security
Semantic links
Share
Shopping Cart
Shoutbox
Site Identity
Slideshow
Smarty Template
Social Networking
Spam protection (Anti-bot CATPCHA)
Spellcheck
Spreadsheet
Staging and Approval
Stats
Survey
Syntax Highlighter (Codemirror)
Tablesorter
Tags
Task
Tell a Friend
Terms and Conditions
Theme
TikiTests
Timesheet
Token Access
Toolbar (Quicktags)
Tours
Trackers
TRIM
User Administration
User Files
User Menu
Watch
Webmail and Groupmail
WebServices
Wiki History, page rename, etc
Wiki plugins extends basic syntax
Wiki syntax text area, parser, etc
Wiki structure (book and table of content)
Workspace and perspectives
WYSIWTSN
WYSIWYCA
WYSIWYG
XMLRPC
XMPP




Useful Tools