Loading...
 
Skip to main content

Robots.txt Enhancements

  • Tracker item actions
Trackers
Items
Status
Closed
Subject
Robots.txt Enhancements
Version
12.x
14.x
15.x
Category
  • Feature request
  • Less than 30-minutes fix
  • Patch
Feature
Search engine optimization (SEO)
Resolution status
Fixed or Solved
Submitted by
drsassafras
Volunteered to solve
drsassafras
Lastmod by
drsassafras
Rating
(0)
Description

I have completed an enhanced robots.txt file.

Initial Website XML Maps I was making were returning Just under 500 results. The Sitemap now returns 45 results, one of which is an error, that I have filed a separate bug report for. No pages are duplicated, all pages with relevant content are crawled.

Of note is that I have re-enabled the file gallery being crawled. It was returning large number of files and was disabled to prevent pressure on the web server. Results can now be indexed without the massive duplication that existed before.

There is a section in the file that can be commented out to prevent duplicated pages when SEF URL's is enabled. Applied automatically, it will prevent the entire website from being crawled.

Of importance is also that I have only tested this for the feature set that I am using. It WILL create a better crawling and therefor better Search Engine ranking with other features, but may not take it as far as one could be achieved.

Using this robots.txt file not only decreases server load, but also leads to better search engine rankings.

Files
  1. Robots
Priority
25
Demonstrate Bug on Tiki 19+
Demonstrate Bug (older Tiki versions)
Ticket ID
5223
Created
Thursday 01 May, 2014 03:58:25 UTC
by drsassafras
LastModif
Saturday 06 July, 2024 10:21:44 UTC


Show PHP error messages