TRIM

Current state

What soon/maybe

• For now, it requires SSH access but it is designed so it could eventually work for FTP/SFTP as well. (planned for June 2009) some functions are now available via FTP!

Who is behind this?

• Louis-Philippe Huberdeau
• Marc Laporte
• Louis-Martin Richard (tester)
• Philippe Back (plans to add a nicer front-end GUI)
• amette
• Rodrigo Primo (added Wordpress support)
• You? (We are looking for Beta-testers!!)

To do

• update code to use tiki.org instead of tikiwiki.org

Known Bugs

• TRIM: don't kill templates_c/index.php
• MacOS needs this path -> /opt/local/bin/php -d memory_limit=256M scripts/addinstance.php
• On first backup, I get this error message "rm: cannot remove `/home/user/trim/trim/backup/1/manifest.txt': No such file or directory" (note: this has no side effects, warning only.)
• Perform database setup... -> use stars for 2 password fields
• my fedora has no php5 command - only a php command make instance can not be executed
• On update, need space between instance name and URL

Which instances do you want to update?
[0] profiles.tikiwiki.orghttp://profiles.tikiwiki.org  lphuberdeau@tikiwiki.org

Nice to have

• If TRIM tries to make an instance where there is a directory "images", it will causes conflicts and create problems.
  • TRIM could check that no conflicting directories exist

deleted files, not useful to report

Roadmap

Phase 1

• Disable site during upgrade (via .htaccess) Done
• Update DB (1.x to 1.y sql script) Done
• Handle both 1.9 and 1.10
• Cron job for automatic check and report on file integrity (what about cache files?) Done
• Something to indicate TRIM is working (sometimes, it is busy for many minutes and we are just left to wait, not knowing if it's stalled) ... may not be possible
  • Ok. Then, please indicate (in general) what TRIM is doing. Something like "TRIM will now transfer all files from SourceForge to TRIM (master) and then upload them to your instance. As long as you are not getting an error message, TRIM is hard at work! This operation could take 10-20 minutes or even a few hours, depending on server performance and network speed." Done
• run fixperms.sh Done
• Be more robust to various paths for php5 (ex.: using "which php" to find it) Done (using variables in Makefile so modification only required once)

Phase 2

• For TRIM to handle backups Done
• Make instance should suggest some values (ask for SSH host name and SSH user first). If you just type "Enter", it picks this value. These defaults should be editable somewhere in a config file. It also serves as a good example for the proper format. The default format which works for Fantastico hosting:
  • Web root : /home/$sshuser/public_html
  • Web URL : http://$sshhost
  • Working directory : /home/$sshuser/trim_working (TRIM should create with appropriate rights) Done
• Admin interface to edit and delete instances (maybe a ssl protected web interface?) or here
• To create database, database user and permissions from this user when installing Tiki. Could be in tiki-install.php as well
  • TRIM will need to be indicated the database login with sufficient rights to do this
  • Use this new info to populate db/local.php
  • Maybe pick profile from TRIM? (equivalent to tiki-install.php?) Done
• Nicer error message (from .htaccess when site is being updated) Done
• Backup
  • Cron job
  • Option to backup things outside of Tiki (ex.: mail directory, access logs, etc)
  • To be able to handle symlinks
    • /home/user/www is a symlink of /home/user/public_html and if I indicate /home/user/www as the web root of the instance, TRIM backup will not backup files) bug or feature enhancement request?
• bundled _htaccess
  • Rename the bundled _htaccess to .htaccess for instances running on Apache
• major upgrades
  • Handle transition from CVS to SVN (Sylvie has an unreleased script)
  • 1.9.x to 1.10.x, which requires 2 .sql operations
• Easier to manage in shell interface
  • document so people know to type things like C 0-52
  • "Next page" when too much info in one screen (sometimes hundreds of lines of scrollback), but maybe this is a bad idea because it'll prevent unattended operations. hmmm
  • Add a note before entering a passphrase in addinstance.php : "If you enter a passphrase, you will need to enter it every time you run TRIM, and thus, automatic, unattended operations (like backups, file integrity checks, etc.) will not be possible."
  • If mysqladmin fails (ex.: bad login, not mysql but postgres, etc), it should offer to re-enter value or to abort database creation. In this last case, the system should indicate something like : "TRIM was unable to create your database with the information you provided. You should create it via your admin panel (cPanel, etc) and provide the information when you point your browser to tiki-install.php"
• Indicate which warnings and error messages not to worry about (could be a link to a wiki page)
  • time stamp 2008-06-02 22:33:48 is 3938 s in the future
  • errors on setup.sh (when insufficient perms)
  • "warning: locate: warning: database /var/lib/slocate/slocate.db' is more than 8 days old" Nothing to do with TRIM
  • ssh: profiles.tikiwiki.org: Name or service not known Seems solved--
  • Make convert should handle 2.2 to 3.0 This is done by make upgrade
• Make it possible to have unattended operations. This is useful for server farms like http://tiki.cat.io/
• Plan for Tiki Suite.

Phase 3

• Should be a way to set a different folder/disk for backups (since this will become huge!)
• TRIM should add db/lock file
• TRIM should populate {$lastup}, which appears in templates/tiki-bot_bar.tpl. However, lastup should be (de)activatable and filterable by group.
  • Normal case is that I would activate and only let admins view it, but in some cases, Anonymous could see.
• Set password at install (or anytime!)should be in tiki-install.php as well
  . recreate admin use if deleted?
• Investigate DB creation with cPanel and ISPConfig
• Once everything is nice and stable with SSH, add SFTP/FTP support (done) so we could use on hosts like Mosso Cloud sites
• In addition to check and backup, having an automatic update script option (useful to always keep test sites like demo.tiki.org up to date. Unattended updates won't be able to deal with SVN conflicts. Maybe just send a warning email?
• Check used/available disk space. (% of used space) (Tiki misbehaves when disk is full and often we forget to check for that)
  • Report if less than x Megs left or x% (email warning)
  • Check on master and instances.
  • This seems unreliable on shared hosting
• Repair
  • Attempt to do all stuff which is can break sometimes
    • Repair table in MySQL
    • Clear Tiki cache

Phase 4

• Backup
  • Have backups generate a "live" version of Tiki.
    • So a web agency could have a locally running backup of all their Tikis with yesterday's data.
      • Insures an easy way to test the automated backup system
      • Staff can continue to work if server or Internet connection is down.
      • Data could be in a USB Tiki format so Tiki admins could copy/sync to a mobile drive to have a backup in their pocket at all times.
    • Could also be to setup read-only mirrors
      • Tiki would need a read-only mode. So when users log in, they are informed that any changes here will be overwritten at the next sync.
      • These mirrors could be used for DNS3, DNS4, DNS5, etc.
  • Move an instance from one server to another (backup & restore in one step?)
    • This could include (This is the manual procedure I do) closing the old site ($site_closed eq 'y'), setting $site_closed_msg="This site has moved to: http://..." or "This site has moved to a new server. If you are seeing this message, it's because the DNS has not yet propagated for you. Please try again in a few hours or use this temporary address: http://..." before the backup and opening the site on the new server ($site_closed eq 'n').
  • Save to Amazon S3 and similar online storage services
  • Backup of the database should handle multitiki like doc/devtools/backupdb.sh
  • Investigate if scripts like automysqlbackup, rsync-incr or rsnapshot have any concepts, features or code that would be useful for us.
• Profiles
  • Investigate if/how TRIM could help maintaining: OpenSourceCMS type demo to test/develop and show off profiles
  • Hourly build of each profile: http://profiles.tiki.org/profile_name
  • Investigate any synergy with the profile manager to apply database change to one, many or all instances.
    • Turn off unsafe, buggy or unused features
    • Add watch all wiki pages or watch all trackers to multiple Tikis
    • Use existing Tiki functions for mass operations. Ex.: add a Tiki user on dozens of Tikis, etc (In this case, it's important to use existing Tiki functions rather than alter the database directly)
• Management
  • Add branch & lastup when listing instance.
    • Sort by branches & date, so as to see who is most out of date, who needs a security release, etc.
• Install
  • At install, a maintenance.html (or install.html should be shown instead of files/folders)

Phase 5

• Add a check for domain name expiry
  • This is useful because the person hosting the Tiki may not be listed in the DNS registries and won't be informed of DNS changes (expiry, domain theft, etc)
  • Some Tikis can have several domain names (domain parking)
  • Will this be easy provided the anti-spammer protections?
  • Warn by email (both to TRIM manager and instance contact)
    • 30 days before expiry
    • Ideally, on any change to the entry (for domain theft)
  • See script by Jean-François Pelletier, started at Codefest
• Uptime/performance/content testing
  • Some of this, like spell checking and link testing, may be better done as a Tiki feature instead, so as to be able to access all content (web crawler will have problem with perms), and be usable without TRIM, but be "triggerable" via TRIM.
    • Check domain for presence of Google Analytics code
    • Check to see if Tiki is running properly (check presence of keyword like http://host-tracker.com/ and/or changes like http://www.changedetection.com/log/org/americana/index_log.html#changelist)
    • Check for the presence of a customized 404 apache error page (we often to forget to set this up)
    • Check response time
    • Check for broken links
    • Check spelling
    • Check missing images
    • Check for missing plugins (especially for non-bundled plugins which cause error when migrating text from site to another)
    • Check for standards compliance
• Server monitoring
  • From future server admin panel
  • Interact with other tool (Nagios, etc.)

Phase 6

• When suspicious files are detected, in addition to deleting them, it should be possible to send them to a non Web accessible "quarantine" for further analysis or even optionally send to the security squad.
• Could enhancements to _htaccess be merged to .htaccess? (ex.: rename .htaccess to _htaccess just before svn/cvs up) so if I have a modified it, I still benefit from community enhancements.
• Test/document on Cygwin so users could manage remote Tikis via their Windows PC or Mac. Or a VMware appliance
• Initially built for Linux-Apache-MySQL-PHP (LAMP). Later, we will adjust TRIM to work with different setups.
• Investigate a TRIM-like application that works directly on the remote instance. A small php script which fetches the latest version of Tiki and installs it. (Could be impossible because of limited file permissions that php scripts, with Apache rights, that have limited permissions). mods have this issue.
• import mysql data a bit like BigDump
• Possibly automated running of testing feature.
• Investigate if we should make additional use of .htaccess in various directories (only useful for Apache)

Related

• TikiTests
• https://svn.bryght.com/hostmaster and http://groups.drupal.org/hm2/project-goals
• http://mu.wordpress.org/
• XWiki Enterprise Manager is typically used to manage public or private wiki farms.

Wishlist

	Rating	Subject	Importance	Category	Created
	(1)	Prepare for SourceForge hosted Apps	9 high	Feature request Community projects	2008-11
	(1)	OpenSourceCMS type demo to test/develop and show off profiles	8	Feature request Community projects Dogfood on a *.tiki.org site	2008-01
	(1)	templates_c cleaning is too aggressive and deletes system files	8	Bug: Error	2008-05
	(1)	Round Robin / Redundancy / Disaster planning for all *.tikiwiki.org content	6	Feature request Community projects Dogfood on a *.tiki.org site	2007-07
	(0)	Setting admin password in the installer, with option to force change at first login	6	Feature request	2009-05
	(1)	TRIM should replace PECL's ssh2 extension with phpseclib	6	Feature request	2010-02
	(1)	Secdb automatic check with cron job	5	Feature request	2007-09
	(0)	tiki-install.php to create mysql user, mysql database and assign permissions	1 low	Feature request	2008-03

Alias

• Remote Management
• Backup
• Backup Script
• Restore
• Manage
• Upgrade Script
• Check Security
• Tiki Remote Instance Manager
• TikiWiki Remote Instance Manager