Over the last seven days, GrabPERF has been running in a very unstable state. This appears to be directly related to the use of InnoDB as the DB engine on a few of the larger tables in the database. [InnoDB changeover discussed here]

Today, the system was offline for several hours before I noticed that the DB had failed to restart properly. Apparently the system simply decided that the InnoDB engine had gone away.

All tables have now been switched back to the good old MyISAM and all the headaches that come with that.

On the weekend, I upgraded the database engine for GrabPERF to Mysql 5.1 and switched the main data table from MyISAM to InnoDB.

The switch to InnoDB was done because of the locking issues that were occurring during long queries, especially when doing ad-hoc analysis. The row-level (versus table-level) locking of InnoDB has removed most of these issues.

I have been seeing some strange behavior with the new engine. As a result of this, I will be re-starting the database engine twice a day. There should be no degradation, as this is simply a daemon re-start, not a machine re-start.