In 2004-2005, as a lark, I created my own Web performance measurement system, using PERL, PHP and MySQL. In August 2005, I managed to figure out how to include remote agents.

I dubbed it…GrabPERF. An odd name, but an amalgamation of “Grab” and “Performance” that made sense to my mind at the time. I also never though that it would go beyond my house, a couple of basement servers, and a cable modem.

In the intervening three years, I have managed to:

• scale the system to handle over 250 individual measurements
• involve nine remote measurement locations
• move the system to the Technorati datacenter
• provide key operational measurement data to system visitors

Although the system lives in the Technorati datacenter and is owned by them, I provide the majority of the day-to-day maintenance on a volunteer basis, if only to try and keep my limited coding skills up.

But this post is not about me. It’s about GrabPERF.

Thanks to the help of a number of volunteers, I have measurement locations in the San Francisco Bay Area, Washington DC, Boston, Portugal, Germany and Argentina.

While this is a good spread, I am still looking to gather volunteers who can host a GrabPERF measurement location. The areas where GrabPERF has the most need are:

• Asia-Pacific
• South Asia (India, Pakistan, Bangladesh)
• UK and Continental Europe
• Central Europe, including the ancestral homeland of Polska

It would also be great to get a funky logo for the system, so if you are a graphic designer and want to create a cool GrabPERF logo, let me know.

The current measurement system requires Linux, cURL and a few add-on Perl modules. I am sure that I could work on other operating systems, I just haven’t had the opportunity to experiment.

If you or your organization can help, please contact me using the GrabPERF contact form.

Tags: Argentina, Asia-Pacific, Bangladesh, Boston, cable modem, Central Europe, Germany, GrabPERF, GrabPERF.org, India, internet performance measurement data, Linux, mysql, operating systems, Pakistan, performance measurement, Perl, PHP, Polska, Portugal, San Francisco, United Kingdom, Washington DC, web, Web Performance, Web performance measurement data, Web performance measurement system

One of the performance hits that the GrabPERF system has is the dynamic generation of the main page. The nature of the SQL calls and the underlying PHP makes it scale exponentially past a certain number of measurements.

Last night, Kevin Burton made a grand suggestion: generate a static page on a regular schedule.

Duh!

Today, I wrote the script that does this. The performance of the main page has adjusted accordingly.

Yikes!

UPDATE: Ian Holsman reminded that if I use cURL, I can use the exiting PHP to build the pages without a PERL script.

I. AM. AN. IDIOT.

Now, bedtime.

Technorati Tags: GrabPERF, Web+performance, PHP, MySQL, Dynamic+pages, Static+pages, HTML

Tags: ARIN, bbc, blog, GrabPERF, hits, HTML, HTTP, IE, IM, improvement, IP, it, LAN, measurement, mysql, Om, One, Pages, performance, performance improvement, Perl, PHP, pr, Technorati, update, web, Web+performance

I was trying yesterday to debug an issue that was appeared to be affecting the PubSub Agent — yes, I re-started it at the request of their sysadmin.

The issue was that it was showing data that appeared to have no relationship with the data appearing from all of the other measurement locations. I tried blocking it off using IPTables, MySQL restrictions, etc.

This afternoon, I figured out the problem.

Yesterday, I had been using this query to diagnose the problem:

select
       *
from
    data
where
    agent_id = 11
    and date between subdate(now(),interval 30 minute) and now()
order by
    date DESC

And everytime I looked, there was new data. I couldn’t seem to stop the agent from delivering data. Then I had a flash.

select
       count(*)
from
    data
where
    agent_id = 11
    and date > now()

157,000 rows of data. From the future.

Thankfully I know the guy who is responsible for keeping the PubSub servers running, and he is going to adjust the time, etc. But he makes no promises about how long the agent machine will stay running.

I apologize for the bad data.

Technorati Tags: GrabPERF, PubSub, Web Performance, The Future

Tags: app, ARIN, data, GrabPERF, HTTP, IE, IM, IP, it, live, Mac, measurement, mysql, Om, performance, pr, run, server, Technorati, the other, web, Web Performance

After a few month hiatus, I am starting to code for GrabPERF again. I need to exercise my brain; as I am a hobbyist code mangler, I have to take on a project every now and then to keep my not-so-l33t skillz honed.

The change to the agent is one of efficiency. The current production agent opens two database connections to run tests: one to retrieve the test configuration data; the other to insert the results of the tests. This means I loop through one set of database query results while doing inserts inside the loop on a second database connection.

This is stupid.

The new code opens a single connection to the database, retrieves the test configuration, dumps the results to an array of arrays, then inserts the data on the same connection. This is more efficient, as I use persistent connections and compression to MySQL to improve performance.

I have this running as TEST AGENT 1 from the Technorati #2 site.

Let me know if you see any madness…outside of Washington DC, and specifically with GrabPERF.

Technorati Tags: GrabPERF, PERL, MySQL

Tags: compress, compression, current, data, database, GrabPERF, GrabPERF.org, HTTP, IE, IM, it, mysql, Om, One, performance, Perl, persistent, persistent connections, pr, run, Technorati, testing, the other, Washington DC

I updated the GrabPERF database to the latest version of MySQL. Please let me know it you notice any unusual behaviour…

…from GrabPERF.

Technorati Tags: GrabPERF, MySQL, upgrade

Tags: data, database, GrabPERF, HTTP, it, mysql, Om, Technorati, update

MySQL 5.0 has a Blackhole Storage Engine. [here]

Cool. And if you make me grumpy, all of your performance data will be stored there!
Via: Jeremy Zawodny

Tags: blog, cool, data, HTML, HTTP, mysql, Om, performance

For reasons that I have yet to discover, my outbound connection to the Internet appears to have been down between the times listed in the title.

As I have said before, my connection is less than “professional grade”, i.e. “don’t do this at home”.

If I find a cause, I will update.

I apologize for the incovenience.

UPDATE: Ouch.

Found a MONDO huge problem with mysqldump in MySQL 5.0.15 — it locked the tables. Totally.

I backup the database by taking a dump 4 times a day, 12 hours apart on two machines.

The Web server machine, where I was testing MySQL 5.0.15 prior to the production upgrade, is using the version mysqldump that shipped with that.

I need to read the docs and disable the table locks.

Tags: app, data, database, EAD, GrabPERF, IE, IM, Internet, IP, it, Mac, mysql, Om, outage, pr, server, testing, update, web, Web server

Wait! Didn’t he say he had done this already?

Yup. But forgot one stupid little step. You have to re-compile PHP to work with the new MySQL 5.0 libraries. I tried to restart Apache and it barfed all over the PHP module with MySQL library module mismatch errors…or something like that.

Done and the system is now using the right MySQL libraries.

Usual caveats about system weirdness apply. Operator weirdness is your own issue.

Tags: apache, app, EAD, IE, it, mysql, Om, One, PHP

Ok, so I lied; sue me.

The main database is now running MySQL 5.0.15.

Please report any weirdness you see with any of my applications.

Tags: app, application, data, database, IE, it, mysql, run

Ok Everyone, if you start seeing some odd behaviour from my sites (GrabPERF, Newest Industry, Pierzchala.com), it’s because I am doing some testing to see if the scripts, processes, and applications I have built on top of MySQL 4.1 can stand the upgrade.

I will not be upgrading the primary database system until all of this testing is complete.

Tags: app, application, data, database, GrabPERF, IE, IM, IP, it, mysql, Newest Industry, Om, One, pr, testing