There's a recent article in Computerworld by Robert Wenig entitled How to avoid Web application pitfalls. Robert takes on the issue of customer experience in Web site performance. He says:

Questions such as "Is the network up?" and "Are the pages loading quickly?" provide only limited visibility into the success or failure of an application. No one assumes the customer perspective. Does the application deliver the right information? Which users are affected by application failures, who are they, and how much is it costing the business? With so much invested in the success of mission-critical Web applications, why are we still relying on outdated success metrics such as page download speed and system uptime? Are these measurements really telling us how technology is enabling business?

From How to avoid Web application pitfalls - Computerworld
Referenced Thu Mar 04 2004 13:54:29 GMT-0700

Too often the NOC will say "well, I can ping the machine and the database is up...I don't understand what the problem is." That's OK, that's what they're supposed to do and that's what they're tools tell them. They don't have the perspective to look at the product. When I was a CTO, we struggled to get a handle on this. I think in the end we figured it out and the answer was a position we called the Product Operations Engineer. The job of the product operations engineer is to work in the operations side of the house and look at the performance and operation of products and services from the customer's standpoint.

The product operations engineer is usually a senior system administrator who's responsible for making sure the product works. As such, they're caught in the middle. They must work with Engineering and are the chief interface between Engineering and the operations group. In addition, they also serve as a primary point of reference for the NOC and Technical support. They must also work closely with product managers. Until they get the product working right and the technical support and NOC team trained, they lose a lot of sleep.

I took some time a while back and wrote all this up in a paper on Tiered Support. The section on performance metrics is still weak and needs to be fleshed out.