JP has been wond'ring aloud about sub-linear and super-linear complexity growth and how it might apply to systems. He's referencing a construct from this remarkable TED talk by Geoffrey West:

West is presenting the surprising math of cities and corporations. The talk is well worth 18 minutes of your life, but if you're impatient, the point that matters in the discussion to follow is that cities (empirically) get more of both the good and the bad things as they get larger. They are super-linear with a slope of about 1.15. Meaning that as a city gets bigger it will have more crime and more innovation than a strictly one-to-one linear slope would suggest. The bigger a city gets, the worse and better it gets.

Companies, on the other hand have economies of scale--and that can be a bad thing. They are sub-linear with a slope of 0.75. A company that gets twice as big doesn't have twice the costs--remember the slope of the line is 0.75, so it will have 1.5 times the cost. Unfortunately, it will also have 1.5 times the innovation, not twice. As a company gets bigger, you need more and more to achieve less and less. Ugh.

JP is wondering how this relates to IT systems.

As a result, soon after TED, I've been trying to formulate a similar argument about super-linearity and sub-linearity from a systems perspective: which costs scale up faster than the rate at which a system "grows", and which costs scale down faster? How do we measure system "growth"? Like West and his team found in the relationships between city populations and many other city measures, is there an equivalent for complex systems?

Most importantly, what is the role of collaboration in all this? How does all this tie up with the Jane Jacobs view of neighbourhoods and sidewalks and boulevards and parks? What does that mean for our understanding of stacks and ecosystems? Of course I have anchors and frames, I know what I want these answers to be, but I want to be able to prove them. Scientifically. Because the data in the research by people like Hagel, Seely Brown and West has some fascinating pointers.

From More Wond'ring Aloud -- confused of calcutta
Referenced Wed Sep 14 2011 16:46:18 GMT-0600 (MDT)

Meanwhile, Dion Hinchcliffe is responding to JP with some interesting ideas.

This fractal aspect of user systems, Web 2.0, and SOA is one that I deeply explored in the 2005-2007 timeframe, and my ideas on this even made the cover story of the SOA/Web Services Journal at one point. This is when we began to see successful, composite systems sprouting up "in the wild" that were eminently natural and highly successful, such as Amazon's, Flickr's, and many others. Innovative new open API-based businesses had definitively emerged and shown us --- in fact, virtually proved to anyone willing to observe --- that business/technology ecosystems could be routinely produced that were far more successful than the ones virtually any traditional business had managed to produce for itself internally with methods like SOA. It was clear something important and new was happening and we tried to learn from it.

...

The Web has also shown us that complexity, as important as it is to address and resolve the many inherent vagaries of dealing with the real world, is largely the enemy, whenever it exists needlessly. Our assumptions of where the complexity should be, in the transports for example, was wrong. It was in the ecosystem itself and we paid dearly for half a decade (and running) based on that misapprehension. But we've learned. SOAP usage has almost completely shifted to REST, complex WS-* stacks have largely moved on to simple standards and methods like Web-Oriented Architecture.

From Exceeding the Benefits of Complexity? A Fractal Model for the Social Business Era | On Web Strategy | Dion Hinchcliffe
Referenced Thu Sep 15 2011 15:02:08 GMT-0600 (MDT)

Dion is making the point that the Web 2.0 systems we see being built for consumer-side Internet services like Amazon and Flickr scale better and are less complex than systems built by IT shops behind the firewall. Dion goes on to reference John Hagel lamenting that "the cultural gap between the enterprise crowd and the Internet crowd is still much too large and is still hindering the move forward for many organizations." Dion, rightly, sees the difference being technology choice: Web services like SOAP and the WS-* specs proved to be too heavyweight to use. Here's a diagram he uses to illustrate this:

So, why am I telling you all this? Like JP and Dion, I think there's more than a mere metaphor between the city/corporation insights and the ways we build software today. I'd argue that social systems, including cities are models for the techniques and technologies we ought to be using. Networked systems scale better and are more flexible than hierarchical systems. They can absorb complexity better without suffering from the debilitating effects of tight coupling that hierarchical systems create. But I'm ready to go further than I think JP and Dion are.

While API-based systems are more flexible than the moribund systems that enterprise IT shops create, they don't go far enough. They are still request-response based. Request-response based systems are hierarchical and create unnecessary coupling. Cities and other social systems do not operate solely using a request-response model.

The dual of request-response is events. Event-driven systems exhibit more of the network architecture that make cities flexible and robust. Let me use the following scenario to illustrate.

Suppose you want a system set up in your living room so that when you get a phone call, your Tivo automatically pauses what you're watching.

In a request-response system the Tivo would have some kind of API and I'd configure my phone--probably by installing an app and pairing it with the Tivo--so that when a phone call came in, it would send the "pause" command to the Tivo. Not so bad. All we need is an API on the Tivo.

But suppose I want more. I want my stereo to mute the sound when I get a phone call. Another API and another app. But I have a DVR from Comcast in the bedroom and I want that to pause too. Another API and another app. I switch out receivers--better find the right app. Buy a new Android phone and you've got to do it all over again. What about the house phone? It doesn't have apps. Sheesh!

Do you see what's going on? I'm creating a hierarchy with my phone at the center. Installing and configuring an app isn't so bad for the first, but do you really want to install and configure apps for every thing (device, appliance, car, and so on) you own? It doesn't scale and the complexity goes through the roof.

Now, imagine an evented system. The phone merely raises and event to an event bus that says "Phil's getting a phone call." The Tivo listens for events and knows that it pauses (that's the natural behavior for a Tivo when the viewer gets distracted by something else) when there's an inbound call. The stereo knows to pause. Lights know to come on. And so on. The phone doesn't know these other systems are there or what their actions will be. The phone doesn't have to understand their APIs, what commands they understand, or be configured when they are added. Any configuration happens on the device itself. If a new DVR shows up, everything else can ignore it. If the stereo component goes away, everything else goes on about their business.

Here's the magic. In a request-response system the requester is obligated to understand the semantics of the APIs it deals with. In event-driven system, the responsibility for semantics falls on the event processor--the receiver. This is called semantic encapsulation. Turning semantic responsibility around leads to looser coupling.

When you think about it, that's how cities work. When a new person moves in the neighborhood, all the neighbors don't have to be configured to interact with them. If you want to start a new business, there are some things you need to know and some conventions you'd better follow, but you do it in relative independence. There is loose coupling between the actors in the system and the semantic responsibilities are encapsulated in the actors--just like our DVR in the evented living room.

Of course, event-driven systems aren't a panacea. Events can be messy and figuring out the flow of control is tricky because there's no central command-and-control system like there is in a hierarchical system. Event-driven systems can be hard to debug when something unexpected happens. Nevertheless, I think there is an architectural imperative in force that will lead to more and more systems being event-driven. We're already starting to see the importance of events in the flows of information that social tools create. Notifications, activity streams, and the like are becoming more important and the natural way to deal with them isn't request-response systems or, in most cases, long-lived sockets, but events.

Events create a networked pattern of interaction with decentralized decision making. Because new players can enter the event system without others having to give permission, be reconfigured, or be reprogrammed, they grow organically. This is how the Internet was meant to be. There's no reason why we have to create hierarchical systems on top of the Internet. Our current interaction patterns on the Web are a legacy of client-server computing--itself a legacy of the corporate structure of the industrial age. Hierarchical interaction isn't the ultimate expression of the Internet, but merely a stepping stone to more powerful systems of the future. I'm convinced they'll be event-based.

I'm writing a book about this called The Live Web. If you'd like advanced access to the PDF, let me know and I'll invite you to the Dropbox.