Lately I've been exploring APIs that push data and the use of personal data. While we're proponents of the Evented API specification, there are already a number of APIs that push data in some way. One of those is Foursquare.com. The Foursquare push API will call a URL anytime a user checks in. (Foursquare calls it the "real-time" API, but I'm coming to dislike that term as a description for this functionality.) They post data to a URL of your choosing. Turns out, with the Sky API accepts event signal URLs (ESLs) that meet the Evented API Specification and Foursquare can use properly formatted ESLs with their Push API. Consequently, Foursquare's Push API works perfectly with KRL.

Once I've checked in and Foursquare has pushed the checkin data to an ESL being watched by a KRL application, it's easy enough to store that data and then retrieve it from the same ruleset. KRL has identity built-in along with a concept of stateful variables that persist from invocation to invocation. These variables, called entity variables store data persistently on an entity-by-entity basis without any work on the developer's part.

But I wanted something more--a personal data store. I wanted the ruleset processing the checkin to see the Foursquare checkin event and put the data somewhere where other rulesets could use it. A KRL module with an associated rule turns out to be a good way to do this. The rule responds to a pds:new_location_available event and stores that data away. Other rulesets can access to the data using a function provided as a module. This ruleset just looks like any other application in my personal even network, seeing relevant events and doing it's job--but also functions as a KRL module for retrieving the data values.

The final result looks something like the diagram below.

(click to enlarge)

Whenever I checkin at Foursquare, a foursquare:checkin event is raised to my personal event network. One of the apps, I've installed, the Foursquare Processor ruleset, is listening for events of that domain and type. The Foursquare Processor manipulates the data that Foursquare has sent and raises a pds:new_location_available event. Another app I've installed, the Personal Data Manager ruleset, is listening for that event and merely stores the Foursquare data away in an entity variable.

Later, I visit a Web page, exampley.com in this case, and a rule in another app I've installed, the PDS Inspector ruleset, retrieves the latest Foursquare checkin information from my Personal Data Manager using a function, get_location(), and displays it.

You can see these apps installed in my personal event network in the following screenshot. The apps that make this work are the ones in the red rectangle.

(click to enlarge)

"Wait a minute!" I can hear you saying. "That's just the 'Manage Apps' page in your Kynetx browser extension!" Precisely. Installing an app in my Kynetx account is the same thing as installing it in my personal event network. My Kynetx account is showing me the apps that I've got installed in my personal event network and it's where I control how they behave. If you have a Kynetx account, you already have a personal event network.

A few points to make:

• None of these apps knows about the other ones. They are loosely coupled. None of them know what's going to happen when they raise their respective events because an event isn't a request. The behavior emerges from the particular suite of apps I have installed.
• This behavior isn't hard coded for me. These apps work generally across anyone's personal event network. Because these are stored in entity variables, if you install these apps in your personal event network, then you'll see your data, not mine. When you look at the code below, you won't see any code that makes that happen. That's because the personal event network infrastructure is handling all of the identity information below the ruleset level. KRL developers get multi-tenanted applications for free.
• Personal data is being managed by a separate ruleset allowing for indirection in how the data is used. Storing the checkin and seeing the checkin are asynchronous processes. Only apps I've installed in my personal event network can see the personal data in the Personal Data Manager. To see the data, apps have to use the Personal Data Manager module and thus disclose their intent to use personal data. This is not a final answer to a personal data store since it doesn't provide for explicit user permissioning, but it's getting a lot closer.

This demonstration shows a personal event network responding to Foursquare checkin events and storing the information about the checkin in a personal data service using a manager application that is loosely coupled and privacy respecting. Pretty cool, huh?

The Gory Details

What follows are the gory details of how it all works. Keep reading if you're into that sort of thing.

The Foursquare Real-Time API and KRL Events

Foursquare provides a method for creating an OAuth consumer. As part of that process, you'll need to register your application, providing the Web site URL and a callback URL. After you register the application, you need to edit it to provide a push URL. This is where you put the Sky API ESL:

https://cs.kobj.net/sky/event/<token>/fakeeid/foursquare/checkin

The token identifies the personal event network that the event is sent to. It's unique to the particular application raising the event. This creates a revokable, one-to-one connection between the Foursquare OAuth consumer and the personal event network. The token generator is still in beta; if you'd like to play with this, contact me and I'll tell you how to get to it. Here's what the Foursquare OAuth consumer page looks like when you're done:

(click to enlarge)

You have to use OAuth to tie a Foursquare account to this Foursquare consumer. I just did it manually (cutting and pasting URLs into my browser) so that my Foursquare account was tied to this consumer. We're not quite using the Foursquare API the way they envisioned it. They think of a single OAuth consumer handling multiple accounts and thus a single push URL being used to push checkin data for every user who has authorized the OAuth consumer to see their acccount.

Instead, I'm tying the push URL (an ELS) to an OAuth consumer and then only tying a single user account to each OAuth consumer. This allows the Foursquare OAuth consumer to function as an endpoint for a single personal event network. The following diagram shows this difference.

(click to enlarge)

I'd like to see Foursquare allow a user to associate a push URL directly with their account to save the trouble of creating serperate OAuth consumers and going through the OAuth process, but this works fine for now. The consumer detail page (shown above) contains a button for testing the push URL. You can do this before you go through the OAuth process. The biggest problem I had with it is that it didn't show the return value or response code and I was having a few problems debugging it, so that would have been helpful.

Once you've made these links, you have a Foursquare endpoint raising checkin events to your personal event network whenever you checkin with the Foursquare app on your smart phone:

(click to enlarge)

The Foursquare Processor

The Foursquare processor is a KRL ruleset that is listening for foursquare:checkin events. The rule is selected when a foursquare:checkin event is raised. If you look cartefully at the event signal URL we gave to Foursquare that's shown above, you'll see that we've encoded that event domain (foursquare) and type (checkin) in the URL.

Here's the rule:

rule fs_checkin {
  select when foursquare checkin
  pre {
    // decode the JSON to get the data structure
    checkin = event:attr("checkin").decode();          
  }
  noop();
  fired {
    raise pds event new_location_available with
       key = "foursquare" and
       value = 
         {"venue": checkin.pick("$..venue.name"),
          "city": checkin.pick("$..location.city"),
          "shout": checkin.pick("$..shout", true).head(),
          "createdAt": checkin.pick("$..createdAt")
         }
  }
}

I'm picking apart the JSON structure that Foursquare sends and just looking at four values: the venue name, the city, the shout, and the checkin time. There's lots more data there, but this was sufficient for my purposes.

The rule's sole result is to raise another event, pds:new_location_available. Any other ruleset in the personal event network might be listening for this event. Note that this rule isn't controlling who will see the event or what they're supposed to do with it if they do see it. This is an example of semantic encapsulation that aids the loose coupling of personal event network applications.

The Personal Data Manager

The Personal Data Manager (PDM) ruleset is perhaps the most interesting of the three rulesets presented here because it's functioning as a ruleset to process event and a module for retrieving data. The PDM ruleset has a rule that is listening for the pds:new_location_available event that the Foursquare Processor ruleset raises.

rule add_location_item {
  select when pds new_location_available
  noop();
  always {
    set ent:location{event:attr("key")} 
        event:attr("value");
    log "Saw " + event:attr("key") + " data";
    raise pds event new_location_item_added 
      with key = event:attr("key");
  }
}

This rule does three things when it sees a salient event:

1. Stores the value of the event attribute named value in an entity variable named ent:location. Ent:location is a map and the key is taken from the event attribute named key.
2. Writes a log message that it saw data with a given key.
3. Raises a pds:new_location_item_added event.

The first is the primary result we want from this rule: the data gets stored in an entity variable. As I mentioned above, because this is an entity variable it will be persistently stored for the entity that owns the personal event network where this rule is running. In other words, the personal event network has a built-in method for storing personal data without the programmer ever having to worry about the identity and database issues involved in making it work.

The third result is interesting because it ensures that this rule is not a dead-end leaf on the event tree. In the case of this demonstration, there isn't any ruleset listening for the pds:new_location_item_added event in my personal event network; but there could be. Next week I might install an app that needs to know when my location changes and it will see this event and do whatever it does. Ensuring that there are no dead-ends is a good practice that aids loose coupling.

In addition to having a rule that responds to events and stores data, the PDM ruleset is also a KRL module:

meta {
  name "Personal Data Manager"
  provides get_item, get_location
}

global { 
  get_location = function (k) {
    ent:location{k};
  };
}

The module provides two functions: get_item() (not shown) and get_location(). Get_location() simply accesses and returns the value stored with a given key, k, in the location entity variable, ent:location.

Note that modules form a closure over persistent variables used in the ruleset because of the static scoping or persistent variables in modules. That means that when a rule stores something in a persistent variable in a module and a provided function retrieves it, they're talking about the same piece of data. This is the magic that allows a ruleset to act as a personal data manager.

The Personal Data Inspector

The Personal Data Inspector ruleset is simply a convinience for testing the system we've put in place and seeing that it all works. This app is a pretty standard KRL ruleset for modifying a Web page. The ruleset uses the PDM ruleset as a module so that it has access to the location data.

Here's the code for the entire ruleset:

ruleset a16x138 {
  meta {
    name "PDS Inspector"
    use module a16x137 version "dev" alias pds
  }

  dispatch {
    domain "exampley.com"
  }
  
  rule show_location_data {
    select when pageview ".*"
    pre {
      fs_location = pds:get_location("foursquare");
      v = fs_location.pick("$..venue");
      c = fs_location.pick("$..city");
      s = fs_location.pick("$..shout");
      t = "Phil last seen at #{v} in #{c}";
      body = 
        (s.typeof()) eq "str" => t + ' saying "#{s}"'
                               | t;
    }
  notify("Where's Phil?", body) with sticky = true;
}

The operation of the ruleset is simple:

• The ruleset uses the PDM ruleset (a16x137) aliasing it with the name pds.
• The dispatch section declares exampley.com as a salient domain so that the Kynetx Browser Extension (KBX) knows to run this ruleset on that domain.
• The rule uses the function pds:get_location() function from the PDM module with the key "foursquare" to get the data for the last checkin, picks it apart, and places a notification box on the Web page with the data about the last checkin.

With that, I see a notifcation box like this whenever I visit exampley.com:

(click to enlarge)

This notification is happening aynchronously from the checkin data being stored. The Foursquare Processor ruleset stores the data whenever I check in. The PDS Inspector ruleset shows the last checking location whenever I visit this particular Web page. The PDM rule is sitting between then providing data storage for the shared data that these two rulesets use.

Conclusion

This demonstration has shown three primary ideas:

1. Foursquare can be configured as an endpoint for a personal event network.
2. A KRL ruleset can function as a data manager for personal data within the personal event network.
3. A personal event network can function as an entity-specific event bus, allowing loosely coupled, semantically isolated applications to function together on an owner's behalf.

These ideas are each an important building block in creating the vision for personal event networks that function on behalf of their owners, protecting personal data and allowing applications to use shared personal data.