Fuse as a Microservice Architecture

Phil Windley // Tue Jul 1 11:32:00 2014 // architecture cloudos fuse krl kynetx microservice picos

Summary

Microservices provide a powerful pattern for programming picos with KRL. This post describes microservices and shows how we can view rules within the Fuse system as microservices for a vehicle. We give a detailed, technical example of microservice interaction within Fuse and of a specific rule.

I recently ran across the idea of microservices. I don't know why it took me so long to run across it, since they've been discussed for a few years and there are many article written about them (see the end of this post for a link list). They have, like anything new, many different definitions, but I've settled on a few characteristics that I think differentiate a microservice from other styles of architecture:

Organized around a single business capability
Small, generally less than 1000 lines and code and usually much smaller
Event-based and asynchronous
Run in their own process
Independently deployable
Decentralized data storage

What struck me as I reviewed various material on microservices is how much the philosophy and architectural style match what I've been preaching around persistent compute objects (picos). As I worked through the ideas, I came to the conclusion that since you can view each pico as an event bus, we can view each rule installed in the pico as a microservice.

With this lens, Fuse can be seen as an extensible, microservice architecture for connected cars.

Background

Fuse is a connected-car platform. I've written extensively on Fuse in this blog. For the purposes of this post, it's important to understand the following:

Fuse uses Carvoyant to manage devices and provide an API from which we use to get vehicle data. The Carvoyant API is a well-designed RESTful API that uses OAuth for user authorization.
Picos use a set of pre-built services that collectively I call CloudOS to manage things like creating and destroying picos, pico-to-pico subscriptions, storing profiles, etc.
Rules are collected into rulesets that can share function definitions.
Each ruleset has a separate persistent key-value store from every other ruleset.
Rules are programmed in a language called KRL.
When we create a pico for a vehicle, the pico is automatically endowed with an event bus that connects all rules installed in the pico.
CloudOS provides a ruleset that functions as a persistent data store for the entire pico called the PDS. The PDS provides a standard profile for each pico. Fuse stores all of the vehicle's configuration and identity information in the pico profile.
Other vehicle data is stored by the individual service. For example, the trip service stores information about trips, the fuel service stores information about fuel purchase, and so on.
Rules can use HTTP to interface with other Web-based APIs.

Not only do we create a pico for each vehicle, but we also create one for each owner, and one per owner to represent the fleet. They are organized as shown below.

This organization provides essential communication pathways and establishes a structural representation of ownership and control.

Example Interactions

Microservices should be designed to be tolerant of failure. Let me walk through one place that shows up in Fuse. As mentioned above, Carvoyant provides an API that allows Fuse to interact with the OBD II device and it's data. To do that, we have to mirror the vehicle, to some extent, in the Carvoyant API. Whenever we create a vehicle in Fuse, we have to create one in Carvoyant. But there's a small problem: before the vehicle can be created at Carvoyant, the owner needs a Carvoyant account and that account needs to be linked (via OAuth) to their Fuse account.

One way to solve this problem would be to ensure that a vehicle can't be added to Fuse unless a Carvoyant account is active. Another way is to be more tolerant and let users add vehicles any time, even before they've created their Carvoyant account, and sync the two accounts pro actively. This has the added benefit of working the other direction as well. If someone happens to already have a Carvoyant account, they can add their vehicle to Fuse and the two accounts will sync up.

The following diagram shows a few of the microservices (rules) in the vehicle pico that help perform this task.

(click to enlarge)

As I mentioned above, the vehicle configuration is stored in the PDS profile. Whenever the profile is updated, the PDS raises a pds:profile_updated event. So anytime the user or a process changes any data in the vehicle profile, this event will be raised.

Any number of rules might be listening to that event. One rule that listens for that event is carvoyant initialize vehicle (lower left of the preceding diagram). Carvoyant initialize vehicle is a fairly complicated service that ensures that any vehicles in Fuse and Carvoyant are represented, if possible, in the other service. We'll examine it in more detail below. When it's done, carvoyant initialize vehicle raises the fuse:vehicle_uninitialized if the vehicle is changed.

Also, when the carvoyant initialize vehicle rule contacts Carvoyant to either update or create a vehicle, it does so with an http:post(). Picos are designed to automatically raise an event with the results of the post when it completes. The initialization OK rule (lower center) listens for that event (only with an HTTP 200 status) and store various information about the vehicle at Carvoyant. This is what establishes a link between the vehicle pico and the vehicle at Carvoyant.

Independently, the initialize vehicle rule (upper left in the diagram) is listening for the fuse:vehicle_uninitialized event. Initialize vehicle primarily functions to raise other events and thus sets off a collection of activities that are necessary to bring an uninitialized vehicle into the system and make it function. Since it's now been connected to Carvoyant, the vehicle needs to

subscribe to events from the Carvoyant system including events that communicate the ignition status, fuel level, battery level, and any diagnostic codes,
retrieve the current status of various vehicle systems, and
retrieve specific data about the vehicle's geoposition.

The initialize vehicle rule doesn't ask for all these on it's own, it merely signals that they're needed by raising events. Other services respond by carrying out one simple task.

For example, the update vehicle data rule in the upper right makes calls to the Carvoyant API to gather data from the vehicle, communicates that data to the fleet pico (i.e., raises an external event from the standpoint of the vehicle), and raises various events including the pds:new_data_available event.

The PDS add item rule (lower right corner) is listening for the pds:new_data_available event. It takes the data in the event and stores it in the vehicle PDS. We chose to store the vehicle data in the PDS instead of the update vehicle data rule because it's widely used and is more widely available in the PDS.

Of course, these are only a few of the rules that make up the entire system. There are at least 60 in the vehicle pico and many more in the fleet. Most of these rules are quite simple, but still perform a vital task.

One important note: each vehicle is represented by a unique pico and this has it's own copy of each of these rules, running against the unique data for that vehicle. The rule doesn't know or care about any vehicles except the one that it's installed in. As such, it is a dedicated service for that specific vehicle. In a system with 10,000 vehicles, there will be 10,000 independent carvoyant initialize vehicle microservices running, listening for pds:profile_update events from the PDS in the pico in which it runs.

A Detailed Look at A Service Rule

The carvoyant initialize vehicle rule performs initialization, but it has to be flexible. There are three basic scenarios:

The Fuse vehicle pico has a representation in Carvoyant and thus any changes that have been made in Fuse merely need to be reflected back to Carvoyant.
The Fuse vehicle pico has no Carvoyant representation but there is a Carvoyant vehicle that matches the Fuse vehicle in specific attributes (e.g. the vehicle identification number) and thus should be linked.
The Fuse vehicle pico has no Carvoyant representation and nothing matches, so we should create a vehicle at Carvoyant to represent it.

Here's the code for the rule

rule carvoyant_init_vehicle {

  select when carvoyant init_vehicle
	   or pds profile_updated

   pre {
    cv_vehicles = carvoyantVehicleData();
    profile = pds:get_all_me();
    vehicle_match = cv_vehicles
		      .filter(function(v){
			       v{"vin"} eq profile{"vin"}  
			     }).head();

    existing_vid = ent:vehicle_data{"vehicleId"} || profile{"deviceId"};

    // true if vehicle exists in Carvoyant with same vin and not yet linked
    should_link = not vehicle_match.isnull() 
	       && ent:vehicle_data{"vehicleId"}.isnull();

    vid = should_link                            => vehicle_match{"vehicleId"} 
	| ent:vehicle_data{"vehicleId"}.isnull() => "" 
	|                                           existing_vid;

    config_data = get_config(vid); 
    params = {
	"name": event:attr("name") || profile{"myProfileName"} || "Unknown Vehicle",
	"deviceId": event:attr("deviceId") || profile{"deviceId"} || "unknown",
	"label": event:attr("label") || profile{"myProfileName"} || "My Vehicle",
	"vin": event:attr("vin") || profile{"vin"} || "unknown",
	"mileage": event:attr("mileage") || profile{"mileage"} || "10"
    };
  }

  if( params{"deviceId"} neq "unknown"
   && params{"vin"} neq "unknown"
    ) then
  {
    carvoyant_post(config_data{"base_url"},
		   params,
		   config_data
	    )
	with ar_label = "vehicle_init";
  }

  fired { 
    raise fuse event vehicle_uninitialized 
      if should_link || event:name() eq "init_vehicle";
    log(">> initializing Carvoyant account with device ID = " + params{"deviceId"});
  } else {
    log(">> Carvoyant account initializaiton failed; missing device ID");
  }
}

This code is fairly dense, but straightforward. There are four primary pieces to pay attention to:

The select statement at the beginning declaratively describes the events that this rule listens for: carvoyant:init_vehicle or pds:profile_updated
The pre block (or prelude) retrieves the current list of vehicles Carvoyant knows about and filters them to see if any of them match the current vehicle. Then it calculates the vehicle ID (vid) based on that information. The calculated value of vid corresponds to the three scenarios listed above. The code also computes the parameters to post to Carvoyant.
The action block is guarded by a condition that ensures that the deviceId and vin aren't empty. Assuming the guard condition is met, the rule POSTs to carvoyant (the action carvoyant_post has all the logic for making the POST with appropriate OAuth access tokens).
Finally, if the rule fired (i.e. it was selected and it's guard condition was true) the postlude raises the fuse:vehicle_uninitialized event. The raise statement is further guarded to ensure that we don't signal that vehicle is uninitialized event when updating an existing vehicle.

Discussion

There are a few points I'd like to make about the preceding discussion and example:

Service contracts must be explicit and honored. Because we lean heavily on independent services, knowing which services raise which events and what those events mean is critical. There have been times that I've built services that didn't quite connect because I had two events that meant the same thing and each service was looking for a different event.

Self-healing allows misses. Picos don't guarantee event delivery. So consider what happens if, for example, the initialization OK rule misses the http:post event and the Carvoyant vehicle information fails to get stored? The carvoyant initialize vehicle rule is tolerant of failure in that the system will merely find the vehicle again later and link it then.

Idempotency is important to loose coupling. There are multiple places where it's easier to just raise an event and rely on the fact that running a service again does no harm. The logic gets very complicated and tightly coupled when one service has to determine if it should be raising an event because another service isn't designed idempotently.

Scaling by adding services. The carvoyant initialize vehicle rule ensures that Fuse vehicles are linked to their representation at Carvoyant, but it doesn't sync historical data, such as trips. What if something gets disconnected by a bad OAuth token? Because of the microservice architecture, we can add other rules later that look at historic data in the Carvoyant system and sync the Fuse vehicle with that data. These new services can be added as new needs arise without significant changes to existing services

Picos and CloudOS provide infrastructure for creating microservices in the Internet of Things. Picos provide a closure around the services and data for a given entity. In the case of Fuse, a vehicle. By representing an entity and allowing multiple copies to be created for like entities, picos provide persistenly available data and services for things even when theyre not online.

Asynchronous, one-way communication is the rule. Note that two-way communication is limited in the preceding example. Rules are not waiting for returned results from other rules. Neither is it right to think of rules "calling" each other. While the KRL system does make some guarantees about execution order, much of the execution proceeds as dictated by the event-flow, which can be conditional and change with circumstances.

Picos cleanly separate data. Picos, representing a specific entity, and microservices, representing a specific business capability within the pico, provide real benefit to the Internet of Things. For example, if you sell you car, you can transfer the vehicle pico to the new owner, after deleting the trip service, and its associated data, leaving untouched the maintenance records, which are stored in the maintenance service.

A microservice architecture means picos are extensible. Microservices provide an easily extensible API, feature set, and data model for the pico. By changing the mix of services in a given pico, the API, functionality, and data model of that pico change commensurately. There is nothing that requires that every vehicle pico be the same. They can be customized by make, model, owner, or any other characteristic.

Conclusion

Viewing rules as microservics within a pico have given me a powerful way to view pico programming. As I've folded this way of thinking into my programming, I've found that many of the ideas and patterns being developed for microservices can equally be applied to KRL programming within a pico.