The pico engine creates and manages picos.¹ Picos (persistent compute objects) are internet-first, persistent, actors that are a good choice for building reactive systems—especially in the Internet of Things.

Pico engine is the name we gave to the node.js rewrite of the Kynetx Rules Engine back in 2017. Matthew Wright and Bruce Conrad have been the principal developers of the pico engine.

The 2017 rewrite (Pico Engine 0.X) was a great success. When we started that project, I listed speed, internet-first, small deployment, and attribute-based event authorization as the goals. The 0.X rewrite achieved all of these. The new engine was small enough to be able to be deployed on Raspberry Pi's and other small computers and yet was significantly faster. One test we did on a 2015 13" Macbook Pro handled 44,504 events in over 8000 separate picos in 35 minutes and 19 seconds. The throughput was 21 events per second or 47.6 milliseconds per request.

This past year Matthew and Bruce reimplemented the pico engine with some significant improvements and architectural changes. We've released that as Pico Engine 1.X. This blog post discusses the improvements in Pico Engine 1.X, after a brief introduction of picos so you'll know why you should care.

Picos

Picos support an actor model of distributed computation. Picos have the following three properties. In response to a received message,

1. picos send messages to other picos—Picos respond to events and queries by running rules. Depending on the rules installed, a pico may raise events for itself or other picos.
2. picos create other picos—Picos can create and delete other picos, resulting in a parent-child hierarchy of picos.
3. picos change their internal state (which can affect their behavior when the next message received)—Each pico has a set of persistent variables that can only be affected by rules that run in response to events.

I describe picos and their API and programming model in more detail elsewhere. Event-driven systems, like those built from picos, can be used to create systems that meet the Reactive Manifesto.

Despite the parent-child hierarchy, picos can be arranged in a heterachical network for peer-to-peer communication and computation. As mentioned, picos support direct asynchronous messaging by sending events to other picos. Picos have an internal event bus for distributing those messages to rules installed in the pico. Rules in the pico are selected to run based on declarative event expressions. The pico matches events on the bus with event scenarios declared in the event expressions. Event expressions can specify simple single event matches, or complicated event relationships with temporal ordering. Rules whose event expressions match are scheduled for execution. Executing rules may raise additional events. More detail about the event loop and pico execution model are available elsewhere.

Each pico presents a unique Event-Query API that is dependent on the specific rulesets installed in the pico. Picos share nothing with other picos except through messages exchanged between them. Picos don't know and can't directly access or affect the internal state of another pico.

As a result of their design, picos exhibit the following important properties:

• Lock-free concurrency—picos respond to messages without locks.
• Isolation—state changes in one pico cannot affect the state in other picos.
• Location transparency—picos can live on multiple hosts and so computation can be scaled easily and across network boundaries.
• Loose coupling—picos are only dependent on one another to the extent of their design.

Pico Engine 1.0

Version 1.0 is a rewrite of pico-engine that introduces major improvements:

• A more pico-centric architecture that makes picos less dependent on a particular engine.
• A more module design that supports future improvements and makes the engine code easier to maintain and understand.
• Ruleset versioning and sharing to facilitate decentralized code sharing.
• Better, attribute-based channel policies for more secure system architecture.
• A new UI written in React that uses the event-query APIs of the picos themselves to render.

One of our goals for future pico ecosystems is build not just distributed, but decentralized peer-to-peer systems. One of the features we'd very much like picos to have is the ability to move between engines seamlessly and with little friction. Pico engine 1.X better supports this roadmap.

Figure 1 shows a block diagram of the primary components. The new engine is built on top of two primary modules: pico-framework and select-when.

The pico-framework handles the building blocks of a Pico based system:

• Pico lifecycle—picos exist from the time they're created until they're deleted.
• Pico parent/child relationships—Every pico, except for the root pico, has a parent. All picos may have children.
• Events—picos respond to events based on the rules that are installed in the pico. The pico-framework makes use of the select_when library to create rules that pattern match on event streams.
• Queries—picos can also respond to queries based on the rulesets that are installed in the pico.
• Channels—Events and queries arrive on channels that are created and deleted. Access control policies for events and queries on a particular channel are also managed by the pico-framework
• Rulesets—the framework manages installing, caching, flushing, and sandboxing rulesets.
• Persistence—all picos have persistence and can manage persistent data. The pico-framework uses Levelup to define an interface for a LevelDB compatible data store and uses it to handle persistence of picos.

The pico-framework is language agnostic. Pico-engine-core combines pico-framework with facilities for rendering KRL, the rule language used to program rulesets. KRL rulesets are compiled to Javascript for pico-framework. Pico-engine-core contains a registry (transparent to the user) that caches compiled rulesets that have been installed in picos. In addition, pico-engine-core includes a number of standard libraries for KRL. Rulesets are compiled to Javascript for execution. The Javascript produced by the rewrite is much more readable than that rendered by the 0.X engine. Because of the new modular design, rulesets written entirely in Javascript can be added to a pico system.

The pico engine combines the pico-engine-core with a LevelDB-compliant persistent store, an HTTP server, a log writer, and a ruleset loader for full functionality.

Wrangler

Wrangler is the pico operating system. Wrangler presents an event-query API for picos that supports programatically managing the pico lifecycle, channels and policies, and rulesets. Every pico created by the pico engine has Wrangler installed automatically to aid in programatically interacting with picos.

One of the goals of the new pico engine was to support picos moving between engines. Picos relied too heavily on direct interaction with the engine APIs in 0.X and thus were more tightly coupled to the engine than is necessary. The 1.0 engine minimizes the coupling to the largest extent possible. Wrangler, written in KRL, builds upon the core functionality provided by the engine to provide developers with an API for building pico systems programmatically. A great example of that is the Pico Engine Developer UI, discussed next.

Pico Engine Developer UI

Another significant change to the pico engine with the 1.0 release was a rewritten Developer UI. In 0.X, the UI was hard coded into the engine. The 1.X UI is a single page web application (SPA) written in React. The SPA uses an API that the engine provides to get the channel identifier (ECI) for the root pico in the engine. The UI SPA uses that ECI to connect to the API implemented by the io.picolabs.pico-engine-ui.krl ruleset (which is installed automatically in every pico).

Figure 2 shows the initial Developer UI screen. The display is the network of picos in the engine. Black lines represent parent-child relationships and form a tree with the root pico at the root. The pink lines are subscriptions between picos—two-way channels formed by exchanging ECIs. Subscriptions are used to form peer-to-peer (heterachical) relationships between picos and do no necessarily have to be on the same engine.

When a box representing a pico in the Developer UI is clicked, the display shows an interface for performing actions on the pico as shown in Figure 3. The interface shows a number of tabs.

• The About tab shows information about the pico, including its parent and children. The interface allows information about the pico to be changed and new children to be created.
• The Rulesets tab shows any rulesets installed in the pico, allows them to be flushed from the ruleset cache, and for new rulesets to be installed.
• The Channels tab is used to manage channels and channel policies.
• The Logging tab shows execution logs for the pico.
• The Testing tab provides an interface for exercising the event-query APIs that the rulesets installed in the pico provide.
• The Subscriptions tab provides an interface for managing the pico's subscriptions and creating new ones.

Because the Developer UI is just using the APIs provided by the pico, everything it does (and more) can be done programatically by code running in the picos themselves. Most useful pico systems will be created and managed programmatically using Wrangler. The Developer UI provides a convenient console for exploring and testing during development. The io.picolabs.pico-engine-ui.krl ruleset can be replaced or augmented by another ruleset the developer installs on the pico to provide a different interface to the pico. Interesting pico-based system will have applications that interact with their APIs to present the user interface. For example, Manifold is a SPA written in React that creates a system of picos for use in IoT applications.

Come Contribute

The pico engine is an open source project licensed under a liberal MIT license. You can see current issues for the pico engine here. Details about contributing are in the repository's README.

In addition to the work on the engine itself, one of the primary workstreams at present is to complete Bruce Conrad's excellent work to use DIDs and DIDComm as the basis for inter-pico communication, called ACA-Pico (Aries Cloud Agent - Pico). We're holding monthly meetings and there's a repository of current work complete with issues. This work is important because it will replace the current subscriptions method of connecting heterarchies of picos with DIDComm. This has the obvious advantages of being more secure and aligned with an important emerging standard. More importantly, because DIDComm is protocological, this will support protocol-based interactions between picos, including credential exchange.

If you're intrigued and want to get started with picos, there's a Quickstart along with a series of lessons. If you want support, contact me and we'll get you added to the Picolabs Slack.

Notes

1. The pico engine is to picos as the docker engine is to docker containers.

Photo Credit: Flowers Generative Art from dp792 (Pixabay)