Federated Multi-Tenant Service Architecture for an Internet of Things

The scenario we call the Internet of Things (IoT) is an inflection point in the development of information local and global infrastructure. We know cloud computing as a commoditization of primary infrastructure resources (also `things') for flexible datacentre hosting. The facilitation of a common platform for the next generation of global commerce presents a challenge of both technological and human dimensions. Not only do we have to solve the matter of technology at scale, we must also solve the matter of human dignity and participation. This is a challenge that spans every layer of the software and networking stacks, yet can be described in general terms without the need to specific implementations. That is our goal in this (revised) draft. Only a few new ideas are needed to synthesize this infrastructure, however several old technology practices must be deprecated for scaling and security considerations. A platform for society as a whole must be vendor agnostic at its root, and must leave ample space for vendor specific creativity on top. What distinguishes IoT from past scenarios is the prolific contact surface it will expose to the physical world, embedding devices pervasively in our close environments, and touching every part of human life. At the time of writing, IoT has barely begun to emerge in domestic and industrial settings; however, choices we make now could help or hinder the development of an adequate platform over the coming decades. The proposed architecture not only scales up to large numbers, it also scales down to small devices of low capability; from the largest installations to the smallest, and from the tiniest amounts of data, to vast data-stores collected by scientific computing at the limits of possibility.

The term "PROMISE", "PROMISES" in this document are to be interpreted as described in Promise Theory When used, the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

A hardware or software agent that is IP addressable, via a TCP/IP capable interface. A hardware or software agent with an IP address (prefix and subnet) that is fixed over the timescale of application service interactions. A hardware or software agent whose IP address location can change on the timescale of application service interactions. Any agent that promises to respond to requests, from external parties, and perform services of any kind, on a timescale that we may call the application service timescale. A collection of agents housed as tenants within a single host device, each offering different services, with potentially different timescales. An agent that consumes data from an application service, requested either by imposed query or by promised schedule. A set of interfaces that expose remote agents to accessing and overriding of the local autonomy of agents. A full function device (FFD), with an IP address, that can present its own service gateway or interface to the IP network. A reduced function device (RFD), with no IP address, that attaches to a host gateway device as a peripheral, over an arbitrary network (USB, PCIe, CANbus, Profibus, ModBus, wireless sensor network, etc). Devices are addressable, only through the gateway service. This includes portmapped devices. Any network (IP or non-IP) that is non-IP routed, i.e. contained within a host endpoint as part of a black box, e.g. isolated NAT, device bus, serial channels. An agent that consumes a service from another agent, and provides a new service based on the consumed service, e.g. a router, encrypter, compressor, etc. A unilateral policy assessment of one agent by another, concerning its reliability in honouring promises. Trust is not necessarily a transitive property. A device is said to have partial connectivity if it is unavailable for intervals of time, e.g. due to loss of connectivity, mobility, or power napping.

A platform that brings computing closer to users, away from specialized datacentres, must be based on plausible assumptions. We assume that devices live in a partially connected environment, of limited reliability: they MUST be fault tolerant to loss of communications, both with other devices, in the course of providing application services, and with trusted sources of information. The minimum level of interdependency is recommended to facilitate this. For a nascent Internet of Things, our focus is naturally drawn to the specialized leaf devices, where data may be produced or consumed. It will take many years to commoditize these sensors and actuators, and their local communication architectures. However, these are only half the picture. `Thing' devices, by design, also communicate with online services deployed `higher up', or `Northbound' in the system, to offload analysis and decision-making. Their physical capabilities thus place them into two broad categories: These are assumed to connect by an IP (or layer 3) addressable underlay network. Connectivity is assumed end-to-end, without reference to tunnels or software-defined overlays. Policy-based routing is assumed to be provided end-to-end, and fully decoupled from the hardware and software running on devices. Routing paths may be managed through a namespace registration abstraction which we call workspaces. Segregation and firewalling of certain network regions may be included as part of external network governance design, but will not be considered here. These include bare sensors and actuators, which do not possess sufficient onboard resources or software interfaces, may attach to hosting standalone devices that act as a gateway and IP (or layer 3) endpoint on their behalf. These pass-through devices transformers, converters, encapsulation services, etc

IP Endpoint +------------------+ +------------------+ | RFD / Peripheral |--+ +------------------+ | +------------------+ +------| FFD / Standalone |--> IP or L3 endpoint +------------------+ | +------------------+ | RFD / Peripheral |--+ +------------------+ ]]> Devices may be standalone (FFD), with service interfaces, or hosted peripherals (RFD), where data are exposed through service interfaces from other buses, e.g. USB, CANbus, MODbus, Profibus, etc. Standalone devices are full stack devices that provide data oriented services to data clients Stand-alone devices and transducers can vary considerably in their processing, memory, and connectivity constraints. This architecture assumes a minimum resource level at the stand-alone device, but the device must support `full stack' implementations. In practice, this implies that they contain an embedded OS (e.g. Linux), and are capable of running an agent providing secure service and connectivity interfaces.

Centralization of intent is a natural form of coordination. However, centralization of command and control (e.g. by API) is not practical in environments where the density of devices and overlapping concerns reaches the level of a pervasive Internet of Things. Legacy technology is pervasively centralized and top-down in nature: requests for domain names and name services, IP address assignment, change management of information records, cloud controllers, service entry points, etc. The barriers to scalable autonomous activity are high. The trend has been to delegate these activities to sub-authorities (multiple queues) to limit scaling bottlenecks, which improves queue latency but not minimum transaction time. In the future, resource management can profit by propagating intentions and desires from the bottom-up, i.e. from the many points of service consumption, with localization to minimize queueing and flooding contention of the independent needs.

Infrastructure ownership is an important issue in a multi-tenant consumer environment. While some devices can be centrally managed by providers (regardless of owner), many devices in an Internet of Things will be owned by private individuals who will permit management by centralized services. Devices may be managed by: This applies in particular to personal consumer electronics, phones, cars, domestic appliances, etc, where users need to retain trusted ownership of their personal belongings. This applies to managed services, factory machinery, fleet vehicles, set-top boxes placed in the home, power controllers, etc, where users do not need to interact with the devices on a management level, but there is an advantage to placing a device as a local presence in a smart environment.

Federation of intent, aka multi-tenancy or diversity, all point to the need for Special Interest Groups (SIG) or workgroups, who specialize within organizations to develop expertise. Software architectures following this pattern are sometimes called microservice architectures. We shall introduce the notion of `workspaces' as a federated infrastructure abstraction designed to wrap one or more of these specialized services under an umbrella abstraction that is easy to understand and work with. A goal of the workspace is to expose only the working parts that need to interact with consumers, e.g. in the same way that one does not expose the inner components of a television or a car. Federation is desirable along a number of lines: Geographic partitioning (location) Separation of timescales (fast and slow) By special interest group (functional) See sections below for further information.

Although user-facing devices, deployed in the field, may be separate from the agencies processing their sensory data, or feeding them guidance (e.g. as policies), it becomes increasingly impractical to transport data over long distances between leaf devices and `cloud' services as the density of deployed devices grows. The logical outcome is therefore a decentralization of the processing cloud itself, so as to bring all necessary resources close to the field-deployed data sources themselves. To scale such a distribution, the data services will naturally associate with private workspaces, which bound the scope of data generated by Things.

Workspaces may be thought of as a modernization and generalization of the familiar network domain concept. Workspaces go beyond namespacing, to include federation, collaboration, and segmentation of services. Currently, name domains are typically linked to simple directory services (DNS, Active Directory, LDAP etc) for name-address mapping. These are assigned from some top-down agency, either within an organization or even beyond it, at a regional level. The demands of multi-tenant environments, where shared resources and separate business-processes mix and compete, make these older services less than optimal, though not inherently flawed. It is awkward to separate independent collaborative activities and then manage their interactions on a need-to-know/need-to-do basis, without involving multiple human interventions. Cloud APIs bring some improvement, by exposing arbitrary capabilities to remote operation, but the remoteness also brings risks and inefficiencies by exposing an attack surface, and from lack of situational awareness of actual state. Workspaces are related to the more familiar notion of namespaces in information technology; however, namespaces refer mainly to priority name-referencing of objects, without necessitating underlying resource access segmentation. Workspaces MUST support multi-tenant separation of concerns within a hosted hardware resource space. Today, workspace-like facilities are commonly offered as user logins on computer operating systems or online services, and quasi-workspace-like facilities are offered by virtual private networks, and VLANs, etc, in networking. However, resource management platforms do not yet bring the same level of flexibility to infrastructure. Typically, resources are managed by top-down agencies who design and grant resource usage manually, leading to the insertion of a human processing timescale into the flow of automation.

Workspaces need to be able to promise segmentation and privacy. This involves some basic capabilities: Authorization and user identification for resource access control, within each workspace. Multiple compatible user identities in case of a member node belonging to several workspaces. Resource separation that is respected by all levels of the software interactions. This requires the support of operating systems. i.e. segmentation at the network, application, and operating system level. For example, Linux containers, systemd containment, network virtualization, etc. Service discovery, and registration of capabilities. Resource mobility and availability tracking. Scaling of information on a need-to-know/need-to-propagate basis, avoiding costly data consistency protocols that broadcast data updates, and lead to waiting.

At the time of this revision, many of the properties of workspaces are being explored under the aegis of cloud application cluster managers. Google's Kubernetes , for example, is presently the most ambitious of these. It is plausible to imagine extending such a system to cover the workspace proposal for both cloud and IoT scenarios. For a collaborative Internet of Things, where interests span many issues from manufacturer interests, to personal ownership, service provider concerns, functional responsibility, and security, etc, the technologies for inter-group collaboration need to be modernized to support logical segmentation, authenticated access, instrumented delegation, shared name-service information, as well as private naming, all across a converged palette of resources: compute, network, storage and sensor-actuators. This is somewhat reminiscent, but not identical, of the goals of Named Data Networking (NDN) , which promotes the semantics of space above the details of its interconnectivity. Workspaces imply scope; i.e. they may or may not be private, but they must be self-contained and separable, in the manner of namespaces. Workspaces may or may not be associated with multiple tenants; but they are associated with multiple functional or organizational issues. Workspaces represent a context for human activity, respecting separation of concerns. We need not prejudge what aspects of future activity need to embed software-based technologies, but it seems safe to assume no limits. Example workspaces might include: online stores and services, offices, the home, a children's playground, a squash court, a shop, a factory floor, buildings, districts, cities, an emergency communications channel, a subsystem of hot and cold water pipes, a hotel dining room, a personal drinks cabinet, etc. Ubiquitous computing (the Internet of Things) is all about how networked devices support a wider variety of workspaces than industrial scale central services. As the density of device resources (compute, storage, sensors, actuators) in a workplace or home environment increases, isolation of regions, and mapping of resources to responsible or interested parties become more difficult problems, both to implement and to understand. A detailed description of workspaces will be given separately .

The following characteristics describe compatible policy update processes: Devices subscribe to policy from a trusted workspace source, download changes to the policy model when they can, and cache it locally so that it is always available. Local agents implement cached policy, without any dependence on remote communication, and in a fault tolerant fashion. The failure to keep one promise should have minimal impact on the ability to keep others. By verifying promises continuously, the agent that runs on each standalone device will know (or be able to calculate) its operational context, and can decide which promises are needed from the policy model, and whether or not to keep the promises. This scales O(1), i.e. without bottleneck. Each promise that documents and intended outcome of the system is verified and measured in the process, providing immediate and statistical feedback to policy designers about the success of the policy in describing a stable desired outcome.

The properties we are looking for in workspaces, suggest an architecture based on the principles of promise theory; such a promise-oriented architecture is described implicitly in and . It lays out a generic `bottom up' management concept, in which devices each have the responsibility for their own state and roles. It resembles Service Oriented Architecture (SOA) superficially, without reference to specific technologies, implementations or protocols, and relates to the modern notion of microservices By formulating architecture from the bottom up, one can easily account for multi-contextual concerns, from developer concerns about realtime software updates (Continuous Delivery and DevOps etc), to operational service scaling, governance, and security, in a way that top-down schemes cannot easily achieve. The relationship between a generic promise-oriented architecture and the concept of a workspace is that the former provides a necessary and sufficient basis for implementing the latter. Workspaces are expected to be a friendly interface to the underlying promise architecture, separating interior and exterior promises cleanly and intuitively.

A promise-oriented architecture communicates (e.g. intent and data) by authenticated publish-subscribe (aka "pull") methods, for security and predictability. In a workspace, devices MUST not accept control commands imposed upon them by remote "push" methods, as this exposes a security risk and may lead to inconclusive results during uncoordinated pushes (multithreaded access). In the vernacular usage of "control plane" and "data plane", control is asserted through agreed service level policies, and data are exchanged within services to carry out functions. Every standalone device operates autonomously, with policy guidance from its owner, without direct external intervention. To form a workspace, any standalone device can give up that autonomy to a trusted manager, offering policy updates as a service. Workspaces separate interior and exterior promises about resources and their accessibility: they are a priori opaque from the exterior, and transparent from the interior. By joining a workspace, any device (whether a cloud server or an IoT embedded FFD, subordinates itself to a bounded policy domain with a private namespace. Policy determines whether a given member of the workspace will expose public service entry points, or will be entirely anonymous to exterior agents. This is very close to the community namespace idea currently being implemented in the Kubernetes cloud cluster manager . One can imagine a meta-cluster of such cloud clusters, which are designed to tolerate partial network reliability, as forming a basis for the Internet of Things alongside more traditional high reliability datacentre environments. Currently, cluster managers assume top-down ownership, rather than autonomous self-management, and expose more complexity through APIs than is appropriate for average engineers.

Rough correspondence between contemporary cloud clusters and proposed workspace concept: workspaces are principally a bottom-up, self-service collaboration over multiple clusters with more diverse hardware and software.

All devices provide services with varying degrees of sophistication. Peripheral devices serve data or actuators to host devices, and standalone devices expose functions to one another as software services. Each server plays a role to be composed into the wider system. Services may be used both for basic infrastructure support, and for driving user applications. No limitations need be stated about applications. Each fully functional, standalone device is free to host any application services. The result is superficially similar to the Service Oriented Architecture , but without reference to a specific technology or methodology. In modern parlance, the model is an example of microservices . Data collection services are also best implemented with pull methods, for resource-light scalability and security. However, extremely limited application devices might initially struggle to support this mode of operation. Service scaling is a task for workspaces. Public (exterior) services can be provided in a standardized manner, through accessible points of entry, whose name information is propagated publicly, analogous to a DNS directory. Workspaces can hide internals (e.g. vendor or implementation specific details, private internal services, load sharing parallelism. Interior name services deal with the registration and propagation of information between workspace members, on a need-to-know basis. This is never visible to the exterior network.

The basic atom of bottom-up policy is a promise. Each promise consists of three things: i.e. a resource that will affect a change by keeping its promise to the system, e.g. a file, a process, a transaction, a measurement, device settings, etc. i.e. the desired-outcome that is achieved when the promise is kept. This SHOULD be implemented in a convergent, idempotent manner, (see , ). in which the promise applies, based on time, location, type and group membership of the devices referred to in the model.

In a promise architecture, every device is contextually evaluated and integrated from the bottom up, according to the promises is keeps, e.g. the services it provides, its behaviours and properties, etc. Thus every device is modelled by its individual degree of agency to act as a proxy for human intent (policy). Standalone devices are assumed to be equipped with policy-keeping software agents. Peripheral devices, such as sensors or actuators, are assumed to be integral parts of the standalone devices, and hence maintainable by the their software agents. NO system MUST push changes or data to such agents ad hoc, without a documented promise to accept; thereafter, `fault tolerance' demands that we reject the word `must' from most descriptions, and replace it with `promise of best effort', as to reply on perfect behaviour leads to brittle systems with unrealistic expectations. For human safety in a rapidly expanding sphere of human involvement, the only `must' is for each agent to be stable and self-correcting, subject to the guidance of policy.

The following characteristics describe the cooperation between agents: Standalone devices promise to bootstrap to some trusted bootservice, i.e. register to one or more workspaces. Standalone devices promise to refuse direct commands imposed from network peers (as mentioned above). Policy consists of a collection of promises that apply in labelled contexts, each of which describes a unique desired end-state. Promises are kept in a convergent manner, so that all promise-keeping actions lead to the desired end-state, no matter what the initial state of the device. Agents that live on every device have drivers/renderers and make all changes without remote communication.

Each policy agent promises to maintain a context evaluator that computes a set of classifying `tags' or `labels' that characterize the state of the agent. This is updated every time the agent verifies policy, as its state may change as a result of repairs. These may be used as conditionals for distributed policy-based decision-making. Contextual labels characterize the device, its environment, and its location and time. The labels can then be used in policy to make certain promises apply only in specific contexts. When promises, within a policy, are tagged by issue or context, agents can select those that apply to its condition, within a larger trust relationship implied by policy sourcing. This simplifies logic and promotes stability, as evidenced by experience with software agents .

Policy change can be initiated from within a workspace, subject to a defined quality assurance, or fit-for-purpose review. Thus change of infrastructure may be instigated from the bottom-up also, as a self-service request. Human operators (owners or managers) decide on a policy model for all devices in an organization or policy group. This may be informed by the feedback about the success rate of previously kept promises. The changes are edited into a model, which consists of a collection of promises that should be kept by all resources on all devices. Changes are checked and tested before publishing. Once changes are approved, they are published by a policy service for download at the convenience of the standalone device.

Infrastructure stability is supported by a separation of systems into agencies that act in alignment with specific, separable timescales. Separation of fast and slow timescales avoids tight coupling and associated complex behaviours and should be considered a priority for maintaining safe, stable systems for human dependence. Systems scale along two broad lines, which a promise-oriented architecture helps to resolve: Workload timescales concern the quantitative activity of the system: how fast requests are handled, how quickly service is delivered, and promises are kept. Semantics are normally the concern of software engineers and system designers. This facilitates functional understanding. It is a form of human interface or knowledge management. It is sometimes at odds with the needs of dynamical scaling. Changes to semantics should generally be slow compared to the workload related dynamical activity, in order to maintain functional stability. Cooperative design of workspaces may observe this principle to foster functional stability and workload efficiency.

A number of functional roles are required to maintain a service lifecycle in a distributed environment. Making these roles self-managed within each workspace is how one scales the diversity of human intent and concerns. Roles are defined by the kinds of promises kept by devices: To provide trusted need-to-know data and local contacts so that clients can begin working within a policy domain. To accept essential directory information on trust in order to join a local policy domain. To deliver current policy from an authorized source, appropriate for each client (tenancy terms) from its global perspective To subscribe to the policy, selectively, depending on context from its local perspective. data server (aka ``Thing'') To offer a catalogue of data streams to different tenants This includes sensors, actuators. To subscribe to the policy, selectively, depending on context from its local perspective. Manufacturer User Description service is promised by all Things providing a URI that points to a description of the device, its serial number characteristics, service details etc. Identity clients promise to make use of data schemas and encodings involved in the interpretation of data pertaining to the device. A mutual binding service of published-subscribed registry information, enabling mutual discovery of interior services.

The roles in each collaborative workspace. Devices at the bottom of the figure typically coordinate through workspace services hosted in the "cloud" or any nearby compute resource. Efficiency suggests avoiding long data paths, instead moving computational resources closer to data collection points. Bootstrapping new devices into a workspace represents the beginning of a device lifecycle. Devices must begin with the location of a known bootstrap server. Devices must also promise to advertise their nature and capabilities, called `identification'. This may include Manufacturer Usage Description (MUD) identifiers .

So far, much as been said on how the application devices provide services via promises, and how system intent can be described and orchestrated via policy. There is also a connectivity (transport) fabric for these devices that operates on a set of promises that underly the described service framework, i.e. the network. Each network endpoint can be seen as providing its own set of promises that are used by other network elements to deliver routing and switching capabilities . The simplest form of SDN is simply name registration and route management. Intent driven networking is becoming more relevant as Software Defined Networking (SDN) deployments proliferate. In the described IoT architecture, service policies that describe the IoT system intent can be used as an input to derive partial network policies (e.g. Group Based Policy or some other model-based approach), with modulation by other data discovered from bootstrapping, etc. The figure below illustrates the relationship between the service and network layer policies for IoT.

Service policy could be partially rendered as an SDN baseline for simplifying dependency management. The simplest form of SDN is simply name registration and route management.

The architecture, described in this draft, enables densely clustered IT resources to form arbitrary self-service communities that span local or wide area networks. This is decouples a logical patchwork of segments on top of a plain end-to-end IP network. By basing on principles of fault-tolerance, including publish-subscribe dissemination semantics, this may be scaled, without bottleneck, by only the well-known methods currently employed by the World Wide Web. IPv6 and successors will play a key role in recapturing network simplicity from the many workarounds that have been stacked on top of IPv4 and its limitations. However, currently missing are adequate directory services to support a transparent workspace concept. The present Internet architecture is still geared principally towards a shared single-tenant, top-down management model, with host authority at the top. Top down methods require the leaf domains to trust (and hence always be exposed to attack from) the layers high up in the network. However, shrink-wrapping workspace boundaries closer around their private resources, this management can be simplified, speeded up, and become less exposed.

The issues discussed and laid out in this draft address key issues of scalability, fault tolerance, separation of concerns, and federation of intent within networked information systems. The platform, described here, is a synthesis of well-known techniques, and is deliberately aligned with the needs of agile commercial spaces, as well as large industrial distributions, and small domestic needs. We purposely leave open vendor specific concerns, which can easily fit into the described architecture, on top of this common set of principles. Interest in using IT to stimulate smart spaces (homes, buildings, vehicles, cities, etc., necessitates a scalable approach to interactive services, in which the service clients are not only humans but sensors and actuators. This cannot be scaled reliably without the segmentation of spacetime itself. Centralized cloud controllers, as we understand them today, cannot plausibly manage stable services for a society to rely on. We propose extending the notion of cloud and IoT to become a single seamless vision, without centralization as the core paradigm.

We are grateful for helpful conversations with K. Burns, M. Dvorkin, D. Maluf, and E. Lear.

With a pervasive contact surface onto both the Internet and the real world, security is obvious a major concern. Experience with pervasive frameworks like , as well as theoretical studies of pull-based architectures, suggest that the promise-oriented pull-only architecture can reduce the exposure to denial of service attacks and data-based overflow attacks, by rejecting all external data sent without invitation. Moreover, the tie-in between service and network policy reduces the likelihood of errors in policy across the layers. Workspaces can play a role too here, as a shrink-wrapping of service scope around minimal set of endpoints, thus reducing the logical contact surface for data communications, and publishing information purely on a need-to-know basis. We take is for granted that workspace data are encrypted with workspace authorized credentials.