RPS Data Persistence (CMDB) Software Design

Last updated on March 24, 2021.

Last Reviewed and Approved on PENDING REVIEW

The data persistence feature of the Rapid Provisioning System (RPS) provides the underlying data storage needs of the system. The RPS Configuration Management Database (CMDB) maintains the metadata that is needed by RPS to targetable containers of items, the resources to be applied to those items, tasks and task maps for orchestrating task execution on those items, and the node hierarchy in which these items reside.

The figure below shows a high-level relationship of how these meta-data types interact for the purposes of automations.

RPS CMDB – ER Database Diagram

The detailed entity relationship diagram documentation for the RPS CMDB is available at: RPS > Trunk > Documents > Architecture > RPSEntityRelationshipDiagram.vsd

Note

This document is intended to outline the physical database layout.

Term	Definition
Automation	A single automated task that is executed by the RPS system.
CMDB	Configuration Management Database (custom for RPS).
DSC	Microsoft PowerShell Desired State Configuration software package.
Device	Any computer, router or other networked device which can be targeted by an automated risk.
Node	A running instance of the RPS system.
RPS	Rapid Provisioning System.
Resource	Any item of resource that can be applied to a target through a running task.
Target	Any container or item on a container that can be directly targeted for task execution by RPS.
Task	Any workflow/runbook that can be executed on a target by RPS.
TMS	Task Management Service - hosts & runs runbooks, used by RPS.

Metadata Storage – RPS Tables

This section will describe the tables needed to persist data for the RPS API. This document will further describe how each table is related and leveraged by the RPS API.

RPS coordinates automations by leveraging the CMDB for target, task, resource, and node configuration metadata. This data represents the containers/items to target, resources to assign, tasks to assign, orchestrate & automate, and hierarchal nodes to locate and synchronize. This data is all stored in a very generic format to offer maximum flexibility and extensibility for any user of the solution. Breaking down data into a generic data model allows for any type of target, resource, and task information to be stored and utilized within the RPS Database for coordinated automation and synchronization. The table below briefly describes the current tables represented within the RPS CMDB:

Data Type	Function
Container	A Container is metadata about a TargetItem that is a top-level parent of a set of target Items. Task maps are only assigned to target items that are containers.
ItemProperty	An ItemProperty is a key/value property bag item that contains all custom metadata for an RPS item allowing for extensibility. The ItemProperty table is used by TargetItem, TargetGroup, ResourceItem, ResourceGroup, Node, TaskItem, TaskAssignment, and ResourceAssignmentStatus.
LocalConfig	A LocalConfig is metadata about a local RPS instance. An example is the identification of the local RPS node.
Node	A Node is metadata about an identifiable location that is a running instance of RPS. Nodes can have parent/child relationships. Changed data is synchronized between related nodes.
ResourceAssignmentStatus	A ResourceAssignmentStatus is metadata about a resource assignment to a targetable item and its current state. Resource assignments are defined and managed by individual tasks (workflows).
ResourceGroup	A ResourceGroup is metadata about a logical grouping of resource items. A ResourceGroup has a set of ResourceItem members to simplify resource management and assignments.
ResourceItem	A ResourceItem is metadata about a resource item that is not an executable task. A resource item is either a local or global item of resource that is applied to a targetable item by a task workflow/runbook. An example of a resource item is an operating system patch or a configuration template.
TargetGroup	A TargetGroup is metadata about a logical grouping of target items. A TargetGroup has a set of TargetItem members to simplify task/resource assignments.
TargetItem	A TargetItem is metadata about a device item or container. Target items can have parent/child relationships. A top-level target item is called a ‘Container’ since it always contains a set of targetable items. Target items can have tasks assigned to be executed on them via automation. Target items can have resources assigned to be applied to them. An example of a target item is a router or a switch.
TaskAssignment	A TaskAssignment is metadata about a task assignment to a targetable item and its current state. Task assignments are sometimes direct assignments of a task to a target item. Task assignments are also generated through assignments of a task map to a container. TMS uses task assignment state to control automation.
TaskAssignmentDependency	A TaskAssignmentDependency is metadata about a task assignment dependency. Each instance defines the parent/child dependency between two task assignments. These dependencies are generated from task map definition dependencies during a task map assignment to a target. These dependencies are used by TMS to control task execution order during automation.
TaskAssignmentUserAction	A TaskAssignmentUserAction is metadata about a user action associated with a task assignment when task assignment state is ‘PendingUserAction’.
TaskItem	A Task is metadata about a task workflow. A task is a workflow that is executed by TMS. Tasks are assigned to target items directly or through task maps for coordinating task execution on target items.
TaskMap	Identifies a set of steps, what order they should be performed in, and what target items those steps apply to.
TaskMapAssignment	A TaskMapAssignment is metadata about a task map assignment to a target. A task map assignment contains information about the assignment of a task map to a target item or group of target items.
TaskMapDefinition	A TaskMapDefinition is metadata about a task map definition. A task map definition defines which task item is assigned to which target item. An optional property bag filter can be associated.
TaskMapDefDependency	A TaskMapDefDependency is metadata about a task map definition dependency. Each instance defines the parent/child dependency between two task map definitions. These dependencies are used by TMS to control task execution order during automation.
TaskMapDefFilter	A TaskMapDefFilter is metadata about a task map definition filter. A filter is used to further define which target items a task item is being assigned to by specifying which property bag key/value pairs are present on a target item.
TaskState	A TaskState is an enumeration of task states used for task assignments by the TMS controller.

RPS CMDB Relationships

The detailed design diagram for database entity relationships (ER) on how database entities are related to each other can be found here: RPS > Trunk > Documents > Architecture > RPSEntityRelationshipDiagram.vsd

The CMDB database tables have the following cardinality in relation to other tables:

• A Node can have zero-or-more child Nodes, and zero-or-one parent Node.
• A Node can contain zero-or-more TargetItems.
• A Node can contain zero-or-more local ResourceItems.
• A Node can have zero-or-more ItemProperties.
• A LocalConfig can reference one local Node.
• A TargetItem can have zero-or-more child TargetItems, and zero-to-one parent TargetItems.
• A TargetItem can have zero-or-more ItemProperties.
• A TargetItem can belong to one Node.
• A TargetItem can belong to zero-or-more TargetGroups.
• A TargetItem can belong to zero-or-more ResourceAssignments.
• A TargetItem can belong to zero-or-more TaskAssignments.
• A TargetGroup can have zero-or-more TargetItems.
• A TargetGroup can have zero-or-more ItemProperties.
• A ResourceItem can have zero-or-more child ResourceItems, and zero-or-one parent ResourceItem.
• A ResourceItem can belong to one local Node or be global.
• A ResourceItem can have zero-or-more ItemProperties.
• A ResourceItem can belong to zero-or-more ResourceGroups.
• A ResourceItem can belong to zero-or-more ResourceAssignments.
• A ResourceGroup can have zero-or-more ResourceItems.
• A ResourceGroup can have zero-or-more ItemProperties.
• A ResourceAssignmentStatus can reference one ResourceItem.
• A ResourceAssignmentStatus can reference one TargetItem.
• A ResourceAssignmentStatus can have zero-or-more ItemProperties.
• A TaskItem can belong to zero-or-more TaskMapDefinitions.
• A TaskItem can belong to zero-or-more TaskAssignments.
• A TaskItem can have zero-or-more ItemProperties.
• A TaskMap can have zero-or-more TaskMapDefinitions.
• A TaskMap can have zero-or-more TaskMapAssignments.
• A TaskMapDefinition can belong to one TaskMap.
• A TaskMapDefinition can be associated with one TaskItem.
• A TaskMapDefinition can have zero-or-more dependent child TaskMapDefinitions, and zero-or-more dependent parent TaskMapDefinitions.
• A TaskMapDefinition can have zero-or-more TaskMapDefFilters.
• A TaskMapDefFilter can be associated with one TaskMapDefinition.
• A TaskMapAssignment can reference one TaskMap.
• A TaskMapAssignment can have zero-or-more TaskAssignments.
• A TaskAssignment can reference one TaskMapAssignment.
• A TaskAssignment can reference one TaskItem.
• A TaskAssignment can reference one TargetItem.
• A TaskAssignment can have zero-or-more dependent child TaskAssignments, and zero-or-more dependent parent TaskAssignments.
• A TaskAssignment can have zero-or-more ItemProperties.
• A TaskAssignmentUserAction can reference one TaskAssignmentStatus.

RPS CMDB Detail Descriptions

Nodes

Nodes provide a physical location for a set of containers, target items and local resource items that is globally identifiable by the synchronization of data between nodes. Custom node properties are stored in an item property bag.

For example, the defined hierarchy of nodes are assigned containers, then also task and resource assignments. A parent node can automatically provide a child node through synchronization data describing a patch or other release, and the controller can begin to execute the release as soon as the data is available.

Containers – Automations Scope

Containers provide a scoping mechanism to properly ‘group’ items together for automations purposes. It is defined as the top-most parent in a TargetItem’s hierarchy. When a parent/child hierarchy of TargetItems is established, the highest-level item is automatically derived as the “Container”. Custom Container properties are stored in an item property bag.

When executing TaskMaps that require more complex execution chains, the container provides the scope that is representative of how to coordinate those tasks across devices. This enables the capability to define many similar sets of devices with different scopes for automation purposes.

Consider a series of systems that are interdependent, such as a set of servers on which different pieces of software reside. Software or devices may require that certain components are configured before others. The container offers a scoping mechanism for how those dependencies relate.

Other examples include:

• A datacenter containing servers. Representing the datacenter as a TargetItem, then adding child TargetItems.
• A vehicle containing equipment. Representing the vehicle as an TargetItem, then adding computer devices as child TargetItems.

Both of these offer the scope necessary to automate their TargetItems as related to each other.

Important

A TargetItem may only belong to one container. Containers cannot belong to other containers, as they are derived from the top-most parent item in a parent TargetItem / child TargetItem hierarchy.

TargetItems – Target Device Data

TargetItems represent the data about an individual item and its properties. A target item is simply a data representation of the device that requires automation. This could be physical or logical. A target item belongs to a container for the purposes of scope and executions. Custom TargetItem properties are stored in an item property bag.

Examples include:

• A computer or server
• A router or switch
• A ‘vehicle’, logically vehicle may be physical, it is not a ‘device’ that we automate against, however there may be a model in which that is the case.

Note

TargetItems also maintain a parent-child relationship to each other, leaving the highest-level parent always automatically defined as the container. This offers the ability to structure item relationships around their relationships to other items while maintaining a scope automatically. As this scoping mechanism is logical in nature, it can be defined using whatever logic best suits the business case. (E.g., a “Vehicle” containing vehicles, or a “Datacenter” containing servers, or a “Computer” containing software.) The figure below shows three sample target items, as related to each other. Each has its own separate list of properties, but all exist within the context of the Container, while the container itself is the top-most parent defined in the hierarchy of items.

TargetGroups

TargetGroups offer the ability to relate TargetItems into a bucket for administrative purposes. TargetGroups are used to group together all items of a certain type, or all items based on a specific criterion. This provides administrative capabilities to assign tasks to a large number of TargetItems with whatever common criteria are required by the business case. (E.g., “All Servers” or “All Dell devices”, etc.) Custom TargetGroup properties are stored in an item property bag.

Important

That TargetGroups do NOT offer the same scoping functionality that Containers do for the purposes of coordinating Tasks. TargetGroups provide an administrative way to manage a bundle of TargetItems. Each is treated individually and iteratively whenever an action is taken against the group, such as assigning tasks.

ResourceItems - Associative Resource Data

ResourceItems represent a reusable, associative piece of data for broader consumption than an individual item or container. For example, storing a list of all patches as required to be applied to Windows Servers, then associating that data to all Windows Servers in our TargetItem data. Custom ResourceItem properties are stored in an item property bag.

Resources provide this mechanism by representing the patch data as a ResourceItem (what patch, where the files are located, etc.). These details are the properties of the ResourceItem. This resource can then be associated to all of the relevant devices via a ResourceAssignment. Providing a one-to-many relationship of that single Resource to as many TargetItems as deemed applicable.

The figure below shows a simple ResourceItem and an example of the resource being ‘associated’ to various TargetItems or Containers.

ResourceGroups

ResourceGroups offer identical functionality to TargetGroups in regard to ResourceItems. They provide a logical grouping mechanism for grouping ResourceItems together for use and consumption. As with TargetGroups, the ResourceGroup does not offer any scoping functionalities and is merely an administrative function. Custom ResourceGroup properties are stored in an item property bag.

One benefit to ResourceGroups however, given their flexibility, is that they can be used to piece together configurations from smaller sets of ResourceItems. Configurations can be stored as Resources, then added to various ResourceGroups to create different configuration sets for different purposes.

ResourceAssignment

ResourceAssignments offer a correlation mechanism between ResourceItems and TargetItems and subsequently store a status about that relationship. For example, defining a patch resource, and assigning it to a device allows for both the existence of that correlation as well as its status. Custom ResourceAssignment properties are stored in an item property bag.

It is important to note however, the action of associating ResourceItems to TargetItems has no outward effect on automation. No execution request is performed, and no action is taken simply by the action of creating this assignment. The assignment instead creates the reference between that TargetItem so it may be leveraged in code. This provides both management of what ResourceItems are assigned to what TargetItems as well as an optional AssignmentStatus for this relationship.

Using this mechanism is optional, as there may be cases where a ResourceItem does not have or require a status of the Assignment. As a result, use of ResourceAssignments are to be leveraged as needed. While one may exist, it may contain null or useless data.

Additionally, a ResourceAssignment is not required to make use of a ResourceItem in code. Storing ‘global’ information in a ResourceItem without assigning it to anything is an expected use-case.

The diagram below and provides context to the resulting list of ResourceAssignments that would be generated if the displayed associations were created.

ResourceItem	Target	Status
ResourceItem1	TargetItem1	Complete
ResourceItem1	TargetItem2	Incomplete
ResourceItem1	Container1	NULL
ResourceItem 1	TargetItem3	MiscStatus

TaskItems

A TaskItem is an instance of a runbook registered to the RPS database. This represents a runbook that has been imported into TMS for execution and is being added to RPS to be made available for coordinated execution by RPS. Custom TaskItem properties are stored in an item property bag.

Runbooks can exist inside of TMS without being registered to RPS for control, coordination, and execution by the automation's solution. This distinction registers an item for coordinated execution by RPS.

Task Maps - Coordinating Pieces of Automations

Automations requirements vary by need from installing software to configuring or installing firmware on devices that don’t necessarily run common operating systems. This makes automations tasks in general quite difficult to author due to the complexity of dependencies that exist when analyzing any specific goal. Coding these dependencies in begins to create a complex web of code that can be tedious to manage as well as a difficult to update or change.

By coordinating componentized bits of code, each task can be isolated, re-used where able, and coordinated in the appropriate order or executed individually if needed. This removes the concept of authoring automations into monolithic scripts by coordinating the smaller pieces.

Each automation action’s state is represented by an assignment record (see TaskAssignments) containing information about the TaskItem and TargetItem in the assignment. The status of that action/item relationship controls the flow of execution.

The RPS toolset provides this capability by managing state data about these executions and their intended target device. This allows the code to be implemented and updated quickly, while maintaining this complex web of requirements. Additions and changes become simply managing the order in which these activities are executed or making updates to the pieces that require an update, without fear of breakage elsewhere.

Finally, it is important to note that when executing a TaskMap a scoping mechanism was designed to allow automations that require cross-device dependencies to function. This mechanism is the “Container” as defined in the Device Metadata section of this document. Containers offer the “highest level” relationship between a set of intended targets. When developing TaskMaps which have these types of cross-device dependencies, those dependency checks are always performed within the scope of their parent Container.

TaskMap Definitions – Filtering on Criteria

When defining tasks for execution in a TaskMap, an optional filtering mechanism has been added to support choosing between items of similar types.

As shown in the figure above, when defining a TaskItem to be run within a TaskMap, the definition may contain a Target ‘type’ of device as well as provide criteria from that TargetItem’s Properties to isolate specific devices. For example, if multiple “Target1” items existed in a Container, Property filters could be used to choose a specific instance of that item that meets only those criteria.

For example, a Vehicle may contain multiple computing units, but a specific computing unit may be required as a target. This method of target definition allows for control of which TargetItems receive assigned tasks.

Task and TaskMap Assignments

When a TaskItem or TaskMap is assigned to a TargetItem or Container for execution, an assignment record is generated in the RPS Database. This assignment record represents that execution request and persists the status of that request so the TaskMap can be coordinated as it is defined.

Tracking/reacting to this status data provides the core mechanism for how task maps operate as well as basic reporting information on whether or not a task was successful.

TaskMapAssignment records are persists a copy of the originating TaskMap and TaskMapDefinition along with the dependencies between definitions for automation sequencing. The TaskAssignment records are used to identify TaskItem and TargetItem execution status. Custom TaskAssignment properties are stored in an item property bag.

References

RPS Task State Diagram

The detailed design diagram for task states and how they are processed can be found here:

RPS > Trunk > Documents > Operations > TaskStateDiagram v1.10.vsd

Note

This document is intended to outline how task states are processed by the controller.

RPS CMDB – ERD Database Diagram

The detailed design diagram for database entity relationships (ER) can be found here:

RPS > Trunk > Documents > Architecture > RPSEntityRelationshipDiagram.vsd

Note

This document is intended to outline how database entities are related to each other.

More Resources

• RPS Software Design
• RPS Configuration Management (DSC) Software Design