The nodlin agent | nodlin.com

What is the nodlin agent?

The nodlin agent will handle specific types of node. It is a logical grouping of node types that are generally coupled in some subject area (e.g. finance, project management etc).

By ‘coupling’ we mean that there is likely to be some dependency between the node types.

For example, a ‘software development’ agent may have a node type ‘project’, ’task’, ‘bug’, ‘feature’ etc. A ‘finance’ agent may have ‘monthly balance’, ‘cost’, ‘revenue’ etc.

Each of these node types will have its own logic and form interface. The logic for a type may refer to details on any connected type. It is likely that there is specific logic applied between types in the same agent.

In the ‘software development’ agent, a ’task’ may refer to a ‘bug’ or a ‘feature’. A ‘bug’ may refer to a ’task’ or a ‘project’. On any changes to a node, those related nodes will re-compute their own state.

This does not mean that the agent is limited to a single type. It is possible that the ‘finance’ agent may have specific logic to process costs associated with a ‘project’ or a ’task’ and therefore you can connect these concepts across your business.

Note that the ‘finance’ agent in the above example may not have to specifically understand a ’task’ node to extract associated cost detail. The finance agent may define that it will extract any ‘adjustment’ property (representing a credit or debit on specific date) from any connected node. Any new type you want to introduce that you require to work with the ‘finance’ agent would just specify this as one of its properties.

Another consideration are the node ‘forms’ (see One Interface for your applications). When building a form it is useful to have buttons (shortcuts) that enable the user to create other related nodes. For example a ‘feature’ in a software project, may have a button to create a ‘bug’ node. You will want the ‘bug’ node type to be available in nodlin.
If the ‘bug’ type is part of the same agent as the ‘feature’ type then this will be successful.

There are no specific limitations on what makes up an agent.

Agents come in 3 forms

Agents can come in 3 forms, internal, external and scripted:

Internal - the agent is built into nodlin.
- These agents provide general utility and other agents can utilise in support.
- Examples are workflow, expression (and chart), note etc.
External - the agent is a separate application
- Agents that are developed outside of nodlin and provide a list of types they can manage.
- These can be secured and access information from internal business systems if required.
- An API is provided to support development of external agents (in golang).
- External business systems can utilise the API to provide additional capabilities. For example if the external application requires to provide some workflow functionality, they can create nodes externally and track responses from nodlin. It therefore provides a flexible way to extend application functionality, allow the user to have a single UI, and save on development costs.
- These can be defined and execute in the nodlin cluster (if containerised).
- The openAI and FRED (federal reserve data) are open source and can be used as examples (and template) to aid in building external agents.
Scripted - the agent is a script that is executed by nodlin
- Execute python like scripts.
- These can be developed as one-of scripts for a node, or named and shared with the wider population.
- These can vary in complexity (see finance script package for complex example).
  - The language allows for definition of forms, node image, logical processing, create and link to new nodes, review associated actions and FYI’s (and create) etc.
  - Internal nodlin API’s can be access (to create images) and create a chart (chartJS API library).
  - The only limitation is that the script can only access detail from itself and related nodes. It cannot access the internet or external API’s

Agents must be stateless

Nodlin can be configured to run as many instances of an agent as required to cater for volume. This enables us to scale horizontally to process volume. Given that multiple instances of the same agent may be running, it is important that they process operations using only the current node state and the state of immediate connected nodes. No state should exist outside of the node detail on record.

External agents may of course maybe recording some state in some operational store that is shared between multiple external instances of the agent. If there is a single instance running of the external agent then this is not an issue. If multiple instances of an external agent are running and sharing the same operational store then this is ok - but consider the scale issues in relation to this.

Note that scripted internal agents only have access to the current recorded node state. There is not mechanism to record state outside of the node detail.

Logic & special considerations

Developing logic to handle a specific nodlin type can be straight forward, but you must consider the following in more complex cases.

A ‘state’ of a node (determined by the logic) is determined from:

the detail input by the user (provided through the node form)
the state of any connected nodes you require for input

Nodlin will request the agent to re-compute when the node is created (so create ’empty), updated (through the user input form), or based on any event raised by a connected node. Events raised are just labels with no payload.

Number of calls to re-compute is non-deterministic

The events raised in nodlin do not carry a payload (it would not work otherwise!).

If you imagine a scenario with a hierarchy of tasks (a work-breakdown-structure, WBA), 100 levels deep, and 100,000 of tasks. If there are constant updates to say a ’task estimate’ that is aggregated at the top level, the node at the top of the WBA will be constantly updated. Nodlin will propagate this change. To manage this, nodlin will de-duplicate the events (which is why we do not allow a payload on an event). An additional ‘update lag’ is applied to ‘hot’ nodes, to capture more events from connected nodes, thereby reducing volume of updates.

The number of times the agent is therefore called to re-compute is non-deterministic.

Use of the ‘clock’ for current time

Due to the potential delay in processing an operation (where a lag is applied to update), the current date/time when used in node logic should be cognisant of this potential for a lag to have applied in update.

Loops

Loops are allowed (and required) as a node can represent a complex data type. In the WBS example, a task may have a sub-task relationship to another task. There is a bi-directional relationship here. The sub-task will want to know the if its parent task is marked as ‘done’, and the parent task will want to know if its sub-tasks are all ‘done’.

The state of the nodes are expected to converge in a finite number of iterations. A limit is set by the administrator of the nodlin domain (default 20) on the number of times a node can be re-computed as a result of a single user operation. If this limit is exceeded the node is identified as ‘blocked’ due a loop and reported. The UI will show the inconsistency between related node versions.

Nodlin is not supposed to be used as a simulation engine (although it does have a limited capability to do so). It is not efficient as a simulation engine. It is expected that if this is required, and external agent could provide the simulation capability and post the result to nodlin for display.

Prefer commutativity & idempotence, and idempotent operations

Nodlin handles versioning of nodes. When a node is requested to re-compute nodlin will provide a payload containing the current state of the node, its version, and payload & versions of any related node it requires for processing. The version of the node being updated must be the same as when this is committed (otherwise the whole operation will re-try), this is a guardrail. The nodlin scheduler will reduce the occurrence of this by not allowing >1 concurrent operations to be in flight for the same node. There are no guarantees on this due to failure in execution where the operation fails to respond with an error status so therefore the version and re-try guardrail.

A node will record the version of the nodes on which it is dependent on the relationship - and if these are different it is visible on the UI (a red edge). This will denote that a node is not-consistent with a related node. This is visible to the user when there are delays in processing operations on related nodes.

Events are just ‘optimisations’

An operation can raise different event labels (not just ‘updated’) events for optimisation purposes. You may want to avoid any processing for specific events you know will not change the result. Say a task is ‘complete’ and informs the parent task. When the parent task processes the operation, the status of the sub-task may have changed (e.g. to not complete). The parent cannot infer that the task is complete, the event just indicates that at some prior point that it was ‘complete’. The payload provided of the current state of the task and sub-task will be visible in the payload provided to the operation.

This allows the operation to ignore events which are not important, but it is possible that ones that are not important may be processed. Events are therefore just optimisations, and any detail used in processing must be based on the payload and not inferred from the event label.

Design to allow duplicate operations

Exactly-once processing is a myth. Your operation may be requested by nodlin to execute >1’ve for the same change to a node through the form, or from a change to a related node. If an agent processes an operation but the response is lost, then the same operation will be re-tried. The resulting state of an operation should be the same (idempotent).

There is no transactionality across nodes

A node update (and any resulting event or action) is transactional. Transactions cannot be applied across nodes.

The first update to a node ‘wins’ where multiple users attempt to update the same node (no clock checking). Locks will be taken out on user edit - but race conditions may still occur.

Warning

The more complex server side version checking and lock function in the scheduler is implemented. The client side request to lock for update is pending.

Avoid hard constraints

You should avoid hard constraints. An constraint is a condition that must be true for the agent to be able to process a node.
For example, say you wanted a ’task’ node and you wanted to avoid over allocation to a particular member of a team.

You can have these as soft constraints. For example a warning on a node for display and can compute a delta of an over assigned effort. This can be used to inform the user. However, the constraint cannot be enforced.

The sub-task nodes are independent transactions, and when netted at the parent you may find that the net of both tasks lead to an overallocation of a resource. This overallocation is on record (both sub-tasks have been saved). The operation processing the parent task should warn and track the overallocation, but cannot fail the operation.

Where this is a concern, an external agent can be used to enforce a constraint. An external agent could have its own transactional data store and not allow an update to a node if the constraint is not met. This type of constraint however cannot be enforced internally by nodlin (or by scripted logic).