Why Nodlin?
Nodlin provides the high level user interface that can cover all aspects of managing your business.
Does your organisation look like this?
_diagram a mix of systems and people in an organisaation - with email, chat, meetings being the glue
- transciption & mutation…
- need to establish best practice (SWOT, Objective hierarchies etc)
- information overload
- email and chat are ‘catchall’ tools
- expensive development costs in integrating disparate systems, or
- human capital costs in import/export, translation
- basing decisions on inconsistent and timely information
- errors in translation
Where are we?
Applications are designed around processing information efficiently and maintaining consistency.
Internally, an application has a data model (’entities’ in technical terms) in which this information is recorded. In cases where information is not ‘structured’ you can provide commentary, attachments etc.
The above example shows entities that are recorded in different applications that are typical in an IT organisation.
Applications tend to grow over time and add new features, to bring in other entities to track and provide consistency. Systems tend to overlap over time, they either directly record the same detail, or an entity is a translation of some other entity in another system.
There is always a boundary to an application. In order to improve efficiencies we tend to provide import/export capabilities, or directly connect the systems using technology. Where this is done, inconsistencies in the data can arise, latency, and questions regarding the source of truth.
Where systems are not connected, people intervene and transcribe detail using unstructured tools such as email, chat, attachments etc that all undermine the consistency of data across the organisation.
Significant cost has been spent to integrate and provide tooling to support bring together disparate data elements to provide more structure (Microsoft loop, Notion etc) although these all have limitations (and boundaries), and result in overlapping capabilities.
This results in significant costs and risks to a business:
| Problem | Impact |
|---|---|
| Information is translated & summarised between different applications | human cost in translation, inconsistency, errors in translation, a lag in updates, decisions based on incomplete or bad data |
| Continual new features to applications to encompass more details in order to improve consistency | overlapping capabilities between applications, identifying the source of truth |
| Unstructured tools used to transfer & record detail (e.g. email, chat, word, spreadsheet) | information overload, duplication, difficulty in seeing the big picture, high storage requirements, data inconsistencies |
To repeat, applications are great at maintaining consistency efficiently. Where we have high volumes of data (say millions of transactions) application efficiency and consistency is essential.
In many cases where users are involved however, eventual consistency is OK.
{{% callout note %}} Eventual consistency means that those related data elements may not be immediately up to date with related detail, but will eventually be correct. {{% /callout %}}
{{% callout note %}} Example record of tech spend
All IT projects in a large organisation will rollup to a single ‘budget’ for technology. A project will consist of a large hierarchy of tasks (a typical work-breakdown-structure, or WBS). Time spent at the bottom of this structure will be recorded as a cost, and will rollup to a teams cost, which will result in a groups cost, division cost, up to the ultimate firm technology budget.
The numbers typically take time to flow up the organisation. Typically there will be a sequence of weekly status reports, summarised to monthly and say quarterly or annually at the top of the organisation. The costs of executing these steps is high, accuracy difficult to gauge, and there is a huge time lag on visibility.
This is an example where we accept the ’eventual consistency’ of detail. {{% /callout %}}
Organisations are a network of systems and people. We utilise too many unstructured communication tools, information overload, and live with the lack of visibility where people are involved to relate disparate details.
put in some details on emails sent daily, chat messages etc, red flag
To tackle the problems in the human communication flows, we establish Standard Operating Procedures, and document workflows to ensure that people follow defined practice. These SOPs are like the computer code but for humans! These practices fill in the automation gaps. Although these are very flexible, they do however introduce issues in audit, ensuring compliance and delays.
How can we improve this?
Where do we want to be?
Organisations are a network of systems and people. We need a model where:
- we can handle the large volumes of transactional data efficiently and consistently, and
- we handle the lower volume ‘human’ interactions in a more structured, auditable and efficient manner
Resolving the first issue is down to established software engineering practice. This continues to improve, although there are significant challenges in dealing with the consistency of detail between disparate transactional stores where human involvement is required to cleanup the gaps. the size of an organisation is based on the how well the large transactional software applications handle these issues (e.g. operations departments in banks). Note the risk in that where operations are sized to correct the gaps - if the software goes wrong, they cannot correct it.
note: I can refer to a link to another note: how organisations fail in using persistent queues to share state. Events are history, and people fill in the gaps. SAGA patterns to undo etc - if we want to automate - we need state brokers. I can post this to the tech group channels, and medium. A link back to nodlin will support interest…
idea: have a diagram of systems and people. systems speaking to each other. people being the informal…. sometimes with SOPs. Different levels of experience and skill with the people. We are however trying to get the whole organisation working as one to an established principle and objective. Informal communication channels are email and chat. Little formal tools exist.
The second presents challenges in approach. Its quick to establish and update SOPs to reflect gaps (although application of these may take time and training). We need an approach where:
- information requirements can be easily defined,
- information can be related to other information to maintain consistency, and
- the behaviour (or logic/code) can be defined to maintain consistency of this information
We need something like a spreadsheet, that operates on a multi-user organisational scale, that works with data as opposed to numbers!
In an ideal world all this information would be immediately consistent. This would bring in the complexities that the traditional large volume transactional data stores handle. We are currently OK with eventual consistency as employees are involved in the flow. With this acceptable limitation, Nodlin provides a solution these 3 challenges.
How Nodlin resolves the problem
Nodlin models the world as one large network dependency graph. The nodes in the graph represent the data elements, and these nodes can change in relation to each other.
The following example represents key data elements that would typically exist in an IT organisation:
This information would typically be maintained across different applications. The business may state objectives (typically in an unstructured format, powerpoint or document). Meeting these objectives would lead to software requirements, projects and work-breakdown-structures (tasks). This work has a cost impacting budgets. Just for example the diagram introduces entities typical in incident management, leading to bug reports, requests and additional work. The diagram shows these data elements as nodes - with dependent relationships (the edges).
The application that focusses on managing a set of data elements (e.g jira handling projects, tasks and stories) no longer exists.
The logic that is applied in handling a task, or project, or objective etc is defined with the data element itself. This logic can reference details in its related nodes. For example, a ’task’ node can review any related sub task nodes to check if they are complete, assigned effort and so forth.
When a node is updated, Nodlin will ensure that any related nodes are also updated. This progresses until all updates have been completed. The one large network dependency graph will eventually be consistent.
Revisiting each of our requirements in turn:
1. information requirements can be easily defined
Nodlin allows for flexible definition of detail for a node. These can be defined in simple python like scripts, or provided by other external programs that use the Nodlin interface. The user input forms are also data driven (JSON) that describe the input fields required.
2. information can be related to other information to maintain consistency
The user can add nodes to the graph, or a node can add other related nodes automatically. Nodlin will ensure that changes get propogated to other related nodes for assessment. Nodlin will control the update processing until the graph network is consistent.
3. the behaviour (or logic/code) can be defined to maintain consistency of this information
Nodlin has the concept of a node Type where this code (behaviour) is defined. For example a Task Type will define the logic for handling the Task detail. The definition of this Type can include what other nodes it can be related to, what Actions can be performed, and what Events it it interested in. As with point 1, this behaviour can be defined in simple python like scripts or in external programs. For a more detailed description on how this logic is written and how this is different from an Application, see BBBBBBBBB.
As noted at the start, Nodlin is only one large network graph. There will be many nodes, clusters etc in this graph and handling the updates to ensure consistency is at the heart of Nodlin. More detail on this is BBBBBB.
Obviously viewing or managing the organisation as 1 network graph is not sensible or technically feasible. A Nodlin
(derived from ‘Nodes and Lines’!) is used as a term to reflect a particular perspective onto this one large network graph.
A Node does not exist IN a Nodlin (just a reference).
A Nodlin is simply a container of references to Nodes in this large network. A Node may be referenced in many different Nodlins. A manager may be looking at the project nodes across an organisation, and developers may be looking at 1 of these single project nodes and its related task nodes.
A Nodlin, apart from containing a simple contextual view of relevant nodes, can also present this information in a variety of ways. It can present as a graph, a board (aka Kanban style), an infographic style, a gantt chart, or even add to a free form excalidraw view.
Other Opportunities
Nodlin is a general framework for displaying detail in different contexts and keeping this detail up to date. This detail is visible in real-time to all participants involved. Individuals can focus on particular data points (nodes) and contextualise this information as appropriate from their perspective.
Aside from the technical aspects of managing these disparate pieces of information, viewing this detail in a graph can be a more natural way of presenting information in ‘context’. Mind maps are often utilised to help focus and organise work. Nodlins ‘graph’ view is the primary contextual view and provides a better perspective than a document. Humans have the ability to better recognise and work with this contextual style of information.
Various ‘chat’ tools (in teams, Slack, ….) are ‘streams’ of conciousness, with no controlled organisation. These tools have their place, but as a catchall for supporting non-structured communication they can become overbearing. ‘Tags’ can be placed in these streams to linkup different parts of the conversation, but this is not an efficient method of tackling the underlying problem.
Example: ‘chat’ becoming unmanagable
In one example I was involved in an outage, where the impacts across the organisation became quite significant with multiple groups involved. Ultimately we had to create different chat ‘streams’ to coordinate the different activities of the different groups involved. Investigating potential causes and impacts, whilst others trying to report on the overall status and resolution of the problem. Trying to assess ‘where we were at’ became a long exercise of trawling multiple chat streams.
Structuring this analysis (with detail pertinent to types: event, possible cause, investigation, impact etc) can be structured and this detail linked and presented in an overall view or ‘map’ of resolution.
These features of Nodlin can therefore be applied to support a wide range of problems:
| Problem | How can Nodlin help? |
|---|---|
| Standardizing workflows | Nodlin defines standardised Node types that can constitute a workflow (task order, decision points, approval etc). These can be utilised functionally by other nodes and applied to ensure audited and transparent steps. Automate your SOP’s! |
| Reducing email and chat | Email and chat are the ‘catchall’ for informal communication where automated solutions do not exist. Define appropriate node types to capture detail that can be standardised. In addition this removes the duplication (no more ‘forwarding’) and tagging emails for a ‘response’. |
| Reduce information overload | Both chat and email can get out of control. Having a single, structured source of detail, where Actions and FYI’s can be assigned for attention, will focus the attention on what is important. The Nodlin ‘graph’ view is also a more contextualised view of information as opposed to the ‘stream of consciousness’ in chat streams. |
| How to utilise an appropriate AI | AI is a fast growing area with different many models having different capabilities and requirement to augment different informational flows. Nodlin provides a simple example (an external Agent) that utilises OpenAI. This can be used as a template for an organisation to define other capabilities and integrate into Nodlin workflows. |
| Reducing tech cost$ with multiple web based portals | Tech will often introduce bespoke portals to support functions (e.g. requests for action). Nodlin provides a simpler route to introduce new capabilities without costly development. Users also have a one-stop tool for utilising these capabilities. |
| Crafting a document | Mind maps are a technique applied to help structure and support a document. A document is a graph, albeit constrained to a tree, and there is an order to the sections. Nodlin presents this in the ‘book’ view context. This is easier to work with compared to the summary ‘page’ view often provided. |
| Organisational knowledge management | If an organisation has many datapoints across multiple systems, making this information available to Nodlin will enable the user to investigate and link required detail. External systems can provide a nodlin interface and present detail as a node from which users can include and expand for further analysis. |
| Too much powerpoint, visio etc. | Information that is placed in a powerpoint or Visio document, or any document, becomes stale. Documents represent a ‘snaphot’ of information and thinking at a point in time. This may be what is required and has its uses. For status reports (where it has an operational use) it is questionable. Nodlin allows for the presentation of the latest available detail. |
| Slow transactional systems with high cost | Systems that handle data that can be eventually consistent but in a transactional fashion will perform slower from the users perspective. Taking Jira as an example, this catch-all task/epic/resource tool has a very extensive data model and user perceived performance can be slow in high volume settings. A node payload in Nodlin is generally small, and can be processed in realtime. The impact of this across other nodes may ultimately take longer (to become globally consistent), but the initial responses will be negligible. |
Obviously managing the 1 network graph has complexities….See for more detail
How do we ensure everything gets updated?
Note that there is 1 graph…
The logic or behaviour around a data element, is encoded on the node itself.
When detail changes in a node, the logic is applied on those related nodes. In this way updates will flow throughout the larger network graph, eventually enabling all data elements to be consistent.
If we can accept eventual consistency, then we do not need to be bound by the application constraint (to ensure consistency).
We need a method of enabling different pieces of information (aka entities), to be related, and for these related pieces of information to be eventually consistent.
- formalise the handling of exceptions between these systems
The larger the organisation, the more important it is to
Ultimately we should be looking to reduce the unstructured email and chat communications.
There are always exceptions…
In an ideal world we
We need to utilise a model for software where disparate detail can be
We require a tool that is flexible in being able to:
- provide
Is this possible?
As we formalise and structure these communication patterns…
Can these problems be resolved?
software implemented import/export, OLE, etc. technology integrate systems
Tools like EXCEL are great, used for lots of business cases. Changing data in a cell changes all the related cells.
Excel however is a personal tool. They have limited capabilities to share. Shared filesystems are generally used, and at worst sent as attachments in email.
Woundn’t it be great if you could relate your data, and have this update other related information?
Applications are limited
….they manage the consistency across various data points, but always a boundary, and we fall back into import/export.
How Nodlin works
All the functionality (behaviour) is
“Its like the power of the spreadsheet but for data”
These small programs are scheduled by Nodlin. A change to one, will immediately schedule those connected nodes…
Unlike excel the results will not be immediately consistent (or you have to wait for a re-calculation)….
Nodlin will never be as fast as excel…. but….
This all happens extremely fast…
Focus on…
Big systems that maintain consistency of data are required.
User based systems however are quite different, this realtime consistency is not always required.
Applying this big system consistency philosophy to user based apps presents issues. Users operate across boundaries of applications….
context picture of app…. getting bigger to control the whole organisation? - is this the target state? OR, maintain the logic around the small data elements themselves, and connect them up?
