Defining and Designing a Data Strategy
In our previous article, we explored the topic of explainability and why understanding a
model’s decisions matters. We discussed two key qualities of a good explanation:
- Plausibility: the explanation should make intuitive sense to a human reader.
- Faithfulness: the explanation should accurately reflect how the model actually arrived
at its decision.
Explanations give us important insights into how well the model was trained. If we notice that there’s a mismatch between the model’s predictions and the input information that it uses, or if the model’s reasoning process seems counterintuitive despite a correct answer, then there’s a good chance that there’s something wrong with the model.
But what happens when the problem isn’t the model at all? What if the model’s predictions are confusing, not because there was a flaw in the way we trained the model or how we’re explaining it, but because the data the model learned from was already inconsistent? What if the model isn’t malfunctioning, but merely exposing a flaw in the information we gave it?
And what if this inconsistent data is all we have?
The Plight of Inconsistency
Inconsistency is a natural part of almost every organization's data. Businesses evolve,
processes change, regulations are updated, and customer expectations shift. As a result, the information we store today may not perfectly match the information we stored a year ago.
Consider a simple example: a company updates its customer support process every few
months. The current procedure might require support agents to verify a customer's identity before discussing account details. An older version of the process might not include that step at all.
Neither document is necessarily “wrong”, they simply reflect the truth at different points in
time. This kind of situation is especially common with Standard Operating Procedures
(SOPs). SOPs are living documents that change as organizations adapt to new requirements and improve their workflows. New versions are created, while older versions are archived for compliance, auditing, or historical reference. Over time, organizations can accumulate hundreds or even thousands of versions of similar documents.
Now imagine training a large language model on this collection of documents. The model
may encounter ten different versions of a process called “Customer Contact Procedure.” Some versions contain additional steps. Others reorganize the workflow. A few may even contain instructions that have since been deprecated.
Unlike a human employee, the model does not automatically know which version is the
current one. It simply learns patterns from everything it sees. As a result, the model may
develop an inconsistent understanding of the organization's procedures. When asked how a process works, it might combine information from multiple versions or produce answers that are partially outdated.
At that point, the inconsistency is no longer confined to the data. It has become embedded in the model itself. And unfortunately, there’s only so much that can be done to optimize the model architecture or training to prevent it from learning the inconsistent signals, since these signals are an integral part of the data itself.
The Temptation of Data Cleanup
A reasonable solution to this problem is to try to eliminate inconsistencies in the data before the model ever sees it.
Organizations can maintain strict version control, carefully documenting every change and ensuring that only the latest approved procedures are included in training datasets. In theory, this would successfully avoid inconsistent data making it into the model’s training dataset. In practice, however, things are rarely that simple.
First of all, maintaining detailed documentation introduces additional administrative
overhead, as well as additional data. We’re not just creating a new document to replace an
older one anymore, we’re also creating a document that explains all the changes we made, and this document might also have a predecessor that we need to keep track of. The more processes we have, and the more changes they undergo, the more difficult it will be to store and analyze this data.
But more importantly, it might not be a good idea to throw out historical versions of
documents. These older versions often contain valuable information that organizations still need. For example:
-
How has a process evolved over the past three years?
-
Which version of a procedure was active when a particular event occurred?
-
Was an employee following the current process, or an earlier version?
These questions require access to historical knowledge, not just the latest version. Discarding older documents may improve our data’s consistency, but it also risks losing context that could be important later.
So, it’s clear that hanging on to older, inconsistent documents is good for helping humans understand their data. But what if our goal is to build an automated system that can understand and process this data, similarly to how a human would?
We already know that the secret isn’t to simply feed an LLM all the data and hope that it will resolve the inconsistencies on its own (it won’t!) But there are many potential ways of preparing inconsistent data for a machine learning system, and there’s no single “correct” method to choose. Rather, the best method depends on how we want the human and the LLM to interact.
The Zen of Letting the User Observe
At Spryfox, we’re building a system that not only detects inconsistencies in process documents, but also helps users discover those inconsistencies for themselves.
We begin with a simple question: What happens when a user describes a real-world situation and wants to know whether it complies with company procedures?
Our system first retrieves a collection of relevant process documents using a Retrieval-Augmented Generation (RAG) pipeline. Importantly, we do not restrict retrieval to only the latest versions of a process. Instead, the system can surface both current and historical versions, allowing users to see how procedures have evolved over time.
The next step is compliance assessment. For each retrieved process, a specialized compliance checker model evaluates whether the user’s situation complies with that version of the procedure.
To ensure that the model itself is not biased by inconsistent process definitions, we trained it on a carefully curated dataset of situation-process pairs and corresponding compliance judgments. Importantly, the training data contained only a single authoritative version of each process. In addition, the domains represented in the training data do not overlap with those used during evaluation. This allows us to assess whether the model can generalize compliance reasoning rather than simply memorizing specific procedures.
The result is a system that produces a separate compliance judgment for every retrieved process version.
This design choice is deliberate. Rather than deciding which version is “correct” on behalf of the user, we first expose the underlying reality of the data. Users can see that multiple versions of a process exist, understand how they differ, and observe how their situation relates to each version.
Before treating inconsistency as a problem to be solved, we treat it as information that deserves to be understood.
This becomes particularly important when process changes are subtle. Small wording changes can have surprisingly large consequences.
Imagine two versions of a customer onboarding procedure. Both contain a step requiring identity verification. However, the newer version adds a short clause requiring verification to be completed before account access is granted. A user’s situation may comply with the older version while failing the newer one, even though the two procedures appear almost identical at first glance.
In our system, users can drill down further and inspect which individual process steps contributed to a compliance judgment. This makes it easier to identify precisely where differences between versions begin to matter.
Using Simple NLP to Surface Differences
Interestingly, some of the most useful tools for this task are not large language models at all.
One example is TF-IDF, short for term frequency-inverse document frequency. The underlying idea is simple: identify which words are unusually important within a document compared to the rest of the collection.
Words such as “the”, “and”, or “process” appear almost everywhere and therefore carry little distinguishing information. In contrast, words such as “verification”, “approval”, or “escalation” may occur only in specific procedures and therefore tell us much more about what makes a process unique.
Suppose we compare two versions of a procedure:
-
Version 1: “Contact the customer and record the issue.”
-
Version 2: “Contact the customer, verify their identity, and record the issue.”
TF-IDF would highlight terms such as verify and identity because they are distinctive additions in the second version. If a user's situation includes identity verification, those terms may help explain why the situation complies with Version 2 but not Version 1.
The strength of methods like TF-IDF is that they provide a transparent view of the data itself. They help users understand where documents differ and which concepts appear most important within those differences.
However, there is an important limitation.
From Discovery to Model Explanation
While techniques such as TF-IDF can reveal inconsistencies within process documents, they do not explain how the compliance checker arrived at its decision.
The compliance checker was trained on its own dataset and learned its own internal representations. There is no guarantee that the words highlighted by TF-IDF are the same pieces of information the model relied upon when making a compliance judgment.
In other words, we may have an explanation of the data without having an explanation of the model.
This led us to an interesting question:
Can we obtain an explanation from the compliance checker itself while simultaneously learning something about the inconsistencies between process versions?
In our previous article on explainability, we argued that text-based explanations alone are often insufficient. Simply asking a model to explain its reasoning in natural language does not necessarily tell us how the decision was actually made. To obtain more faithful explanations, we need to look inside the model.
One place to start is with the model's internal scores, often referred to as logits.
What are Logits?
When a language model produces an output, it does not immediately choose a word. Instead, it first assigns a numerical score to every possible token in its vocabulary. These scores are called logits.
A logit is not yet a probability. Rather, it is an unnormalized preference score. The model later transforms all logits into probabilities using a mathematical operation called softmax. Tokens with higher logits receive higher probabilities and are therefore more likely to be selected.
For example, imagine a model is considering how to complete the sentence:
The customer must first ______.
The model might internally assign high logits to words such as “verify”, “contact”, and “authenticate”, while assigning much lower logits to unrelated words. After these scores are converted into probabilities, the most likely token is chosen as the output.
The interesting question is whether these internal scores contain clues about the model's reasoning process.
Suppose the word “contact” appears primarily in Step 1 of a procedure, while the word “archive” appears in Step 5. If the model consistently assigns greater importance to tokens associated with Step 1 when making compliance decisions, could that tell us which parts of the procedure influenced the model most strongly?
This is an appealing idea, but the answer remains unclear.
Why This is Difficult
The first challenge is that logits are not automatically explanations. A high logit tells us that the model prefers a particular output token, but it does not tell us why that preference exists. Additionally, it’s not clear whether there’s a meaningful ordering or pattern present when we look at the model’s “top ten” or “top fifty” ranked token preferences.
Let’s say that for a particular data sample containing a process and a user scenario, the model assigns the highest logit to the token “Yes”, indicating a positive compliance judgment. Now let’s say that the model also assigns a similarly high logit to the token “contact” in Step 1 of the sample process. Does this mean that the model considers Step 1 to be important to its prediction (the “Yes” token)? Currently, there is no scientific consensus about this.
The second challenge is that processes are not composed of isolated words. A single process step will often span multiple sentences, with meaning emerging from the interaction of many tokens rather than any individual one. As a result, examining the logit associated with a single token may tell us very little about how the model evaluated an entire process step.
What we ultimately need are methods that connect model logits, confidence scores, and human-understandable process components in a meaningful way.
Looking Toward Calibration
Questions like these are closely related to the field of model calibration.
Broadly speaking, calibration studies the relationship between a model's internal confidence and its actual performance. A well-calibrated model should be uncertain when a task is difficult and confident when it is likely to be correct. In other words, when the model outputs a correct token, we want the logit of that token to be higher than when a model outputs an incorrect token.
In theory, a well-calibrated model’s internal ranking of possible output tokens will reflect how important or relevant the model believes those tokens are to its output (the token with the highest logit). For example, if we saw that the model’s top choice was “Yes”, and we saw that the other tokens within the top five or ten all have something to do with Step 1 of the input process, then this might tell us that the model considers Step 1 to be important.
But this is a theory that must first be tested. At Spryfox, we are exploring whether calibration techniques can help bridge the gap between logits and explanation: Can we find meaningful correlations between the ordering of the model’s token preferences and the specific process steps it relied on? Is there a way to optimize the model’s training such that its logits are more reflective of this information? And can we do so in a way that remains faithful to the model’s actual behavior?
These questions are what guide us on the way to finding a reliable, model-internal way to explain the compliance checker model. In the case of inconsistent data, this will help us do a fine-grained analysis of individual process steps that will shed light on which version of a step the model is using.
This will bring us closer to transforming data inconsistency from a source of confusion to a source of insight.
Outlook: Fixing an Inconsistent Model from Within
Currently, our end goal is to have a model that makes consistent compliance judgments, based on the most recent, correct version of a process. When a potential customer comes along, we want to be able to train such a model for them, despite any data inconsistencies they may have.
But what if we encounter a different scenario: What if the customer already has a model, but that model was trained on a version of their data that still contained inconsistencies? And what if it’s too expensive to retrain this model, both in terms of money as well as available compute resources? This is a situation that often comes up in industry settings, particularly for small organizations: A company has made a large investment in training a model, but once that model turns out to be flawed, it’s impossible to start again from scratch.
What can we do in this case? Can we somehow fix an already existing model, perhaps surgically “edit out” the outdated knowledge without affecting the rest of it?
Fortunately, there is a growing body of research in exactly this direction, aptly named “model editing”. Recent works in this area have found that it is indeed possible to alter specific facts in a language model’s knowledge, such as the city where the Eiffel Tower is located 1.
Moreover, it has been demonstrated that this can be done at scale, with large amounts of facts2.
However, no works have yet applied this to the case of process knowledge, which is bound to be more complex, both due to the complex linguistic structure of business processes, as well as the many different versions of a process that might exist.
Additionally, recent works have cautioned about a pitfall of model editing, namely that even simple edits might cause the model’s overall performance to deteriorate3. It’s clear that model editing is a balancing act: finding a method that is thorough enough to remove all traces of outdated knowledge, while at the same time precise enough to preserve everything else.
In our future posts, we will go deeper into the details of how we’re developing model editing methods at Spryfox.
Checklist for Inconsistent Data
As we've seen, the goal isn’t to prevent data inconsistencies altogether (doing so would be unrealistic). Processes change, requirements shift, and new knowledge replaces old knowledge. The challenge isn’t to eliminate these differences, but to manage them in a way that preserves their context.
When data is organized thoughtfully, both humans and machine learning systems can understand which version of a process applies, how it has changed over time, and why those changes occurred. In other words, data inconsistency doesn't have to become model inconsistency.
Citations
[1] Meng et al., 2022a. “Locating and Editing Factual Associations in GPT.” NeurIps 2022.
[2] Meng et al., 2022b. “Mass-Editing Memory in a Transformer”. ICLR, 2022.
[3] Yang et al., 2024. “The Fall of ROME: Understanding the Collapse of LLMs in Model
Editing.” EMNLP 2024.
