A brownfield integration can look successful long before its data is safe to use. A Modbus register responds, an OPC UA subscription produces values, and a legacy export reaches the platform. Those facts prove that bytes crossed the boundary. They do not prove that a value belongs to the right asset, uses the expected engineering unit, represents source time, or carries a quality state that permits it to drive alarms and analytics.
This article examines a reproducible lab case rather than claiming a customer production result. We created 20 deliberately dirty telemetry records from three expected source contracts and one unknown source, then ran a deterministic audit over identity, unit, range, time, cadence, and quality. Eight records were accepted and 12 were quarantined. The 40% acceptance ratio is a property of this synthetic fixture, not a defect rate for real factories. The transferable conclusion is that brownfield programs need separate gates for “collected” and “consumable,” with provenance and replay preserved whenever a record fails.
1. The constraint: three connected interfaces without one shared contract
The fixture compresses a familiar retrofit into three feeds. Compressor discharge pressure comes from a Modbus register, oven temperature comes from an OPC UA node, and panel energy arrives through a minute-level legacy export. Their update intervals, clock ownership, and quality semantics differ. A raw Modbus value usually does not carry its engineering unit or acquisition time. An OPC UA DataValue can carry a StatusCode, sourceTimestamp, and serverTimestamp. A legacy file may expose only an operator-maintained column name and export time.
If an ingest API requires only value + timestamp, those differences disappear at the first normalization step. The data is not truly unified; its evidence has been erased. A pressure value of 6.9 may mean bar or kPa. An oven timestamp may come from the PLC, the gateway, or the platform. A familiar point name may have been copied to the wrong machine. Once that ambiguity reaches a state store, an energy report, or an alarm rule, downstream consumers can no longer reconstruct what happened.
The experiment therefore defines a minimum contract for each source: stable source_key, expected asset_id, canonical metric, engineering unit, plausible range, and sampling interval. The contract does not model the entire plant. It only decides whether a record has enough evidence to enter a trusted state layer. For the larger system boundary, see Layering the Industrial Telemetry Pipeline. For the identity and semantic contract behind industrial tags, see Industrial Tag Modeling and Governance.
2. Audit result: 12 of 20 records require quarantine, not silent repair
The local script reads the JSONL fixture line by line and evaluates nine issue types. It accepted eight records and quarantined 12. The quarantined records produced 14 issue observations because one record can be both missing and non-good, while another can have both an unexpected unit and an out-of-range value.
| Check | Count | Why silent repair is unsafe |
|---|---|---|
| Missing value | 2 | null may mean a communication failure, shutdown, or missing export column; zero is not a neutral substitute |
| Unit mismatch | 2 | Converting bar, kPa, Fahrenheit, or Celsius requires source and version evidence |
| Range violation | 2 | The value may be a real process exception or a mapping/unit error |
| Cadence gap | 2 | Gaps change window statistics, rates of change, and event order |
| Non-good quality | 2 | A numeric payload is not automatically usable for control or alarms |
| Clock skew | 1 | Gateway receipt time cannot answer source-event ordering |
| Asset mapping mismatch | 1 | A valid value on the wrong asset is harder to detect than a missing value |
| Duplicate timestamp | 1 | Idempotency and conflict rules are required; arrival order is not enough |
| Unknown source | 1 | An unapproved point must not become a production tag automatically |
The correct lesson is not to add as many correction rules as possible. If ingest silently multiplies every bar value by 100, fills missing samples with the previous value, and binds unknown points through fuzzy name matching, dashboards become more complete while provenance becomes weaker. Brownfield governance should preserve the raw record and failure reason, then make correction a versioned and reviewable decision.
The inline visual treats source, unit, time, quality, and asset mapping as separate evidence, instead of hiding rejected records behind a single score.
3. Root-cause review: defects spread through identity, value, and time
3.1 Identity defects can place a correct value on the wrong asset
One pressure record retains the known modbus-1/40001 source but changes its asset from compressor-07 to compressor-08. Its value, unit, and range all appear normal. A platform that trusts the target asset field would contaminate another compressor's history without producing an obvious gap. This failure commonly appears after point-list copying, PLC address changes, gateway-template reuse, or asset renaming.
Source address, canonical metric, and asset binding must remain separate, versioned facts. Discovery can propose a candidate mapping; it should not overwrite a confirmed binding. When the two disagree, the system should retain the original source_key and quarantine the record. That policy creates manual work, but it prevents a plausible value from silently rewriting another asset's state.
3.2 Unit and range checks must be evaluated together
The fixture includes 6.9 bar for pressure and 363.5 F for temperature. Both are mathematically convertible, but the platform cannot tell whether the device changed configuration, the gateway selected a different template, or an operator only edited a column header. A unit change without a mapping version and activation time makes historical continuity impossible to explain.
A range violation also cannot decide the final truth on its own. The 356 C oven value exceeds the fixture contract's 300 C ceiling. It may be a sensor fault, a real process excursion, or a unit/configuration error. The quality gate may prevent it from advancing current trusted state, but it should not delete the raw observation. Separating a raw stream, quarantine stream, and trusted state layer lets the platform preserve facts while controlling what downstream consumers treat as valid.
3.3 Time quality controls windows, ordering, and causal interpretation
One oven record has 32 seconds of difference between source and gateway timestamps. Another repeats the same source timestamp. Thirty-two seconds may be irrelevant for daily aggregation, but it can reverse event order in a ten-second stream, change a rate-of-change calculation, or distort alarm duration. A single timestamp field cannot express source time, receipt time, processing time, and clock ownership.
OPC UA separates sourceTimestamp, serverTimestamp, and StatusCode because a value needs usability and time context. Sparkplug requires UTC timestamps on data metrics and defines metric name, alias, and datatype through BIRTH messages. Flattening those fields into a float and a server timestamp discards quality evidence that the protocol already supplied.
4. Implementation: put the quality gate before trusted state
Quality evaluation must happen before a record advances device state, alarms, reports, or model features. Raw observations first enter a replayable path. The active contract evaluates each observation. Passing records can advance trusted state and downstream data products; failing records enter quarantine with the rule version, reason codes, and raw payload intact. This sequence allows a rule change to replay history without pretending that the original release was correct.
flowchart LR
A("Raw observation"):::blue --> B("Quality observation"):::cyan
B -->|Contract passes| C("Trusted state and data products"):::green
B -->|Identity, unit, time, or quality fails| D("Quarantine stream"):::orange
D --> E("Owner confirms root cause"):::violet
E --> F("Versioned mapping or rule"):::slate
F --> G("Shadow replay and diff"):::cyan
G -->|Pass| H("Approved release"):::green
G -->|Still uncertain| D
H --> B
classDef blue fill:#EAF4FF,stroke:#3B82F6,color:#16324F,stroke-width:2px;
classDef cyan fill:#E9FBF8,stroke:#14B8A6,color:#134E4A,stroke-width:2px;
classDef orange fill:#FFF3E8,stroke:#F08A24,color:#7C3F00,stroke-width:2px;
classDef violet fill:#F4EDFF,stroke:#8B5CF6,color:#4C1D95,stroke-width:2px;
classDef green fill:#ECFDF3,stroke:#22C55E,color:#14532D,stroke-width:2px;
classDef slate fill:#F8FAFC,stroke:#64748B,color:#1F2937,stroke-width:2px;
Automatic correction is appropriate only when the evidence is explicit and the operation is reversible. If a source contract states that a register always uses a 0.1 kPa scale and the versioned transform has been validated, the edge can apply it automatically. If the unit change came from an operator-maintained spreadsheet, the system lacks enough evidence. A transformation that cannot explain its source and activation time should remain quarantined even when its output looks plausible.
5. Ownership and metrics: a quality score is not an action model
A dashboard saying “data quality: 92” cannot identify who must act. Operational governance separates quality dimensions and owners. The connectivity team owns receipt success and transport delay. A process or equipment owner confirms engineering units and valid ranges. Asset governance owns source-to-asset mappings. The platform team owns rule versions, quarantine backlog, replay reproducibility, and release evidence. Each issue type closes with different proof.
Useful launch metrics include acceptance ratio, quarantine ratio, unknown-source count, mapping conflicts, unit conflicts, source-time skew percentiles, overdue tags, and quarantine age. Acceptance ratio cannot stand alone. A team can make it rise by disabling rules. A safer view combines closure time by reason, the diff produced after replay, and whether dependent alarms and reports pass acceptance again.
Alarm consumers should not treat every numeric payload as process truth. A non-good, overdue, or mapping-pending record can create a data-quality event, but it should not share the same lifecycle as a confirmed process alarm. The separation between event facts, alarm episodes, and operator workflow is discussed in Why Alarms and Events Should Be Modeled Separately.
6. What the experiment proves—and what it does not
Repeated execution produces the same result: 20 inputs, eight accepted records, 12 quarantined records, and deterministic counts for nine checks. This shows that the minimum contract blocks the fixture's known untrustworthy records from trusted state. It also demonstrates why one record may fail multiple quality dimensions. The report creates a regression baseline: after changing a rule, reviewers can compare the quarantine set rather than accepting “the script ran” as evidence.
The experiment does not connect to a physical PLC, gateway, broker, or time-series database. It does not simulate network loss, device reboot, backfill bursts, or hot rule deployment. The 40% acceptance ratio is not a production defect rate. The five-second skew threshold and 1.5-times cadence threshold are not universal standards. A real deployment must derive limits from process windows, alarm semantics, acquisition mode, and consumer SLOs, then measure false quarantine and missed defects under shadow traffic.
Production acceptance must also separate “the rule detects the fixture” from “the system continuously governs field data.” Deterministic tests establish the first claim. The second requires operational evidence: quarantine backlog after rule changes, replay duration, owner closure time, and the difference in alarms and reports before and after repair. Without that evidence, a passing test can authorize shadow validation, but not full data-product release.
ISO 8000-61 treats data-quality management as an ongoing process rather than a one-time cleaning task. NIST guidance on collecting, curating, and reusing manufacturing data likewise emphasizes context, standards, and repeatability. For a brownfield program, “all devices produce data” is therefore a weak completion criterion. A stronger one is that critical data products have explicit contracts, owners, quarantine paths, replay evidence, and agreed quality SLOs.
7. When edge rules are sufficient—and when platform governance is required
For a small, stable installation where units and mappings rarely change, data supports only local display, and one team owns every failure, explicit edge validation plus raw logs and a controlled point list may be enough. Adding a central rule registry, quarantine service, and approval workflow to a closed system with a few dozen points can create more operational cost than value.
When the same source feeds alarms, energy reporting, predictive maintenance, and cross-site analytics—or when mapping, units, firmware, and sampling policies evolve—quality rules become a platform contract. Without versions, ownership, and replay, every downstream team cleans the same data differently and creates incompatible truths. The purpose of platform governance is not another quality dashboard. It is to make one correction explainable, testable, releasable, and reversible.
Conclusion: define consumable before scaling collectable
The first goal of brownfield data governance is not to display every point quickly. It is to ensure that critical records carry identity, unit, time, quality, and version evidence before they enter state, alarms, and analytics. Collection answers whether something arrived. A quality contract answers whether it can be used, for which purpose, and how the system can recover when it fails.
Separating raw, quarantine, and trusted paths adds rule maintenance, owner review, and replay cost. It also turns hidden contamination into a visible, assignable, and reversible problem. For industrial platforms expected to run for years and serve multiple consumers, that cost is usually lower than explaining incorrect alarms, energy totals, and model outputs after the fact.
References
- OPC Foundation — OPC UA Part 4, DataValue
- Eclipse Foundation — Sparkplug Specification 3.0.0
- ISO 8000-61:2016 — Data quality management process reference model
- NIST — Recommendations for Collecting, Curating, and Re-Using Manufacturing Data
- NIST — Foundations of Information Governance for Smart Manufacturing