From cases to a service view
A case owner needs the current agreement for one learner. A service manager may want a view across thousands of courses: where cases remain open, how long resolution takes and which centres need help. That introduces derived data, refresh times and potentially distributed processing.
Start with the decision and the required freshness. Record expected volume, arrival rate, variety, growth and recovery needs. These determine which processing choices are worth examining.
Medallion architecture
Medallion architecture organises data into layers with progressively more preparation for use. Bronze retains received data, silver provides cleaned and validated records, and gold serves particular business needs. Databricks describes the pattern and Microsoft documents its use in Fabric.
For the cancellation report, an illustrative arrangement is:
| Layer | Contents | Checks before use |
|---|---|---|
| Bronze: received | Case extracts with source identity, version and receipt time. | Check that the expected extract arrived and can be traced to its source. Retain receipt and failure details. |
| Silver: validated | One current record per case, with recognised states and a defined centre reference. | Reconcile duplicate or conflicting versions; check required references. Hold unresolved records for correction and report the gap. |
| Gold: prepared for use | Currently resolved cases by centre, with a cutoff and report-rule version. | Reconcile counts to the accepted case set; show freshness, exclusions and unresolved issues. The service owner accepts the report’s meaning and suitability for use. |
Each layer needs an owner, named consumers and explicit quality rules. A gold report can answer the wrong question if “resolved” has been misunderstood. Link its measures to the business definition and retain the path back to individual cases and source extracts.
If a resolved case is reopened, retain the changed input, revise the accepted case version and recompute the affected report. Record which rules ran and which outputs changed. An unresolved source conflict should remain visible to the report consumer; totals alone can hide missing cases.
Layering can make checks, reuse and recovery easier to locate. Maintaining extra copies and transformations also adds storage, processing, delay and support work. Compare those costs with the consumers and recovery needs. Agree access and retention at each layer, including how corrections and removal reach derived data. Retained raw data needs its own protection and retention period.
How the choices fit
| Choice | What it describes |
|---|---|
| Medallion layers | How data is prepared, checked and offered to consumers. |
| Batch or streaming | How selected inputs or ongoing arrivals are processed. |
| MapReduce | A model for dividing processing and combining results. |
| Lake, warehouse or lakehouse | Storage and analytical capabilities. |
A medallion flow can use batch, streaming or both. Bronze, silver and gold describe data preparation; conceptual, logical and physical describe levels of architectural detail. A conceptual model can explain all three layers. Architecture cycles revisit their definitions, checks and implementation as needs change.
MapReduce
MapReduce is a model for processing data in parallel. The map step produces intermediate key/value pairs; values with the same key are grouped, then the reduce step combines them. Dean and Ghemawat described Google’s implementation in their 2004 paper.
For this illustration, centre means the centre responsible for the cancelled course. An alternative elsewhere keeps that original attribution. Take one current record per case at an agreed cutoff:
| Case | Centre | Current state |
|---|---|---|
| A | North | Resolved |
| B | North | Open |
| C | South | Resolved |
To count currently resolved cases by centre, the map step emits (North, 1)
for A and (South, 1) for C. Group by centre and sum: North has one, South has
one. The same calculation could run in a simple database query; distributed
execution becomes a choice when the workload and operating requirements justify it.
The definition still matters. If C is reopened in a later snapshot, South’s current count becomes zero. A versioned list of centres lets the report show that zero explicitly. Retain the input version, cutoff and transformation version so the reported number can be explained and corrected.
Batch and streaming
Batch processing works on a selected, bounded input. Streaming processing handles an ongoing arrival of records or events. A stream needs rules for when a result is useful enough to show and how later information changes it. Apache Beam’s guide explains bounded and unbounded inputs and event time, processing time and late data.
A nightly snapshot might be enough for planning. A live operational view may need updates sooner, including reopening and correction events. The team must agree how quickly those changes should appear and how users can see the view’s freshness.
Architectural choices
A report, a search index and a step using artificial intelligence (AI) may use the same source records for different purposes. Define what each consumer needs. Its required freshness, access and handling of corrections shape the processing contract.
| Choice | Question and trade-off |
|---|---|
| Partitioning | How is work divided? A popular centre may create an uneven share of the processing. |
| Copies and derived views | Which record supplies each fact, and how do corrections reach reports? Local reads may be faster while reconciliation takes more work. |
| Time and late arrivals | Is a result grouped by when something happened or when it was received? Later records may require a revised result. |
| Retry and replay | Can the same input be processed again without counting it twice? Keep input identity and an agreed replacement or correction rule. |
| Schema and meaning changes | Which readers understand a new field or state? Preserve definitions and transformation versions through the transition. |
Stores and processing
A data lake can retain data in varied formats; a warehouse organises analytical data for querying. Lakehouse approaches bring some warehouse features to lake storage. Microsoft’s architecture overview describes these patterns alongside batch and stream processing.
For any arrangement, identify the authoritative inputs, derived outputs, access, retention and recovery responsibilities. Data architecture defines meaning and quality; systems, software and technology views develop the interfaces, processing behaviour and deployment.
Methodology configuration
Your method may describe a pipeline, data flow, analytical service or data product. Map the name to its inputs, outputs, owner, consumers and operating promises. Relate its delivery and review to your existing cycles.
If your method uses landing, integration and serving layers, compare their purpose with bronze, silver and gold. Map the contents, quality rules and responsibilities before treating the names as equivalent.
Use the processing record to compare options, expected freshness, corrections and recovery. The cycles page explains how findings return to design decisions, and configuration guidance covers tailoring.