Offline-First Data Architecture for a National Postal Service

Think about a postal branch as an autonomous data node. It can accept payments, register shipments, deliver parcels, and close operational shifts without a connection to the central platform. When connectivity returns, the branch uploads locally recorded operations and receives the central data it is authorized to use.

A national postal network may contain roughly 10k branches, ranging from small offices with one computer to large locations with dozens of workstations. It also includes mobile branches operating over weak cellular connections. Some locations experience severe packet loss, some remain disconnected for months, and some may never reconnect because the building, vehicle, or equipment has been destroyed.

These conditions rule out a conventional application that depends on a permanently available central database. Every branch requires local storage and operational autonomy. A dedicated synchronization and recovery layer connects the independent branches to the central platform.

Offline-first data architecture for a national postal service

The important idea in the diagram is that the system has two replication dimensions.

Horizontal replication keeps data available inside one physical location. Central database nodes replicate among themselves, and database nodes inside one branch can replicate over the branch network.

Vertical replication moves selected data between organizational levels. It connects the central platform with each branch, but it does not copy the complete national dataset into every location.

Architecture At A Glance

The main parts are the Central Platform, the Synchronization and Recovery Layer, and the independent Postal Branches.

The central platform contains the complete national view. Each branch stores enough shared reference data and branch-specific operational data to continue working independently.

The synchronization layer is responsible for:

This separation is important. The databases remain responsible for durable storage and local availability. The synchronization layer is responsible for organizational routing, data authority, throttling, and recovery.

Horizontal Replication Inside A Location

Horizontal replication happens inside one physical location.

The central platform runs a redundant database cluster so the loss of one database node does not stop the national system.

A branch is also treated as an isolated data location. A small branch may have one practical database node, while a larger office may use several local nodes. Workstations continue using the local database even when the connection to the central platform disappears.

The critical boundary is that horizontal replication never crosses from one branch into another. Each branch is an independent data island. Branches do not exchange operational records directly and one branch must never see another branch’s payments, shipments, or shifts.

Vertical Replication Between Organizational Levels

Vertical replication crosses the boundary between the central platform and a branch.

It is selective by design. There are two separate data paths because shared reference data and branch operational data have different owners.

Shared Reference Data

Reference data includes address directories, branch locations, working hours, service definitions, price lists, temporary closures, relocated offices, and replacement locations.

The central platform is authoritative for this information. The direction is strictly:

Central Platform → Synchronization Layer → Postal Branches

A branch can use this data while offline, but it does not publish local reference-data changes back to the central platform. A branch closure, relocation, or new working schedule must come from the authoritative central source.

Branch Operational Data

Operational data includes payments, subscriptions, accepted and delivered shipments, cargo operations, and shift opening or closing events.

This flow is bidirectional:

Branch → Synchronization Layer → Central Platform

and:

Central Platform → Synchronization Layer → Owning Branch

The first direction uploads work performed locally. The reverse direction delivers an authorized central correction or a centrally created record back to the branch that owns it.

Every operational document carries a branch ownership key. The synchronization layer uses that value as an isolation boundary. A record belonging to one branch must never be distributed to every other branch.

This is the difference between controlled synchronization and blind database copying. Each class of data travels only through its allowed path.

Why Central Changes Are Captured Once

The most obvious implementation would open one central change reader for every branch and let every reader filter its own data.

At national scale, that design becomes a trap. It creates thousands of long-lived central readers, repeated database work, large connection pools, and a reconnection storm whenever the central system or a major network provider has a problem.

Instead, a small fixed set of shared readers captures central changes once. Shared reference changes and branch-scoped operational changes remain independent because their routing and failure rules are different.

Captured changes are placed in a durable delivery journal. Branch delivery workers consume that journal at their own pace. Every branch has its own delivery position, lag state, retry state, and recovery state.

The central database therefore reads each source change once instead of repeating the same work for thousands of branches. Delivery still requires work for every eligible target, but slow branches no longer multiply the cost of reading the central change history.

Keeping the central paths independent also improves failure isolation. A malformed branch ownership value must not stop shared address or working-hours updates for the entire postal network.

Branch-To-Central Flow

The reverse direction cannot be shared in exactly the same way because every branch is an independent source.

The synchronization layer supervises a connection to every registered branch. It watches the configured operational data and uploads eligible changes to the central platform.

Before forwarding a document, the service validates its branch ownership. This prevents a damaged, compromised, or incorrectly configured branch from publishing records on behalf of another location.

Most branch work is append-oriented. A branch creates a payment, shipment, or shift event and rarely changes it afterward. That practical property keeps real write conflicts exceptional rather than normal.

Normal Operation Sequence

Daily branch work is completed against the local branch database. The operator does not wait for a round trip to the central platform before a payment, shipment, or delivery can be recorded.

Normal data synchronization sequence for a postal branch

The synchronization layer detects locally committed operations and sends them to the central platform in the background. After the central platform stores the operation, the branch records the confirmed synchronization position.

The reverse path distributes new reference data and authorized central corrections. Each update is routed only to the branches allowed to receive it and is then committed to the local database for use by branch applications.

What Happens During A Network Failure

The branch does not stop when the central connection disappears. It continues writing to its local database.

The synchronization supervisor treats a network failure as an expected operating state. It closes failed resources, waits with increasing retry delays and randomized timing, and tries again. Randomization matters because a large provider outage can disconnect thousands of branches at once. They must not all reconnect in the same second.

Short interruptions can be recovered incrementally from stored change positions and delivery checkpoints. Longer interruptions require a different path.

Some branches may remain offline for several months. Keeping unlimited incremental history for every possible outage would make central retention unbounded. The system therefore keeps a finite delivery journal and detects when a branch has fallen behind the oldest retained change.

At that point, partial replay is unsafe. The branch switches to snapshot recovery and is rebuilt from an authoritative central state before incremental delivery resumes.

Snapshot Recovery

Snapshot recovery supports three closely related situations:

The recovery process establishes a fixed central boundary, copies only the data the branch is authorized to receive, prepares the required indexes, records the completed boundary, and then replays changes that happened after that point.

Large reference datasets are copied through temporary staging areas. Data is transferred in bounded batches, indexes are prepared, and only a completed dataset replaces the branch’s active version. Operators must not see a half-copied address directory.

The copy is resumable. Confirmed batches and completed datasets are checkpointed, so restarting the synchronization service continues from durable progress instead of copying several gigabytes from the beginning.

The operational goal is straightforward: install a replacement computer, securely assign it to the branch, start synchronization, and obtain a working postal workstation. The minimum data required for branch operations should arrive first. Less important history can continue downloading in the background.

Conflict Resolution

The business workload makes conflict resolution simpler than it would be in a collaborative editing system.

Branches mostly add records. Existing operational documents are changed only in exceptional situations, and workstation clocks are synchronized with a trusted time source whenever connectivity permits.

Documents carry creation and modification timestamps. The application-level conflict policy is that the latest valid modification replaces the older document as a whole instead of attempting a field-by-field merge.

For a record created during a long outage, the creation time represents when the operation happened at the branch. A separate central receipt time represents when the central platform actually received it. These timestamps answer different operational questions and should not be confused.

Time synchronization does not make clocks perfect, especially after months offline, so implausible future timestamps still require validation. In practice, however, append-oriented data and rare edits make timestamp conflicts an edge case rather than the dominant synchronization path.

Bandwidth And Worker Limits

The central network link is finite even when it has substantial capacity. Database reads, open connections, processing time, memory, and storage input and output are finite too.

The synchronization layer controls both global and per-branch capacity. It limits how many delivery workers may run concurrently and applies backpressure when the central platform reaches its configured ceiling.

The branch side has an equally important limit. Synchronization must not consume the entire connection because the same link is needed for payments and normal postal work. A meaningful share of branch bandwidth must always remain available for operational traffic.

The system combines those limits with:

The objective is not to make every branch appear real-time at any cost. It is to move data as quickly as conditions allow without preventing the branch from doing its actual job.

Data-Loss Objective

When connectivity is available, new operational data should reach the central platform within seconds.

That is a best-effort objective rather than an unconditional guarantee. A branch on an unstable cellular connection cannot have the same synchronization latency as a branch on a stable wired connection.

The reason to transfer data quickly is not only user experience. A disconnected mobile branch may be destroyed before its next successful upload. Every confirmed transfer reduces the amount of business data that can disappear with the hardware.

When no central connection exists, the local branch database is the durable source. When even a narrow channel becomes available, recent critical operations should be transferred before large reference-data refreshes or historical recovery work.

Why Built-In Database Replication Was Not Enough

We initially evaluated a database with built-in replication and eventual consistency. It looked attractive because much of the offline synchronization mechanism appeared to be included in the product.

The problem appeared at national scale and with our data shape. Revision processing, change feeds, and view indexes made the system operationally sensitive. A missing, stale, or incorrectly designed index could turn a selective operation into expensive document processing.

That load appeared in both places where we could least afford it:

With thousands of synchronization relationships, a small indexing mistake is no longer a local performance bug. It can become a network-wide incident. The prototype became too slow under the required synchronization patterns, so we rejected generic built-in replication as the complete solution.

The final design keeps storage concerns in the database while routing, throttling, recovery, and data authority belong to the dedicated synchronization layer. A database does not naturally understand that a national address directory belongs to the central platform while a payment belongs to exactly one branch.

Why The Separation Works

The architecture works because each layer owns one kind of responsibility:

The most important property is failure isolation. One branch with a severely damaged connection should become one lagging branch, not a slow central platform and not thousands of other slow branches.

Final Thought

What I like about this architecture is that it accepts the physical world instead of pretending the network is reliable.

A branch is allowed to be offline. A connection is allowed to lose packets. A workstation is allowed to disappear permanently. The central platform is allowed to have finite capacity. Recovery is not an exceptional manual procedure; it is one of the normal synchronization paths.

Horizontal replication keeps each location available on its own. Vertical replication moves only the data allowed to cross organizational boundaries. Shared readers keep central database work bounded, per-branch workers isolate bad connections, and snapshots make very long outages recoverable.

The result is not a perfectly real-time system. It is something more useful for a national postal service: a system that keeps working when the network, hardware, and sometimes the branch itself do not.