The flash sale is a commercial mechanism that scales surprisingly poorly. For a single retailer running a single sale on a single site, the engineering is, as I have argued elsewhere, demanding but tractable. For a brand portfolio running coordinated sales across a network of sites — a multi-store group, a franchise network, a parent brand running concurrent promotions on its directly-owned site and on its wholesale partners’ sites — the engineering acquires a class of complications that are absent from the single-site case and are, in my observation, almost universally underestimated at the planning stage.
Unified Flash Sale Network is a tool I built, originally as a fork of the Mapper throughput identification system, to address the multi-site case explicitly. The fork was undertaken after a particularly chastening project for a UK brand portfolio in which a coordinated flash sale across four sites resulted in approximately eleven per cent of orders being committed against inventory that, by the time the orders settled, had already been sold elsewhere in the network. This post is an account of why the multi-site case is harder than the single-site case, what the tool does to address it, and what remains an open problem despite the tool’s existence.
Why the single-site solution does not extend
The transactional pattern that makes single-site flash sales correct — the row-level lock on the product’s stock record, held for the duration of the order’s stock decrement — depends on the existence of a single canonical record for the product’s stock level. For a single site, the record exists in the site’s database; the lock is acquired against that record; the decrement is performed; the lock is released. The mechanism is well-understood and, when implemented correctly, prevents the overselling failure mode.
For a brand portfolio in which several sites share a single physical inventory, no single canonical record exists. Each site has its own database; each database has its own copy of the product’s stock level; the copies are typically synchronised via a periodic update from the inventory management system, with a synchronisation window that is, in the businesses I have worked with, in the range of one to fifteen minutes. The copies are therefore not authoritative; they are stale by design. A row-level lock acquired against any one of them is a lock against a stale value and provides no protection against the case in which a different site has, in the same window, already decremented the underlying physical inventory.
The result, during a coordinated flash sale, is that each site acts in good faith on the basis of its local stock record and the network as a whole oversells the inventory by the cumulative quantity of the simultaneous orders. The eleven per cent figure mentioned above was not, in the project that produced it, the consequence of any individual site’s failure; each of the four sites had implemented the row-level lock pattern correctly. The failure was structural, in the sense that no individual site had visibility of the network’s aggregate state, and the network had no mechanism for resolving the conflict at the moment it occurred.
The Mapper origins of the architecture
Mapper, the tool that Unified Flash Sale Network was forked from, was originally built to address a different problem in the multi-site space — the identification of throughput bottlenecks across a network of sites with shared back-end services. Mapper instruments each site’s request pipeline at a small set of well-defined points, aggregates the timing data across the network, and produces a real-time view of which sites are saturating which back-end resources. The tool is useful for capacity planning and for identifying the source of intermittent slowdowns; it does not, in its original form, do anything related to inventory.
The architectural component that the fork preserves from Mapper is the aggregation layer. Each site reports its actions to the aggregation layer in real time; the aggregation layer maintains a network-wide view of the relevant state; individual sites query the aggregation layer to make decisions that depend on network-wide rather than site-local state. For Mapper, the relevant state is request timing; for Unified Flash Sale Network, the relevant state is stock levels and pending orders. The pattern is the same; the data is different.
The two-phase commit, slightly modified
The transactional pattern that the tool implements is a modified two-phase commit between the local site and the aggregation layer. When an order arrives at a site during an active flash sale, the site does not immediately decrement its local stock record. Instead, the site sends a “reserve” request to the aggregation layer, identifying the SKU and the quantity. The aggregation layer maintains a network-wide stock record that is updated in real time from the inventory management system and adjusted by the cumulative reservations from the network’s sites; if the requested quantity is available against the network-wide record, the aggregation layer reserves the quantity and returns a reservation token; if it is not, the aggregation layer rejects the request and returns the available quantity for the site to handle as it sees fit.
The site, having received the reservation token, completes the order locally and sends a “commit” request to the aggregation layer with the reservation token; the aggregation layer converts the reservation into a confirmed decrement and updates the network-wide record accordingly. If the local order fails — the customer abandons the checkout, the payment is declined, the local site times out — the site sends a “release” request that converts the reservation back into available stock. Reservations that are not committed or released within a configurable timeout are released automatically by the aggregation layer.
The pattern is not architecturally novel; variations of two-phase commit have been studied in distributed database literature for several decades. The aspect that is novel, in the context of brand-portfolio retail commerce, is the requirement that the protocol be tolerant of the operational conditions of a flash sale — high request volumes, occasional network partitions between sites and the aggregation layer, and the presence of legacy site code that cannot be modified to participate in the protocol. The tool’s implementation includes a fall-through mode for legacy sites, in which the site’s local stock decrement is reported to the aggregation layer asynchronously after the fact rather than synchronously beforehand; the fall-through mode does not provide the same correctness guarantees as the participating mode, but it is, in my experience, considerably better than the alternative of operating the legacy site outside the network entirely during the sale.
What the tool does not solve
Two operational problems remain even after the tool has been deployed correctly, and they are worth describing because they are, in my experience, the problems that the brands using the tool most frequently surface in post-sale reviews.
The first is that the aggregation layer itself becomes, during a high-traffic sale, a single point of failure for the network. If the aggregation layer is unavailable — for any reason, network partition or hosting outage or saturation under the sale’s request volume — the participating sites are forced into the fall-through mode, with the corresponding loss of correctness guarantees. The mitigation is to host the aggregation layer with substantially more capacity than the sites it serves and to deploy it in a redundant configuration; this is straightforward in principle and considerably more expensive in practice than the brands using the tool typically anticipate.
The second is that the tool does not address the case in which the inventory management system itself is the point of contention. If the inventory management system updates the aggregation layer with a synchronisation delay that exceeds the duration of the sale, the aggregation layer’s network-wide record is itself stale and the protocol provides protection against between-site contention but not against the system-wide inventory error. The mitigation is to instrument the inventory management system to publish updates to the aggregation layer in real time rather than on a periodic schedule; this requires modification to the inventory management system, which is, in many of the brands I have worked with, a vendor product over which the brand has no direct engineering authority.
An advisory close
For a single-site retailer, the existing tooling described in Easy Flash Sale is, in my experience, sufficient for the great majority of operational requirements. For a brand portfolio with shared inventory, the tooling does not extend, and the failure modes that the multi-site case introduces are operationally serious enough that they cannot be ignored at the planning stage. Unified Flash Sale Network is the tool I have come to recommend for these cases; it is not, in any sense, a finished or polished product, but it is the only tool I am aware of that addresses the multi-site coordination problem with the correctness guarantees that a brand portfolio’s inventory deserves.
It is recommended that any brand portfolio planning a coordinated flash sale of meaningful scale either deploy the tool, or accept that the sale will produce a non-trivial overselling rate and budget the customer-service consequences accordingly. The middle path — running the sale without the tool and hoping that the overselling rate will be tolerable — is the path that produced the eleven per cent figure described at the top of this post, and is, in my view, not a path that anyone with a duty of care to the brand should choose.