Black Friday week, considered as an engineering challenge, is unlike any other trading window in the retail year. The traffic patterns are sharper than the rest of the year by approximately a factor of ten on the median site I have worked with; the conversion patterns are tighter (visitors spend less time in consideration and more time at the checkout); the operational tolerance for failures is essentially zero, since each minute of platform downtime represents a quantity of revenue that, in absolute terms, exceeds the typical day’s total. The combination of these properties produces an engineering environment in which interventions that are operationally reasonable for the rest of the year become, in the seventy-two hours that surround the sale, decisively unreasonable; and interventions that are operationally indefensible for the rest of the year become, in the same window, the only reasonable choice.
Across the four years I spent at Praxis Digital 3D, I led peak-trading engineering for approximately one hundred and seven retail sites; the work was the most concentrated technical activity of each year, and the failures and successes that emerged from it have informed essentially every procurement conversation I have had with retail clients in the period since. This post is an account of the failure modes that consistently re-emerge, the interventions that have proven reliable, and the procurement decisions that determine whether the trading window produces revenue or operational disaster.
The failure modes that re-emerge each year
The dominant pattern, observed across approximately every one of the hundred-plus sites I have worked with, is that the first failure each November is rarely the failure the engineering team had spent the previous year preparing for. The team has, by hypothesis, optimised the platform for the predictable kinds of pressure — database query saturation, web-server thread exhaustion, cache invalidation storms — and the platform has, accordingly, been improved against those specific failure modes. The first failure is, in the great majority of cases, in some entirely different layer of the platform that the optimisation work had not addressed: a third-party tag whose vendor has fallen over under the surge of pageviews; a captcha service whose API rate limit had been adequate for ordinary trading but is exhausted within minutes of the sale’s launch; a customer-service tool that the marketing team had recently integrated and which is, on the day, the proximate cause of the platform’s failure to render product pages.
The pattern is not, in any abstract sense, surprising; the surprising aspect is the consistency with which the pattern repeats year after year despite the engineering team’s improving understanding of it. The reason for the consistency, in my analysis, is that the third-party-failure mode is a class of failure that the typical engineering team has limited authority over. The team can profile and optimise the platform’s own code; the team cannot, in most cases, audit and prepare every third-party service the marketing team has integrated, and the marketing team’s third-party additions are made on a continuous basis throughout the year rather than concentrated in periods that the engineering team can review.
The mitigation that has, in my experience, proven effective is the imposition of a third-party-tag freeze in the four weeks preceding the sale. The freeze prevents the marketing team from adding new tags to the platform, and requires that any tag that has been added in the previous quarter be reviewed against an explicit set of peak-trading criteria — the vendor’s stated capacity, the integration’s failure-mode behaviour, the platform’s tolerance for the integration’s potential failure. The freeze is, in operational terms, an unpopular intervention; the conversations with the marketing team are, each year, somewhat tense; the freeze nevertheless reduces the year-on-year incidence of third-party-mediated failures by approximately seventy per cent on the data I have available.
The infrastructure interventions that work
Beyond the third-party-tag freeze, the infrastructure interventions that have, across the period of the work, consistently produced measurable improvement on peak-trading capacity are, in approximate priority order, the following.
The first is aggressive caching of product and category pages, with cache durations longer than would be reasonable for the rest of the year. The conventional caching configuration tolerates a small staleness window in exchange for fresher data; the peak-trading configuration tolerates a larger staleness window in exchange for considerably reduced database load. A retailer whose product-detail pages cache for sixty minutes during normal trading should, in my experience, cache for several hours during peak trading; the staleness implications are operationally manageable provided the inventory and price-change pipelines bypass the cache, and the database load reduction is decisive for the platform’s tolerance of the traffic surge.
The second is the pre-warming of the cache before the sale opens. The cache, by hypothesis, is empty at the moment the sale launches; the first wave of visitors will therefore receive cache misses and will produce database load proportional to the cache miss rate. The pre-warming pattern is straightforward: a synthetic visitor performs a structured walk of the catalogue’s most popular pages in the hours preceding the sale, ensuring that the cache contains the relevant entries at the moment of the launch. The intervention is, again, not architecturally novel; the operational discipline required to actually execute it is, in my experience, the variable that separates the engineering teams that successfully launch peak-trading sales from those that do not.
The third is the explicit capacity reservation with the underlying hosting infrastructure. The dominant hosting platforms permit reserved capacity to be requested for specific time windows; the request is rarely free but the cost is, in absolute terms, modest compared to the consequences of the reservation having not been made when the sale launches. The retailers I have worked with who have made the reservation have, with very few exceptions, not had hosting-capacity failures during the trading window; the retailers who have relied on the auto-scaling behaviours of the hosting platform have, with some regularity, encountered scaling delays that produced visible degradation in the first thirty minutes of the sale.
The fourth is the static fallback for the homepage and the most popular landing pages. The fallback is a fully-rendered static HTML version of each page, served from the CDN with no involvement from the application server, configured to be activated in response to a specific signal from the platform’s monitoring infrastructure. The fallback removes essentially all dynamic functionality from the affected pages; visitors can read the page but cannot personalise it, see real-time inventory, or interact with any feature that depends on the application server. The fallback is, in operational terms, an emergency mechanism; it is the difference between a platform that visibly fails and a platform that is visibly degraded, and the visible failure is decisively worse for the brand than the visible degradation.
The procurement decision that matters
The single decision that has, in my observation, the largest impact on the peak-trading outcome is the decision of when to begin the engineering work for the next year’s sale. The conventional procurement pattern treats peak-trading engineering as a discrete project completed in October of each year; the pattern that consistently produces better outcomes treats the engineering as a year-round process with a specific intensification in the final quarter.
The reason for the difference is that the third-party-tag accumulation, the database-schema drift, the cache-configuration entropy, and the dependency-version drift that produce the platform’s vulnerability to the November surge all accumulate continuously throughout the year. A platform that is engineered each October is, in some sense, retrofitted against the surge it is about to encounter; a platform that is engineered continuously is, by hypothesis, in a closer-to-ready state when the surge arrives. The continuous-engineering pattern is more expensive in operational terms; the discrete-engineering pattern produces, in approximately one year in three on the data I have collected, the year-over-year failure that the continuous pattern almost universally avoids.
The recommendation is not, in any abstract sense, surprising; the procurement conversations I have nevertheless rarely treat the recommendation as actionable, principally because the budget allocation for engineering work is governed by an annual cycle that does not align with the continuous nature of the work the recommendation requires. The retailers who have successfully made the alignment have, on the data, considerably better peak-trading outcomes than those who have not.
An advisory close
Black Friday week is, in commercial terms, the year’s most consequential trading window for the great majority of UK retailers; in engineering terms, the window is the year’s most demanding test of the platform’s accumulated decisions. The retailers who pass the test consistently have, in my observation, made the engineering investment continuously rather than seasonally; the retailers who pass the test occasionally have, by the same observation, treated the test as a discrete project to be prepared for in the final quarter and have correspondingly produced inconsistent outcomes.
It is recommended that any retailer scoping engineering investment for the coming year do so with explicit reference to the peak-trading capacity the platform should be capable of, and with explicit budget allocation for the continuous engineering work the recommendation requires. The alternative — preparing the platform in the final quarter for the surge it will encounter in November — is, in my experience of one hundred and seven sites, the pattern that produces the year-over-year failures the trade literature routinely documents.