Designing Systems That Survive Growth

Most systems don’t fail because the first version was wrong. They fail because growth changed the shape of the problem, and nothing in the design could bend. Here is what we harden early and what we leave flexible, with the projects that taught us.

1. 01Start from the load that would hurt

   Name the load that would actually break things: search across a huge catalog, large uploads, month-end reports or webhook bursts. Design for that first.

   On AutoBuffy, search by make, year and model slowed down as the parts tables grew. Caching alone didn’t help, so we added Elasticsearch, refined the database structure and added indexes. On Cad Cam Masters, large 3D design files were stalling checkout, so uploads moved to a separate background pipeline on AWS.

2. 02Keep heavy work away from the people using the app

   AutoBuffy’s admin uploads used to slow the store down for shoppers. We split the storefront and admin onto separate servers, and added load balancers and auto-scaling. Files now go straight from the browser to S3 instead of passing through the server. Cercle moved heavy work to Sidekiq background jobs for the same reason.

3. 03Separate stable contracts from replaceable insides

   APIs, event formats and data ownership should be boring and versioned. Internal modules, queues and caches should stay easy to replace. BuffyHub connects payment gateways, shipping providers and inventory systems through APIs and webhooks, so each connection can change without rewriting the rest.

4. 04Make failure visible early

   Retries, timeouts and monitoring show you where a system breaks before customers do. On Truck-4, sign-in has fallbacks and monitoring, so it stays available when an upstream service fails. SonarQube and the CI pipeline catch problems across its services before release.

5. 05Split into services only when you must

   A clear seam you can cut later is cheaper than a microservices platform you have to staff forever. Truck-4 did need microservices, because its services depended on each other and had to scale separately. They talk through REST and RabbitMQ, run on Kubernetes and deploy with Helm. Most products don’t need that on day one.

6. 06Fix the slow query before buying a bigger server

   On ATC, analytics built with LINQ across many tables slowed every dashboard. We rewrote them as optimized SQL with caching, which cut the database load and sped up the pages.