How I transformed experimentation development from tedious manual processes into a seamless, production-integrated workflow
As developers, we’ve all been there: copying code between environments, manually refreshing browsers, juggling multiple build processes, and wrestling with deployment pipelines just to test a simple button color change. After years of building A/B tests and personalization campaigns across multiple clients, I decided enough was enough. Here’s how I built a developer experience that makes experimentation feel as smooth as local development should be.
The Problem: Death by a Thousand Manual Steps
Traditional A/B testing development is painful. You write code locally, build it (or not?), upload it to a 3rd party tool, save, refresh the production site, hope it works, make a change, and repeat. The feedback loop is measured in minutes, not seconds. Multiply this across six different clients with different configurations, and you’re looking at a productivity nightmare.
The breaking point came when I realized I was spending more time managing code bases and deployments than actually crafting experiments. Something had to change.
The Solution: Production-Integrated Development
What if experimentation development felt like modern frontend development? What if you could:
- Test your changes instantly on live production sites
- See real data and user flows during development
- Get automatic page refreshes when code changes
- Switch between different client environments seamlessly
- Manage experiment lifecycles with simple CLI commands
That’s exactly what I built.
Architecture: The Perfect Trinity
1. The Development Hub
A Vite-powered build system that manages multiple different client configurations, each with their own experiments, personalizations, and tracking setups. The key insight was treating each client as a separate build target while sharing common utilities and patterns.
2. The Chrome Extension
Here’s where it gets interesting. Instead of deploying to staging servers, I built a Chrome extension that intercepts network requests in real-time. It can either redirect production script requests to your localhost server or inject your local scripts directly into production pages.
The extension handles two modes:
- Redirect Mode: “Hey production site, instead of loading your script, load mine from localhost”
- Injection Mode: “Hey production site, you don’t have this script yet, but here it is from my dev server”
3. The Live Reload System
A WebSocket server that watches for file changes and broadcasts reload signals. When you save a file, it rebuilds the appropriate client bundle, restarts the preview server, and tells the Chrome extension to refresh all matching tabs across all your browser instances.
The Developer Experience in Action
Here’s what a typical development session looks like:
# Start development for a client
npm run watch -- dev_client
# This spins up:
# - File watcher with smart debouncing
# - Vite build process for the specific client
# - Preview server on localhost
# - WebSocket reload server on port 5678
I enable the appropriate rule in my Chrome extension popup, navigate to the production website, and suddenly I’m developing directly against live data. Every code change triggers an automatic browser refresh across all tabs.
It’s like having hot module replacement, but for production websites.
Experiment Lifecycle Management
Managing experiments became a breeze with purpose-built CLI tools:
Creating Experiments
npm run new:experiment
An interactive prompt walks through client selection, experiment type, and configuration. It scaffolds the proper directory structure, updates the tracking files, formats everything with Prettier, and adds files to Git. No more copy-pasting boilerplate or forgetting to update import statements.
Archiving Completed Tests
npm run remove:experiment
When experiments conclude, a smart archival system moves folders to organized archive directories, cleans up all import references, and maintains the codebase automatically. It even handles the tedious task of parsing and rebuilding the experiment registry files.
The Technical Magic
Smart File Watching
The file watcher uses debouncing to prevent rebuild spam while maintaining responsive feedback. It only watches source directories and ignores build artifacts, keeping CPU usage minimal even with large codebases.
Multi-Instance Synchronization
WebSocket connections work across multiple Chrome instances and tabs. Make a change in your editor, and every relevant tab refreshes simultaneously, whether you have one browser window or ten.
Automatic Formatting Integration
Every generated file passes through Prettier automatically. The remove-experiment script even reformats the files it modifies. Consistency is maintained without developer intervention.
Performance Considerations
The system is surprisingly lightweight:
- WebSocket connections use minimal bandwidth
- The Chrome extension caches URL patterns for efficient request matching
- File watching is scoped to source directories only
- Build processes are client-specific, avoiding unnecessary work
Page load impact is negligible. The extension operates at the browser level, and the WebSocket connection adds maybe 1KB of JavaScript to development builds.
Seamless Deployment Pipeline
The development experience extends seamlessly into production deployment. When an experiment is ready for QA:
git push origin main
git checkout -b release-client
git push origin release-client
That’s it. Each client has a dedicated Cloudflare Pages project connected to their release branch. Push to the branch, and within minutes your script is live on a production CDN with a custom domain like growthbook-toolbox.client.com.
No build servers to manage, no deployment pipelines to configure, no manual upload processes. The same Git workflow developers use for everything else handles production deployments automatically.
Real-World Impact
This setup transformed how I approach experimentation development:
- Development velocity increased exponentially – The feedback loop went from minutes to seconds
- Context switching eliminated – No more juggling multiple environments or deployment processes
- Real data testing – Developing against production data reveals edge cases immediately
- Multi-client efficiency – Switching between client configurations takes seconds, not setup time
- Zero-friction deployment – Git push to production deployment in under 2 minutes
- Reduced cognitive overhead – The system handles the tedious parts automatically
Lessons Learned
Start with Developer Pain Points
The best developer tools solve real frustrations. Every feature in this system addresses a specific pain point I experienced repeatedly.
Integration Creates Magic
Each component is useful individually, but their integration creates capabilities greater than the sum of parts. The Chrome extension alone is helpful; combined with live reload and smart builds, it becomes transformative.
Automate the Tedious Stuff
Developers shouldn’t spend mental energy on repetitive tasks. CLI tools that handle scaffolding, cleanup, and formatting free up cognitive resources for actual problem-solving.
Test with Real Data
There’s no substitute for testing against production data and user flows. Staging environments never quite match the complexity of real-world usage.
Looking Forward
This system has become the foundation for all experimentation work. The principles extend beyond A/B testing – any development workflow that involves testing code against external systems could benefit from similar integration patterns.
The key insight is that developer experience isn’t just about pretty UIs or faster builds. It’s about reducing friction at every step, eliminating context switches, and creating tight feedback loops that keep you in flow state.
The Bottom Line
Building great developer experiences requires treating developer productivity as a first-class concern. This experimentation setup saves hours every week and makes the work genuinely enjoyable instead of tedious.
The complete pipeline – from localhost development against live data, through automated CLI tools, to Git-triggered production deployments – eliminates friction at every stage. When developers can focus on crafting great experiments instead of wrestling with tooling and deployment processes, everyone wins.
The magic isn’t in any single component – it’s in how they work together to create a seamless experience from first line of code to production deployment.
This workflow was built iteratively over months of daily use, solving real problems as they emerged. The result is a system that feels invisible in the best possible way – it just works, and gets out of your way so you can focus on building great experiences.
