Feature Flags: Progressive Release and Safe Experimentation

Feature flags decouple deployment from release: you ship code to production with new behavior hidden, then enable it for specific users, regions, or percentages — and disable it instantly if it breaks, without redeploying. This guide covers flag types, a working implementation, progressive rollout with decision gates, and the lifecycle discipline that keeps flags from becoming debt.

Overview

Every feature flag creates two code paths where there was one. That buys you safety — deploy on Friday, release on Tuesday — but each flag is a temporary fork in your codebase that someone has to test, monitor, and eventually remove. Teams that treat flags as scaffolding ship faster; teams that treat them as architecture drown in flag debt.

What follows assumes you can already deploy on demand and have somewhere to put flag state (a database row, a config service, or a dedicated flag platform).

One clarification upfront: a feature flag is not a substitute for testing, and it’s not remote configuration. Flags are a release mechanism — they control which of two already-tested code paths runs. If you’re using them to avoid testing the disabled branch, or to tune constants that should live in config, you have the wrong tool for both jobs.

When to Use

• Deploying incomplete changes without exposing them to users
• Rolling out gradually while watching error rates and business metrics
• A/B testing changes with real traffic
• Emergency kill switches for risky paths (payments, migrations)
• Merging to main daily instead of maintaining long-lived feature branches

If what you need is experimentation rather than release control — hypothesis testing with statistical analysis — the A/B testing guide covers that layer.

Core Concepts

Concept	Description
Feature Flag	A conditional check that enables or disables a code path
Kill Switch	A flag that instantly disables a change in production
Progressive Rollout	Gradually increasing the percentage of users who see a change
Targeted Flag	A flag enabled for specific users, groups, or regions
Evaluation Context	The attributes (user ID, plan, region) a flag evaluates against
Flag Lifetime	The period from creation to permanent removal from code
Flag Debt	Accumulated stale flags that clutter code and configuration

Feature Flag Types

The type drives everything else — lifetime, ownership, monitoring, and removal procedure:

Type	Use Case	Lifetime
Release flag	Hide incomplete changes during development	Short (days to weeks)
Experiment flag	A/B testing and data-driven decisions	Medium (weeks to months)
Operational flag	Circuit breakers, rate limits, debug modes	Long (months to permanent)
Permission flag	Capability access by user tier	Permanent (use RBAC instead)
Kill switch	Emergency disable for risky changes	Short (removed after stabilization)

Two rules from that table: tag every flag with its type at creation — the removal policy follows from it — and treat permission flags as a smell. Flags that gate access permanently are authorization wearing a different hat.

Targeting Rules and Evaluation Context

Percentage rollout is only one axis. Flags resolve against an evaluation context — the attributes you pass in: user ID, plan tier, country, device, app version. A typical rule set evaluates in order:

1. Kill override — flag off for everyone wins over everything
2. Explicit targets — specific user IDs or an allowlist
3. Segment rules — plan = 'enterprise' or country = 'DE'
4. Percentage rollout — deterministic bucketing over the remaining traffic
5. Default — what nobody matching gets

Keep the context small and stable. Every attribute you add is a dimension your flags can diverge on — “enabled for EU mobile users on app v3” is three axes of surprise. And pass a consistent user identifier: flags evaluated on email in one service and user_id in another produce users who see both experiences at once.

Implementing a Flag System

Build vs buy:

Option	Best For	Examples
Open-source	Self-hosted, full control	Unleash, Flagsmith, Flipt
SaaS	Quick setup, managed scale	LaunchDarkly, Split, Optimizely
Custom build	Simple cases, tight integration	In-app config + database
Config files	Static flags, no runtime changes	YAML/JSON configs

If you build your own, target the OpenFeature contract — a vendor-neutral evaluation API — so migrating to a managed provider later doesn’t mean rewriting call sites.

from dataclasses import dataclass
from typing import Optional
import hashlib

@dataclass
class FeatureFlag:
    name: str
    enabled: bool
    rollout_percentage: float = 100.0
    target_users: Optional[list[str]] = None

class FeatureFlagManager:
    def __init__(self):
        self.flags = {}

    def register(self, flag: FeatureFlag):
        self.flags[flag.name] = flag

    def is_enabled(self, flag_name: str, user_id: str = None) -> bool:
        flag = self.flags.get(flag_name)
        if not flag or not flag.enabled:
            return False

        if flag.target_users and user_id:
            return user_id in flag.target_users

        if flag.rollout_percentage < 100 and user_id:
            # Deterministic hash: same user always lands in the same bucket
            digest = hashlib.md5(f"{flag.name}:{user_id}".encode()).hexdigest()
            return int(digest, 16) % 100 < flag.rollout_percentage

        return True

ffm = FeatureFlagManager()
ffm.register(FeatureFlag("new-dashboard", enabled=True, rollout_percentage=10))

if ffm.is_enabled("new-dashboard", user_id="user-123"):
    show_new_dashboard()
else:
    show_legacy_dashboard()

The md5 bucketing is the part people get wrong. random.random() < 0.10 gives a different answer on every call — the same user flips between old and new experiences mid-session, and your 10 % ramp is actually 10 % of requests, not users. Hashing flag.name + user_id makes the assignment deterministic, so ramping from 10 % to 50 % keeps the original 10 % in the treatment group.

TypeScript version of the same contract (for backends or edge workers — use a non-crypto hash like FNV-1a or a crypto.subtle digest, the point is determinism):

import { createHash } from "node:crypto";

interface FeatureFlag {
  enabled: boolean;
  rolloutPercentage: number;
  targetUsers?: string[];
}

function isEnabled(flagName: string, flag: FeatureFlag | undefined, userId?: string): boolean {
  if (!flag?.enabled) return false;
  if (flag.targetUsers && userId) return flag.targetUsers.includes(userId);
  if (flag.rolloutPercentage < 100 && userId) {
    const hash = createHash("md5").update(`${flagName}:${userId}`).digest("hex");
    return parseInt(hash, 16) % 100 < flag.rolloutPercentage;
  }
  return true;
}

Progressive Rollout with Decision Gates

Increase exposure in stages, and define the criteria for advancing before you start:

ROLLOUT_STAGES = [
    {"name": "dev-team",    "percentage": 0,   "target_users": ["dev1", "dev2", "qa1"]},
    {"name": "1-percent",   "percentage": 1,   "target_users": None},
    {"name": "10-percent",  "percentage": 10,  "target_users": None},
    {"name": "50-percent",  "percentage": 50,  "target_users": None},
    {"name": "full",        "percentage": 100, "target_users": None},
]

A progression that works in practice:

1. Internal only — dev team and QA, verify the feature works at all
2. Beta ring — friendly early adopters, verify UX
3. 1 % — catch crashes and load problems cheaply
4. 10 % — validate error rates and p99 at scale
5. 50 % — confirm business metrics hold
6. 100 % — then stabilize 1-2 weeks before removing the flag

The decision gates between stages are what separates this from “turn it up and hope”. Before each step up, check: error rate parity (enabled vs disabled cohorts), p99 latency delta, and at least one business metric (activation, conversion — whatever the flag touches). If any gate fails, you don’t roll back a deployment — you flip the flag, which takes seconds.

Flags for Database Migrations

Flags are the runtime half of the expand-contract pattern for schema changes. The safe sequence: add the new column/table, deploy code that dual-writes to old and new behind a flag, backfill, flip the flag to read from the new location, then — in a later deploy — remove the flag and drop the old column.

Each step is reversible because the flag controls which representation is authoritative, not the schema. If the new read path misbehaves, you flip back; the data is still correct in both places. Trying to do the same thing with deploys alone forces the “big migration” weekend.

Kill Switches

Flags that exist purely to disable a path instantly:

def process_payment(order):
    if not feature_flags.is_enabled("payment-processing-v2"):
        return process_payment_v1(order)

    try:
        return process_payment_v2(order)
    except Exception:
        if feature_flags.is_enabled("payment-auto-fallback"):
            return process_payment_v1(order)
        raise

Guidelines that matter: every risky change ships behind a kill switch by default; flag values are cached in-process so evaluation adds microseconds, not a network call; someone drills the kill path (a switch nobody has ever flipped is a hope, not a control); and the runbook says which flags are kill switches and who may flip them.

One caveat on auto-fallback like the example: catching Exception and retrying against v1 can mask a real bug and double-charge a partially-completed payment. Only auto-fallback for operations that are safe to retry, and alert every time it triggers — silent fallback is silent failure.

Also worth deciding early: who can flip a kill switch. If flipping requires a deployment or three levels of approval, it’s not a kill switch — it’s a change request. Grant the on-call rotation the ability to flip safety flags without a deploy, and audit every flip so the flag log doubles as an incident timeline.

Monitoring Flag Performance

Metric	Why It Matters
Evaluation latency	Flag checks add overhead to every request
Error rate by flag state	Is the enabled path actually worse?
Cohort engagement	Compare on/off groups for the metric the flag touches
Conversion impact	Business effect of the change
Flag age	Flags past their expiry are debt, not features

panels:
  - title: "Flag Evaluation Rate"
    query: 'rate(feature_flag_evaluations_total[5m])'
  - title: "Active Kill Switches"
    query: 'feature_flag_enabled{name=~".*-kill-switch"}'
  - title: "Stale Flags"
    query: 'time() - feature_flag_last_modified > 7776000'  # >90 days

The error rate by flag state panel earns its keep on day one: it turns “did the rollout break something?” into a chart instead of a guess.

Experiment Flags Need Different Plumbing

An experiment flag isn’t just a release flag with a dashboard. For the data to mean anything you need an exposure event — logging “user X saw variant Y at time T” — because that’s what defines your cohorts. Without it you’re comparing whoever happened to convert, not who was exposed.

Two rules follow: don’t repurpose a release flag as an experiment mid-flight (the 1 % canary cohort contaminates your test group), and keep experiment measurement separate from rollout monitoring — one answers “is it safe”, the other “is it better”.

Flag Lifecycle and Debt

Flags have a lifecycle whether you manage it or not — unmanaged just means the debt accumulates silently:

Operationally:

# Cleanup workflow
# 1. List flags enabled >30 days unchanged
# 2. Confirm the feature is stable and fully rolled out
# 3. Ticket the removal — flag owner decides, not "someone"
# 4. Remove the conditional, keep only the enabled path
# 5. Remove flag from config, deploy, verify no regressions

Rules that prevent debt: set an expiry date on every release and experiment flag when it’s created (30-60 days); cap active flags per service — healthy codebases stay under ~20-30, and past that every test matrix and code path review gets harder; archive removed flags in a changelog so “when did we enable X?” has an answer.

Naming helps more than it looks. new-dashboard-v2-final tells nobody anything; {area}-{behavior}-{ticket} (checkout-retry-logic-4821) makes the flag greppable, ties it to an owner, and makes the removal diff obvious. If you can’t tell from the name whether a flag should still exist, it probably shouldn’t.

Feature Flags and Trunk-Based Development

Flags are what make merging incomplete work safe. Instead of a feature branch living for three weeks and drifting from main, you merge small pieces behind a disabled flag — main stays releasable, the branch never rots, and review happens in reviewable chunks.

The discipline that makes it work: the disabled path has to stay compilable and tested. Merging dead code “because it’s behind a flag” rots just like a branch does — when you finally enable it, it doesn’t compile or quietly breaks the path around it. CI should build and smoke-test the flagged path even when it’s off.

Server-Side vs Client-Side Flags

Where a flag evaluates changes what it’s good for:

Aspect	Server-side	Client-side
Latency	Evaluation at request start	SDK fetch + local evaluation
Payload exposure	Hidden path never ships to client	Disabled code is in the bundle — and readable
Consistency	Authoritative per request	Can drift until SDK refreshes
Good for	Payments, business logic, experiments	UI variants, theme, layout

Two practical consequences: never put a dangerous path behind only a client-side flag (the code is inspectable in the browser), and for experiments, evaluate server-side or you get flicker — the old UI rendering before the flag loads.

Getting Flag State to the Application

How values reach your code is an availability decision: a remote call per evaluation means the flag service is on your critical path — if it’s down, everything evaluates to default (which is why the safe default matters). Common patterns, in order of operational weight:

• In-process SDK with polling or streaming — the SDK maintains a local copy; evaluation is a dict lookup. Most production setups land here.
• Sidecar agent — a local process (like AWS AppConfig’s agent) holds flag state and your app calls localhost. Same benefit, polyglot-friendly.
• Remote evaluation — simplest to start; acceptable for low-traffic tools, risky as a per-request dependency.

Whatever the mechanism, stale flag state is normal — decide how stale is acceptable. A kill switch that takes five minutes to propagate is a “pause switch”, which is sometimes exactly what you need and sometimes useless.

One Flag, One Decision Point

The single most valuable convention: each flag gets exactly one named decision point — a helper like features.newDashboard() — called from everywhere it’s needed. What you don’t do is scatter if flags["new-dashboard"] across twelve files, because removing the flag then becomes a graph-traversal problem and half the call sites disagree on the default.

# features.py — the only place flag names live
def new_dashboard_enabled(user_id: str) -> bool:
    return ffm.is_enabled("new-dashboard", user_id)

When the flag retires, you delete the helper’s flag check and inline the True branch — one diff, no archaeology.

What Works

• One flag per change, never nested conditionals — combinatorial flags can’t be tested.
• Safe default: if the flag system is down, the flag evaluates to the proven behavior, not the new one.
• Evaluate once per request, cache the result, propagate through context — don’t re-evaluate mid-request.
• Unit-test both paths, and smoke-test the disabled path in CI — that’s the state most users see. (Slack’s May 2020 outage cascaded precisely because a flag-on code path had never run under real production load.)
• Owner, description, type, and expiry on every flag from day one.

Common Mistakes

• Leaving flags in code indefinitely — stale flags are dead branches, stale config, and surprise interactions. Past ~30 days at 100 %, a flag is debt.
• Math.random() bucketing — inconsistent per-user experience and polluted experiment data; use deterministic hashing.
• Scattered flag checks — no single decision point means removal requires a codebase expedition.
• Flags as access control — long-lived permission gating belongs in RBAC, not the flag store.
• Testing only the enabled path — the disabled branch is what production actually runs most of the time.
• Flag interdependencies — “enable B only if A” creates a test matrix nobody verifies; keep flags orthogonal.
• Silent auto-fallback — fallback paths that don’t alert hide the failure they were built to survive.

See Also

• Canary Deployment Guide
• A/B Testing Guide
• Blue-Green Deployment Guide
• Feature Toggles — Martin Fowler (canonical taxonomy of flag types)
• OpenFeature specification (vendor-neutral flag evaluation API)