Hire Software Developers 7
Back to blogs

The Release Checklist for a SaaS Team Without QA

November 9, 2026
One change spreading to every server at once while a small canary ring stays protected, the risk a release checklist is built to catch

The Release Checklist for a SaaS Team Without QA

"If this had been flag protected, the issue would have been caught in staging." Google Cloud wrote that about its own outage of 12 June 2025 [R16]. A Service Control feature added on 29 May lacked appropriate error handling and had no feature flag. It rolled out region by region, and the code path that failed never ran during the rollout [R16].

The fix Google committed to was a flag rule: changes to critical binaries must be "feature flag protected and disabled by default" [R16]. Nobody had asked, in time, whether the change would break for everyone at once. That question belongs on your release checklist. Most of what teams usually put there doesn't.

A release checklist is the short list of questions a team answers before and after every deploy, each one earned by a past failure. A smoke test is the automated subset that checks the main functionality of the deployed build works at all [R1, R12].

Disclosure: yousource publishes this and sells QA On Demand, which lists "Release validation & smoke" among its services [P1].

Key takeaways

  • DORA found "no evidence" that a more formal, external change approval process is associated with lower change fail rates [R14].
  • In DORA's 2024 report, the Medium cluster had a 10% change fail rate, lower than the High cluster's 20% [R4].
  • CrowdStrike (July 2024), Google Cloud (June 2025) and Cloudflare (November and December 2025) each had an outage where a change reached everyone before anyone saw it fail [R15, R16, R18, R19].
  • The WHO surgical checklist pilot cut inpatient deaths from 1.5% to 0.8%, but a mandated rollout across 101 Ontario hospitals showed no significant change [R24, R25].
  • DORA's 2025 report says AI's benefits depend on small batches, and that frequent use of rollback amplifies AI's positive effect on team performance [R8, R9].

What is a release checklist, and what is smoke testing?

A release checklist is the short list of questions your team answers before and after every deploy, each one earned by a past failure. Smoke testing is the cheapest automated check that a freshly deployed build works at all.

Google's SRE book notes that checklists are used "to reduce failure and ensure consistency and completeness" [R12]. That's the whole job. Once a checklist also has to prove diligence or assign blame, it stops doing it.

The ISTQB glossary defines a smoke test as "A test suite that covers the main functionality of a component or system", run "to determine whether it works properly before planned testing begins" [R1]. It isn't a smaller regression suite. Regression asks whether anything that used to work has stopped working. Smoke asks whether this build, in this environment, works right now: can a user log in, load the main screen and do the one thing they pay for? It has to finish in minutes, because it runs on every deploy. Which tests belong in the full run is a separate decision, covered in your regression suite is mostly noise.

Don't put a sign-off line on your release checklist

The first line most teams write is "who signs off?". Without a QA team, that usually means the CTO, or whoever is online.

DORA studied this. Heavyweight change approval has "a negative impact on software delivery performance", and "no evidence was found to support the hypothesis that a more formal, external review process" was associated with lower change fail rates [R14]. DORA recommends peer review captured in your platform, plus automation, and says to "invest in making it quicker and safer to make changes" [R14].

Change fail rate is the share of deploys that need immediate intervention, usually a rollback or a hotfix [R3]. DORA's 2024 clusters [R4]:

DORA 2024 clusterChange fail rateShare of respondents
Elite5%19%
High20%22%
Medium10%35%
Low40%25%

Read the Medium row twice. Medium teams had a lower change fail rate than High teams [R4]. Shipping more often doesn't make each deploy safer on its own, and neither does a signature.

Checklists have a mixed record of their own. In the WHO surgical checklist pilot across 8 hospitals, as reported by the Harvard Gazette, major complications fell from 11% to 7% and inpatient deaths from 1.5% to 0.8% [R24]. The checklist "requires only a few minutes to complete" [R24]. Then Ontario mandated surgical checklists in 101 hospitals. As reported by HospiMedica, operative mortality (in hospital or within 30 days) went from 0.71% to 0.65% and complications from 3.86% to 3.82%, both statistically insignificant [R25]. The lead author, David Urbach, said: "One problem might be mandating the use of a checklist" [R25].

A checklist works when people actually run it. A mandated form that gets ticked from memory does not. Keep the human lines few, make each one earn its place, and automate the rest.

The question three big outages failed: does it reach everyone at once?

Google's SRE book sets the bar for any checklist line: "Every question's importance must be substantiated, ideally by a previous launch disaster" [R12]. You don't have to wait for your own. Three companies have published theirs.

CrowdStrike, July 2024. The Channel File 291 root cause analysis found that "Template Instances should have staged deployment" [R15]. Content that has "passed canary testing" now gets "successively promoted to wider deployment rings or rolled back if problems are detected", with bake-in time per ring [R15].

Google Cloud, June 2025. The rollout was staged by region, but the new code wasn't behind a flag, so the failing path never ran during it [R16]. The red-button fix was ready to roll out about 25 minutes after the incident began [R17].

Cloudflare, November and December 2025. On 18 November, a Bot Management feature file that is "published to our entire network" every few minutes carried a bad version across the network [R18]. Cloudflare called it its "worst outage since 2019" [R18]. On 5 December, about 28% of all HTTP traffic was affected for roughly 25 minutes [R19]. That change went through a global configuration system that "does not perform gradual rollouts" [R19]. It reached the entire fleet within seconds [R19]. Cloudflare's follow-up plan is to "treat configuration the same way that we treat code", with controlled rollouts for every config change and automatic rollback when a step fails [R20].

None of these were cases of nobody testing. Each was a change that reached everyone before anyone could see it fail.

Charity Majors, co-founder and CTO of Honeycomb, goes further. Staging can't usefully reproduce production's size and chaos, she argues, and "Your deploys are the cause of nearly all your failures" [R33]. Her answer is automated canaries and "Making rollbacks wicked fast and reliable" [R33]. Of testing before release, she also says: "We should never, ever stop doing this" [R33]. Google's outage shows how a careful staged rollout can still miss a broken code path [R16]. Her insistence on testing before release is why the smoke test stays on the list.

The release checklist: 10 lines in three stages

Use this as your pre-deployment checklist and your post-deploy check in one. Each line traces to an incident or finding above. Copy it, then delete any line your team can't justify.

Before merge

  1. Is the change small enough to review and roll back on its own? "Large changes are difficult to review, test, and integrate safely" [R10].
  2. Is new behaviour behind a flag that's off by default, with a date to remove it? That's Google Cloud's remediation [R16]. Release toggles "should generally not stick around much longer than a week or two" [R32].
  3. Is the database migration backward-compatible and reversible? Split breaking schema changes into expand, migrate and contract [R23]. GitLab holds migrations that can't easily be rolled back until after several successful deploys [R21].

Before deploy

  1. Do you know the rollback path, and has someone used it recently? DORA calls swift undo "a critical enabler of speed and resilience" [R9].
  2. Does the change reach a small slice of users first? A canary is "a partial and time-limited deployment of a change in a service and its evaluation" [R11]. Google's flag frameworks usually ramp to "between 1 and 10 percent" of users [R12].
  3. Is any config change getting the same review and staged rollout as code? Cloudflare learned this twice [R18, R19, R20].
  4. Did the automated tests pass on this exact build? Google's "Push on Green" teams deploy every build that passes all tests [R13].

After deploy

  1. Did the smoke test run against the deployed URL and pass? Automatically, on every deploy (next section).
  2. Was any flaky smoke result treated as a failure until someone explained it? Playwright marks a test that fails first and passes on retry as "flaky", not "passed" [R29].
  3. Is someone watching errors and latency for a set window, with a kill switch ready? Cloudflare's remediation includes "Enabling more global kill switches for features" [R18]. For what to watch, see what to monitor in production.

There's no "who signs off" line. Peer review already happened at merge [R14].

Then size it to what a bad release costs

Ten lines is a starting point, not a standard. At yousource every project comes with its own requirements, and the cost of getting a release wrong decides how heavy the gate should be. For example, a mobile wallet and an agriculture IoT application don't carry the same expectations. In a wallet, a bad release can touch people's money, so the smoke test has to cover the payment paths, and rollback has to account for transactions that already went through. In an IoT application the software runs on devices out in the field, so a small first slice (line 5) matters more than shipping fast.

So before you copy the list, write one line above it: what happens to a user if this release is wrong? Then add or drop lines to match the answer.

Make the smoke test run itself on every deploy

Automated smoke testing needs two pieces: a tagged subset and a trigger. In Playwright, tags "must start with @ symbol", and --grep runs the subset [R28]:

test('user can log in and see the dashboard', { tag: '@smoke' }, async ({ page }) => {
  // your smoke steps
});
npx playwright test --grep @smoke

For the trigger, Playwright's CI docs show a GitHub Actions job on deployment_status that "will start the tests after a GitHub Deployment went into the success state" [R27]:

on: deployment_status
jobs:
  smoke:
    if: github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest
    env:
      PLAYWRIGHT_TEST_BASE_URL: ${{ github.event.deployment_status.target_url }}
    steps:
      # check out, install dependencies and browsers as in your existing CI job
      - run: npx playwright test --grep @smoke

Services like Vercel use this pattern, according to the docs [R27]. GitHub won't trigger the run when a deployment status is set to inactive [R30].

Still want a human gate on production? Check your plan first. GitHub environments allow up to six required reviewers, only one of whom has to approve, and can stop people approving their own deploy [R31]. But required reviewers and wait timers only work on public repositories unless you're on GitHub Enterprise [R31]. Plenty of private SaaS repos don't have the gate their team assumes.

At the far end of this, GitLab.com deploys "up to 12 times" a day through Staging Canary, Production Canary, Staging Main and Production Main, with "QA validation" at multiple checkpoints [R21].

AI-sized PRs need a smaller release gate

AI coding tools make line 1 harder to pass. DORA's 2024 report estimated a 7.2% drop in delivery stability "for every 25% increase in AI adoption" [R6]. Its hypothesis: teams forgot "the importance of small batch sizes" [R6].

The 2025 report found AI now improves throughput, but "it still increases delivery instability" [R7]. "With a high degree of certainty", it says, AI's benefits depend on small batches, and its recommendation is blunt: "Enforce the discipline of working in small batches" [R8]. Teams that use rollback features more often also see a bigger boost to team performance from AI [R9].

So the gate gets smaller as the diffs get bigger. A 2,000-line AI-assisted PR fails line 1 before anyone opens a browser. Split it, flag it, ship the pieces.

Where an outside QA engineer fits in release validation

The checklist is yours. Nobody outside the team can set your rollback path or your flag policy. What a team without QA usually lacks is an owner for the smoke suite: someone to keep it short, keep it green for the right reasons and chase the flaky results.

That's one of the services on yousource's QA On Demand. The page describes "Release validation & smoke" as "Smoke and readiness checks before every deploy — each ship reaches production proven" [P2]. Playwright and GitHub Actions are on its tooling list [P7]. Single stream is $3,495/mo and Dual stream is $6,795/mo, with a "3-day turnaround per task" [P3, P4]. You can "Try one real task first" [P6].

Whoever owns it, start with your last three incidents. Write one checklist line for each, then cut any line no incident ever earned.

Frequently asked questions

What should be on a release checklist?

Lines that past failures earned: a small change, a flag that's off by default, a reversible migration, a known rollback path, a staged rollout, config reviewed like code, a passing smoke test on the deployed build, and someone watching after deploy [R10, R16, R19, R23]. Then size it to what a bad release would cost your users: a mobile wallet needs a heavier gate than an internal dashboard.

What's the difference between a smoke test and a regression test?

A smoke test checks that the main functionality of a deployed build works at all, and it runs on every deploy [R1]. A regression test checks that existing behaviour hasn't broken, and its quieter tests can run less often [P9].

Who should sign off on a release if we don't have a QA team?

Nobody needs to sign the deploy itself. DORA found no evidence that a more formal, external review process lowers change fail rates [R14]. Use peer review at merge, automated checks and a staged rollout.

Do feature flags make every release safe?

No. Flags add complexity and carry a cost, and several together create "a combinatoric explosion of possible toggle states" [R32]. Release flags "should generally not stick around much longer than a week or two" [R32].

Sources

QA On Demand
back to top
Book 15 min with Albert
Smiling man with short dark hair and glasses wearing a black suit, white shirt, and black tie against blue background.
Tell Albert what you're shipping.
He'll read this before joining the call. Phone number comes next, on the calendar step.
↳ info@you-source.com
↳ 4-hour response
Please wait while we retrieve meeting schedules.
Oops! There's a problem with your request. We're working on fixing it. Please try again later.