DevOps CI/CD interview questions

A well-designed build artifact is immutable once published, produced from a specific reproducible source commit, and promoted unchanged across environments. It should NOT bundle environment-specific secrets or config; those are injected at runtime so the same artifact can be safely deployed everywhere.

Common build-artifact formats in CI/CD pipelines include Docker container images, compiled JAR or WAR files, and ZIP archives of binaries and assets. Git commit messages are source-control metadata, not build artifacts.

Pipeline-as-code means defining CI/CD pipeline configuration in version-controlled text files such as YAML or Groovy, rather than configuring it manually through a web UI. This lets teams review, branch, and reproduce their build and deployment workflows exactly like application code.

In a Docker CI/CD deployment, the steps proceed in strict dependency order: first clone the source repository, then build the image from the Dockerfile, push it to a container registry, pull it on the deployment target, and finally start a container from it. Each step requires the previous step's output — you cannot build without source, push without a build, pull without a push, or run without a local image.

The evaluation cycle of a single automated deployment gate is: the preceding stage completes and triggers the gate; the gate collects data from configured data sources; the gate evaluates the collected data against defined criteria; the gate produces a pass or fail result; and if all criteria pass, the pipeline proceeds to the next stage. Each step strictly depends on the previous one — you must collect data before evaluating it, and evaluate before reporting a result.

False. Build artifacts like compiled binaries, JAR files, and Docker images should be stored in a dedicated artifact repository (e.g., Artifactory, Nexus, or a container registry), not committed to Git. Committing artifacts bloats the repository with large binary files, creates unnecessary merge conflicts, and duplicates data that can always be reproduced by rebuilding from source.

True. The same immutable build artifact that passed tests in staging should be promoted directly to production. Rebuilding from source for production risks producing a subtly different artifact due to changes in build environment, dependency versions, or transient state, which would break the guarantee that the tested software is exactly the shipped software.

A manual approval gate is a deployment checkpoint that requires explicit human approval before a build can be promoted to the next pipeline stage, adding a manual decision point to an otherwise automated release flow.

In CI/CD pipelines, a quality gate automatically blocks deployment when a measurable signal — such as test pass rate, code coverage, or static-analysis score — falls below the configured threshold.

In a canary deployment, the progressive rollout is governed by automated health gates that continuously monitor signals such as error rate and latency; if those metrics breach their thresholds, the pipeline triggers an automatic rollback to the previous stable version.

Pipeline-as-code lets you version, review, and branch pipeline changes alongside application code. Because the pipeline definition lives in a committed file (e.g., Jenkinsfile or .gitlab-ci.yml), every modification goes through the same code-review and history workflow as source code, giving you traceability, reproducibility, and team collaboration that a web-UI configuration cannot match.

The canonical CI/CD pipeline stage order is Source → Build → Test → Deploy. The pipeline first checks out code from version control, then compiles and packages the artifact, then runs automated tests against that artifact, and finally releases the validated artifact to the target environment. Each stage depends on the output of the prior stage.

Continuous delivery and continuous deployment both keep the codebase continuously releasable — every change is automatically built, tested, and proven deployable. The one difference is the production gate. Continuous delivery makes each change release-ready but a human approves the final push to production. Continuous deployment removes that gate: any change that passes the automated pipeline ships to production with no manual step. Continuous integration is the upstream practice both build on — frequent merges with automated testing on each commit.

The practice is called "build once, promote everywhere" (also referred to as promoting immutable artifacts or single-source artifact promotion). One key reason it is preferred is that it guarantees the exact binary tested in staging is the one deployed to production, eliminating drift caused by non-deterministic builds (e.g., differing dependency versions, toolchain updates, or environment-specific build flags) that could introduce defects between environments.

In a CI/CD pipeline, a canary gate samples live telemetry — such as error rate and p99 latency — for a fixed observation window after routing a small percentage of production traffic to the new release. The pipeline proceeds to full rollout only if every metric threshold stays within its defined limit throughout that window; otherwise the release is automatically rolled back.

A change to the shared library's default branch can modify every consuming pipeline with no commit in any application repository. Because library 'shared-pipeline-lib' specifies no pinned version, Jenkins resolves the library from its default branch at each build, so the library and the Jenkinsfile are not coupled to the same commit and the library's changes are reviewed in its own repo, not the application's.

The correct order is Source → Build → Package → Publish. A commit triggers the pipeline; source is compiled and linked into a binary; the binary is bundled with its runtime dependencies into a deployable artifact; and the packaged artifact is pushed to a container registry. Each stage depends on the direct output of the one before it, forming a strict prerequisite chain.

It re-deploys the exact previously-running artifact (app:git-2b7e08) with no rebuild, so the rollback is fast and reproducible — you get back the bytes that were known-good in prod

Gate architecture: Each pipeline has five stages with both team-configurable and org-mandated gates.

1. Build stage: Org-mandated gate — container image build + SAST scan + dependency vulnerability scan (fail on Critical CVEs). Team gate — unit tests with ≥80% coverage delta threshold.
2. Integration stage: Org-mandated gate — consumer-driven contract tests against the service's published API spec (Pact or equivalent). The gate fails if any registered consumer's contract is broken. Team gate — integration tests against a shared ephemeral environment.
3. Staging stage: Org-mandated gate — load test reaching 2× expected peak QPS; p99 latency must stay within SLO. Team gate — smoke tests and synthetic transaction checks.
4. Pre-prod (canary launch): The service is deployed to 5% of production traffic. Canary gate evaluates error rate (must not exceed baseline + 0.5 percentage points), p99 latency (must not exceed baseline × 1.2), and business KPI delta (e.g., checkout conversion rate must not drop more than 1% relative to control). If any metric breaches its threshold, the gate triggers an automatic rollback to the previous version. Metrics that are informative but not safety-critical (e.g., cache hit rate, memory usage trending up but within limits) trigger a hold-and-notify — deployment pauses at 5%, alerts the team, and waits for manual promotion or rollback.
5. Production (progressive rollout): After canary passes, traffic is ramped 25% → 50% → 100% with the same canary gate re-evaluated at each step. A failure at any step rolls back to the last known-good percentage.

Cross-service dependency gating: Every service publishes an OpenAPI/Protobuf spec in a central schema registry. Before a service can deploy to production, the gate runs backward-compatibility checks — if the new spec removes a required field or changes a type in a breaking way, the gate fails unless all downstream consumers have already deployed a compatible version. The dependency graph is maintained by a service catalog; the gate queries it to identify consumers. Consumer-driven contract tests run at the integration stage and must all pass; the gate blocks if any registered consumer's contract is broken by the new version.

Decentralization with enforced baseline: Gates are defined as code in a shared gate-library repository. Each team's pipeline imports a 'base-gate-profile' that contains the org-mandated gates (SAST, CVE scan, contract tests, load test, canary SLO gate). Teams can add team-specific gates on top but the base profile is enforced by a policy controller (OPA/Conftest) that validates every pipeline definition at commit time — if the base gates are missing or disabled, the pipeline config is rejected by the CI system and cannot run. This gives teams autonomy to extend while guaranteeing the safety floor.

Gate taxonomy and automation boundary:

Automated gates (no human action required, evidence captured automatically):

• Build gate: compilation + unit tests + coverage threshold (≥80%). Pass/fail is deterministic.
• Security gate: SAST, DAST, dependency CVE scan, container image scan. Fail on Critical/High.
• IaC gate: Terraform plan validation + policy-as-code check (no public S3 buckets, encryption required, etc.).
• Integration gate: contract tests + integration tests against a staging environment mirroring production config.
• Compliance gate: automated check that the deployment package matches the approved change ticket (e.g., Jira issue linked in the commit), and that all required automated gates have passed with their evidence stored.

Human-approval gate (the only manual step):

• Production release authorization: a designated approver (not the builder) reviews the compiled evidence package — test results, scan reports, change-ticket linkage, artifact hash — and explicitly approves promotion to production. This gate exists because SOX requires management authorization for changes to production financial systems, and a human must attest that the change was reviewed.

Separation of duties enforcement: The CI/CD platform enforces two disjoint RBAC roles: 'Deployer' (can trigger pipeline runs, merge code, push artifacts through staging) and 'Approver' (can authorize production promotion). The pipeline's production-promotion step checks the identity of the approver against the identity of the user who triggered the current pipeline run. If they match, the step is blocked and returns an error. Additionally, pull requests require at least one review from a member of the Approver group before merge, and the PR system prevents self-review. These are enforced by the platform (e.g., pipeline engine identity checks + SCM branch-protection rules), not by verbal policy.

Compliance evidence capture: Every gate — automated and manual — writes a structured evidence record to an immutable, append-only audit store (e.g., a dedicated evidence repository or a tamper-evident log service). Each record contains: gate name, gate version/config hash, input artifact hash, pass/fail result, detailed output (test report, scan findings), timestamp (UTC, from a trusted time source), and the identity that triggered or approved it. Records are correlated by a deployment ID that links every gate result to the specific artifact hash and change ticket. An auditor can query by deployment ID or date range and receive a complete, ordered timeline of every gate executed, its result, and who acted. The audit store uses hash-chaining (each record references the hash of the previous record) to make tampering detectable.

Bottleneck reduction: (1) The human approver never runs tests or scans manually — by the time approval is requested, all automated gates have passed and the evidence package is pre-assembled. The approver reviews a summary dashboard with green/red indicators and drill-down links, taking 2–5 minutes. (2) Production rollout uses canary delivery: the approver authorizes a 5% canary, automated health gates evaluate metrics for 15 minutes, and if they pass, the rollout auto-promotes to 100% without a second human approval. The human is in the loop once, not at every step. (3) For low-risk changes (documentation, non-functional config via feature flags), an expedited path auto-approves if the change classifier determines the diff touches only flag-config files — the compliance gate still records all evidence and the auto-approval decision with its rationale.

Breaking-change detection and expand-contract enforcement:

The gate includes a migration analyzer that parses each migration script (or diffs the before/after schema) and classifies every DDL operation:

• Breaking: DROP COLUMN, DROP TABLE, RENAME COLUMN, type narrowing (VARCHAR(255) → VARCHAR(50)), adding NOT NULL constraint without a default, removing a default.
• Non-breaking: ADD COLUMN (nullable or with default), ADD INDEX (using CONCURRENTLY), widening types, adding a table.

When the gate detects a breaking change, it checks the migration history and the currently-deployed application code version. For a DROP COLUMN, the gate requires that: (a) a prior migration added the replacement column, (b) the currently-deployed code version no longer references the old column (verified by checking that the deployed artifact's schema-config or ORM mapping doesn't include it), and (c) at least one full deployment cycle has passed since the expand migration. If any condition is unmet, the gate blocks the migration with a message like 'Contract phase blocked: column X still referenced by deployed code version Y.' This enforces expand-then-contract by making the gate stateful — it tracks which expand migrations have been deployed and only permits the corresponding contract migration after code deployment is verified.

Production-scale performance validation:

The gate does not trust staging data volume. Instead, it runs the migration against a production-sized data clone — a logical replica of production created from the latest PITR snapshot, restored to a disposable compute instance. The migration is executed against this clone with instrumentation: the gate captures wall-clock time, lock-wait time, and whether any statement holds an AccessExclusiveLock for more than a configurable threshold (e.g., 5 seconds). If the migration exceeds the time threshold or acquires a blocking lock, the gate fails and suggests an alternative (e.g., use CREATE INDEX CONCURRENTLY, or batch the UPDATE). For very large tables, the gate also runs EXPLAIN ANALYZE on destructive statements to estimate row-scan costs. The clone is sized to match production's largest table row counts, closing the staging-vs-production gap.

Migration-application deployment ordering:

The gate enforces ordering through a deployment manifest that each service submits with its pipeline. The manifest declares the migration type (expand, contract, or none) and the code compatibility level (backward-compatible with old schema, requires new schema). The gate enforces:

1. If the migration is 'expand' (additive): migration runs first, then new code is deployed. Old code remains compatible because the migration is additive.
2. If the migration is 'contract' (removing old schema): the gate verifies the currently-deployed code is the backward-compatible version from the expand phase, then runs the contract migration, then (optionally) deploys code that assumes the old schema is gone.
3. If no migration: standard code deployment. The pipeline engine refuses to proceed if the manifest's declared ordering doesn't match the gate's classification — e.g., if a team labels a DROP COLUMN migration as 'expand,' the analyzer catches the mismatch and blocks.

Rollback safety:

The gate classifies each migration as reversible or irreversible at analysis time:

• Reversible: ADD COLUMN (nullable), ADD INDEX, CREATE TABLE. These can be rolled back with DROP, and no committed data is lost (the column was new).
• Irreversible: DROP COLUMN (data in that column is destroyed), DROP TABLE, TRUNCATE, type narrowing (data may be truncated).

Policy: When a deployment fails after a migration has been applied, the system checks the migration's reversibility classification. If reversible, the migration is rolled back and the previous code version is redeployed. If irreversible, the system rolls back the code to the previous version but leaves the schema change in place — rolling back the migration would cause data loss. The incident is flagged for a forward-fix: the team must fix and redeploy rather than revert the schema. The gate also requires that all contract-phase migrations are preceded by a verified backup or that the data being removed has been archived — the gate checks for an archive artifact (e.g., a dump of the column's data stored in object storage) before permitting an irreversible migration to proceed.

Build the deployment artifact once, run fast feedback gates (lint, unit tests) first, then parallelize independent slower checks (integration tests, SAST) after those pass, and promote the same image through staging and production. This follows fail-fast sequencing and build-once-promote-everywhere: you get early signal on cheap failures, avoid redundant rebuilds, and guarantee the tested artifact is the one deployed.

The deployment pipeline progresses through these gates: (1) Commit gate — CI builds the artifact, runs unit tests and SAST; (2) Integration gate — deploys to a staging cluster and runs integration and contract tests; (3) Canary gate — deploys to 1% of traffic in a single AZ in the primary region with a 5-minute soak; (4) Progressive rollout gates — traffic shifts through 5%, 25%, 50%, and 100% in the primary region with a 10-minute soak and gate evaluation at each stage; (5) Multi-region gates — repeat stages 3-4 region-by-region in a defined order (primary → secondary → edge regions).

Complete gate pipeline from commit to production (c1): Every change starts at the commit gate, where CI builds the container image, runs unit tests, executes SAST/DAST scans, and publishes a signed artifact to the registry. Only a passing commit gate triggers the integration gate, which deploys the artifact to a staging cluster that mirrors production topology (minus scale) and runs integration tests, consumer-driven contract tests, and schema-compatibility checks. A failing integration gate blocks the artifact from ever reaching production traffic. Only after both pre-deployment gates pass does the system proceed to the canary gate, ensuring no code reaches live traffic without automated pre-deployment validation.

Canary SLI selection and baseline comparison (c2): Each service declares its SLIs via a policy file: error rate (HTTP 5xx / total requests), p99 latency, and saturation (CPU and connection pool utilization). The baseline is the previous deployed version running concurrently in the same environment. The platform collects metrics for both the canary and baseline cohorts using the same observability stack (e.g., Prometheus with canary/baseline labels injected by the service mesh). The comparison is relative: canary error rate must not exceed baseline by more than the service-defined tolerance.

Statistical rollback decision boundary (c3): The platform uses sequential hypothesis testing (specifically, a sequential probability ratio test, SPRT) on the error-rate delta between canary and baseline. Parameters: minimum sample size of 10,000 requests per cohort before evaluation begins; significance threshold of p < 0.01 for rollback (null hypothesis: canary error rate ≤ baseline); evaluation window of 60 seconds sampled every 10 seconds. If at any sample the SPRT rejects the null hypothesis (canary is worse), the system triggers automatic rollback. If after the full soak period the SPRT has not rejected the null and the canary is not worse, the gate promotes to the next traffic stage. A secondary check on p99 latency uses a non-inferiority test: canary p99 must not exceed baseline p99 by more than 10ms with 95% confidence. If either check fails, rollback fires. The 10-second sampling interval combined with the SPRT's sequential nature means that once statistical power is reached (minimum sample size met), a regression is detected and rollback initiated within the 90-second window.

Progressive traffic-shifting stages with soak gates (c4): Traffic shifts via the service mesh load balancer: 1% (5-min soak) → 5% (10-min soak) → 25% (10-min) → 50% (10-min) → 100% (10-min). At each stage boundary, the gate evaluates all SLIs using the same SPRT. If any gate fails, traffic is immediately shifted back to 0% for the canary version (full rollback to previous version) and the deployment is marked failed with an alert to the owning team. The soak times are minimums; the gate cannot advance until both the soak time has elapsed and the SPRT has not triggered rollback.

Blast radius containment (c5): Two mechanisms limit impact. First, sticky-session routing: users assigned to the canary cohort are pinned via a cookie or consistent hash so that the canary population is stable and does not rotate new users in during evaluation — this means at most 1% of users are exposed at the first canary stage. Second, single-AZ canary: the initial canary deploys to one AZ in one region, so a canary failure affects at most one AZ's worth of traffic in one region, and the mesh health checks drain the canary instances within seconds. On rollback, the service mesh removes canary endpoints from the load balancer pool; existing canary sessions are redirected to baseline instances. For stateful services, session draining waits for in-flight requests to complete (up to a 30-second drain timeout) before terminating canary instances.

Per-service customization with enforced platform invariants (c6): Services define their gate policy in a YAML file versioned alongside their code: custom SLIs (e.g., a payment service might add a 'payment success rate' SLI), custom thresholds (tolerance for error-rate delta, latency budget), and pre-deploy checks (e.g., a database migration dry-run). The platform loads this policy but applies a platform-level override layer that enforces non-negotiable invariants: the canary stage cannot be skipped or disabled; minimum soak times cannot be set below platform-defined floors (5 min for canary, 10 min for progressive stages); the SPRT significance threshold cannot be weakened below p < 0.01; and the rollback action is always platform-controlled (services cannot override the rollback trigger). If a service's policy conflicts with an invariant, the platform rejects the policy at the integration gate and the deployment cannot proceed. This model lets services customize their sensitivity while the platform guarantees that no service can weaken safety below the floor.

Multi-region rollout failure handling (c7): During multi-region rollout, each region runs its own independent canary and progressive stages. If region B's canary fails, region B rolls back immediately and independently — its traffic returns to the previous version without waiting for any cross-region coordination. The platform then pauses the rollout to all remaining un-deployed regions (region C, D, etc.) to prevent propagating a potentially broken build. Region A (already promoted to 100%) is not automatically rolled back. Instead, the platform raises a high-severity alert and surfaces the region B failure analysis (which SLIs failed, magnitude of regression, error signatures) to the owning team, who must explicitly decide whether to rollback region A. This alert-and-halt approach is chosen over automatic multi-region rollback because the failure signal in region B may be region-specific (e.g., a regional dependency outage, a data residency edge case) and would not necessarily reproduce in region A. An unnecessary multi-region rollback of healthy regions carries its own blast radius — it disrupts users currently served by the new (and apparently healthy) version in region A and consumes deployment capacity. The platform provides a one-click rollback for region A and, if the team identifies the failure as a global code defect, a single action to rollback all regions simultaneously.

Use a content-addressable storage (CAS) model keyed on the hash of inputs (source + toolchain + env vars + transitive deps), not on branch/commit alone. Bazel and Buck2 exemplify this: the build graph is hermetically sealed (all inputs explicitly declared), outputs are stored by their content hash, and remote execution nodes share a CAS-backed cache. A cache hit on node A is therefore deterministic on node B because the hash captures the full input set—toolchain version, compiler flags, env var values, and dependency hashes—making locality irrelevant.

Reproducibility means the build is a pure function of declared inputs: the same inputs produce byte-identical output regardless of time, worker, or environment. Reproducibility is the prerequisite—without it, cache keys derived from inputs are unreliable. Hermeticity is the enforcement mechanism: all inputs are explicitly declared and ambient state (wall clock, network, env vars, filesystem layout) is sandboxed or forbidden. Together, hermeticity guarantees that the input hash captures everything, which makes reproducibility provable and cache keys trustworthy.