Question

Scenario #1: Zombie Pods Causing NodeDrain to Hang

Answer 1

output:

Cluster Management K8s v1.23, On-prem bare metal, Systemd cgroups

Summary: Node drain stuck indefinitely due to unresponsive terminating pod.

What Happened:

A pod with a custom finalizer never completed termination, blocking kubectl drain. Even after the pod was marked for deletion, the API server kept waiting because the finalizer wasn’t removed.

Diagnosis Steps:

Checked kubectl get pods --all-namespaces -o wide to find lingering pods.
Found pod stuck in Terminating state for over 20 minutes.
Used kubectl describe pod to identify the presence of a custom finalizer.
Investigated controller logs managing the finalizer – the controller had crashed.

Root Cause:

Finalizer logic was never executed because its controller was down, leaving the pod undeletable.

Fix/Workaround:

kubectl patch pod  -p '{"metadata":{"finalizers":[]}}' --type=merge

Lessons Learned:

Finalizers should have timeout or fail-safe logic.

How to Avoid:

Avoid finalizers unless absolutely necessary.
Add monitoring for stuck Terminating pods.
Implement retry/timeout logic in finalizer controllers.

Answer 2

output:

Cluster Management K8s v1.24, GKE, heavy use of custom controllers

Summary: API server crashed due to flooding by a malfunctioning controller creating too many custom resources.

What Happened:

A bug in a controller created thousands of Custom Resources (CRs) in a tight reconciliation loop. Etcd was flooded, leading to slow writes, and the API server eventually became non-responsive.

Diagnosis Steps:

API latency increased, leading to 504 Gateway Timeout errors in kubectl.
Used kubectl get crds | wc -l to list all CRs.
Analyzed controller logs – found infinite reconcile on a specific CR type.
etcd disk I/O was maxed.

Root Cause:

Bad logic in reconcile loop: create was always called regardless of the state, creating resource floods.

Fix/Workaround:

• Scaled the controller to 0 replicas.
• Manually deleted thousands of stale CRs using batch deletion.

Lessons Learned:

Always test reconcile logic in a sandboxed cluster.

How to Avoid:

Implement create/update guards in reconciliation.
Add Prometheus alert for high CR count.

Answer 3

output:

Cluster Management K8s v1.21, Self-managed cluster, Static nodes

Summary: A rebooted node failed to rejoin the cluster due to kubelet identity mismatch.

What Happened:

After a kernel upgrade and reboot, a node didn’t appear in kubectl get nodes. The kubelet logs showed registration issues.

Diagnosis Steps:

Checked system logs and kubelet logs.
Noticed --hostname-override didn't match the node name registered earlier.
kubectl get nodes -o wide showed old hostname; new one mismatched due to DHCP/hostname change.

Root Cause:

Kubelet registered with a hostname that no longer matched its node identity in the cluster.

Fix/Workaround:

• Re-joined the node using correct --hostname-override.
• Cleaned up stale node entry from the cluster.

Lessons Learned:

Node identity must remain consistent across reboots.

How to Avoid:

Set static hostnames and IPs.
Use consistent cloud-init or kubeadm configuration.

Answer 4

output:

Cluster Management K8s v1.25, Bare-metal cluster

Summary: etcd ran out of disk space, making API server unresponsive.

What Happened:

The cluster started failing API requests. Etcd logs showed disk space errors, and API server logs showed failed storage operations.

Diagnosis Steps:

Used df -h on etcd nodes — confirmed disk full.
Reviewed /var/lib/etcd – excessive WAL and snapshot files.
Used etcdctl to assess DB size.

Root Cause:

Lack of compaction and snapshotting caused disk to fill up with historical revisions and WALs.

Fix/Workaround:

bash
CopyEdit
etcdctl compact 
etcdctl defrag
• Cleaned logs, snapshots, and increased disk space temporarily.

Lessons Learned:

etcd requires periodic maintenance.

How to Avoid:

Enable automatic compaction.
Monitor disk space usage of etcd volumes.

Answer 5

output:

Cluster Management K8s v1.26, Multi-tenant cluster

Summary: Critical workloads weren’t getting scheduled due to incorrect node taints.

What Happened:

A user added taints (NoSchedule) to all nodes to isolate their app, but forgot to include tolerations in workloads. Other apps stopped working.

Diagnosis Steps:

Pods stuck in Pending state.
Used kubectl describe pod – reason: no nodes match tolerations.
Inspected node taints via kubectl describe node.

Root Cause:

Lack of required tolerations on most workloads.

Fix/Workaround:

• Removed the inappropriate taints.
• Re-scheduled workloads.

Lessons Learned:

Node taints must be reviewed cluster-wide.

How to Avoid:

Educate teams on node taints and tolerations.
Restrict RBAC for node mutation.

Answer 6

output:

Cluster Management K8s v1.22, EKS

Summary: Continuous pod evictions caused by DiskPressure due to image bloating.

What Happened:

A new container image with many layers was deployed. Node’s disk filled up, triggering kubelet’s DiskPressure condition. Evicted pods created a loop.

Diagnosis Steps:

Checked node conditions: kubectl describe node showed DiskPressure: True.
Monitored image cache with crictl images.
Node /var/lib/containerd usage exceeded threshold.

Root Cause:

Excessive layering in container image and high pull churn caused disk exhaustion.

Fix/Workaround:

• Rebuilt image using multistage builds and removed unused layers.
• Increased ephemeral disk space temporarily.

Lessons Learned:

Container image size directly affects node stability.

How to Avoid:

Set resource requests/limits appropriately.
Use image scanning to reject bloated images.

Answer 7

output:

Cluster Management K8s v1.20, On-prem

Summary: One node dropped from the cluster due to TLS errors from time skew.

What Happened:

TLS handshakes between the API server and a node started failing. Node became NotReady. Investigation showed NTP daemon was down.

Diagnosis Steps:

Checked logs for TLS errors: “certificate expired or not yet valid”.
Used timedatectl to check drift – node was 45s behind.
NTP service was inactive.

Root Cause:

Large clock skew between node and control plane led to invalid TLS sessions.

Fix/Workaround:

• Restarted NTP sync.
• Restarted kubelet after sync.

Lessons Learned:

Clock sync is critical in TLS-based distributed systems.

How to Avoid:

Use chronyd or systemd-timesyncd.
Monitor clock skew across nodes.

Answer 8

output:

Cluster Management K8s v1.23, Azure AKS

Summary: An app spamming Kubernetes events slowed down the entire API server.

What Happened:

A custom controller logged frequent events (~50/second), causing the etcd event store to choke.

Diagnosis Steps:

Prometheus showed spike in event count.
kubectl get events --sort-by=.metadata.creationTimestamp showed massive spam.
Found misbehaving controller repeating failure events.

Root Cause:

No rate limiting on event creation in controller logic.

Fix/Workaround:

• Patched controller to rate-limit record.Eventf.
• Cleaned old events.

Lessons Learned:

Events are not free – they impact etcd/API server.

How to Avoid:

Use deduplicated or summarized event logic.
Set API server --event-ttl=1h and --eventRateLimit.

Answer 9

output:

Cluster Management K8s v1.24, DigitalOcean

Summary: CoreDNS pods kept crashing due to a misconfigured Corefile.

What Happened:

A team added a custom rewrite rule in the Corefile which had invalid syntax. CoreDNS failed to start.

Diagnosis Steps:

Checked logs: syntax error on startup.
Used kubectl describe configmap coredns -n kube-system to inspect.
Reproduced issue in test cluster.

Root Cause:

Corefile misconfigured – incorrect directive placement.

Fix/Workaround:

• Reverted to backup configmap.
• Restarted CoreDNS.

Lessons Learned:

DNS misconfigurations can cascade quickly.

How to Avoid:

Use a CoreDNS validator before applying config.
Maintain versioned backups of Corefile.

Answer 10

output:

Cluster Management K8s v1.18, On-prem, Flannel CNI

Summary: Misaligned pod CIDRs caused overlay misrouting and API server failure.

What Happened:

A new node was added with a different pod CIDR than what Flannel expected. This broke pod-to-pod and node-to-control-plane communication.

Diagnosis Steps:

kubectl timed out from nodes.
Logs showed dropped traffic in iptables.
Compared --pod-cidr in kubelet and Flannel config.

Root Cause:

Pod CIDRs weren’t consistent across node and Flannel.

Fix/Workaround:

• Reconfigured node with proper CIDR range.
• Flushed iptables and restarted Flannel.

Lessons Learned:

CNI requires strict configuration consistency.

How to Avoid:

Enforce CIDR policy via admission control.
Validate podCIDR ranges before adding new nodes.

Answer 11

output:

Cluster Management K8s v1.22, On-prem with kube-proxy in IPTables mode

Summary: Services became unreachable due to overlapping custom IPTables rules with kube-proxy rules.

What Happened:

A system admin added custom IPTables NAT rules for external routing, which inadvertently modified the same chains managed by kube-proxy.

Diagnosis Steps:

DNS and service access failing intermittently.
Ran iptables-save | grep KUBE- – found modified chains.
Checked kube-proxy logs: warnings about rule insert failures.

Root Cause:

Manual IPTables rules conflicted with KUBE-SERVICES chains, causing rule precedence issues.

Fix/Workaround:

• Flushed custom rules and reloaded kube-proxy.
bash
CopyEdit
iptables -F; systemctl restart kube-proxy

Lessons Learned:

Never mix manual IPTables rules with kube-proxy-managed chains.

How to Avoid:

Use separate IPTables chains or policy routing.
Document any node-level firewall rules clearly.

Answer 12

output:

Cluster Management K8s v1.20, kubeadm cluster

Summary: New nodes couldn’t join due to a backlog of unapproved CSRs.

What Happened:

A spike in expired certificate renewals caused hundreds of CSRs to queue, none of which were being auto-approved. New nodes waited indefinitely.

Diagnosis Steps:

Ran kubectl get csr – saw >500 pending requests.
New nodes stuck at kubelet: “waiting for server signing”.
Approval controller was disabled due to misconfiguration.

Root Cause:

Auto-approval for CSRs was turned off during a security patch, but not re-enabled.

Fix/Workaround:

bash
CopyEdit
kubectl certificate approve 
• Re-enabled the CSR approver controller.

Lessons Learned:

CSR management is critical for kubelet-node communication.

How to Avoid:

Monitor pending CSRs.
Don’t disable kube-controller-manager flags like --cluster-signing-cert-file.

Answer 13

output:

Cluster Management K8s v1.21 → v1.23 upgrade, kubeadm

Summary: Upgrade failed when static control plane pods weren’t ready due to invalid manifests.

What Happened:

During upgrade, etcd didn’t come up because its pod manifest had a typo. Kubelet never started etcd, causing control plane install to hang.

Diagnosis Steps:

Checked /etc/kubernetes/manifests/etcd.yaml for errors.
Used journalctl -u kubelet to see static pod startup errors.
Verified pod not running via crictl ps.

Root Cause:

Human error editing the static pod manifest – invalid volumeMount path.

Fix/Workaround:

• Fixed manifest.
• Restarted kubelet to load corrected pod.

Lessons Learned:

Static pods need strict validation.

How to Avoid:

Use YAML linter on static manifests.
Backup manifests before upgrade.

Answer 14

output:

Cluster Management K8s v1.24, AWS EKS, containerd

Summary: Application pods generated excessive logs, filling up node /var/log.

What Happened:

A debug flag was accidentally enabled in a backend pod, logging hundreds of lines/sec. The journald and container logs filled up all disk space.

Diagnosis Steps:

df -h showed /var/log full.
Checked /var/log/containers/ – massive logs for one pod.
Used kubectl logs to confirm excessive output.

Root Cause:

A log level misconfiguration caused explosive growth in logs.

Fix/Workaround:

• Rotated and truncated logs.
• Restarted container runtime after cleanup.
• Disabled debug logging.

Lessons Learned:

Logging should be controlled and bounded.

How to Avoid:

Set log rotation policies for container runtimes.
Enforce sane log levels via CI/CD validation.

Answer 15

output:

Cluster Management K8s v1.21, production cluster with HPA and PDB

Summary: kubectl drain never completed because PDBs blocked eviction.

What Happened:

A deployment had minAvailable: 2 in PDB, but only 2 pods were running. Node drain couldn’t evict either pod without violating PDB.

Diagnosis Steps:

Ran kubectl describe pdb – saw AllowedDisruptions: 0.
Checked deployment and replica count.
Tried drain – stuck on pod eviction for 10+ minutes.

Root Cause:

PDB guarantees clashed with under-scaled deployment.

Fix/Workaround:

• Temporarily edited PDB to reduce minAvailable.
• Scaled up replicas before drain.

Lessons Learned:

PDBs require careful coordination with replica count.

How to Avoid:

Validate PDBs during deployment scale-downs.
Create alerts for PDB blocking evictions.

Answer 16

output:

Cluster Management K8s v1.23, self-hosted control plane

Summary: Controller-manager crashed on startup due to outdated admission controller configuration.

What Happened:

After an upgrade, the --enable-admission-plugins flag included a deprecated plugin, causing crash.

Diagnosis Steps:

Checked pod logs in /var/log/pods/.
Saw panic error: “unknown admission plugin”.
Compared plugin list with K8s documentation.

Root Cause:

Version mismatch between config and actual controller-manager binary.

Fix/Workaround:

• Removed the deprecated plugin from startup flags.
• Restarted pod.

Lessons Learned:

Admission plugin deprecations are silent but fatal.

How to Avoid:

Track deprecations in each Kubernetes version.
Automate validation of startup flags.

Answer 17

output:

Cluster Management K8s v1.24, Velero-based etcd backup

Summary: A partial etcd restore led to stale object references and broken dependencies.

What Happened:

etcd snapshot was restored, but PVCs and secrets weren’t included. Many pods failed to mount or pull secrets.

Diagnosis Steps:

Pods failed with “volume not found” and “secret missing”.
kubectl get pvc --all-namespaces returned empty.
Compared resource counts pre- and post-restore.

Root Cause:

Restore did not include volume snapshots or Kubernetes secrets, leading to an incomplete object graph.

Fix/Workaround:

• Manually recreated PVCs and secrets using backups from another tool.
• Redeployed apps.

Lessons Learned:

etcd backup is not enough alone.

How to Avoid:

Use backup tools that support volume + etcd (e.g., Velero with restic).
Periodically test full cluster restores.

Answer 18

output:

Cluster Management K8s v1.25, Corporate proxy network

Summary: Nodes failed to pull images from DockerHub due to incorrect proxy environment configuration.

What Happened:

New kubelet config missed NO_PROXY=10.0.0.0/8,kubernetes.default.svc, causing internal DNS failures and image pull errors.

Diagnosis Steps:

kubectl describe pod showed ImagePullBackOff.
Checked environment variables for kubelet via systemctl show kubelet.
Verified lack of NO_PROXY.

Root Cause:

Proxy config caused kubelet to route internal cluster DNS and registry traffic through the proxy.

Fix/Workaround:

• Updated kubelet service file to include proper NO_PROXY.
• Restarted kubelet.

Lessons Learned:

Proxies in K8s require deep planning.

How to Avoid:

Always set NO_PROXY with service CIDRs and cluster domains.
Test image pulls with isolated nodes first.

Answer 19

output:

Cluster Management K8s v1.22, Bare-metal, bonded NICs

Summary: Flapping interface on switch caused nodes to be marked NotReady intermittently.

What Happened:

A network switch port had flapping issues, leading to periodic loss of node heartbeats.

Diagnosis Steps:

Node status flapped between Ready and NotReady.
Checked NIC logs via dmesg and ethtool.
Observed link flaps in switch logs.

Root Cause:

Hardware or cable issue causing loss of connectivity.

Fix/Workaround:

• Replaced cable and switch port.
• Set up redundant bonding with failover.

Lessons Learned:

Physical layer issues can appear as node flakiness.

How to Avoid:

Monitor NIC link status and configure bonding.
Proactively audit switch port health.

Answer 20

output:

Cluster Management K8s v1.24, DaemonSet-based node config

Summary: A DaemonSet used for node labeling overwrote existing labels used by schedulers.

What Happened:

A platform team deployed a DaemonSet that set node labels like zone=us-east, but it overwrote custom labels like gpu=true.

Diagnosis Steps:

Pods no longer scheduled to GPU nodes.
kubectl get nodes --show-labels showed gpu label missing.
Checked DaemonSet script – labels were overwritten, not merged.

Root Cause:

Label management script used kubectl label node key=value --overwrite, removing other labels.

Fix/Workaround:

• Restored original labels from backup.
• Updated script to merge labels.

Lessons Learned:

Node labels are critical for scheduling decisions.

How to Avoid:

Use label merging logic (e.g., fetch current labels, then patch).
Protect key node labels via admission controllers.

Answer 21

output:

Cluster Management K8s v1.24, AWS EKS with Cluster Autoscaler

Summary: The cluster was rapidly scaling up and down, creating instability in workloads.

What Happened:

A misconfigured deployment had a readiness probe that failed intermittently, making pods seem unready. Cluster Autoscaler detected these as unschedulable, triggering new node provisioning. Once the pod appeared healthy again, Autoscaler would scale down.

Diagnosis Steps:

Monitored Cluster Autoscaler logs (kubectl -n kube-system logs -l app=cluster-autoscaler).
Identified repeated scale-up and scale-down messages.
Traced back to a specific deployment’s readiness probe.

Root Cause:

Flaky readiness probe created false unschedulable pods.

Fix/Workaround:

• Fixed the readiness probe to accurately reflect pod health.
• Tuned scale-down-delay-after-add and scale-down-unneeded-time settings.

Lessons Learned:

Readiness probes directly impact Autoscaler decisions.

How to Avoid:

Validate all probes before production deployments.
Use Autoscaler logging to audit scaling activity.

Answer 22

output:

Cluster Management K8s v1.21, self-managed

Summary: A namespace remained in “Terminating” state indefinitely.

What Happened:

The namespace contained resources with finalizers pointing to a deleted controller. Kubernetes waited forever for the finalizer to complete cleanup.

Diagnosis Steps:

Ran kubectl get ns -o json – saw dangling finalizers.
Checked for the corresponding CRD/controller – it was uninstalled.

Root Cause:

Finalizers without owning controller cause resource lifecycle deadlocks.

Fix/Workaround:

• Manually removed finalizers using a patched JSON:
bash
CopyEdit
kubectl patch ns  -p '{"spec":{"finalizers":[]}}' --type=merge

Lessons Learned:

Always delete CRs before removing the CRD or controller.

How to Avoid:

Implement controller cleanup logic.
Audit finalizers periodically.

Answer 23

output:

Cluster Management K8s v1.23, managed GKE

Summary: CoreDNS stopped resolving names cluster-wide after a config update.

What Happened:

A platform engineer edited the CoreDNS ConfigMap to add a rewrite rule, but introduced a syntax error. The new pods started crashing, and DNS resolution stopped working across the cluster.

Diagnosis Steps:

Ran kubectl logs -n kube-system -l k8s-app=kube-dns – saw config parse errors.
Used kubectl describe pod to confirm CrashLoopBackOff.
Validated config against CoreDNS docs.

Root Cause:

Invalid configuration line in CoreDNS ConfigMap.

Fix/Workaround:

• Rolled back to previous working ConfigMap.
• Restarted CoreDNS pods to pick up change.

Lessons Learned:

ConfigMap changes can instantly affect cluster-wide services.

How to Avoid:

Use coredns -conf locally to validate changes.
Test changes in a non-prod namespace before rollout.

Answer 24

output:

Cluster Management K8s v1.25, self-managed, containerd

Summary: A sudden spike in image pulls caused all nodes to hit disk pressure, leading to massive pod evictions.

What Happened:

A nightly batch job triggered a container image update across thousands of pods. Pulling these images used all available space in /var/lib/containerd, which led to node condition DiskPressure, forcing eviction of critical workloads.

Diagnosis Steps:

Used kubectl describe node – found DiskPressure=True.
Inspected /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/.
Checked image pull logs.

Root Cause:

No image GC and too many simultaneous pulls filled up disk space.

Fix/Workaround:

• Pruned unused images.
• Enabled container runtime garbage collection.

Lessons Learned:

DiskPressure can take down entire nodes without warning.

How to Avoid:

Set eviction thresholds properly in kubelet.
Enforce rolling update limits (maxUnavailable).

Answer 25

output:

Cluster Management K8s v1.20, CSI storage on vSphere

Summary: PVCs were stuck in Pending state due to existing orphaned PVs in Released state.

What Happened:

After pod deletion, PVs went into Released state but were never cleaned up due to missing ReclaimPolicy logic. When new PVCs requested the same storage class, provisioning failed.

Diagnosis Steps:

Ran kubectl get pvc – saw Pending PVCs.
kubectl get pv – old PVs stuck in Released.
CSI driver logs showed volume claim conflicts.

Root Cause:

ReclaimPolicy set to Retain and no manual cleanup.

Fix/Workaround:

• Manually deleted orphaned PVs.
• Changed ReclaimPolicy to Delete for similar volumes.

Lessons Learned:

PV lifecycle must be actively monitored.

How to Avoid:

Add cleanup logic in storage lifecycle.
Implement PV alerts based on state.

Answer 26

output:

Cluster Management K8s v1.22, managed AKS

Summary: Workloads failed to schedule on new nodes that had a taint the workloads didn’t tolerate.

What Happened:

Platform team added a new node pool with node-role.kubernetes.io/gpu:NoSchedule, but forgot to add tolerations to GPU workloads.

Diagnosis Steps:

kubectl describe pod – showed reason: “0/3 nodes are available: node(s) had taints”.
Checked node taints via kubectl get nodes -o json.

Root Cause:

Taints on new node pool weren’t matched by tolerations in pods.

Fix/Workaround:

• Added proper tolerations to workloads:
yaml
CopyEdit
tolerations:
- key: "node-role.kubernetes.io/gpu"
operator: "Exists"
effect: "NoSchedule"

Lessons Learned:

Node taints should be coordinated with scheduling policies.

How to Avoid:

Use preset toleration templates in CI/CD pipelines.
Test new node pools with dummy workloads.

Answer 27

output:

Cluster Management K8s v1.21, on-prem, private registry

Summary: New nodes failed to join the cluster due to container runtime timeout when pulling base images.

What Happened:

The internal Docker registry was down during node provisioning, so containerd couldn't pull pauseand CNI images. Nodes stayed in NotReady state.

Diagnosis Steps:

journalctl -u containerd – repeated image pull failures.
Node conditions showed ContainerRuntimeNotReady.

Root Cause:

Bootstrap process relies on image pulls from unavailable registry.

Fix/Workaround:

• Brought internal registry back online.
• Pre-pulled pause/CNI images to node image templates.

Lessons Learned:

Registry availability is a bootstrap dependency.

How to Avoid:

Preload all essential images into AMI/base image.
Monitor registry uptime independently.

Answer 28

output:

Cluster Management K8s v1.19, kubeadm cluster

Summary: Several nodes went NotReady after reboot due to kubelet failing to start with expired client certs.

What Happened:

Kubelet uses a client certificate for authentication with the API server. These are typically auto-rotated, but the nodes were offline when the rotation was due.

Diagnosis Steps:

journalctl -u kubelet – cert expired error.
/var/lib/kubelet/pki/kubelet-client-current.pem – expired date.

Root Cause:

Kubelet cert rotation missed due to node downtime.

Fix/Workaround:

• Regenerated kubelet certs using kubeadm.
bash
CopyEdit
kubeadm certs renew all

Lessons Learned:

Cert rotation has a dependency on uptime.

How to Avoid:

Monitor cert expiry proactively.
Rotate certs manually before planned outages.

Answer 29

output:

Cluster Management K8s v1.24, custom scheduler deployment

Summary: kube-scheduler pod failed with panic due to misconfigured leader election flags.

What Happened:

An override in the Helm chart introduced an invalid leader election namespace, causing the scheduler to panic and crash on startup.

Diagnosis Steps:

Pod logs showed panic: cannot create leader election record.
Checked Helm values – found wrong namespace name.

Root Cause:

Namespace specified for leader election did not exist.

Fix/Workaround:

• Created the missing namespace.
• Restarted the scheduler pod.

Lessons Learned:

Leader election is sensitive to namespace scoping.

How to Avoid:

Use default kube-system unless explicitly scoped.
Validate all scheduler configs with CI linting.

Answer 30

output:

Cluster Management K8s v1.23, self-hosted Calico

Summary: DNS resolution broke after Calico CNI update due to iptables policy drop changes.

What Happened:

New version of Calico enforced stricter iptables drop policies, blocking traffic from CoreDNS to pods.

Diagnosis Steps:

DNS requests timed out.
Packet capture showed ICMP unreachable from pods to CoreDNS.
Checked Calico policy and iptables rules.

Root Cause:

Calico’s default deny policy applied to kube-dns traffic.

Fix/Workaround:

• Added explicit Calico policy allowing kube-dns to pod traffic.
yaml:
egress:
- action: Allow
destination:
selector: "k8s-app == 'kube-dns'"

Lessons Learned:

CNI policy changes can impact DNS without warning.

How to Avoid:

Review and test all network policy upgrades in staging.
Use canary upgrade strategy for CNI.

Answer 31

output:

Cluster Management K8s v1.22, on-prem, kubeadm

Summary: Authentication tokens failed across the cluster due to node clock skew.

What Happened:

Token-based authentication failed for all workloads and kubectl access due to time drift between worker nodes and the API server.

Diagnosis Steps:

Ran kubectl logs and found expired token errors.
Checked node time using date on each node – found significant drift.
Verified NTP daemon status – not running.

Root Cause:

NTP daemon disabled on worker nodes.

Fix/Workaround:

• Re-enabled and restarted NTP on all nodes.
• Synchronized system clocks manually.

Lessons Learned:

Time synchronization is critical for certificate and token-based auth.

How to Avoid:

Ensure NTP or chrony is enabled via bootstrap configuration.
Monitor time drift via node-exporter.

Answer 32

output:

Cluster Management K8s v1.24, multi-zone GKE

Summary: Zone-aware workloads failed to schedule due to missing zone labels on some nodes.

What Happened:

Pods using topologySpreadConstraints for zone balancing failed to find valid nodes because some nodes lacked the topology.kubernetes.io/zone label.

Diagnosis Steps:

Pod events showed no matching topology key errors.
Compared node labels across zones – found inconsistency.

Root Cause:

A few nodes were manually added without required zone labels.

Fix/Workaround:

• Manually patched node labels to restore zone metadata.

Lessons Learned:

Label uniformity is essential for topology constraints.

How to Avoid:

Automate label injection using cloud-init or DaemonSet.
Add CI checks for required labels on node join.

Answer 33

output:

Cluster Management K8s v1.23, OpenShift

Summary: API latency rose sharply due to thousands of watch connections from misbehaving clients.

What Happened:

Multiple pods opened persistent watch connections and never closed them, overloading the API server.

Diagnosis Steps:

Monitored API metrics /metrics for apiserver_registered_watchers.
Identified top offenders using connection source IPs.

Root Cause:

Custom controller with poor watch logic never closed connections.

Fix/Workaround:

• Restarted offending pods.
• Updated controller to reuse watches.

Lessons Learned:

Unbounded watches can exhaust server resources.

How to Avoid:

Use client-go with resync periods and connection limits.
Enable metrics to detect watch leaks early.

Answer 34

output:

Cluster Management K8s v1.21, self-managed with local etcd

Summary: Entire control plane crashed due to etcd disk running out of space.

What Happened:

Continuous writes from custom resources filled the disk where etcd data was stored.

Diagnosis Steps:

Observed etcdserver: mvcc: database space exceeded errors.
Checked disk usage: df -h showed 100% full.

Root Cause:

No compaction or defragmentation done on etcd for weeks.

Fix/Workaround:

• Performed etcd compaction and defragmentation.
• Added disk space temporarily.

Lessons Learned:

Etcd needs regular maintenance.

How to Avoid:

Set up cron jobs or alerts for etcd health.
Monitor disk usage and trigger auto-compaction.

Answer 35

output:

Cluster Management K8s v1.24, Rancher

Summary: A user accidentally deleted the kube-root-ca.crt ConfigMap, which many workloads relied on.

What Happened:

Pods mounting the kube-root-ca.crt ConfigMap failed to start after deletion. DNS, metrics-server, and other system components failed.

Diagnosis Steps:

Pod events showed missing ConfigMap errors.
Attempted to remount volumes manually.

Root Cause:

System-critical ConfigMap was deleted without RBAC protections.

Fix/Workaround:

• Recreated ConfigMap from backup.
• Re-deployed affected system workloads.

Lessons Learned:

Some ConfigMaps are essential and must be protected.

How to Avoid:

Add RBAC restrictions to system namespaces.
Use OPA/Gatekeeper to prevent deletions of protected resources.

Answer 36

output:

Cluster Management K8s v1.22, Azure AKS

Summary: A critical deployment was unschedulable due to strict nodeAffinity rules.

What Happened:

nodeAffinity required a zone that did not exist in the cluster, making all nodes invalid.

Diagnosis Steps:

Pod events showed 0/10 nodes available errors.
Checked spec.affinity section in deployment YAML.

Root Cause:

Invalid or overly strict requiredDuringScheduling nodeAffinity.

Fix/Workaround:

• Updated deployment YAML to reflect actual zones.
• Re-deployed workloads.

Lessons Learned:

nodeAffinity is strict and should be used carefully.

How to Avoid:

Validate node labels before setting affinity.
Use preferredDuringScheduling for soft constraints.

Answer 37

output:

Cluster Management K8s v1.25, self-hosted

Summary: A stale mutating webhook caused all deployments to fail due to TLS certificate errors.

What Happened:

The admission webhook had expired TLS certs, causing validation errors on all resource creation attempts.

Diagnosis Steps:

Created a dummy pod and observed webhook errors.
Checked logs of the webhook pod – found TLS handshake failures.

Root Cause:

Webhook server was down due to expired TLS cert.

Fix/Workaround:

• Renewed cert and redeployed webhook.
• Disabled webhook temporarily for emergency deployments.

Lessons Learned:

Webhooks are gatekeepers – they must be monitored.

How to Avoid:

Rotate webhook certs using cert-manager.
Alert on webhook downtime or errors.

Answer 38

output:

Cluster Management K8s v1.19, kubeadm

Summary: After 1 year of uptime, API server certificate expired, blocking access to all components.

What Happened:

Default kubeadm cert rotation didn’t occur, leading to expiry of API server and etcd peer certs.

Diagnosis Steps:

kubectl failed with x509: certificate has expired.
Checked /etc/kubernetes/pki/apiserver.crt expiry date.

Root Cause:

kubeadm certificates were never rotated or renewed.

Fix/Workaround:

• Used kubeadm certs renew all.
• Restarted control plane components.

Lessons Learned:

Certificates expire silently unless monitored.

How to Avoid:

Rotate certs before expiry.
Monitor /metrics for cert validity.

Answer 39

output:

Cluster Management K8s v1.22, containerd switch

Summary: kubelet failed to start after switching from Docker to containerd due to incorrect CRI socket path.

What Happened:

The node image had containerd installed, but the kubelet still pointed to the Docker socket.

Diagnosis Steps:

Checked kubelet logs for failed to connect to CRI socket.
Verified config file at /var/lib/kubelet/kubeadm-flags.env.

Root Cause:

Wrong --container-runtime-endpoint specified.

Fix/Workaround:

• Updated kubelet flags to point to /run/containerd/containerd.sock.
• Restarted kubelet.

Lessons Learned:

CRI migration requires explicit config updates.

How to Avoid:

Use migration scripts or kubeadm migration guides.
Validate container runtime on node bootstrap.

Answer 40

output:

Cluster Management K8s v1.24, multi-tenant namespace setup

Summary: Cluster workloads failed after applying overly strict resource quotas that denied new pod creation.

What Happened:

A new quota was applied with very low CPU/memory limits. All new pods across namespaces failed scheduling.

Diagnosis Steps:

Pod events showed failed quota check errors.
Checked quota via kubectl describe quota in all namespaces.

Root Cause:

Misconfigured CPU/memory limits set globally.

Fix/Workaround:

• Rolled back the quota to previous values.
• Unblocked critical namespaces manually.

Lessons Learned:

Quota changes should be staged and validated.

How to Avoid:

Test new quotas in shadow or dry-run mode.
Use automated checks before applying quotas.

Answer 41

output:

Cluster Management K8s v1.21 to v1.22, custom CNI plugin

Summary: Cluster upgrade failed due to an incompatible version of the CNI plugin.

What Happened:

After upgrading the control plane, CNI plugins failed to work, resulting in no network connectivity between pods.

Diagnosis Steps:

Checked kubelet and container runtime logs – observed CNI errors.
Verified CNI plugin version – it was incompatible with K8s v1.22.

Root Cause:

CNI plugin was not upgraded alongside the Kubernetes control plane.

Fix/Workaround:

• Upgraded the CNI plugin to the version compatible with K8s v1.22.
• Restarted affected pods and nodes.

Lessons Learned:

Always ensure compatibility between the Kubernetes version and CNI plugin.

How to Avoid:

Follow Kubernetes upgrade documentation and ensure CNI plugins are upgraded.
Test in a staging environment before performing production upgrades.

Answer 42

output:

Cluster Management K8s v1.22, AWS EKS

Summary: Privileged containers were able to run despite Pod Security Policy enforcement.

What Happened:

A container was able to run as privileged despite a restrictive PodSecurityPolicy being in place.

Diagnosis Steps:

Checked pod events and logs, found no violations of PodSecurityPolicy.
Verified PodSecurityPolicy settings and namespace annotations.

Root Cause:

PodSecurityPolicy was not enforced due to missing podsecuritypolicy admission controller.

Fix/Workaround:

• Enabled the podsecuritypolicy admission controller.
• Updated the PodSecurityPolicy to restrict privileged containers.

Lessons Learned:

Admission controllers must be properly configured for security policies to be enforced.

How to Avoid:

Double-check admission controller configurations during initial cluster setup.
Regularly audit security policies and admission controllers.

Answer 43

output:

Cluster Management K8s v1.24, GKE

Summary: StatefulSet pods were rescheduled across different nodes, breaking persistent volume bindings.

What Happened:

Node pool scaling in GKE triggered a rescheduling of StatefulSet pods, breaking persistent volume claims that were tied to specific nodes.

Diagnosis Steps:

Observed failed to bind volume errors.
Checked StatefulSet configuration for node affinity and volume binding policies.

Root Cause:

Lack of proper node affinity or persistent volume binding policies in StatefulSet configuration.

Fix/Workaround:

• Added proper node affinity rules and volume binding policies to StatefulSet.
• Rescheduled the pods successfully.

Lessons Learned:

StatefulSets require careful management of node affinity and persistent volume binding policies.

How to Avoid:

Use pod affinity rules for StatefulSets to ensure proper scheduling and volume binding.
Monitor volume binding status when scaling node pools.

Answer 44

output:

Cluster Management K8s v1.20, bare metal

Summary: Kubelet crashed after running out of memory due to excessive pod resource usage.

What Happened:

The kubelet on a node crashed after the available memory was exhausted due to pods consuming more memory than allocated.

Diagnosis Steps:

Checked kubelet logs for OOM errors.
Used kubectl describe node to check resource utilization.

Root Cause:

Pod resource requests and limits were not set properly, leading to excessive memory consumption.

Fix/Workaround:

• Set proper resource requests and limits on pods to prevent memory over-consumption.
• Restarted the kubelet on the affected node.

Lessons Learned:

Pod resource limits and requests are essential for proper node resource utilization.

How to Avoid:

Set reasonable resource requests and limits for all pods.
Monitor node resource usage to catch resource overuse before it causes crashes.

Answer 45

output:

Cluster Management K8s v1.23, multi-cluster federation

Summary: DNS resolution failed between two federated clusters due to missing DNS records.

What Happened:

DNS queries failed between two federated clusters, preventing services from accessing each other across clusters.

Diagnosis Steps:

Used kubectl get svc to check DNS records.
Identified missing service entries in the DNS server configuration.

Root Cause:

DNS configuration was incomplete, missing records for federated services.

Fix/Workaround:

• Added missing DNS records manually.
• Updated DNS configurations to include service records for all federated clusters.

Lessons Learned:

In multi-cluster setups, DNS configuration is critical to service discovery.

How to Avoid:

Automate DNS record creation during multi-cluster federation setup.
Regularly audit DNS configurations in multi-cluster environments.

Answer 46

output:

Cluster Management K8s v1.21, GKE with Horizontal Pod Autoscaler (HPA)

Summary: Horizontal Pod Autoscaler did not scale pods up as expected due to insufficient resource limits.

What Happened:

The Horizontal Pod Autoscaler failed to scale the application pods up, even under load, due to insufficient resource limits set on the pods.

Diagnosis Steps:

Observed HPA metrics showing no scaling action.
Checked pod resource requests and limits.

Root Cause:

Resource limits were too low for HPA to trigger scaling actions.

Fix/Workaround:

• Increased resource requests and limits for the affected pods.
• Manually scaled the pods and monitored the autoscaling behavior.

Lessons Learned:

Proper resource limits are essential for autoscaling to function correctly.

How to Avoid:

Set adequate resource requests and limits for workloads managed by HPA.
Monitor autoscaling events to identify under-scaling issues.

Answer 47

output:

Cluster Management K8s v1.22, Azure AKS

Summary: The control plane became overloaded and slow due to excessive audit log volume.

What Happened:

A misconfigured audit policy led to high volumes of audit logs being generated, overwhelming the control plane.

Diagnosis Steps:

Monitored control plane metrics and found high CPU usage due to audit logs.
Reviewed audit policy and found it was logging excessive data.

Root Cause:

Overly broad audit log configuration captured too many events.

Fix/Workaround:

• Refined audit policy to log only critical events.
• Restarted the API server.

Lessons Learned:

Audit logging needs to be fine-tuned to prevent overload.

How to Avoid:

Regularly review and refine audit logging policies.
Set alerts for high audit log volumes.

Answer 48

output:

Cluster Management K8s v1.23, bare metal

Summary: Resource fragmentation due to unbalanced pod distribution led to cluster instability.

What Happened:

Over time, pod distribution became uneven, with some nodes over-committed while others remained underutilized. This caused resource fragmentation, leading to cluster instability.

Diagnosis Steps:

Checked node resource utilization and found over-committed nodes with high pod density.
Examined pod distribution and noticed skewed placement.

Root Cause:

Lack of proper pod scheduling and resource management.

Fix/Workaround:

• Applied pod affinity and anti-affinity rules to achieve balanced scheduling.
• Rescheduled pods manually to redistribute workload.

Lessons Learned:

Resource management and scheduling rules are crucial for maintaining cluster stability.

How to Avoid:

Use affinity and anti-affinity rules to control pod placement.
Regularly monitor resource utilization and adjust pod placement strategies.

Answer 49

output:

Cluster Management K8s v1.21, AWS EBS

Summary: Cluster backup failed due to a misconfigured volume snapshot driver.

What Happened:

The backup process failed because the EBS volume snapshot driver was misconfigured, resulting in incomplete backups.

Diagnosis Steps:

Checked backup logs for error messages related to volume snapshot failures.
Verified snapshot driver configuration in storage class.

Root Cause:

Misconfigured volume snapshot driver prevented proper backups.

Fix/Workaround:

• Corrected snapshot driver configuration in storage class.
• Ran the backup process again, which completed successfully.

Lessons Learned:

Backup configuration must be thoroughly checked and tested.

How to Avoid:

Automate backup testing and validation in staging environments.
Regularly verify backup configurations.

Answer 50

output:

Cluster Management K8s v1.22, custom Docker registry

Summary: Deployment failed due to image pulling issues from a custom Docker registry.

What Happened:

A deployment failed because Kubernetes could not pull images from a custom Docker registry due to misconfigured image pull secrets.

Diagnosis Steps:

Observed ImagePullBackOff errors for the failing pods.
Checked image pull secrets and registry configuration.

Root Cause:

Incorrect or missing image pull secrets for accessing the custom registry.

Fix/Workaround:

• Corrected the image pull secrets in the deployment YAML.
• Re-deployed the application.

Lessons Learned:

Image pull secrets must be configured properly for private registries.

How to Avoid:

Always verify image pull secrets for private registries.
Use Kubernetes secrets management tools for image pull credentials.

Answer 51

output:

Cluster Management K8s v1.20, AWS EKS

Summary: Ingress controller configuration caused high network latency due to inefficient routing rules.

What Happened:

Ingress controller was handling a large number of routes inefficiently, resulting in significant network latency and slow response times for external traffic.

Diagnosis Steps:

Analyzed ingress controller logs for routing delays.
Checked ingress resources and discovered unnecessary complex path-based routing rules.

Root Cause:

Inefficient ingress routing rules and too many path-based routes led to slower packet processing.

Fix/Workaround:

• Simplified ingress resource definitions and optimized routing rules.
• Restarted ingress controller to apply changes.

Lessons Learned:

Optimizing ingress routing rules is critical for performance, especially in high-traffic environments.

How to Avoid:

Regularly review and optimize ingress resources.
Use a more efficient ingress controller (e.g., NGINX Ingress Controller) for high-volume environments.

Answer 52

output:

Cluster Management K8s v1.21, GKE

Summary: Node draining took an unusually long time during maintenance due to unscheduled pod disruption.

What Happened:

During a scheduled node maintenance, draining took longer than expected because pods were not respecting PodDisruptionBudgets.

Diagnosis Steps:

Checked kubectl describe for affected pods and identified PodDisruptionBudget violations.
Observed that some pods had hard constraints on disruption due to storage.

Root Cause:

PodDisruptionBudget was too strict, preventing pods from being evicted quickly.

Fix/Workaround:

• Adjusted PodDisruptionBudget to allow more flexibility for pod evictions.
• Manually evicted the pods to speed up the node draining process.

Lessons Learned:

PodDisruptionBudgets should be set based on actual disruption tolerance.

How to Avoid:

Set reasonable disruption budgets for critical applications.
Test disruption scenarios during maintenance windows to identify issues.

Answer 53

output:

Cluster Management K8s v1.19, Azure AKS

Summary: Cluster became unresponsive after deploying a large number of pods in a single batch.

What Happened:

The cluster became unresponsive after deploying a batch of 500 pods in a single operation, causing resource exhaustion.

Diagnosis Steps:

Checked cluster logs and found that the control plane was overwhelmed with API requests.
Observed resource limits on the nodes, which were maxed out.

Root Cause:

The large-scale deployment exhausted the cluster’s available resources, causing a spike in API server load.

Fix/Workaround:

• Implemented gradual pod deployment using rolling updates instead of a batch deployment.
• Increased the node resource capacity to handle larger loads.

Lessons Learned:

Gradual deployments and resource planning are necessary when deploying large numbers of pods.

How to Avoid:

Use rolling updates or deploy in smaller batches.
Monitor cluster resources and scale nodes accordingly.

Answer 54

output:

Cluster Management K8s v1.23, Bare Metal

Summary: Node failed to recover after being drained due to a corrupt kubelet configuration.

What Happened:

After a node was drained for maintenance, it failed to rejoin the cluster due to a corrupted kubelet configuration file.

Diagnosis Steps:

Checked kubelet logs and identified errors related to configuration loading.
Verified kubelet configuration file on the affected node and found corruption.

Root Cause:

A corrupted kubelet configuration prevented the node from starting properly.

Fix/Workaround:

• Replaced the corrupted kubelet configuration file with a backup.
• Restarted the kubelet service and the node successfully rejoined the cluster.

Lessons Learned:

Regular backups of critical configuration files like kubelet configs can save time during node recovery.

How to Avoid:

Automate backups of critical configurations.
Implement configuration management tools for easier recovery.

Answer 55

output:

Cluster Management K8s v1.22, AWS EKS

Summary: Cluster resources were exhausted due to misconfiguration in the Horizontal Pod Autoscaler (HPA), resulting in excessive pod scaling.

What Happened:

HPA was configured to scale pods based on CPU utilization but had an overly sensitive threshold, causing the application to scale out rapidly and exhaust resources.

Diagnosis Steps:

Analyzed HPA metrics and found excessive scaling actions.
Verified CPU utilization metrics and observed that they were consistently above the threshold due to a sudden workload spike.

Root Cause:

HPA was too aggressive in scaling up based on CPU utilization, without considering other metrics like memory usage or custom metrics.

Fix/Workaround:

• Adjusted HPA configuration to scale based on a combination of CPU and memory usage.
• Set more appropriate scaling thresholds.

Lessons Learned:

Scaling based on a single metric (e.g., CPU) can lead to inefficiency, especially during workload spikes.

How to Avoid:

Use multiple metrics for autoscaling (e.g., CPU, memory, and custom metrics).
Set more conservative scaling thresholds to prevent resource exhaustion.

Answer 56

output:

Cluster Management K8s v1.20, GKE

Summary: Application behavior became inconsistent after pod restarts due to improper state handling.

What Happened:

After a pod was restarted, the application started behaving unpredictably, with some users experiencing different results from others due to lack of state synchronization.

Diagnosis Steps:

Checked pod logs and noticed that state data was being stored in the pod’s ephemeral storage.
Verified that application code did not handle state persistence properly.

Root Cause:

The application was not designed to persist state beyond the pod lifecycle, leading to inconsistent behavior after restarts.

Fix/Workaround:

• Moved application state to persistent volumes or external databases.
• Adjusted the application logic to handle state recovery properly after restarts.

Lessons Learned:

State should be managed outside of ephemeral storage for applications that require consistency.

How to Avoid:

Use persistent volumes for stateful applications.
Implement state synchronization mechanisms where necessary.

Answer 57

output:

Cluster Management K8s v1.21, AWS EKS

Summary: Cluster-wide service outage occurred after an automated change removed a critical ClusterRoleBinding.

What Happened:

A misconfigured automation pipeline accidentally removed a ClusterRoleBinding, which was required for certain critical services to function.

Diagnosis Steps:

Analyzed service logs and found permission-related errors.
Checked the RBAC configuration and found the missing ClusterRoleBinding.

Root Cause:

Automated pipeline incorrectly removed the ClusterRoleBinding, causing service permissions to be revoked.

Fix/Workaround:

• Restored the missing ClusterRoleBinding.
• Manually verified that affected services were functioning correctly.

Lessons Learned:

Automation changes must be reviewed and tested to prevent accidental permission misconfigurations.

How to Avoid:

Use automated tests and checks for RBAC changes.
Implement safeguards and approval workflows for automated configuration changes.

Answer 58

output:

Cluster Management K8s v1.19, Bare Metal

Summary: Node overcommitment led to pod evictions, causing application downtime.

What Happened:

Due to improper resource requests and limits, the node was overcommitted, which led to the eviction of critical pods.

Diagnosis Steps:

Checked the node’s resource utilization and found it was maxed out.
Analyzed pod logs to see eviction messages related to resource limits.

Root Cause:

Pods did not have properly set resource requests and limits, leading to resource overcommitment on the node.

Fix/Workaround:

• Added appropriate resource requests and limits to the affected pods.
• Rescheduled the pods to other nodes with available resources.

Lessons Learned:

Properly setting resource requests and limits prevents overcommitment and avoids pod evictions.

How to Avoid:

Always set appropriate resource requests and limits for all pods.
Use resource quotas and limit ranges to prevent overcommitment.

Answer 59

output:

Cluster Management K8s v1.23, Azure AKS

Summary: Pods failed to start because the image pull policy was misconfigured.

What Happened:

The image pull policy was set to Never, preventing Kubernetes from pulling the required container images from the registry.

Diagnosis Steps:

Checked pod events and found image pull errors.
Verified the image pull policy in the pod specification.

Root Cause:

Image pull policy was set to Never, which prevents Kubernetes from pulling images from the registry.

Fix/Workaround:

• Changed the image pull policy to IfNotPresent or Always in the pod configuration.
• Re-deployed the pods.

Lessons Learned:

The correct image pull policy is necessary to ensure Kubernetes can pull container images from a registry.

How to Avoid:

Double-check the image pull policy in pod specifications before deployment.
Use Always for images stored in remote registries.

Answer 60

output:

Cluster Management K8s v1.24, AWS EKS

Summary: Control plane resources were excessively utilized during pod scheduling, leading to slow deployments.

What Happened:

Pod scheduling took significantly longer than expected due to excessive resource consumption in the control plane.

Diagnosis Steps:

Monitored control plane metrics and found high CPU and memory usage.
Analyzed scheduler logs to identify resource bottlenecks.

Root Cause:

The default scheduler was not optimized for high resource consumption, causing delays in pod scheduling.

Fix/Workaround:

• Optimized the scheduler configuration to reduce resource usage.
• Split large workloads into smaller ones to improve scheduling efficiency.

Lessons Learned:

Efficient scheduler configuration is essential for handling large-scale deployments.

How to Avoid:

Optimize scheduler settings for large clusters.
Use scheduler features like affinity and anti-affinity to control pod placement.

Answer 61

output:

Cluster Management K8s v1.22, GKE

Summary: Persistent Volume Claims (PVCs) failed due to exceeding the resource quota for storage in the namespace.

What Happened:

A user attempted to create PVCs that exceeded the available storage quota, leading to failed PVC creation.

Diagnosis Steps:

Checked the namespace resource quotas using kubectl get resourcequotas.
Observed that the storage limit had been reached.

Root Cause:

PVCs exceeded the configured storage resource quota in the namespace.

Fix/Workaround:

• Increased the storage quota in the namespace.
• Cleaned up unused PVCs to free up space.

Lessons Learned:

Proper resource quota management is critical for ensuring that users cannot overuse resources.

How to Avoid:

Regularly review and adjust resource quotas based on usage patterns.
Implement automated alerts for resource quota breaches.

Answer 62

output:

Cluster Management K8s v1.21, AWS EKS

Summary: Pods failed to reschedule after a node failure due to improper node affinity rules.

What Happened:

When a node was taken down for maintenance, the pod failed to reschedule due to restrictive node affinity settings.

Diagnosis Steps:

Checked pod events and noticed affinity rule errors preventing the pod from scheduling on other nodes.
Analyzed the node affinity configuration in the pod spec.

Root Cause:

Node affinity rules were set too restrictively, preventing the pod from being scheduled on other nodes.

Fix/Workaround:

• Adjusted the node affinity rules to be less restrictive.
• Re-scheduled the pods to available nodes.

Lessons Learned:

Affinity rules should be configured to provide sufficient flexibility for pod rescheduling.

How to Avoid:

Set node affinity rules based on availability and workloads.
Regularly test affinity and anti-affinity rules during node maintenance windows.

Answer 63

output:

Cluster Management K8s v1.24, Azure AKS

Summary: Network latency issues occurred intermittently due to misconfiguration in the CNI (Container Network Interface) plugin.

What Happened:

Network latency was sporadically high between pods due to improper settings in the CNI plugin.

Diagnosis Steps:

Analyzed network metrics and noticed high latency between pods in different nodes.
Checked CNI plugin logs and configuration and found incorrect MTU (Maximum Transmission Unit) settings.

Root Cause:

MTU misconfiguration in the CNI plugin caused packet fragmentation, resulting in network latency.

Fix/Workaround:

• Corrected the MTU setting in the CNI configuration to match the network infrastructure.
• Restarted the CNI plugin and verified network performance.

Lessons Learned:

Proper CNI configuration is essential to avoid network latency and connectivity issues.

How to Avoid:

Ensure CNI plugin configurations match the underlying network settings.
Test network performance after changes to the CNI configuration.

Answer 64

output:

Cluster Management K8s v1.19, GKE

Summary: A pod was restarting frequently due to resource limits being too low, causing the container to be killed.

What Happened:

Pods were being killed and restarted due to the container’s resource requests and limits being set too low, causing OOM (Out of Memory) kills.

Diagnosis Steps:

Checked pod logs and identified frequent OOM kills.
Reviewed resource requests and limits in the pod spec.

Root Cause:

Resource limits were too low, leading to the container being killed when it exceeded available memory.

Fix/Workaround:

• Increased the memory limits and requests for the affected pods.
• Re-deployed the updated pods and monitored for stability.

Lessons Learned:

Proper resource requests and limits should be set to avoid OOM kills and pod restarts.

How to Avoid:

Regularly review resource requests and limits based on workload requirements.
Use resource usage metrics to set more accurate resource limits.

Answer 65

output:

Cluster Management K8s v1.22, AWS EKS

Summary: Cluster performance degraded because of excessive logs being generated by applications, leading to high disk usage.

What Happened:

Excessive log output from applications filled up the disk, slowing down the entire cluster.

Diagnosis Steps:

Monitored disk usage and found that logs were consuming most of the disk space.
Identified the affected applications by reviewing the logging configuration.

Root Cause:

Applications were logging excessively, and log rotation was not properly configured.

Fix/Workaround:

• Configured log rotation for the affected applications.
• Reduced the verbosity of the logs in application settings.

Lessons Learned:

Proper log management and rotation are essential to avoid filling up disk space and impacting cluster performance.

How to Avoid:

Configure log rotation and retention policies for all applications.
Monitor disk usage and set up alerts for high usage.

Answer 66

output:

Cluster Management K8s v1.21, GKE

Summary: The cluster experienced resource exhaustion because CronJobs were running in parallel without proper capacity checks.

What Happened:

Several CronJobs were triggered simultaneously, causing the cluster to run out of CPU and memory resources.

Diagnosis Steps:

Checked CronJob schedules and found multiple jobs running at the same time.
Monitored resource usage and identified high CPU and memory consumption from the CronJobs.

Root Cause:

Lack of resource limits and concurrent job checks in CronJobs.

Fix/Workaround:

• Added resource requests and limits for CronJobs.
• Configured CronJobs to stagger their execution times to avoid simultaneous execution.

Lessons Learned:

Always add resource limits and configure CronJobs to prevent them from running in parallel and exhausting cluster resources.

How to Avoid:

Set appropriate resource requests and limits for CronJobs.
Use concurrencyPolicy to control parallel executions of CronJobs.

Answer 67

output:

Cluster Management K8s v1.23, Azure AKS

Summary: Pod scaling failed due to conflicting affinity/anti-affinity rules that prevented pods from being scheduled.

What Happened:

A deployment’s pod scaling was unsuccessful due to the anti-affinity rules that conflicted with available nodes.

Diagnosis Steps:

Checked pod scaling logs and identified unschedulable errors related to affinity rules.
Reviewed affinity/anti-affinity settings in the pod deployment configuration.

Root Cause:

The anti-affinity rule required pods to be scheduled on specific nodes, but there were not enough available nodes.

Fix/Workaround:

• Relaxed the anti-affinity rule to allow pods to be scheduled on any available node.
• Increased the number of nodes to ensure sufficient capacity.

Lessons Learned:

Affinity and anti-affinity rules should be configured carefully, especially in dynamic environments with changing node capacity.

How to Avoid:

Test affinity and anti-affinity configurations thoroughly.
Use flexible affinity rules to allow for dynamic scaling and node availability.

Answer 68

output:

Cluster Management K8s v1.22, AWS EKS

Summary: Cluster became inaccessible due to excessive API server throttling caused by too many concurrent requests.

What Happened:

The API server started throttling requests because the number of concurrent API calls exceeded the available limit.

Diagnosis Steps:

Monitored API server metrics and identified a high rate of incoming requests.
Checked client application logs and observed excessive API calls.

Root Cause:

Clients were making too many API requests in a short period, exceeding the rate limits of the API server.

Fix/Workaround:

• Throttled client requests to reduce API server load.
• Implemented exponential backoff for retries in client applications.

Lessons Learned:

Avoid overwhelming the API server with excessive requests and implement rate-limiting mechanisms.

How to Avoid:

Implement API request throttling and retries in client applications.
Use rate-limiting tools like kubectl to monitor API usage.

Answer 69

output:

Cluster Management K8s v1.20, GKE

Summary: Expansion of a Persistent Volume (PV) failed due to improper storage class settings.

What Happened:

The request to expand a persistent volume failed because the storage class was not configured to support volume expansion.

Diagnosis Steps:

Verified the PV and PVC configurations.
Checked the storage class settings and identified that volume expansion was not enabled.

Root Cause:

The storage class did not have the allowVolumeExpansion flag set to true.

Fix/Workaround:

• Updated the storage class to allow volume expansion.
• Expanded the persistent volume and verified the PVC reflected the changes.

Lessons Learned:

Ensure that storage classes are configured to allow volume expansion when using dynamic provisioning.

How to Avoid:

Check storage class configurations before creating PVs.
Enable allowVolumeExpansion for dynamic storage provisioning.

Answer 70

output:

Cluster Management K8s v1.22, AWS EKS

Summary: Unauthorized users gained access to sensitive resources due to misconfigured RBAC roles and bindings.

What Happened:

An RBAC misconfiguration allowed unauthorized users to access cluster resources, including secrets.

Diagnosis Steps:

Checked RBAC policies and found overly permissive role bindings.
Analyzed user access logs and identified unauthorized access to sensitive resources.

Root Cause:

Over-permissive RBAC role bindings granted excessive access to unauthorized users.

Fix/Workaround:

• Corrected RBAC policies to restrict access.
• Audited user access and removed unauthorized permissions.

Lessons Learned:

Proper RBAC configuration is crucial for securing cluster resources.

How to Avoid:

Implement the principle of least privilege for RBAC roles.
Regularly audit RBAC policies and bindings.

Answer 71

output:

Cluster Management K8s v1.20, GKE

Summary: Pods entered an inconsistent state because the container image failed to pull due to incorrect image tag.

What Happened:

Pods started with an image tag that did not exist in the container registry, causing the pods to enter a CrashLoopBackOff state.

Diagnosis Steps:

Checked the pod events and found image pull errors with "Tag not found" messages.
Verified the image tag in the deployment configuration.

Root Cause:

The container image tag specified in the deployment was incorrect or missing from the container registry.

Fix/Workaround:

• Corrected the image tag in the deployment configuration to point to an existing image.
• Redeployed the application.

Lessons Learned:

Always verify image tags before deploying and ensure the image is available in the registry.

How to Avoid:

Use CI/CD pipelines to automatically verify image availability before deployment.
Enable image pull retries for transient network issues.

Answer 72

output:

Cluster Management K8s v1.22, Azure AKS

Summary: Pods experienced disruptions as nodes ran out of CPU and memory, causing evictions.

What Happened:

During a high workload period, nodes ran out of resources, causing the scheduler to evict pods and causing disruptions.

Diagnosis Steps:

Monitored node resource usage and identified CPU and memory exhaustion.
Reviewed pod events and noticed pod evictions due to resource pressure.

Root Cause:

Insufficient node resources for the workload being run, causing resource contention and pod evictions.

Fix/Workaround:

• Added more nodes to the cluster to meet resource requirements.
• Adjusted pod resource requests/limits to be more aligned with node resources.

Lessons Learned:

Regularly monitor and scale nodes to ensure sufficient resources during peak workloads.

How to Avoid:

Use cluster autoscaling to add nodes automatically when resource pressure increases.
Set appropriate resource requests and limits for pods.

Answer 73

output:

Cluster Management K8s v1.21, AWS EKS

Summary: Services could not discover each other due to DNS resolution failures, affecting internal communication.

What Happened:

Pods were unable to resolve internal service names due to DNS failures, leading to broken inter-service communication.

Diagnosis Steps:

Checked DNS logs and found dnsmasq errors.
Investigated CoreDNS logs and found insufficient resources allocated to the DNS pods.

Root Cause:

CoreDNS pods were running out of resources (CPU/memory), causing DNS resolution failures.

Fix/Workaround:

• Increased resource limits for the CoreDNS pods.
• Restarted CoreDNS pods to apply the new resource settings.

Lessons Learned:

Ensure that CoreDNS has enough resources to handle DNS requests efficiently.

How to Avoid:

Monitor CoreDNS pod resource usage.
Allocate adequate resources based on cluster size and workload.

Answer 74

output:

Cluster Management K8s v1.22, GKE

Summary: Persistent volume provisioning was delayed due to an issue with the dynamic provisioner.

What Happened:

PVCs were stuck in the Pending state because the dynamic provisioner could not create the required PVs.

Diagnosis Steps:

Checked PVC status using kubectl get pvc and saw that they were stuck in Pending.
Investigated storage class settings and found an issue with the provisioner configuration.

Root Cause:

Misconfigured storage class settings were preventing the dynamic provisioner from provisioning volumes.

Fix/Workaround:

• Corrected the storage class settings, ensuring the correct provisioner was specified.
• Recreated the PVCs, and provisioning completed successfully.

Lessons Learned:

Validate storage class settings and provisioner configurations during cluster setup.

How to Avoid:

Test storage classes and volume provisioning in staging environments before production use.
Monitor PV provisioning and automate alerts for failures.

Answer 75

output:

Cluster Management K8s v1.21, Azure AKS

Summary: A deployment rollback failed due to the rollback image version no longer being available in the container registry.

What Happened:

After an update, the deployment was rolled back to a previous image version that was no longer present in the container registry, causing the rollback to fail.

Diagnosis Steps:

Checked the deployment history and found that the previous image was no longer available.
Examined the container registry and confirmed the image version had been deleted.

Root Cause:

The image version intended for rollback was deleted from the registry before the rollback occurred.

Fix/Workaround:

• Rebuilt the previous image version and pushed it to the registry.
• Triggered a successful rollback after the image was available.

Lessons Learned:

Always retain previous image versions for safe rollbacks.

How to Avoid:

Implement retention policies for container images.
Use CI/CD pipelines to tag and store images for future rollbacks.

Answer 76

output:

Cluster Management K8s v1.22, AWS EKS

Summary: The Kubernetes master node became unresponsive under high load due to excessive API server calls and high memory usage.

What Happened:

The Kubernetes master node was overwhelmed by API calls and high memory consumption, leading to a failure to respond to management requests.

Diagnosis Steps:

Checked the control plane resource usage and found high memory and CPU consumption on the master node.
Analyzed API server logs and found a spike in incoming requests.

Root Cause:

Excessive incoming requests caused API server memory to spike, rendering the master node unresponsive.

Fix/Workaround:

• Implemented API rate limiting to control excessive calls.
• Increased the memory allocated to the master node.

Lessons Learned:

Ensure that the control plane is protected against overloads and is properly scaled.

How to Avoid:

Use API rate limiting and load balancing techniques for the master node.
Consider separating the control plane and worker nodes for better scalability.

Answer 77

output:

Cluster Management K8s v1.24, GKE

Summary: Pods failed to restart on available nodes due to overly strict node affinity rules.

What Happened:

A pod failed to restart after a node failure because the node affinity rules were too strict, preventing the pod from being scheduled on any available nodes.

Diagnosis Steps:

Checked pod logs and observed affinity errors in scheduling.
Analyzed the affinity settings in the pod spec and found restrictive affinity rules.

Root Cause:

Strict node affinity rules prevented the pod from being scheduled on available nodes.

Fix/Workaround:

• Relaxed the node affinity rules in the pod spec.
• Redeployed the pod, and it successfully restarted on an available node.

Lessons Learned:

Carefully configure node affinity rules to allow flexibility during pod rescheduling.

How to Avoid:

Use less restrictive affinity rules for better pod rescheduling flexibility.
Test affinity rules during node maintenance and scaling operations.

Answer 78

output:

Cluster Management K8s v1.19, AWS EKS

Summary: The ReplicaSet failed to scale due to insufficient resources on the nodes.

What Happened:

When attempting to scale a ReplicaSet, new pods failed to schedule due to a lack of available resources on the nodes.

Diagnosis Steps:

Checked the resource usage on the nodes and found they were running at full capacity.
Analyzed ReplicaSet scaling events and observed failures to schedule new pods.

Root Cause:

Insufficient node resources to accommodate new pods due to high resource consumption by existing workloads.

Fix/Workaround:

• Added more nodes to the cluster to handle the increased workload.
• Adjusted resource requests and limits to ensure efficient resource allocation.

Lessons Learned:

Regularly monitor cluster resource usage and scale proactively based on demand.

How to Avoid:

Enable cluster autoscaling to handle scaling issues automatically.
Set proper resource requests and limits for pods to avoid resource exhaustion.

Answer 79

output:

Cluster Management K8s v1.21, GKE

Summary: A namespace was missing after performing a cluster upgrade.

What Happened:

After upgrading the cluster, a namespace that was present before the upgrade was no longer available.

Diagnosis Steps:

Checked the cluster upgrade logs and identified that a namespace deletion had occurred during the upgrade process.
Verified with backup and confirmed the namespace was inadvertently deleted during the upgrade.

Root Cause:

An issue during the cluster upgrade process led to the unintentional deletion of a namespace.

Fix/Workaround:

• Restored the missing namespace from backups.
• Investigated and fixed the upgrade process to prevent future occurrences.

Lessons Learned:

Always backup critical resources before performing upgrades and test the upgrade process thoroughly.

How to Avoid:

Backup namespaces and other critical resources before upgrading.
Review upgrade logs carefully to identify any unexpected deletions or changes.

Answer 80

output:

Cluster Management K8s v1.23, Azure AKS

Summary: The Horizontal Pod Autoscaler (HPA) was inefficiently scaling due to misconfigured metrics.

What Happened:

HPA did not scale pods appropriately, either under-scaling or over-scaling, due to incorrect metric definitions.

Diagnosis Steps:

Checked HPA configuration and identified incorrect CPU utilization metrics.
Monitored metrics-server logs and found that the metrics were inconsistent.

Root Cause:

HPA was configured to scale based on inaccurate or inappropriate metrics, leading to inefficient scaling behavior.

Fix/Workaround:

• Reconfigured the HPA to scale based on correct metrics (e.g., memory, custom metrics).
• Verified that the metrics-server was reporting accurate data.

Lessons Learned:

Always ensure that the right metrics are used for scaling to avoid inefficient scaling behavior.

How to Avoid:

Regularly review HPA configurations and metrics definitions.
Test scaling behavior under different load conditions.

Answer 81

output:

Cluster Management K8s v1.21, GKE

Summary: Pods could not start because the image registry was temporarily unavailable, causing image pull failures.

What Happened:

Pods failed to pull images because the registry was down for maintenance, leading to deployment failures.

Diagnosis Steps:

Checked the pod status using kubectl describe pod and identified image pull errors.
Investigated the registry status and found scheduled downtime for maintenance.

Root Cause:

The container registry was temporarily unavailable due to maintenance, and the pods could not access the required images.

Fix/Workaround:

• Manually downloaded the images from a secondary registry.
• Temporarily used a local image registry until the primary registry was back online.

Lessons Learned:

Ensure that alternate image registries are available in case of downtime.

How to Avoid:

Implement multiple image registries for high availability.
Use image pull policies that allow fallback to local caches.

Answer 82

output:

Cluster Management K8s v1.20, AWS EKS

Summary: Pods failed to start because their resource requests were too low, preventing the scheduler from assigning them to nodes.

What Happened:

The pods had very low resource requests, causing the scheduler to fail to assign them to available nodes.

Diagnosis Steps:

Checked pod status and found them stuck in Pending.
Analyzed the resource requests and found that they were too low to meet the node's capacity requirements.

Root Cause:

The resource requests were set too low, preventing proper pod scheduling.

Fix/Workaround:

• Increased the resource requests in the pod spec.
• Reapplied the configuration, and the pods were scheduled successfully.

Lessons Learned:

Always ensure that resource requests are appropriately set for your workloads.

How to Avoid:

Use resource limits and requests based on accurate usage data from monitoring tools.
Set resource requests in line with expected workload sizes.

Answer 83

output:

Cluster Management K8s v1.22, GKE

Summary: HPA failed to scale the pods appropriately during a sudden spike in load.

What Happened:

The HPA did not scale the pods properly during a traffic surge due to incorrect metric thresholds.

Diagnosis Steps:

Checked HPA settings and identified that the CPU utilization threshold was too high.
Verified the metric server was reporting correct metrics.

Root Cause:

Incorrect scaling thresholds set in the HPA configuration.

Fix/Workaround:

• Adjusted HPA thresholds to scale more aggressively under higher loads.
• Increased the replica count to handle the peak load.

Lessons Learned:

HPA thresholds should be fine-tuned based on expected load patterns.

How to Avoid:

Regularly review and adjust HPA configurations to reflect actual workload behavior.
Use custom metrics for better scaling control.

Answer 84

output:

Cluster Management K8s v1.21, AWS EKS

Summary: Pods were evicted due to disk pressure on the node, causing service interruptions.

What Happened:

A node ran out of disk space due to logs and other data consuming the disk, resulting in pod evictions.

Diagnosis Steps:

Checked node resource usage and found disk space was exhausted.
Reviewed pod eviction events and found they were due to disk pressure.

Root Cause:

The node disk was full, causing the kubelet to evict pods to free up resources.

Fix/Workaround:

• Increased disk capacity on the affected node.
• Cleared unnecessary logs and old data from the disk.

Lessons Learned:

Ensure adequate disk space is available, especially for logging and temporary data.

How to Avoid:

Monitor disk usage closely and set up alerts for disk pressure.
Implement log rotation and clean-up policies to avoid disk exhaustion.

Answer 85

output:

Cluster Management K8s v1.22, Azure AKS

Summary: A node failed to drain due to pods that were in use, preventing the drain operation from completing.

What Happened:

When attempting to drain a node, the operation failed because some pods were still in use or had pending termination grace periods.

Diagnosis Steps:

Ran kubectl describe node and checked pod evictions.
Identified pods that were in the middle of long-running processes or had insufficient termination grace periods.

Root Cause:

Pods with long-running tasks or improper termination grace periods caused the drain to hang.

Fix/Workaround:

• Increased termination grace periods for the affected pods.
• Forced the node drain operation after ensuring that the pods could safely terminate.

Lessons Learned:

Ensure that pods with long-running tasks have adequate termination grace periods.

How to Avoid:

Configure appropriate termination grace periods for all pods.
Monitor node draining and ensure pods can gracefully shut down.

Answer 86

output:

Cluster Management K8s v1.20, GKE

Summary: The cluster autoscaler failed to scale up the node pool despite high resource demand.

What Happened:

The cluster autoscaler did not add nodes when resource utilization reached critical levels.

Diagnosis Steps:

Checked the autoscaler logs and found that scaling events were not triggered.
Reviewed the node pool configuration and autoscaler settings.

Root Cause:

The autoscaler was not configured with sufficient thresholds or permissions to scale up the node pool.

Fix/Workaround:

• Adjusted the scaling thresholds in the autoscaler configuration.
• Verified the correct IAM permissions for the autoscaler to scale the node pool.

Lessons Learned:

Ensure the cluster autoscaler is correctly configured and has the right permissions.

How to Avoid:

Regularly review cluster autoscaler configuration and permissions.
Monitor scaling behavior to ensure it functions as expected during high load.

Answer 87

output:

Cluster Management K8s v1.21, AWS EKS

Summary: Pods lost network connectivity after a node reboot, causing communication failures between services.

What Happened:

After a node was rebooted, the networking components failed to re-establish proper connectivity for the pods.

Diagnosis Steps:

Checked pod logs and found connection timeouts between services.
Investigated the node and found networking components (e.g., CNI plugin) were not properly re-initialized after the reboot.

Root Cause:

The CNI plugin did not properly re-initialize after the node reboot, causing networking failures.

Fix/Workaround:

• Manually restarted the CNI plugin on the affected node.
• Ensured that the CNI plugin was configured to restart properly after a node reboot.

Lessons Learned:

Ensure that critical components like CNI plugins are resilient to node reboots.

How to Avoid:

Configure the CNI plugin to restart automatically after node reboots.
Monitor networking components to ensure they are healthy after reboots.

Answer 88

output:

Cluster Management K8s v1.22, GKE

Summary: Unauthorized access errors occurred due to missing permissions in RBAC configurations.

What Happened:

Pods failed to access necessary resources due to misconfigured RBAC policies, resulting in permission-denied errors.

Diagnosis Steps:

Reviewed the RBAC policy logs and identified missing permissions for service accounts.
Checked the roles and role bindings associated with the pods.

Root Cause:

RBAC policies did not grant the required permissions to the service accounts.

Fix/Workaround:

• Updated the RBAC roles and bindings to include the necessary permissions for the pods.
• Applied the updated RBAC configurations and confirmed access.

Lessons Learned:

RBAC configurations should be thoroughly tested to ensure correct permissions.

How to Avoid:

Implement a least-privilege access model and audit RBAC policies regularly.
Use automated tools to test and verify RBAC configurations.

Answer 89

output:

Cluster Management K8s v1.19, AWS EKS

Summary: A pod upgrade failed because it was using deprecated APIs not supported in the new version.

What Happened:

When upgrading to a new Kubernetes version, a pod upgrade failed due to deprecated APIs in use.

Diagnosis Steps:

Checked the pod spec and identified deprecated API versions in use.
Verified the Kubernetes changelog for API deprecations in the new version.

Root Cause:

The pod was using APIs that were deprecated in the new Kubernetes version, causing the upgrade to fail.

Fix/Workaround:

• Updated the pod spec to use supported API versions.
• Reapplied the deployment with the updated APIs.

Lessons Learned:

Regularly review Kubernetes changelogs for deprecated API versions.

How to Avoid:

Implement a process to upgrade and test all components for compatibility before applying changes.
Use tools like kubectl deprecations to identify deprecated APIs.

Answer 90

output:

Cluster Management K8s v1.21, Azure AKS

Summary: A container's high CPU usage was caused by inefficient application code, leading to resource exhaustion.

What Happened:

An application was running inefficient code that caused excessive CPU consumption, impacting the entire node's performance.

Diagnosis Steps:

Monitored the pod's resource usage and found high CPU utilization.
Analyzed application logs and identified inefficient loops in the code.

Root Cause:

The application code had an inefficient algorithm that led to high CPU consumption.

Fix/Workaround:

• Optimized the application code to reduce CPU consumption.
• Redeployed the application with the optimized code.

Lessons Learned:

Application code optimization is essential for ensuring efficient resource usage.

How to Avoid:

Regularly profile application code for performance bottlenecks.
Set CPU limits and requests to prevent resource exhaustion.

Answer 91

output:

Cluster Management K8s v1.20, AWS EKS

Summary: Resource starvation occurred on nodes because pods were over-provisioned, consuming more resources than expected.

What Happened:

Pods were allocated more resources than necessary, causing resource contention on the nodes.

Diagnosis Steps:

Analyzed node and pod resource utilization.
Found that the CPU and memory resources for several pods were unnecessarily high, leading to resource starvation for other pods.

Root Cause:

Incorrect resource requests and limits set for the pods, causing resource over-allocation.

Fix/Workaround:

• Reduced resource requests and limits based on actual usage metrics.
• Re-deployed the pods with optimized resource configurations.

Lessons Learned:

Accurate resource requests and limits should be based on actual usage data.

How to Avoid:

Regularly monitor resource utilization and adjust requests/limits accordingly.
Use vertical pod autoscalers for better resource distribution.

Answer 92

output:

Cluster Management K8s v1.21, GKE

Summary: Pods were not scheduled due to overly strict affinity rules that limited the nodes available for deployment.

What Happened:

The affinity rules were too restrictive, preventing pods from being scheduled on available nodes.

Diagnosis Steps:

Reviewed pod deployment spec and found strict affinity constraints.
Verified available nodes and found that no nodes met the pod's affinity requirements.

Root Cause:

Overly restrictive affinity settings that limited pod scheduling.

Fix/Workaround:

• Adjusted the affinity rules to be less restrictive.
• Applied changes and verified the pods were scheduled correctly.

Lessons Learned:

Affinity constraints should balance optimal placement with available resources.

How to Avoid:

Regularly review and adjust affinity/anti-affinity rules based on cluster capacity.
Test deployment configurations in staging before applying to production.

Answer 93

output:

Cluster Management K8s v1.22, Azure AKS

Summary: Pods failed their readiness probes during initialization, causing traffic to be routed to unhealthy instances.

What Happened:

The pods had a slow initialization time, but the readiness probe timeout was set too low, causing premature failure.

Diagnosis Steps:

Checked pod events and logs, discovering that readiness probes were failing due to long startup times.
Increased the timeout period for the readiness probe and observed that the pods began to pass the probe after startup.

Root Cause:

Readiness probe timeout was set too low for the pod's initialization process.

Fix/Workaround:

• Increased the readiness probe timeout and delay parameters.
• Re-applied the deployment, and the pods started passing readiness checks.

Lessons Learned:

The readiness probe timeout should be configured according to the actual initialization time of the pod.

How to Avoid:

Monitor pod initialization times and adjust readiness probe configurations accordingly.
Use a gradual rollout for new deployments to avoid sudden failures.

Answer 94

output:

Cluster Management K8s v1.19, GKE

Summary: Incorrect path configuration in the ingress resource resulted in 404 errors for certain API routes.

What Happened:

Ingress was misconfigured with incorrect path mappings, causing requests to certain API routes to return 404 errors.

Diagnosis Steps:

Checked ingress configuration using kubectl describe ingress and found mismatched path rules.
Verified the service endpoints and found that the routes were not properly configured in the ingress spec.

Root Cause:

Incorrect path definitions in the ingress resource, causing requests to be routed incorrectly.

Fix/Workaround:

• Fixed the path configuration in the ingress resource.
• Re-applied the ingress configuration, and traffic was correctly routed.

Lessons Learned:

Verify that ingress path definitions match the application routing.

How to Avoid:

Test ingress paths thoroughly before applying to production environments.
Use versioned APIs to ensure backward compatibility for routing paths.

Answer 95

output:

Cluster Management K8s v1.20, AWS EKS

Summary: Node pool scaling failed because the account exceeded resource quotas in AWS.

What Happened:

When attempting to scale up a node pool, the scaling operation failed due to hitting AWS resource quotas.

Diagnosis Steps:

Reviewed the EKS and AWS console to identify quota limits.
Found that the account had exceeded the EC2 instance limit for the region.

Root Cause:

Insufficient resource quotas in the AWS account.

Fix/Workaround:

• Requested a quota increase from AWS support.
• Once the request was approved, scaled the node pool successfully.

Lessons Learned:

Monitor cloud resource quotas to ensure scaling operations are not blocked.

How to Avoid:

Keep track of resource quotas and request increases in advance.
Automate quota monitoring and alerting to avoid surprises during scaling.

Answer 96

output:

Cluster Management K8s v1.21, Azure AKS

Summary: Pods entered a crash loop because a required ConfigMap was not present in the namespace.

What Happened:

A pod configuration required a ConfigMap that was deleted by accident, causing the pod to crash due to missing configuration data.

Diagnosis Steps:

Checked pod logs and found errors indicating missing environment variables or configuration files.
Investigated the ConfigMap and found it had been accidentally deleted.

Root Cause:

Missing ConfigMap due to accidental deletion.

Fix/Workaround:

• Recreated the ConfigMap in the namespace.
• Re-deployed the pods, and they started successfully.

Lessons Learned:

Protect critical resources like ConfigMaps to prevent accidental deletion.

How to Avoid:

Use namespaces and resource quotas to limit accidental deletion of shared resources.
Implement stricter RBAC policies for sensitive resources.

Answer 97

output:

Cluster Management K8s v1.22, GKE

Summary: The Kubernetes API server became slow due to excessive log generation from the kubelet and other components.

What Happened:

Excessive logging from the kubelet and other components overwhelmed the API server, causing it to become slow and unresponsive.

Diagnosis Steps:

Monitored API server performance using kubectl top pod and noticed resource spikes.
Analyzed log files and found an unusually high number of log entries from the kubelet.

Root Cause:

Excessive logging was causing resource exhaustion on the API server.

Fix/Workaround:

• Reduced the verbosity of logs from the kubelet and other components.
• Configured log rotation to prevent logs from consuming too much disk space.

Lessons Learned:

Excessive logging can cause performance degradation if not properly managed.

How to Avoid:

Set appropriate logging levels for components based on usage.
Implement log rotation and retention policies to avoid overwhelming storage.

Answer 98

output:

Cluster Management K8s v1.19, AWS EKS

Summary: Pods failed to schedule because the taints and tolerations were misconfigured, preventing the scheduler from placing them on nodes.

What Happened:

The nodes had taints that were not matched by the pod's tolerations, causing the pods to remain unscheduled.

Diagnosis Steps:

Used kubectl describe pod to investigate scheduling issues.
Found that the taints on the nodes did not match the tolerations set on the pods.

Root Cause:

Misconfiguration of taints and tolerations in the node and pod specs.

Fix/Workaround:

• Corrected the tolerations in the pod specs to match the taints on the nodes.
• Re-applied the pods and verified that they were scheduled correctly.

Lessons Learned:

Always ensure taints and tolerations are correctly configured in a multi-tenant environment.

How to Avoid:

Test taints and tolerations in a non-production environment.
Regularly audit and verify toleration settings to ensure proper pod placement.

Answer 99

output:

Cluster Management K8s v1.20, Azure AKS

Summary: The Kubernetes dashboard became unresponsive due to high resource usage caused by a large number of requests.

What Happened:

The Kubernetes dashboard was overwhelmed by too many requests, consuming excessive CPU and memory resources.

Diagnosis Steps:

Checked resource usage of the dashboard pod using kubectl top pod.
Found that the pod was using more resources than expected due to a large number of incoming requests.

Root Cause:

The dashboard was not scaled to handle the volume of requests.

Fix/Workaround:

• Scaled the dashboard deployment to multiple replicas to handle the load.
• Adjusted resource requests and limits for the dashboard pod.

Lessons Learned:

Ensure that the Kubernetes dashboard is properly scaled to handle expected traffic.

How to Avoid:

Implement horizontal scaling for the dashboard and other critical services.
Monitor the usage of the Kubernetes dashboard and scale as needed.

Answer 100

output:

Cluster Management K8s v1.21, GKE

Summary: Containers kept crashing due to hitting resource limits set in their configurations.

What Happened:

Containers were being killed because they exceeded their resource limits for memory and CPU.

Diagnosis Steps:

Used kubectl describe pod to find the resource limits and found that the limits were too low for the workload.
Analyzed container logs and found frequent OOMKilled events.

Root Cause:

The resource limits set for the container were too low, causing the container to be terminated when it exceeded the limit.

Fix/Workaround:

• Increased the resource limits for the affected containers.
• Re-applied the pod configurations and monitored for stability.

Lessons Learned:

Resource limits should be set based on actual workload requirements.

How to Avoid:

Use monitoring tools to track resource usage and adjust limits as needed.
Set up alerts for resource threshold breaches to avoid crashes.
NETWORKING

# Kubernetes Troubleshooting Scenarios

What Happened:

Diagnosis Steps:

Root Cause:

Fix/Workaround:

Lessons Learned:

How to Avoid:

What Happened:

Diagnosis Steps:

Root Cause:

Fix/Workaround:

Lessons Learned:

How to Avoid:

What Happened:

Diagnosis Steps:

Root Cause:

Fix/Workaround:

Lessons Learned:

How to Avoid:

What Happened:

Diagnosis Steps:

Root Cause:

Fix/Workaround:

Lessons Learned:

How to Avoid:

What Happened:

Diagnosis Steps:

Root Cause:

Fix/Workaround:

Lessons Learned:

How to Avoid:

What Happened:

Diagnosis Steps:

Root Cause:

Fix/Workaround:

Lessons Learned:

How to Avoid:

What Happened:

Diagnosis Steps:

Root Cause:

Fix/Workaround:

Lessons Learned:

How to Avoid:

What Happened:

Diagnosis Steps:

Root Cause:

Fix/Workaround:

Lessons Learned:

How to Avoid:

What Happened:

Diagnosis Steps:

Root Cause:

Fix/Workaround:

Lessons Learned:

How to Avoid:

What Happened:

Diagnosis Steps:

Root Cause:

Fix/Workaround:

Lessons Learned:

How to Avoid:

What Happened:

Diagnosis Steps:

Root Cause:

Fix/Workaround:

Lessons Learned:

How to Avoid:

What Happened:

Diagnosis Steps:

Root Cause:

Fix/Workaround:

Lessons Learned:

How to Avoid:

What Happened:

Diagnosis Steps:

Root Cause:

Fix/Workaround:

Lessons Learned:

How to Avoid:

What Happened: