Vignesh Ragupathy

Kubernetes can run even with misconfigured nodes, with wrong network policies or with missing limits. But it only takes one scaling spike to expose the cracks. I’ve walked into too many projects where clusters “worked” until they didn’t. Kubernetes is unforgiving if you skip the basics. Set it up right once, or pay the cost every single time. 54K+ read my DevOps and Cloud newsletter: https://lnkd.in/gg3RQsRK What do we cover: DevOps, Cloud, Kubernetes, IaC, GitOps, MLOps 🔁 Consider a Repost if this is helpful

613

19 Comments

Scaling the pod vertically WITHOUT restarting is now feasible from Kubernetes v1.33. Normally, if you wanted to give an app (a Pod in Kubernetes) more memory or CPU, you had to restart it. That works fine if the app doesn’t care about restarts, but some apps really don’t like being stopped and restarted, such as databases, big batch jobs, or things that need to stay running smoothly. -- Beta The new "in-place Pod resize" feature lets you adjust how much memory or CPU a Pod is using while it’s still running. Now in version 1.33, it’s considered good enough for regular use and is turned on by default. Before that, you had to enable the InPlacePodVerticalScaling feature gate. -- How Instead of doing a normal kubectl edit on the Pod, you use a special /resize subresource. For example, you can run a patch like: kubectl patch pod mypod --subresource=resize .... This is similar to how other Kubernetes features work. For example: - /status is a subresource you can use to update only the status field of an object without touching its spec. - /scale is a subresource on Deployments or StatefulSets that lets you change the number of replicas without editing the entire manifest. In the same way, /resize is a subresource on Pods that lets you adjust resources in place. Important: This applies to individual Pods. If you run kubectl set resources on a Deployment, StatefulSet, or Job, that still changes the template and triggers new Pods, NOT an in-place change. Increasing CPU is simple, and increasing memory usually works if the node has capacity. Lowering CPU is also easy, but lowering memory is the hardest case and may fail, get deferred, or force a restart depending on the policy. To check how it is going, watch the Pod’s status fields and conditions. -- Resize policy Each container in the Pod spec can define a resizePolicy. Inside this field, you list CPU and memory separately, and for each one you choose a restart policy. The two possible values are: - NotRequired: Kubernetes will try to adjust the resource values in place while the container is running. This is the default. It’s best-effort, if the runtime cannot apply the change safely (especially in cases like reducing memory), it will fail instead of forcing a restart. - RestartContainer: Kubernetes must restart the container to apply the new resource settings. This is useful for applications that only read their resource limits when they start up, such as some JVM-based workloads. resizePolicy: - resourceName: cpu restartPolicy: NotRequired - resourceName: memory restartPolicy: RestartContainer

61

2 Comments

WHAT IS 𝗟𝟰 𝗩𝗦 𝗟𝟳 IN KUBERNETES? Initial answer felt easy: • Service (LoadBalancer) → L4 (Network) • Ingress / Gateway → L7 (Application) But that view is incomplete. I traced the actual traffic flow: Client → 𝗟𝟰 → 𝗟𝟳 → 𝗟𝟰 → Pods That’s when things started making sense. The real model: • 𝗟𝟰 → gets traffic INTO the cluster • 𝗟𝟳 → decides WHERE it should go • 𝗟𝟰 → delivers it to the right pods It’s not L4 vs L7 It’s 𝗟𝟰 → 𝗟𝟳 → 𝗟𝟰 working together Then came the real problem 𝗦𝗨𝗗𝗗𝗘𝗡 traffic spikes. Not gradual. Not predictable. Just instant surge. HPA (CPU-based) reacts only after pods are already under pressure. Result: • Latency spikes • Errors start • Scaling comes too late That’s when things clicked. 𝗦𝗖𝗔𝗟𝗜𝗡𝗚 𝗠𝗨𝗦𝗧 𝗕𝗘 𝗣𝗥𝗢𝗔𝗖𝗧𝗜𝗩𝗘. Using: • Prometheus → captures real traffic signals • KEDA → converts them into scaling triggers • HPA → scales on custom metrics Now scaling is based on: • Request rate • Queue length • External signals → Pods scale 𝗕𝗘𝗙𝗢𝗥𝗘 traffic overwhelms them Add to that: • NetworkPolicies • Readiness / Startup probes • PodDisruptionBudgets • Headroom capacity • Edge rate limiting 𝗦𝗛𝗜𝗙𝗧 𝗜𝗡 𝗧𝗛𝗜𝗡𝗞𝗜𝗡𝗚: Not: “Scale when CPU increases” But: “Scale before traffic hits” I’ve broken this down with a clean architecture (L4 → L7 → L4) and full YAML configs. 𝗗𝗘𝗧𝗔𝗜𝗟𝗘𝗗 𝗗𝗘𝗘𝗣 𝗗𝗜𝗩𝗘 𝗜𝗡 𝗠𝗬 𝗕𝗟𝗢𝗚 (link in comments) ⸻ How are you handling sudden traffic spikes in Kubernetes? #Kubernetes #CloudArchitecture #DevOps #PlatformEngineering #Scalability

14

1 Comment

☸️ Deep Dive: etcd & Raft — The Real Backbone of Kubernetes If you truly want to understand Kubernetes architecture (not just YAML files), you must understand: 👉 etcd 👉 Raft Because without them, your cluster has: ❌ No memory ❌ No consistency ❌ No reliability 🧠 What is etcd? etcd is a distributed, strongly consistent key-value store used by Kubernetes to store cluster state. Everything you create is stored in etcd: ✔ Pods ✔ Deployments ✔ Secrets ✔ ConfigMaps ✔ Nodes ✔ RBAC rules ✔ CRDs Technical foundations: Written in Go Uses gRPC Based on MVCC (Multi-Version Concurrency Control) Uses the Raft consensus algorithm 👉 No etcd = No cluster memory. 🏗 How etcd Fits in Kubernetes Architecture Here’s the real flow inside your cluster: 1️⃣ kubectl sends request 2️⃣ kube-apiserver validates it 3️⃣ API server writes to etcd 4️⃣ Controllers & Scheduler watch for changes ⚠ Important: Only kube-apiserver talks to etcd Never access etcd directly in production 🔍 Why etcd Is Special Unlike many NoSQL databases, etcd guarantees: ✔ Strong consistency ✔ Leader-based replication ✔ Linearizable reads ✔ Fault tolerance It is not eventually consistent. That’s why Kubernetes depends on it for correctness. ⚙ Inside the Raft Protocol (Real Engineering) Raft is a consensus algorithm designed to be understandable and practical. It ensures: 👉 All nodes agree on the same data 👉 Even if some nodes fail 👉 Without data corruption 🧩 Core Concepts of Raft 1️⃣ Leader Election Cluster states: Leader Follower Candidate Process: All nodes start as followers If leader heartbeat stops → election begins A candidate requests votes Majority wins → new leader 💡 Only one leader at a time. 2️⃣ Log Replication When a write happens: Client → Leader Leader appends entry to its log Leader replicates to followers Majority confirms → commit Only then is the write successful. This guarantees consistency. 3️⃣ Quorum (Majority Rule) 3-node cluster → 2 required 5-node cluster → 3 required 👉 Always run an odd number of etcd nodes. 🧨 What Happens If etcd Fails? If etcd is unhealthy: kubectl hangs New pods won’t schedule Controllers stop reconciling Cluster appears frozen Your workloads may still run… But the control plane becomes blind. This is why etcd monitoring is critical. 🛡 Production Best Practices ✔ Use 3 or 5 node etcd cluster ✔ Use SSD (disk latency is critical) ✔ Take regular snapshots ✔ Encrypt communication with TLS ✔ Monitor: Disk I/O latency Leader changes fsync duration DB size Most real outages happen due to: 👉 Slow disks 👉 Disk full 👉 Network latency between members 🔥 The Big Picture Kubernetes is declarative. But what makes declarative systems possible? 👉 A consistent distributed datastore 👉 Backed by a safe consensus protocol That’s etcd + Raft. Without them: ❌ No cluster state ❌ No reconciliation loop ❌ No reliability Most engineers learn: Pods ✅ Helm ✅ CI/CD ✅ Few truly understand: Cluster state & consensus ⚙ #Kubernetes #etcd #Raft #sre

10

2 Comments

As a Platform team running large-scale streaming on EKS, one thing they chose is the ability of system to perform high count of messages with ultra-low latency. Thought of having Kafka? well ... right to some extent, but not perfectly thought. I believe an engine having power of accelerating should be equally important than aerodynamics. The same logic applies here, kafka supports monolithic architecture where compute and storage are tightly coupled in brokers, storing messages on local disk as a distributed commit log. Thus, it's not highly scalable or multi-tenant and not offereing ultra-low latency. I've researched and come across about Pulsar which is new distributed message streaming engine which is created by Apache & open-sourced as well. Pulsar features a multi-layer, stateless architecture that decouples compute (brokers) and storage (Apache BookKeeper), allowing for independent scaling and superior multi-tenancy and tiered storage capabilities. . But it's not "install and forget." Here are some of the initial challenges I faced when configuring on EKS and how I tried to tackle them using a GitOps approach suggested by platform architects & other community members. 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲 𝟭: 𝗢𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝗮𝗹 𝗖𝗼𝗺𝗽𝗹𝗲𝘅𝗶𝘁𝘆 Pulsar has multiple moving parts: Brokers, BookKeepers, Zookeeper, Proxies. Managing their configs, updates, and health on Kubernetes can be daunting. 𝗦𝘂𝗴𝗴𝗲𝘀𝘁𝗲𝗱 𝗙𝗶𝘅: We went all-in on GitOps with FluxCD. Every single configuration, from JVM settings to resource limits, is defined declaratively in a HelmRelease manifest stored in Git. This gives us an auditable, version-controlled, and automated way to manage the entire cluster. 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲 𝟮: 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗧𝘂𝗻𝗶𝗻𝗴 𝗶𝘀 𝗮 𝗗𝗮𝗿𝗸 𝗔𝗿𝘁 Default settings won't get you far. We saw unexplained backlogs and high latency initially. Is it the broker? The bookie? The network? 𝗦𝘂𝗴𝗴𝗲𝘀𝘁𝗲𝗱 𝗙𝗶𝘅: Deep observability. We built comprehensive Grafana dashboards using VictoriaMetrics to track everything: broker/bookie JVM stats, ledger storage latency, message rates, and—crucially—consumer processing latency. This helped us realize many "Pulsar problems" were actually slow consumers. 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲 𝟯: 𝗥𝗲𝘀𝗼𝘂𝗿𝗰𝗲 𝗔𝗹𝗹𝗼𝗰𝗮𝘁𝗶𝗼𝗻 Under-provisioning Pulsar components leads to instability, while over-provisioning burns cash. 𝗦𝘂𝗴𝗴𝗲𝘀𝘁𝗲𝗱 𝗙𝗶𝘅: Profiling and component-specific tuning. Brokers are CPU-heavy, Bookies are I/O and memory-heavy. We defined distinct resource requests/limits for each component in our Helm chart values, ensuring a Bookie gets the fast SSD-backed StorageClass it needs, while a stateless Broker can be scaled more flexibly. Running Pulsar on Kubernetes is a journey, but the right tooling and observability transform it from a complex beast into a reliable, scalable platform. #ApachePulsar #Kubernetes #EKS #PlatformEngineering #GitOps #FluxCD #StreamingData #DevOps #SRE

10

💡 Lessons from the AWS DynamoDB Outage This week’s AWS outage — triggered by a DNS issue in US-EAST-1 that cascaded into DynamoDB and other core services — was a strong reminder of a simple truth: Even the most reliable platforms have failure modes. Many businesses across the globe experienced ripple effects. Some degraded gracefully. Others went dark. The difference came down to architecture resilience — and, more importantly, organizational readiness. A few leadership reflections: 🔹 Operational resilience is a leadership choice. Designing for failure isn’t only a technical decision. It’s about where we invest time, complexity, and cost before incidents happen. 🔹 Dependencies compound risk. We often celebrate velocity and abstraction in cloud architecture — until a core dependency like DynamoDB falters. Leaders need to routinely ask: “What’s our plan if this dependency fails at 2 a.m.?” 🔹 Communication beats perfection. In outages, your team’s clarity matters more than the system’s uptime. Teams that have practiced incident response — with clear ownership and psychological safety — recover faster. 🔹 Postmortems are gold, if we treat them that way. When AWS publishes their root cause analysis, resist the urge to just skim it. Map their failure pattern to your own systems. Every incident, even external, is a free rehearsal. We can’t eliminate outages. But we can choose how prepared, transparent, and adaptive our teams are when they arrive. The real test of reliability isn’t when things are working — it’s how we behave when they don’t. #Leadership #ReliabilityEngineering #AWS #DynamoDB #EngineeringManagement #IncidentResponse #SystemDesign

5

Kubernetes just dropped something useful for anyone running non-trivial clusters — Node Readiness Controller 👇 Traditionally, Kubernetes decides if a node can run workloads using a single “Ready” flag. That’s simple, but in real clusters you often have extra pieces — network agents, storage drivers, GPUs, custom checks — that must be healthy before a node truly should accept pods. The new Node Readiness Controller lets you define fine-grained, declarative readiness conditions so nodes only accept workloads once all required components are actually ready. This helps with bootstrapping complex node setups, enforcing health requirements, and improving reliability. In other words: >> You can have custom readiness gates beyond the built-in Ready condition >> You get more control over heterogeneous nodes (eg GPU vs general workloads) >> It fits better with real-world infra where services must be healthy before scheduling starts If you’re working with clusters that need more than the default “ready/not ready”, this controller is worth a look. https://lnkd.in/gVzbeevy #Kubernetes #cloudnative #containers #devops #infrastructure

28

Over the years of building and scaling infrastructure on AWS, one thing that always felt unnecessarily repetitive was managing NAT Gateways across Availability Zones. If your workloads expanded to a new AZ → you created a new NAT Gateway. If an AZ had no workloads → you wondered why you were still paying for the NAT there. And of course, every change meant updating route tables… again. Last week, AWS quietly fixed this pain. 🚀 AWS introduces Regional Availability Mode for NAT Gateways With this launch, we can finally deploy a single NAT Gateway per VPC that automatically expands and contracts across AZs based on workload presence. No more provisioning NATs per-AZ. No public subnet requirement. No route-table reshuffling. This is one of those changes that instantly simplifies VPC design for DevOps, SRE, and Infra teams. 🧠 Why this matters (from an engineering perspective) 1️⃣ Dynamic, workload-aware scaling The NAT Gateway now “follows” your workloads across AZs. If you launch instances in a new AZ, the regional NAT gateway becomes available there automatically. 2️⃣ Built-in high availability Instead of building HA manually per AZ, the regional mode ensures NAT resilience across your entire region with zero engineering effort. 3️⃣ Cleaner, more secure VPC designs You can run private-only architectures without maintaining a public subnet just for NAT. 4️⃣ Operational overhead drops significantly No more per-AZ NAT deployments No more lifecycle management across AZ boundaries No more route table updates Use AWS IPs or BYOIP seamlessly For teams managing large multi-AZ workloads, this is a massive simplification. 🌏 Availability This feature is live across all commercial AWS regions except GovCloud and China regions. Full announcement: https://lnkd.in/d_YFjm87

2