We are using Kubernetes for application deployment and management.
The following namespaces are used for the different environments:
| Environment | Kubernetes Namespace |
|---|---|
| UAT / Pre-Production | los-ns |
| SIT | los-ns-sit |
Important: Always verify the correct namespace before running Kubernetes commands.
For example:
UAT / Pre-Production:
kubectl get pods -n los-ns
SIT:
kubectl get pods -n los-ns-sit
Check the Kubernetes cluster connection:
kubectl cluster-info
Check the current Kubernetes context:
kubectl config current-context
List available contexts:
kubectl config get-contexts
Switch context if multiple clusters are configured:
kubectl config use-context <context-name>
Check Kubernetes client/server version:
kubectl version
kubectl get pods -n los-ns
kubectl get pods -n los-ns-sit
Get pods with more information:
kubectl get pods -n los-ns -o wide
Get pods in SIT:
kubectl get pods -n los-ns-sit -o wide
The -o wide option provides additional information such as:
To continuously monitor pods:
kubectl get pods -n los-ns -w
For SIT:
kubectl get pods -n los-ns-sit -w
Press Ctrl+C to stop watching.
This is useful while:
To see the main resources in a namespace:
kubectl get all -n los-ns
For SIT:
kubectl get all -n los-ns-sit
This can show:
UAT / Pre-Production:
kubectl get deployments -n los-ns
SIT:
kubectl get deployments -n los-ns-sit
Short form:
kubectl get deploy -n los-ns
Get deployments with additional information:
kubectl get deploy -n los-ns -o wide
When a deployment is not behaving as expected, use:
kubectl describe deployment <deployment-name> -n los-ns
Example:
kubectl describe deployment loan-service -n los-ns
For SIT:
kubectl describe deployment loan-service -n los-ns-sit
Pay attention to:
Deployments create ReplicaSets, which manage the pods.
kubectl get rs -n los-ns
For SIT:
kubectl get rs -n los-ns-sit
If a deployment is not creating the expected pods, check the ReplicaSets:
kubectl describe rs <replicaset-name> -n los-ns
This is one of the most important troubleshooting commands.
kubectl describe pod <pod-name> -n los-ns
Example:
kubectl describe pod loan-service-7d8f6c9d5f-abc12 -n los-ns
For SIT:
kubectl describe pod <pod-name> -n los-ns-sit
Check the bottom Events section carefully.
Common events include:
FailedBackOffFailedMountFailedSchedulingImagePullBackOffErrImagePullUnhealthyReadiness probe failedLiveness probe failedBasic logs:
kubectl logs <pod-name> -n los-ns
For SIT:
kubectl logs <pod-name> -n los-ns-sit
Follow logs in real time:
kubectl logs -f <pod-name> -n los-ns
The -f means follow.
Press Ctrl+C to stop.
If a pod contains multiple containers:
kubectl logs <pod-name> -c <container-name> -n los-ns
Example:
kubectl logs loan-service-7d8f6c9d5f-abc12 \
-c loan-service \
-n los-ns
List containers inside a pod:
kubectl get pod <pod-name> -n los-ns \
-o jsonpath='{.spec.containers[*].name}'
If the container restarted, the previous container's logs can be extremely useful.
kubectl logs <pod-name> --previous -n los-ns
For a specific container:
kubectl logs <pod-name> -c <container-name> --previous -n los-ns
Use this when:
Get the last 100 lines:
kubectl logs <pod-name> --tail=100 -n los-ns
Get the last 500 lines:
kubectl logs <pod-name> --tail=500 -n los-ns
Logs from a specific time period:
kubectl logs <pod-name> --since=1h -n los-ns
Example:
kubectl logs <pod-name> --since=30m -n los-ns
If an application needs to be restarted, use:
kubectl rollout restart deployment/<deployment-name> -n los-ns
Example:
kubectl rollout restart deployment/loan-service -n los-ns
For SIT:
kubectl rollout restart deployment/loan-service -n los-ns-sit
This performs a rolling restart of the deployment.
After restarting or deploying:
kubectl rollout status deployment/<deployment-name> -n los-ns
Example:
kubectl rollout status deployment/loan-service -n los-ns
Expected output should indicate that the rollout has successfully completed.
To see previous deployment revisions:
kubectl rollout history deployment/<deployment-name> -n los-ns
Example:
kubectl rollout history deployment/loan-service -n los-ns
This is useful for identifying previous deployment versions.
If a deployment needs to be reverted to the previous revision:
kubectl rollout undo deployment/<deployment-name> -n los-ns
Example:
kubectl rollout undo deployment/loan-service -n los-ns
After rollback:
kubectl rollout status deployment/loan-service -n los-ns
Note: Rollback should be performed according to the team's deployment/change-management process.
List services:
kubectl get svc -n los-ns
For SIT:
kubectl get svc -n los-ns-sit
Describe a service:
kubectl describe svc <service-name> -n los-ns
Check service endpoints:
kubectl get endpoints <service-name> -n los-ns
If supported by the cluster version, EndpointSlices can also be checked:
kubectl get endpointslice -n los-ns
If a service has no endpoints, investigate:
kubectl get pod <pod-name> -n los-ns --show-labels
This is useful when troubleshooting service-to-pod connectivity.
Check the deployment selector:
kubectl describe deployment <deployment-name> -n los-ns
Make sure the Service selector matches the appropriate pod labels.
kubectl get pods -n los-ns -o wide
This helps identify:
kubectl get pods -n los-ns
Example output:
NAME READY STATUS RESTARTS AGE
loan-service-7d8f6c9d5f-x1 1/1 Running 0 2h
The RESTARTS column is important.
A continuously increasing restart count may indicate:
Events are extremely useful when troubleshooting pod/deployment issues.
kubectl get events -n los-ns --sort-by='.lastTimestamp'
For SIT:
kubectl get events -n los-ns-sit --sort-by='.lastTimestamp'
To see events for a specific pod:
kubectl describe pod <pod-name> -n los-ns
Check the Events section at the bottom.
Running
The pod is currently running.
Check the READY column to confirm containers are ready.
Pending
The pod has not been scheduled or started successfully.
Check:
kubectl describe pod <pod-name> -n los-ns
Look at Events for scheduling/resource problems.
CrashLoopBackOff
The container starts and repeatedly crashes.
First commands to run:
kubectl logs <pod-name> -n los-ns
Then:
kubectl logs <pod-name> --previous -n los-ns
And:
kubectl describe pod <pod-name> -n los-ns
ImagePullBackOff
Kubernetes cannot pull the container image.
Check:
kubectl describe pod <pod-name> -n los-ns
Look for:
ErrImagePull
The image could not be pulled.
Use:
kubectl describe pod <pod-name> -n los-ns
Check the Events section.
Error
The container terminated with an error.
Check:
kubectl logs <pod-name> -n los-ns
And:
kubectl describe pod <pod-name> -n los-ns
If Metrics Server is available:
kubectl top pods -n los-ns
For SIT:
kubectl top pods -n los-ns-sit
Check node resource usage:
kubectl top nodes
This is useful for identifying:
kubectl describe deployment <deployment-name> -n los-ns
Look for:
Requests:
cpu:
memory:
Limits:
cpu:
memory:
Resource problems can result in:
Check the pod:
kubectl describe pod <pod-name> -n los-ns
Look for:
Reason: OOMKilled
You can also inspect the container termination state:
kubectl get pod <pod-name> -n los-ns -o jsonpath='{.status.containerStatuses[*].lastState.terminated.reason}'
If OOMKilled is reported, investigate:
To open a shell:
kubectl exec -it <pod-name> -n los-ns -- /bin/bash
If Bash is not available:
kubectl exec -it <pod-name> -n los-ns -- /bin/sh
Example:
kubectl exec -it loan-service-7d8f6c9d5f-x1 -n los-ns -- /bin/sh
Once inside the container:
ps
env
df -h
ls -la
Exit:
exit
Use
kubectl execmainly for troubleshooting. Avoid manually changing application files or configuration inside a running pod because pods are normally recreated from the deployment configuration/image.
To inspect environment variables:
kubectl exec -it <pod-name> -n los-ns -- env
To inspect the deployment configuration:
kubectl describe deployment <deployment-name> -n los-ns
Environment variables may come from:
List ConfigMaps:
kubectl get configmap -n los-ns
Get a ConfigMap:
kubectl get configmap <configmap-name> -n los-ns
Describe:
kubectl describe configmap <configmap-name> -n los-ns
List secrets:
kubectl get secrets -n los-ns
Describe a secret:
kubectl describe secret <secret-name> -n los-ns
Do not expose secret values in tickets, screenshots, chat messages, or logs.
If there is a suspected secret/configuration issue, involve the Kubernetes/DevOps team rather than modifying secrets directly.
Check which image a deployment is using:
kubectl get deployment <deployment-name> \
-n los-ns \
-o jsonpath='{.spec.template.spec.containers[*].image}'
This is useful for verifying whether the expected application version/image is deployed.
If you know the application/deployment name:
kubectl get pods -n los-ns
Then identify the relevant pods.
You can also filter by label:
kubectl get pods -n los-ns -l app=<app-name>
Example:
kubectl get pods -n los-ns -l app=loan-service
First identify the application label:
kubectl get pods -n los-ns --show-labels
Then:
kubectl logs -l app=<app-name> -n los-ns
For following logs:
kubectl logs -f -l app=<app-name> -n los-ns
For multiple replicas, review whether the issue is occurring on one pod or across all replicas.
Use:
kubectl describe pod <pod-name> -n los-ns
Look for:
Liveness:
Readiness:
Startup:
Common issues:
Readiness probe failed
Liveness probe failed
Startup probe failed
When a probe fails, investigate:
If an application is not accessible, follow this sequence.
kubectl get pods -n los-ns
kubectl logs <pod-name> -n los-ns
kubectl describe pod <pod-name> -n los-ns
kubectl get svc -n los-ns
kubectl describe svc <service-name> -n los-ns
kubectl get endpoints <service-name> -n los-ns
If there are no endpoints, investigate pod readiness and service selectors.
Run:
kubectl get deployment <deployment-name> -n los-ns
Then:
kubectl describe deployment <deployment-name> -n los-ns
Check ReplicaSets:
kubectl get rs -n los-ns
Check pods:
kubectl get pods -n los-ns
Describe the problematic pod:
kubectl describe pod <pod-name> -n los-ns
Finally check logs:
kubectl logs <pod-name> -n los-ns
Recommended sequence:
kubectl get pods -n los-ns
kubectl logs <pod-name> -n los-ns
kubectl logs <pod-name> --previous -n los-ns
kubectl describe pod <pod-name> -n los-ns
Then check:
Run:
kubectl describe pod <pod-name> -n los-ns
Check Events for the exact image pull error.
Verify the image:
kubectl get deployment <deployment-name> \
-n los-ns \
-o jsonpath='{.spec.template.spec.containers[*].image}'
Common causes:
Check pod usage:
kubectl top pods -n los-ns
Sort/check the highest resource consumers.
Then inspect:
kubectl describe pod <pod-name> -n los-ns
Check resource configuration:
kubectl describe deployment <deployment-name> -n los-ns
For memory-related crashes, specifically check for:
OOMKilled
kubectl get pods -n los-ns
kubectl get deploy -n los-ns
kubectl get svc -n los-ns
kubectl get rs -n los-ns
kubectl get configmap -n los-ns
kubectl get secrets -n los-ns
kubectl get events -n los-ns --sort-by='.lastTimestamp'
kubectl get pods -n los-ns-sit
kubectl get deploy -n los-ns-sit
kubectl get svc -n los-ns-sit
kubectl get rs -n los-ns-sit
kubectl get configmap -n los-ns-sit
kubectl get secrets -n los-ns-sit
kubectl get events -n los-ns-sit --sort-by='.lastTimestamp'
When a developer reports:
"My application is not working."
Follow this basic flow:
Application Issue
|
v
Check Namespace
|
v
Check Deployment
|
v
Check Pods
|
v
Check Pod Status
|
+---- Running? --------> Check Logs
|
+---- Pending? --------> Describe Pod + Events
|
+---- CrashLoop? ------> Current + Previous Logs
|
+---- ImagePull? ------> Describe Pod + Image
|
+---- OOMKilled? ------> Check Memory Usage/Limits
|
v
Check Service
|
v
Check Endpoints
|
v
Check Application Health
For most application issues, run the following commands first.
kubectl get deployment <deployment-name> -n los-ns
kubectl get pods -n los-ns
kubectl describe pod <pod-name> -n los-ns
kubectl logs <pod-name> -n los-ns
kubectl logs <pod-name> --previous -n los-ns
kubectl get svc -n los-ns
kubectl get endpoints <service-name> -n los-ns
kubectl get events -n los-ns --sort-by='.lastTimestamp'
Before restarting a deployment:
Restart:
kubectl rollout restart deployment/<deployment-name> -n <namespace>
Monitor:
kubectl rollout status deployment/<deployment-name> -n <namespace>
Then:
kubectl get pods -n <namespace>
Finally check:
kubectl logs <pod-name> -n <namespace>
describe output and Kubernetes Events.--previous logs when a container has restarted.| Purpose | Command |
|---|---|
| List pods | kubectl get pods -n <namespace> |
| Watch pods | kubectl get pods -n <namespace> -w |
| List deployments | kubectl get deploy -n <namespace> |
| List services | kubectl get svc -n <namespace> |
| List all resources | kubectl get all -n <namespace> |
| Describe pod | kubectl describe pod <pod> -n <namespace> |
| Describe deployment | kubectl describe deploy <deployment> -n <namespace> |
| View logs | kubectl logs <pod> -n <namespace> |
| Follow logs | kubectl logs -f <pod> -n <namespace> |
| Previous logs | kubectl logs <pod> --previous -n <namespace> |
| Restart deployment | kubectl rollout restart deployment/<deployment> -n <namespace> |
| Rollout status | kubectl rollout status deployment/<deployment> -n <namespace> |
| Rollout history | kubectl rollout history deployment/<deployment> -n <namespace> |
| Rollback | kubectl rollout undo deployment/<deployment> -n <namespace> |
| Pod resource usage | kubectl top pods -n <namespace> |
| Node resource usage | kubectl top nodes |
| View events | kubectl get events -n <namespace> --sort-by='.lastTimestamp' |
| Enter pod | kubectl exec -it <pod> -n <namespace> -- /bin/sh |
| List ConfigMaps | kubectl get configmap -n <namespace> |
| List Secrets | kubectl get secrets -n <namespace> |
| List ReplicaSets | kubectl get rs -n <namespace> |
| List endpoints | kubectl get endpoints -n <namespace> |
For this Kubernetes environment:
UAT / Pre-Production
Namespace: los-ns
SIT
Namespace: los-ns-sit
kubectl get pods -n los-ns
kubectl logs <pod-name> -n los-ns
kubectl describe pod <pod-name> -n los-ns
kubectl rollout restart deployment/<deployment-name> -n los-ns
kubectl get pods -n los-ns-sit
kubectl logs <pod-name> -n los-ns-sit
kubectl describe pod <pod-name> -n los-ns-sit
kubectl rollout restart deployment/<deployment-name> -n los-ns-sit
When escalating a Kubernetes issue to the DevOps/Infrastructure team, provide:
kubectl describe pod outputThis information helps the infrastructure team investigate the issue faster.
This guide provides the standard Kubernetes troubleshooting commands for developers working with the SIT and UAT/Pre-Production environments.
The most important troubleshooting sequence is:
kubectl get pods -n <namespace>
kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace>
kubectl logs <pod-name> --previous -n <namespace>
kubectl get events -n <namespace> --sort-by='.lastTimestamp'
For a deployment restart:
kubectl rollout restart deployment/<deployment-name> -n <namespace>
kubectl rollout status deployment/<deployment-name> -n <namespace>
Always use the correct namespace:
UAT / Pre-Production → los-ns
SIT → los-ns-sit