Migrating Go Applications to Kubernetes: Real Problems and Solutions
Hey everyone!
Migrating a Go application to Kubernetes seems simple on paper. You create a Dockerfile, deploy it, and youβre done, right?
Wrong.
In practice, you encounter problems that donβt appear in the documentation. Applications that worked perfectly on traditional servers start behaving strangely. Requests that take longer. Connections that drop. Resources that arenβt released.
This post is about those real problems. And about how to actually solve them.
What youβll find here
This guide covers the most common problems Go developers face when migrating to Kubernetes:
- ConfigMaps and Secrets: how to manage configuration
- Health checks: liveness and readiness probes
- Graceful shutdown: shutting down applications correctly
- Service discovery: finding other services
- Networking and DNS: connectivity issues
- Resource limits: CPU and memory
- Logs and observability: what changed
Each section has real problems and practical solutions that work in production.
1. ConfigMaps and Secrets: managing configuration
The problem
Your Go application probably reads configuration from files or environment variables:
1
2
3
4
5
// app.go
config := os.Getenv("DATABASE_URL")
if config == "" {
log.Fatal("DATABASE_URL not configured")
}
In Kubernetes, you canβt simply edit files on the server. You need to use ConfigMaps and Secrets.
The solution
Option 1: Environment variables (simplest)
Create a ConfigMap:
1
2
3
4
5
6
7
apiVersion: v1
kind: ConfigMap
metadata:
name: app-config
data:
DATABASE_URL: "postgres://user:pass@db:5432/mydb"
LOG_LEVEL: "info"
And use it in the Deployment:
1
2
3
4
5
6
7
8
9
10
11
12
13
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
template:
spec:
containers:
- name: app
image: my-app:latest
envFrom:
- configMapRef:
name: app-config
Option 2: Mounted files (more flexible)
For larger configuration files:
1
2
3
4
5
6
7
8
9
10
spec:
containers:
- name: app
volumeMounts:
- name: config
mountPath: /etc/app
volumes:
- name: config
configMap:
name: app-config
For Secrets (sensitive data):
1
2
3
4
5
6
7
apiVersion: v1
kind: Secret
metadata:
name: app-secrets
type: Opaque
stringData:
API_KEY: "your-secret-key"
1
2
3
envFrom:
- secretRef:
name: app-secrets
Common problem: ConfigMap doesnβt update
ConfigMaps mounted as volumes are updated, but your application needs to reload. For environment variables, you need to recreate the Pod.
Solution: You can implement hot reload in your Go application to automatically reload configurations when the ConfigMap changes. In the video below, I show how to do this in practice:
Alternatively, you can use a sidecar like Reloader that does the reload automatically.
2. Health checks: liveness and readiness probes
The problem
Your Go application might be running, but itβs not ready to receive traffic. Or it might be stuck, but Kubernetes doesnβt know.
Without health checks, Kubernetes canβt:
- Know when to restart a stuck container
- Know when the application is ready to receive traffic
- Perform rolling updates safely
The solution
Implement health check endpoints in your application:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
// health.go
func healthHandler(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
w.Write([]byte("OK"))
}
func readinessHandler(w http.ResponseWriter, r *http.Request) {
// Check if ready (DB connected, etc)
if db.Ping() != nil {
w.WriteHeader(http.StatusServiceUnavailable)
return
}
w.WriteHeader(http.StatusOK)
}
Configure the probes in the Deployment:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
spec:
containers:
- name: app
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
Difference between liveness and readiness
1
2
3
4
5
6
7
8
9
10
11
βββββββββββββββββββββββββββββββββββ
β Liveness Probe β
β "Is the app alive?" β
β If fails β restarts the Pod β
βββββββββββββββββββββββββββββββββββ
βββββββββββββββββββββββββββββββββββ
β Readiness Probe β
β "Is the app ready?" β
β If fails β removes from Serviceβ
βββββββββββββββββββββββββββββββββββ
Liveness: detects if the application is stuck and needs to be restarted.
Readiness: detects if the application is ready to receive traffic (DB connected, cache loaded, etc).
Common problem: probes too aggressive
If your probes fail too quickly, Kubernetes will constantly restart your Pod.
Solution: Adjust initialDelaySeconds to give the application time to initialize.
3. Graceful shutdown: shutting down correctly
The problem
When Kubernetes needs to terminate a Pod (rolling update, scale down), it sends a SIGTERM. If your application doesnβt handle this correctly, you can:
- Lose requests in processing
- Not close database connections
- Not save state
- Corrupt data
The solution
Implement graceful shutdown in your Go application:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
// main.go
func main() {
// Create HTTP server
srv := &http.Server{
Addr: ":8080",
Handler: mux,
}
// Channel to receive system signals
sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
// Start server in goroutine
go func() {
if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
log.Fatalf("server failed: %v", err)
}
}()
// Wait for signal
<-sigChan
log.Println("Shutdown initiated...")
// Create context with timeout for shutdown
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
// Graceful shutdown
if err := srv.Shutdown(ctx); err != nil {
log.Fatalf("forced shutdown: %v", err)
}
// Close database connections, etc
db.Close()
log.Println("Shutdown complete")
}
Configure the Pod to give time:
1
2
3
4
5
6
7
8
spec:
containers:
- name: app
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 15"]
terminationGracePeriodSeconds: 30
What happens
1
2
3
4
5
1. Kubernetes sends SIGTERM
2. Your application stops accepting new requests
3. Waits for in-flight requests to finish
4. Closes connections
5. Shuts down gracefully
terminationGracePeriodSeconds: maximum time Kubernetes waits before forcing kill (SIGKILL).
4. Service discovery: finding other services
The problem
On traditional servers, you can use fixed IPs or known hosts. In Kubernetes, Pods have dynamic IPs. How to find other services?
The solution
Kubernetes has internal DNS. Use service names:
1
2
3
4
5
// Instead of:
dbURL := "postgres://user:pass@192.168.1.10:5432/db"
// Use:
dbURL := "postgres://user:pass@postgres-service:5432/db"
Kubernetes DNS resolves automatically:
1
2
3
4
5
6
7
8
9
10
11
12
βββββββββββββββββββββββββββββββββββ
β Service Name β
β postgres-service β
β β β
β Kubernetes DNS β
β β β
β Service IP β
β β β
β Load Balancer β
β β β
β Service Pods β
βββββββββββββββββββββββββββββββββββ
Format: <service-name>.<namespace>.svc.cluster.local
For services in the same namespace, you only need the name: postgres-service
Practical example
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
// config.go
func getDBURL() string {
// Kubernetes DNS
host := os.Getenv("DB_HOST")
if host == "" {
host = "postgres-service" // Service name
}
port := os.Getenv("DB_PORT")
if port == "" {
port = "5432"
}
return fmt.Sprintf("postgres://user:pass@%s:%s/db", host, port)
}
Common problem: DNS doesnβt resolve
If youβre testing locally or in development, Kubernetes DNS doesnβt exist.
Solution: Use environment variables for development:
1
2
3
4
5
6
7
8
9
10
host := os.Getenv("DB_HOST")
if host == "" {
if os.Getenv("KUBERNETES_SERVICE_HOST") != "" {
// In Kubernetes
host = "postgres-service"
} else {
// Local development
host = "localhost"
}
}
5. Networking and DNS: connectivity issues
The problem
Your Go application might not be able to connect to other services. Timeouts, connection refused, DNS doesnβt resolve.
Common problems and solutions
1. DNS doesnβt resolve
1
2
3
4
5
6
7
8
9
10
11
12
// Connectivity test
func testConnection(host string) error {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
conn, err := net.DialContext(ctx, "tcp", host)
if err != nil {
return fmt.Errorf("could not connect: %v", err)
}
conn.Close()
return nil
}
2. Timeouts too short
Adjust timeouts for Kubernetes environment:
1
2
3
4
5
6
7
8
9
10
// HTTP client with adequate timeout
client := &http.Client{
Timeout: 30 * time.Second,
Transport: &http.Transport{
DialContext: (&net.Dialer{
Timeout: 10 * time.Second,
KeepAlive: 30 * time.Second,
}).DialContext,
},
}
3. Connections arenβt reused
Use connection pooling:
1
2
3
4
// For database
db.SetMaxOpenConns(25)
db.SetMaxIdleConns(5)
db.SetConnMaxLifetime(5 * time.Minute)
Network debugging
If something doesnβt work, check:
1
2
3
4
# Inside the Pod
nslookup postgres-service
curl http://postgres-service:5432
ping postgres-service
6. Resource limits: CPU and memory
The problem
Without limits, your application can:
- Consume all node CPU
- Exhaust node memory
- Be killed by OOMKiller
- Affect other Pods
The solution
Configure requests and limits:
1
2
3
4
5
6
7
8
9
10
spec:
containers:
- name: app
resources:
requests:
memory: "128Mi"
cpu: "100m"
limits:
memory: "256Mi"
cpu: "500m"
Requests: guaranteed resources (scheduling)
Limits: maximum it can use
GOMAXPROCS and CPU limits
Go uses GOMAXPROCS based on available CPUs. In containers with CPU limits, this can be problematic.
Solution: Use automaxprocs:
1
2
3
4
5
6
import _ "go.uber.org/automaxprocs"
func main() {
// GOMAXPROCS will be automatically adjusted
// based on container CPU limits
}
Memory limits and GC
With memory limits, Goβs GC needs to work harder:
1
2
3
4
// Adjust GOGC if needed
// GOGC=50 = more aggressive (uses less memory)
// GOGC=100 = default
// GOGC=200 = less aggressive (uses more memory)
Monitor memory usage:
1
2
3
var m runtime.MemStats
runtime.ReadMemStats(&m)
log.Printf("Allocated memory: %d MB", m.Alloc/1024/1024)
7. Logs and observability: what changed
The problem
On traditional servers, logs go to files. In Kubernetes, Pods are ephemeral. Logs are lost when Pods are recreated.
The solution
1. Structured logs (JSON)
1
2
3
4
5
6
7
8
import "github.com/sirupsen/logrus"
logrus.SetFormatter(&logrus.JSONFormatter{})
logrus.WithFields(logrus.Fields{
"user_id": 123,
"action": "login",
}).Info("User logged in")
2. Log to stdout/stderr
Kubernetes automatically captures stdout/stderr:
1
2
3
// Use standard log or logging library
log.Println("Log message")
fmt.Fprintf(os.Stderr, "Error: %v\n", err)
3. Context for tracing
Use context to propagate trace IDs:
1
2
3
4
5
6
7
8
9
10
11
12
13
func handleRequest(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
// Add trace ID to context
traceID := r.Header.Get("X-Trace-ID")
if traceID == "" {
traceID = generateTraceID()
}
ctx = context.WithValue(ctx, "trace_id", traceID)
// Logs include trace ID
log.WithContext(ctx).Info("Processing request")
}
Observability integration
For metrics, use Prometheus:
1
2
3
4
5
6
7
8
9
10
11
12
import "github.com/prometheus/client_golang/prometheus"
var httpRequests = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "http_requests_total",
},
[]string{"method", "endpoint", "status"},
)
func init() {
prometheus.MustRegister(httpRequests)
}
Common problems and quick solutions
| Problem | Cause | Solution |
|---|---|---|
| Pod restarts constantly | Liveness probe failing | Increase initialDelaySeconds |
| Requests lost on deploy | No graceful shutdown | Implement graceful shutdown |
| Canβt connect to other services | DNS doesnβt resolve | Use Service names |
| High memory usage | No limits | Configure memory limits |
| CPU not used efficiently | Wrong GOMAXPROCS | Use automaxprocs |
| Logs lost | Logs in files | Log to stdout/stderr |
| Frequent timeouts | Timeouts too short | Adjust network timeouts |
Conclusion
Migrating to Kubernetes isnβt just about deploying. Itβs about adapting your application to a different environment.
The problems youβll encounter are predictable. And the solutions are known. This guide covers the main ones.
The key is understanding how Kubernetes works and adapting your Go application to this environment. Itβs not hard, but it requires attention to detail.
And when you do it right, you gain:
- Automatic scalability
- High availability
- Zero-downtime deployments
- Native observability
- Simplified management
Itβs worth the effort.
References and sources
Official documentation
- Kubernetes Documentation - Complete documentation
- Kubernetes Best Practices - Official best practices
- ConfigMaps - Configuration management
- Secrets - Secrets management
- Probes - Health checks
Articles and guides
- Kubernetes Patterns - Kubernetes patterns
- 12-Factor App - Principles for cloud-native applications
- Go Best Practices - Effective Go
Tools
- Reloader - ConfigMap hot reload
- automaxprocs - Automatic GOMAXPROCS adjustment
- Prometheus Go client - Prometheus metrics
Example code
- Kubernetes Go client - Official Kubernetes client
- controller-runtime - Framework for operators
