Search for a command to run...
Your n8n troubleshooting handbook for Kubernetes: Address database, queue, and access problems.
I am a student studying in Mumbai University, learning DevOps, looking for opportunities to learn more things by gaining experience at prestigious institutions
No comments yet. Be the first to comment.
A comprehensive guide to deploying n8n workflow automation platform on AKS using first principles and production-grade best practices.
Review the business benefits of n8n on AKS for workflow automation Explore advanced enhancements & future improvements for your production deployment.
A Dev Journey Through Hackathons, Childhood Games, and Pokémon Cards
A Deep Dive into CI/CD, Workflow Architecture, and Secure Automation for Modern Software Development
A comprehensive guide to creating a robust, scalable CI/CD pipeline for a multi-tier Java application using AWS, Kubernetes, Terraform &GitHub Actions
A technical deep dive into how we built a cloud-native system that transforms handwritten prescriptions into structured medication information
A First Principles Approach to Decoding Server Activity and Transforming Data into Actionable Intelligence
This is Part 7 of the "Building a Production-Ready n8n Workflow Automation Platform on Azure Kubernetes Service" series. View the complete series here.
Welcome to Part 7 of our n8n on AKS series! In Part 6, we implemented monitoring and optimization strategies. Even with the best preparation, issues can arise, so today we'll explore comprehensive troubleshooting techniques.
Even the most carefully designed systems encounter issues. This section provides a comprehensive guide to troubleshooting common problems you might encounter with your n8n deployment on AKS.
Symptoms:
Error: connect ETIMEDOUT or Error: connect ECONNREFUSEDDiagnostic Approach:
Check PostgreSQL pod status:
kubectl get pods -n n8n -l app=postgres
Verify PostgreSQL service:
kubectl get svc postgres-service -n n8n
Check database logs:
kubectl logs $(kubectl get pod -l app=postgres -n n8n -o jsonpath='{.items[0].metadata.name}') -n n8n
Test database connection from n8n pod:
kubectl exec -it $(kubectl get pod -l app=n8n -n n8n -o jsonpath='{.items[0].metadata.name}') -n n8n -- \
node -e "const { Pool } = require('pg'); const pool = new Pool({host: 'postgres-service', user: process.env.DB_POSTGRESDB_USER, password: process.env.DB_POSTGRESDB_PASSWORD, database: 'n8n'}); pool.query('SELECT NOW()', (err, res) => { console.log(err || res.rows[0]); pool.end(); })"
Common Resolutions:
Authentication Issues:
Network Issues:
Database Health Issues:
Symptoms:
Error connecting to Redis in n8n logsDiagnostic Approach:
Check Redis pod status:
kubectl get pods -n n8n -l app=redis
Verify Redis service:
kubectl get svc redis-service -n n8n
Check Redis logs:
kubectl logs $(kubectl get pod -l app=redis -n n8n -o jsonpath='{.items[0].metadata.name}') -n n8n
Test Redis connection from n8n pod:
kubectl exec -it $(kubectl get pod -l app=n8n -n n8n -o jsonpath='{.items[0].metadata.name}') -n n8n -- \
node -e "const Redis = require('ioredis'); const redis = new Redis('redis-service'); redis.ping().then(res => { console.log(res); redis.disconnect(); })"
Common Resolutions:
Connection Configuration:
Queue Stuck Issues:
Worker Configuration:
Symptoms:
Diagnostic Approach:
Check certificate status:
kubectl get certificate -n n8n
Examine certificate details:
kubectl describe certificate n8n-tls-secret -n n8n
Check cert-manager logs:
kubectl logs -n cert-manager -l app=cert-manager
Verify the ClusterIssuer status:
kubectl describe clusterissuer letsencrypt-prod
Common Resolutions:
Domain Validation Issues:
Configuration Issues:
Certificate Renewal Issues:
Symptoms:
Diagnostic Approach:
Check Ingress resource status:
kubectl get ingress -n n8n
Verify Ingress controller pods:
kubectl get pods -n default -l app.kubernetes.io/name=ingress-nginx
Check Ingress controller logs:
kubectl logs -n default -l app.kubernetes.io/name=ingress-nginx
Test connectivity to Ingress IP:
curl -v http://<ingress-ip>
Common Resolutions:
DNS Issues:
nslookup or dig to verify DNS resolutionIngress Configuration:
Network Issues:
The following diagram presents a structured approach to troubleshooting n8n deployment issues:
If workflow execution is slow or database operations are taking too long:
Check database load:
kubectl exec -it $(kubectl get pod -l app=postgres -n n8n -o jsonpath='{.items[0].metadata.name}') -n n8n -- \
psql -U postgres -c "SELECT * FROM pg_stat_activity WHERE state = 'active';"
Identify long-running queries:
kubectl exec -it $(kubectl get pod -l app=postgres -n n8n -o jsonpath='{.items[0].metadata.name}') -n n8n -- \
psql -U postgres -c "SELECT pid, now() - query_start AS duration, query FROM pg_stat_activity WHERE state = 'active' ORDER BY duration DESC;"
Check table sizes and index usage:
kubectl exec -it $(kubectl get pod -l app=postgres -n n8n -o jsonpath='{.items[0].metadata.name}') -n n8n -- \
psql -U postgres -d n8n -c "SELECT relname, pg_size_pretty(pg_total_relation_size(relid)) AS total_size FROM pg_catalog.pg_statio_user_tables ORDER BY pg_total_relation_size(relid) DESC;"
If n8n pods are steadily increasing in memory usage:
Get memory usage metrics:
kubectl top pods -n n8n
Check container memory limit and usage:
kubectl describe pod $(kubectl get pod -l app=n8n -n n8n -o jsonpath='{.items[0].metadata.name}') -n n8n
Generate a heap dump (for advanced debugging):
kubectl exec -it $(kubectl get pod -l app=n8n -n n8n -o jsonpath='{.items[0].metadata.name}') -n n8n -- \
node --expose-gc -e "const fs=require('fs'); setTimeout(() => { global.gc(); const heapSnapshot = require('v8').getHeapSnapshot(); const file = fs.createWriteStream('/tmp/heap.json'); heapSnapshot.pipe(file); }, 1000);"
kubectl cp n8n/$(kubectl get pod -l app=n8n -n n8n -o jsonpath='{.items[0].metadata.name}'):/tmp/heap.json ./heap.json
Analyze the heap dump with Chrome DevTools or a memory analyzer.
If services can't communicate with each other:
Test network connectivity between pods:
kubectl exec -it $(kubectl get pod -l app=n8n -n n8n -o jsonpath='{.items[0].metadata.name}') -n n8n -- \
nc -zv postgres-service 5432
kubectl exec -it $(kubectl get pod -l app=n8n -n n8n -o jsonpath='{.items[0].metadata.name}') -n n8n -- \
nc -zv redis-service 6379
Check if network policies are restricting traffic:
kubectl get networkpolicies -n n8n
Verify DNS resolution:
kubectl exec -it $(kubectl get pod -l app=n8n -n n8n -o jsonpath='{.items[0].metadata.name}') -n n8n -- \
nslookup postgres-service
kubectl exec -it $(kubectl get pod -l app=n8n -n n8n -o jsonpath='{.items[0].metadata.name}') -n n8n -- \
nslookup redis-service
For complex issues, it's often helpful to collect comprehensive diagnostic information:
#!/bin/bash
# collect-diagnostics.sh - Collect diagnostic information for n8n deployment
# Create output directory
mkdir -p n8n-diagnostics
cd n8n-diagnostics
# Collect pod information
kubectl get pods -n n8n -o yaml > pods.yaml
kubectl describe pods -n n8n > pods-describe.txt
# Collect logs
for pod in $(kubectl get pods -n n8n -o jsonpath='{.items[*].metadata.name}'); do
kubectl logs $pod -n n8n > $pod-logs.txt
done
# Collect service and endpoint information
kubectl get svc,endpoints -n n8n -o yaml > services.yaml
# Collect ingress and certificate information
kubectl get ingress,certificate -n n8n -o yaml > ingress-cert.yaml
kubectl describe ingress,certificate -n n8n > ingress-cert-describe.txt
# Collect events
kubectl get events -n n8n > events.txt
# Collect resource usage
kubectl top pods -n n8n > pod-resources.txt
kubectl top nodes > node-resources.txt
# Create a tar archive
tar -czf n8n-diagnostics.tar.gz *
echo "Diagnostic information collected in n8n-diagnostics.tar.gz"
| Issue | Check Command | Resolution Strategy | |
| Pod won't start | kubectl describe pod <pod-name> -n n8n | Check events section for errors | |
| Pod crashing | kubectl logs <pod-name> -n n8n | Look for error messages near the end | |
| Service unavailable | kubectl get endpoints <service-name> -n n8n | Verify endpoints exist | |
| Certificate issues | kubectl describe certificate <cert-name> -n n8n | Check events and conditions | |
| Database connection | `kubectl exec -it -n n8n -- env \ | grep DB_` | Verify environment variables |
| Redis connection | `kubectl exec -it -n n8n -- env \ | grep REDIS` | Verify environment variables |
| Ingress not working | kubectl get ingress <ingress-name> -n n8n | Check ADDRESS field has an IP | |
| Resource constraints | kubectl top pods -n n8n | Check for pods near resource limits | |
| Webhook not triggering | `kubectl logs -n n8n \ | grep webhook` | Verify webhook URL and connectivity |
Troubleshooting a production n8n deployment on AKS requires a systematic approach. By understanding:
You can quickly identify and resolve problems, minimizing downtime and ensuring a reliable workflow automation platform.
Remember that many issues can be prevented through proper monitoring and proactive maintenance, as discussed in the previous section. When problems do occur, having these troubleshooting procedures documented will significantly reduce the mean time to resolution.
Armed with these troubleshooting strategies and diagnostic workflows, you can quickly identify and resolve issues in your n8n deployment. Remember that systematic investigation and a good understanding of the architecture are key to efficient problem resolution.
In our final article, we'll summarize what we've accomplished, review the benefits of our architecture, and explore advanced enhancements for the future. [Continue to Part 8: Conclusion and Next Steps]
What troubleshooting techniques have you found most effective for Kubernetes applications? Have you encountered any particularly challenging issues with workflow automation systems? Share your stories in the comments!