k8s/services: add ingress networking portainer llm and project guides
- Nginx ingress + TLS termination (homelab-ca) - Portainer container UI - CoreDNS internal DNS rewrites - DuckDNS DDNS updater - Ollama LLM inference - 8 project-usage guides (team reference)
This commit is contained in:
Story Crater Bot
@@ -0,0 +1,314 @@
|
||||
# Networking: Ingress, TLS & Service Discovery
|
||||
|
||||
**Ingress Controller:** `nginx-ingress` (Nginx)
|
||||
**Load Balancer:** Cilium LB-IPAM (eBPF-based)
|
||||
**TLS CA:** homelab-ca (self-signed, 10-year)
|
||||
**Namespace:** `ingress-nginx`
|
||||
|
||||
## When to Use
|
||||
|
||||
- **Public HTTPS endpoints** — External access via TLS
|
||||
- **Hostname-based routing** — Multiple services on same IP
|
||||
- **TLS termination** — Offload encryption/decryption
|
||||
- **Service discovery** — Internal DNS (CoreDNS)
|
||||
|
||||
## Quick Start
|
||||
|
||||
**1. Create Ingress rule:**
|
||||
```yaml
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: myapp
|
||||
namespace: myapp-ns
|
||||
annotations:
|
||||
cert-manager.io/cluster-issuer: "letsencrypt-prod" # or homelab-ca
|
||||
spec:
|
||||
ingressClassName: nginx
|
||||
tls:
|
||||
- hosts:
|
||||
- myapp.riotpiao.homelab.com
|
||||
secretName: myapp-tls
|
||||
rules:
|
||||
- host: myapp.riotpiao.homelab.com
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: myapp-svc
|
||||
port:
|
||||
number: 8080
|
||||
```
|
||||
|
||||
**2. Deploy:**
|
||||
```bash
|
||||
kubectl apply -f ingress.yaml
|
||||
|
||||
# Wait for cert issuance
|
||||
kubectl get certificate -n myapp-ns
|
||||
# Should show "Ready" after ~30s
|
||||
```
|
||||
|
||||
**3. Test from client:**
|
||||
```bash
|
||||
# Add to /etc/hosts (or use WireGuard)
|
||||
192.168.1.160 myapp.riotpiao.homelab.com
|
||||
|
||||
# Access
|
||||
curl https://myapp.riotpiao.homelab.com
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
| Key | Value |
|
||||
|-----|-------|
|
||||
| Ingress class | `nginx` |
|
||||
| Load balancer type | `LoadBalancer` (Cilium LB-IPAM) |
|
||||
| TLS issuer | `homelab-ca` (ClusterIssuer) |
|
||||
| TLS cert lifetime | 90 days (auto-renewed by cert-manager) |
|
||||
| DNS | CoreDNS (in-cluster), external via `/etc/hosts` or DuckDNS |
|
||||
|
||||
## Common Patterns
|
||||
|
||||
**Ingress with path-based routing:**
|
||||
```yaml
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: api
|
||||
namespace: default
|
||||
spec:
|
||||
ingressClassName: nginx
|
||||
tls:
|
||||
- hosts:
|
||||
- api.riotpiao.homelab.com
|
||||
secretName: api-tls
|
||||
rules:
|
||||
- host: api.riotpiao.homelab.com
|
||||
http:
|
||||
paths:
|
||||
- path: /users
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: users-svc
|
||||
port:
|
||||
number: 3000
|
||||
- path: /orders
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: orders-svc
|
||||
port:
|
||||
number: 3001
|
||||
```
|
||||
|
||||
**Ingress with basic auth:**
|
||||
```bash
|
||||
# Generate htpasswd
|
||||
htpasswd -c auth admin
|
||||
# → prompted for password
|
||||
|
||||
# Create Secret
|
||||
kubectl create secret generic basic-auth --from-file=auth -n default
|
||||
|
||||
# Create Ingress
|
||||
```
|
||||
|
||||
```yaml
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: protected
|
||||
namespace: default
|
||||
annotations:
|
||||
nginx.ingress.kubernetes.io/auth-type: basic
|
||||
nginx.ingress.kubernetes.io/auth-secret: basic-auth
|
||||
nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required'
|
||||
spec:
|
||||
ingressClassName: nginx
|
||||
rules:
|
||||
- host: protected.riotpiao.homelab.com
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: app-svc
|
||||
port:
|
||||
number: 8080
|
||||
```
|
||||
|
||||
**Internal DNS (CoreDNS rewrite):**
|
||||
```yaml
|
||||
# k8s/coredns/coredns-configmap.yaml
|
||||
# Rewrite:
|
||||
# - grafana.riotpiao.homelab.com → grafana.logging (cluster-internal)
|
||||
# - prometheus.riotpiao.homelab.com → prometheus-kube-prom-prometheus.monitoring
|
||||
#
|
||||
# Allows pods to use external URLs but resolve to internal Services
|
||||
```
|
||||
|
||||
**Fixed LoadBalancer IP (Cilium LB-IPAM):**
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: ingress-nginx
|
||||
namespace: ingress-nginx
|
||||
annotations:
|
||||
io.cilium/lb-ipam-ips: "192.168.1.160" # fixed IP
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
selector:
|
||||
app: nginx-ingress
|
||||
ports:
|
||||
- port: 443
|
||||
targetPort: 443
|
||||
protocol: TCP
|
||||
```
|
||||
|
||||
## TLS Certificate Management
|
||||
|
||||
**Automatic renewal (cert-manager):**
|
||||
```yaml
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: myapp-cert
|
||||
namespace: myapp-ns
|
||||
spec:
|
||||
secretName: myapp-tls
|
||||
duration: 2160h # 90 days
|
||||
renewBefore: 360h # renew 15 days before expiry
|
||||
commonName: myapp.riotpiao.homelab.com
|
||||
dnsNames:
|
||||
- myapp.riotpiao.homelab.com
|
||||
issuerRef:
|
||||
name: homelab-ca
|
||||
kind: ClusterIssuer
|
||||
```
|
||||
|
||||
**Check certificate status:**
|
||||
```bash
|
||||
# List certs
|
||||
k get certificate -A
|
||||
|
||||
# View cert details
|
||||
k describe certificate -n myapp-ns myapp-cert
|
||||
|
||||
# View TLS Secret
|
||||
k get secret -n myapp-ns myapp-tls -o json | jq '.data."tls.crt"' | base64 -d | openssl x509 -text
|
||||
|
||||
# Check expiry date
|
||||
k get secret -n myapp-ns myapp-tls -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -enddate
|
||||
```
|
||||
|
||||
## Service Discovery
|
||||
|
||||
**Cluster-internal DNS:**
|
||||
```bash
|
||||
# From any pod, resolve via CoreDNS
|
||||
nslookup grafana.logging.svc.cluster.local # full FQDN
|
||||
nslookup grafana.logging # short form (same namespace)
|
||||
nslookup grafana # if in logging namespace
|
||||
|
||||
# Resolved to ClusterIP (internal only)
|
||||
```
|
||||
|
||||
**External DNS (WireGuard VPN or port-forward):**
|
||||
```bash
|
||||
# Option 1: WireGuard tunnel
|
||||
# Client connects to 10.6.0.1 (WireGuard server on talos-cp-1)
|
||||
# All traffic tunneled to cluster
|
||||
|
||||
# Option 2: Port-forward from jump box
|
||||
make pf-grafana # localhost:3000 → grafana.logging:3000
|
||||
|
||||
# Option 3: Add to /etc/hosts (on home network)
|
||||
192.168.1.160 grafana.riotpiao.homelab.com
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
**Grafana dashboard:** `svc-nginx-ingress`
|
||||
|
||||
**Key metrics:**
|
||||
- `nginx_requests_total` — total requests
|
||||
- `nginx_request_duration_seconds` — latency histogram
|
||||
- `nginx_ingress_upstream_requests_total{status=~"5.."}` — backend errors
|
||||
- `nginx_ssl_expire_time_seconds` — cert expiry countdown
|
||||
|
||||
**Alert on cert expiry:**
|
||||
```yaml
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: cert-expiry
|
||||
namespace: ingress-nginx
|
||||
spec:
|
||||
groups:
|
||||
- name: cert-expiry
|
||||
rules:
|
||||
- alert: CertificateExpiringSoon
|
||||
expr: nginx_ssl_expire_time_seconds < 86400 * 14 # < 14 days
|
||||
annotations:
|
||||
summary: "Certificate {{ $labels.host }} expires in {{ $value | humanizeDuration }}"
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Certificate stuck in "Pending":**
|
||||
```bash
|
||||
# Check cert-manager logs
|
||||
k logs -n cert-manager -f deploy/cert-manager
|
||||
|
||||
# Verify ClusterIssuer exists
|
||||
k get clusterissuer
|
||||
|
||||
# Check ACME order (if using LetsEncrypt)
|
||||
k describe certificate -n myapp-ns myapp-cert
|
||||
```
|
||||
|
||||
**Ingress not exposing service (503 error):**
|
||||
```bash
|
||||
# Verify Service exists and has endpoints
|
||||
k get svc -n myapp-ns
|
||||
k get endpoints -n myapp-ns myapp-svc
|
||||
|
||||
# Check if pods are ready
|
||||
k get pods -n myapp-ns
|
||||
|
||||
# Test pod directly (port-forward)
|
||||
k port-forward -n myapp-ns pod/myapp-0 8080:8080
|
||||
curl http://localhost:8080
|
||||
```
|
||||
|
||||
**DNS resolution fails from pod:**
|
||||
```bash
|
||||
# Test from pod
|
||||
k run -it --rm debug --image=busybox:1.28 --restart=Never -- \
|
||||
nslookup grafana.logging.svc.cluster.local
|
||||
|
||||
# If fails, CoreDNS may be unhealthy
|
||||
k get pods -n kube-system -l k8s-app=kube-dns
|
||||
k logs -n kube-system -l k8s-app=kube-dns
|
||||
```
|
||||
|
||||
**TLS handshake error (cert not trusted):**
|
||||
```bash
|
||||
# Verify TLS cert Secret exists
|
||||
k get secret -n myapp-ns myapp-tls
|
||||
|
||||
# Verify cert is correctly signed by homelab-ca
|
||||
k get secret -n myapp-ns myapp-tls -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text | grep -A 5 "Issuer:"
|
||||
|
||||
# If cert is self-signed (homelab-ca), add to client's trusted roots
|
||||
# Or bypass cert verification (dev only):
|
||||
curl -k https://myapp.riotpiao.homelab.com
|
||||
```
|
||||
|
||||
See `/TROUBLESHOOTING.md` for full incident guide.
|
||||
Reference in New Issue
Block a user