Move K8s sidecar from testing/ to examples/

The K8s sidecar is a standalone example, not a test harness — it has no CI integration unlike testing/sidecar/. Move it to examples/ alongside sidecar-nostr-relay/ where it belongs. Update internal path references in Dockerfile, build.sh, and README.
2026-07-30 19:46:15 +00:00 · 2026-03-19 19:36:39 +00:00
parent 9e63b42bd9
commit 8d51dbd268
6 changed files with 4 additions and 4 deletions
@@ -0,0 +1,176 @@
+# FIPS Kubernetes Sidecar
+
+Run FIPS as a sidecar container in a Kubernetes Pod, injecting the FIPS mesh
+network interface into every other container in the Pod. All containers in
+the Pod share the same network namespace, so once the sidecar creates `fips0`
+it is immediately visible to the app container(s) without any further
+configuration.
+
+## Quick Start
+
+```bash
+# 1. Build the sidecar image
+cd examples/k8s-sidecar
+./scripts/build.sh --tag fips-k8s-sidecar:latest
+
+# 2. Push to your registry (replace with your own)
+docker tag fips-k8s-sidecar:latest registry.example.com/fips-k8s-sidecar:latest
+docker push registry.example.com/fips-k8s-sidecar:latest
+
+# 3. Create the identity secret
+kubectl create secret generic fips-identity \
+  --from-literal=nsec=$(openssl rand -hex 32)
+
+# 4. Deploy
+kubectl apply -f pod.yaml
+```
+
+## How It Works
+
+In Kubernetes, all containers in a Pod share the same network namespace. The
+fips sidecar:
+
+1. Generates `/etc/fips/fips.yaml` from environment variables.
+2. Rewrites `/etc/resolv.conf` to route `.fips` DNS through dnsmasq (which
+   forwards `.fips` names to the FIPS daemon's built-in DNS resolver and
+   everything else to the original cluster DNS).
+3. Applies `iptables` isolation rules so that the pod's physical interface
+   (`eth0`) only carries FIPS transport traffic.
+4. Starts dnsmasq and then `exec`s the FIPS daemon.
+
+The app container starts concurrently and immediately sees `lo`, `eth0`, and
+`fips0`. DNS for `<npub>.fips` names resolves to `fd::/8` addresses via the
+dnsmasq → FIPS daemon pipeline.
+
+```text
+┌────────────────────────────────────────────────────┐
+│ Kubernetes Pod (shared network namespace)           │
+│                                                     │
+│ ┌─────────────────┐   ┌───────────────────────────┐│
+│ │  fips (sidecar) │   │  app container(s)         ││
+│ │                 │   │                           ││
+│ │  fips daemon    │   │  your workload            ││
+│ │  fipsctl        │   │  sees: lo, eth0, fips0    ││
+│ │  dnsmasq        │   │                           ││
+│ └─────────────────┘   └───────────────────────────┘│
+│                                                     │
+│ Interfaces:                                         │
+│   lo    — loopback (unrestricted)                   │
+│   eth0  — pod CNI interface (iptables: FIPS only)   │
+│   fips0 — FIPS TUN (fd00::/8, unrestricted)           │
+└────────────────────────────────────────────────────┘
+```
+
+## Network Isolation
+
+`FIPS_ISOLATE` controls whether the sidecar locks down `eth0`:
+
+| Value | Behaviour |
+|---|---|
+| `false` **(default)** | `fips0` is added alongside normal cluster networking. `eth0` continues to work — services, DNS, other pods, and the internet are all reachable as normal. Application traffic to FIPS mesh peers uses `fips0`. |
+| `true` | All traffic on `eth0` is dropped except FIPS transport (UDP 2121). The app container can **only** communicate via `fips0`. Use this for deployments where the workload must never bypass the mesh. |
+
+> **Common gotcha**: if you deploy the sidecar and find that services or
+> other pods are suddenly unreachable, check that `FIPS_ISOLATE` is not
+> set to `true`. The mesh-only mode is intentionally strict and will break
+> normal cluster connectivity.
+
+## Requirements
+
+| Requirement | Notes |
+|---|---|
+| `NET_ADMIN` capability | Required for TUN creation and iptables |
+| `/dev/net/tun` | HostPath volume mount (see `pod.yaml`) |
+| Linux kernel ≥ 4.9 | Standard on all current distributions |
+| No gVisor / kata-containers | Requires real kernel TUN support |
+
+## Environment Variables
+
+| Variable | Default | Description |
+|---|---|---|
+| `FIPS_NSEC` | *(required)* | Node secret key (hex or nsec1 bech32) |
+| `FIPS_PEER_NPUB` | *(empty)* | Single peer npub |
+| `FIPS_PEER_ADDR` | *(empty)* | Single peer transport address (`host:port`) |
+| `FIPS_PEER_ALIAS` | `peer` | Single peer alias |
+| `FIPS_PEER_TRANSPORT` | `udp` | Single peer transport type |
+| `FIPS_PEERS_JSON` | *(empty)* | JSON array of peer objects (overrides single-peer vars). **Note**: the included parser handles simple host:port addresses only; IPv6 addresses with brackets may not parse correctly. |
+| `FIPS_UDP_BIND` | `0.0.0.0:2121` | UDP transport bind address |
+| `FIPS_UDP_PORT` | *(from `FIPS_UDP_BIND`)* | UDP port for iptables rules |
+| `FIPS_TCP_BIND` | *(disabled)* | TCP transport bind address |
+| `FIPS_TUN_NAME` | `fips0` | TUN interface name |
+| `FIPS_TUN_MTU` | `1280` | TUN MTU |
+| `FIPS_ISOLATE` | `false` | Apply iptables isolation (blocks all eth0 traffic except FIPS transport) |
+| `FIPS_POD_IFACE` | `eth0` | Pod physical interface name |
+| `FIPS_REWRITE_DNS` | `true` | Rewrite `/etc/resolv.conf` |
+| `RUST_LOG` | `info` | FIPS log level |
+
+### Multiple Peers
+
+Use `FIPS_PEERS_JSON` to configure more than one peer:
+
+```yaml
+- name: FIPS_PEERS_JSON
+  value: |
+    [
+      {"npub":"npub1abc...","alias":"gw1","addr":"203.0.113.10:2121","transport":"udp"},
+      {"npub":"npub1def...","alias":"gw2","addr":"198.51.100.5:2121","transport":"udp"}
+    ]
+```
+
+Each object supports the keys: `npub` (required), `addr` (required),
+`alias`, `transport`, `priority`.
+
+Alternatively, mount a hand-crafted `fips.yaml` and set `FIPS_NSEC` to
+anything (the mount takes precedence because the entrypoint writes to
+`/etc/fips/fips.yaml` which is then overridden by the volume):
+
+```yaml
+volumes:
+  - name: fips-config
+    configMap:
+      name: my-fips-config
+containers:
+  - name: fips
+    volumeMounts:
+      - name: fips-config
+        mountPath: /etc/fips/fips.yaml
+        subPath: fips.yaml
+```
+
+### JSON Parsing Limitations
+
+`FIPS_PEERS_JSON` supports parsing with a simple awk-based parser. It handles common `host:port` addresses but does not support IPv6 addresses with brackets (e.g., `[::1]:2121`) or values containing embedded commas/colons. For complex configs, install `jq` in the container and modify the entrypoint, or use a mounted `fips.yaml`.
+
+## Files
+
+| File | Purpose |
+|---|---|
+| `Dockerfile` | Builds the sidecar image |
+| `entrypoint.sh` | Generates config, rewrites DNS, applies iptables, starts fips |
+| `pod.yaml` | Example Pod manifest (single app container + fips sidecar) |
+| `scripts/build.sh` | Compiles FIPS and builds the Docker image |
+
+## Troubleshooting
+
+**`FIPS_NSEC is required`** — The secret was not injected. Verify the
+`secretKeyRef` name and key match the secret you created with `kubectl
+create secret`.
+
+**`fips0` not visible in app container** — Check that the fips sidecar
+started successfully: `kubectl logs <pod> -c fips`. The sidecar must start
+and run the daemon before `fips0` appears. Add a startup probe or
+`postStart` lifecycle hook to your app container if you need to wait.
+
+**iptables errors** — The `NET_ADMIN` capability and `/dev/net/tun` volume
+mount are both required. Verify they are present in the Pod spec.
+
+**`.fips` DNS not resolving** — Check that `FIPS_REWRITE_DNS=true` and that
+dnsmasq started: `kubectl exec <pod> -c fips -- pgrep dnsmasq`. Verify the
+FIPS DNS listener: `kubectl exec <pod> -c fips -- fipsctl show status`.
+
+**CNI uses a different interface name** — Set `FIPS_POD_IFACE` to match your
+CNI's interface name (e.g. `ens3`, `net1` for Multus secondary interfaces).
+
+**gVisor / kata-containers** — These sandboxed runtimes intercept syscalls
+and do not support `AF_PACKET` or `/dev/net/tun` in the same way as a
+standard kernel. Use a standard RuntimeClass for FIPS sidecar pods.