Flaky ingress behavior using ingress-nginx and rootless podman #3451
Description
What happened:
I'm trying to run the basic Ingress example with Ingress Nginx from here. I'm using rootless podman.
Once I create the example services and ingress, I try to do curl localhost:12345/foo/hostname. Note that there's a random port 12345, due to the fact that I pass 0 for the hostPort in the Kind cluster config (see below).
About 10% of the time it works and I get the desired response foo-app. The rest of the time, the curl command hangs indefinitely.
When I look in the nginx controller logs, I see a lot of messages like the following:
2023/12/13 13:51:36 [alert] 353#353: pthread_create() failed (11: Resource temporarily unavailable)
2023/12/13 13:51:36 [alert] 39#39: fork() failed while spawning "cache loader process" (11: Resource temporarily unavailable)
2023/12/13 13:51:36 [alert] 39#39: sendmsg() failed (9: Bad file descriptor)
2023/12/13 13:51:37 [alert] 39#39: worker process 52 exited with fatal code 2 and cannot be respawned
This looks to me like nginx is spawning a bunch of worker threads, and most of them are failing to create properly. Maybe there's some problem with rootless podman?
What you expected to happen:
HTTP requests to the ingress should work reliably.
How to reproduce it (as minimally and precisely as possible):
Just following the Ingress instructions. My exact Kind config is as follows:
Kind config file
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
extraMounts:
- hostPath: /nix/store/mrcy594mjgm5zcckr1f4i901isxiwj0s-binary-cache
containerPath: /binary-cache
readOnly: false
propagation: HostToContainer
kubeadmConfigPatches:
- |
kind: InitConfiguration
nodeRegistration:
kubeletExtraArgs:
node-labels: "ingress-ready=true"
authorization-mode: "AlwaysAllow"
streaming-connection-idle-timeout: "0"
extraPortMappings:
- containerPort: 80
hostPort: 0
Environment:
- kind version: (use
kind version):0.20.0 - Runtime info: (use
docker infoorpodman info):
podman info output
host:
arch: amd64
buildahVersion: 1.32.0
cgroupControllers:
- cpuset
- cpu
- io
- memory
- pids
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: Unknown
path: /nix/store/3bmd0vmvvvrashaxqb1d1apyy7smix3d-conmon-2.1.8/bin/conmon
version: 'conmon version 2.1.8, commit: '
cpuUtilization:
idlePercent: 95.22
systemPercent: 0.75
userPercent: 4.02
cpus: 32
databaseBackend: boltdb
distribution:
codename: stoat
distribution: nixos
version: "23.05"
eventLogger: journald
freeLocks: 2044
hostname: desktop2
idMappings:
gidmap:
- container_id: 0
host_id: 100
size: 1
- container_id: 1
host_id: 3000000
size: 2000000
uidmap:
- container_id: 0
host_id: 1001
size: 1
- container_id: 1
host_id: 3000000
size: 2000000
kernel: 6.1.60
linkmode: dynamic
logDriver: journald
memFree: 6684782592
memTotal: 67134550016
networkBackend: netavark
networkBackendInfo:
backend: netavark
dns:
package: Unknown
path: /nix/store/igpk3cb4dmrr1mpvx5kb5prd1fk8kcss-podman-4.7.2/libexec/podman/aardvark-dns
version: aardvark-dns 1.9.0
package: Unknown
path: /nix/store/igpk3cb4dmrr1mpvx5kb5prd1fk8kcss-podman-4.7.2/libexec/podman/netavark
version: netavark 1.7.0
ociRuntime:
name: crun
package: Unknown
path: /nix/store/hllgilr2bhc6rbdrsbnrpaxyfqlzgqjg-crun-1.12/bin/crun
version: |-
crun version 1.12
commit: 1.12
rundir: /run/user/1001/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
os: linux
pasta:
executable: ""
package: ""
version: ""
remoteSocket:
exists: true
path: /run/user/1001/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
seccompProfilePath: ""
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: /nix/store/igpk3cb4dmrr1mpvx5kb5prd1fk8kcss-podman-4.7.2/libexec/podman/slirp4netns
package: Unknown
version: |-
slirp4netns version 1.2.2
commit: 0ee2d87523e906518d34a6b423271e4826f71faf
libslirp: 4.7.0
SLIRP_CONFIG_VERSION_MAX: 4
libseccomp: 2.5.4
swapFree: 3188670464
swapTotal: 9448923136
uptime: 10h 29m 50.00s (Approximately 0.42 days)
plugins:
authorization: null
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
- ipvlan
volume:
- local
registries:
search:
- docker.io
- quay.io
store:
configFile: /home/tom/.config/containers/storage.conf
containerStore:
number: 1
paused: 0
running: 1
stopped: 0
graphDriverName: overlay
graphOptions: {}
graphRoot: /home/tom/.local/share/containers/storage
graphRootAllocated: 1958014603264
graphRootUsed: 1742862733312
graphStatus:
Backing Filesystem: extfs
Native Overlay Diff: "true"
Supports d_type: "true"
Supports shifting: "false"
Supports volatile: "true"
Using metacopy: "false"
imageCopyTmpDir: /var/tmp
imageStore:
number: 1
runRoot: /run/user/1001/containers
transientStore: false
volumePath: /home/tom/.local/share/containers/storage/volumes
version:
APIVersion: 4.7.2
Built: 315532800
BuiltTime: Mon Dec 31 16:00:00 1979
GitCommit: ""
GoVersion: go1.21.4
Os: linux
OsArch: linux/amd64
Version: 4.7.2
- OS (e.g. from
/etc/os-release):NixOS 23.05 - Kubernetes version: (use
kubectl version):1.27.1 - Any proxies or other special environment settings?: No