Node.js DNS Resolution Failure in Containerized Environments

[sukkola]

Bug Report: Node.js DNS Resolution Failure in Containerized Environments

Summary

Wiki.js containers fail to make HTTPS connections to external services (OAuth2 providers, storage backends, etc.) in certain Kubernetes/containerized environments, despite DNS resolution working at the shell level.

Affected Features:

• OAuth2/OIDC authentication callbacks
• Storage backups to external services
• Any outbound HTTPS connections to external hostnames

Symptoms

From within the Wiki.js container, the following behavior is observed:

# DNS resolution works                                                                                                                                                                                                                   
$ nslookup example.com                                                                                                                                                                                                                   
Server:  10.43.0.10                                                                                                                                                                                                                      
Address: 10.43.0.10#53                                                                                                                                                                                                                   
                                                                                                                                                                                                                                         
Name:    example.com                                                                                                                                                                                                                     
Address: 192.168.1.100                                                                                                                                                                                                                   
                                                                                                                                                                                                                                         
$ node -e "require('dns').resolve4('example.com', (err, res) => console.log(err || 'OK: ' + res[0]))"                                                                                                                                    
OK: 192.168.1.100                                                                                                                                                                                                                        
                                                                                                                                                                                                                                         
# But HTTPS connections fail                                                                                                                                                                                                             
$ node -e "require('https').get('https://example.com/api', {rejectUnauthorized: false}, (res) => console.log('OK: ' + res.statusCode)).on('error', (e) => console.log('Error: ' + e.message))"                                           
Error: getaddrinfo ENOTFOUND example.com                                                                                                                                                                                                 
                                                                                                                                                                                                                                         
# And curl works fine                                                                                                                                                                                                                    
$ curl -s -o /dev/null -w "%{http_code}" https://example.com/api                                                                                                                                                                         
200                                                                                                                                                                                                                                      

Root Cause Analysis

The Problem

Node.js provides two different DNS resolution mechanisms:

1. dns.resolve4() / dns.resolve()

   • Uses Node.js's built-in pure DNS resolver
   • Makes raw DNS queries
   • Works reliably in containerized environments

2. dns.lookup() / HTTPS module

   • Uses the system's getaddrinfo() function
   • Respects system DNS configuration in /etc/resolv.conf
   • Fails in certain Kubernetes/container configurations

The HTTPS module uses dns.lookup() internally, which fails while direct DNS queries work.

Why It Happens

The system's getaddrinfo() can fail in containerized environments due to:

1. Incorrect /etc/resolv.conf configuration

   • May have incorrect nameservers
   • May have invalid search domains that cause excessive queries
   • May have sub-optimal timeout settings
   • May have been generated incorrectly by the container runtime

2. DNS query behavior differences

   • dns.resolve4() - Direct DNS queries, bypasses system resolver
   • dns.lookup() - Uses system resolver with search domain logic
   • Search domains can cause multiple failed queries before finding the real nameserver

3. Timeout and retry issues

   • Default timeout settings may be too high (5+ seconds)
   • Multiple retries on search domain failures cause hanging connections
   • Node.js may cache DNS failures

Testing Instructions

To reproduce and verify the issue:

Test 1: Check if dns.lookup() fails

kubectl exec -it -n wiki <pod-name> -- node << 'EOF'                                                                                                                                                                                     
const dns = require('dns');                                                                                                                                                                                                              
                                                                                                                                                                                                                                         
console.log('Test 1: dns.resolve4() - should work');                                                                                                                                                                                     
dns.resolve4('example.com', (err, addrs) => {                                                                                                                                                                                            
  console.log(err ? 'FAIL: ' + err.message : 'OK: ' + addrs[0]);                                                                                                                                                                         
                                                                                                                                                                                                                                         
  console.log('\nTest 2: dns.lookup() - may fail');                                                                                                                                                                                      
  dns.lookup('example.com', { family: 4 }, (err, addr) => {                                                                                                                                                                              
    console.log(err ? 'FAIL: ' + err.message : 'OK: ' + addr);                                                                                                                                                                           
  });                                                                                                                                                                                                                                    
});                                                                                                                                                                                                                                      
EOF                                                                                                                                                                                                                                      

Expected output showing the bug:

Test 1: dns.resolve4() - should work                                                                                                                                                                                                     
OK: 192.168.1.100                                                                                                                                                                                                                        
                                                                                                                                                                                                                                         
Test 2: dns.lookup() - may fail                                                                                                                                                                                                          
FAIL: getaddrinfo ENOTFOUND example.com                                                                                                                                                                                                  

Test 2: Check HTTPS connection

kubectl exec -it -n wiki <pod-name> -- node << 'EOF'                                                                                                                                                                                     
const https = require('https');                                                                                                                                                                                                          
                                                                                                                                                                                                                                         
https.get('https://example.com/api', { rejectUnauthorized: false, timeout: 5000 },                                                                                                                                                       
  (res) => {                                                                                                                                                                                                                             
    console.log('OK: Status ' + res.statusCode);                                                                                                                                                                                         
    res.destroy();                                                                                                                                                                                                                       
  }).on('error', (e) => {                                                                                                                                                                                                                
    console.log('Error: ' + e.message);                                                                                                                                                                                                  
  });                                                                                                                                                                                                                                    
EOF                                                                                                                                                                                                                                      

Expected output showing the bug:

Error: getaddrinfo ENOTFOUND example.com                                                                                                                                                                                                 

Test 3: Verify curl works

kubectl exec -it -n wiki <pod-name> -- curl -v https://example.com/api                                                                                                                                                                   

Expected output showing curl works:

* Connected to example.com (192.168.1.100) port 443                                                                                                                                                                                      
* ...                                                                                                                                                                                                                                    
< HTTP/1.1 200 OK                                                                                                                                                                                                                        

Test 4: Inspect system DNS configuration

kubectl exec -it -n wiki <pod-name> -- cat /etc/resolv.conf                                                                                                                                                                              

Example output:

nameserver 10.43.0.10                                                                                                                                                                                                                    
search default.svc.cluster.local svc.cluster.local cluster.local                                                                                                                                                                         
options ndots:5 timeout:5 attempts:5                                                                                                                                                                                                     

If you see issues like:

• ndots:5 or higher (causes excessive search domain lookups)
• timeout:5 or higher (slow DNS queries)
• Invalid or too many search domains
• Missing or incorrect nameservers

Then you've identified the DNS configuration problem.

Proposed Solution

For Helm Chart Maintainers

Add optional DNS configuration support to the Helm chart that allows users to customize DNS behavior without imposing it on all deployments:

File: dev/helm/values.yaml

Add a new optional section:

# Pod DNS configuration for containerized environments                                                                                                                                                                                   
# Only applied if explicitly enabled                                                                                                                                                                                                     
# See: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-dns-config                                                                                                                                           
podDnsConfig: {}                                                                                                                                                                                                                         
  # Uncomment to enable custom DNS configuration                                                                                                                                                                                         
  # policy: ClusterFirst                                                                                                                                                                                                                 
  # options:                                                                                                                                                                                                                             
  #   - name: ndots                                                                                                                                                                                                                      
  #     value: "1"                                                                                                                                                                                                                       
  #   - name: timeout                                                                                                                                                                                                                    
  #     value: "2"                                                                                                                                                                                                                       
  #   - name: attempts                                                                                                                                                                                                                   
  #     value: "2"                                                                                                                                                                                                                       
  #   - name: rotate                                                                                                                                                                                                                     
  # searches:                                                                                                                                                                                                                            
  #   - svc.cluster.local                                                                                                                                                                                                                
  #   - cluster.local                                                                                                                                                                                                                    
  # NOTE: Do NOT specify nameservers - let Kubernetes inject them automatically                                                                                                                                                          
  # This ensures compatibility with all cluster configurations                                                                                                                                                                           

File: dev/helm/templates/deployment.yaml

Add DNS configuration to the pod spec if provided:

spec:                                                                                                                                                                                                                                    
  template:                                                                                                                                                                                                                              
    spec:                                                                                                                                                                                                                                
      {{- with .Values.podDnsConfig }}                                                                                                                                                                                                   
      dnsPolicy: {{ .policy | default "ClusterFirst" }}                                                                                                                                                                                  
      dnsConfig:                                                                                                                                                                                                                         
        {{- with .options }}                                                                                                                                                                                                             
        options:                                                                                                                                                                                                                         
          {{- toYaml . | nindent 10 }}                                                                                                                                                                                                   
        {{- end }}                                                                                                                                                                                                                       
        {{- with .searches }}                                                                                                                                                                                                            
        searches:                                                                                                                                                                                                                        
          {{- toYaml . | nindent 10 }}                                                                                                                                                                                                   
        {{- end }}                                                                                                                                                                                                                       
      {{- end }}                                                                                                                                                                                                                         
      containers:                                                                                                                                                                                                                        
        # ... rest of template                                                                                                                                                                                                           

For Users Experiencing the Issue

In your Helm values or Kustomize patch, enable DNS configuration:

podDnsConfig:                                                                                                                                                                                                                            
  policy: ClusterFirst                                                                                                                                                                                                                   
  options:                                                                                                                                                                                                                               
    - name: ndots                                                                                                                                                                                                                        
      value: "1"                                                                                                                                                                                                                         
    - name: timeout                                                                                                                                                                                                                      
      value: "2"                                                                                                                                                                                                                         
    - name: attempts                                                                                                                                                                                                                     
      value: "2"                                                                                                                                                                                                                         
    - name: rotate                                                                                                                                                                                                                       
  searches:                                                                                                                                                                                                                              
    - svc.cluster.local                                                                                                                                                                                                                  
    - cluster.local                                                                                                                                                                                                                      

For Kubernetes Users: Generic Workaround

If your Kubernetes cluster has this issue, apply a generic DNS configuration patch to Wiki.js deployments:

kubectl patch deployment wiki -n wiki --type=json -p='[                                                                                                                                                                                  
  {                                                                                                                                                                                                                                      
    "op": "add",                                                                                                                                                                                                                         
    "path": "/spec/template/spec/dnsPolicy",                                                                                                                                                                                             
    "value": "ClusterFirst"                                                                                                                                                                                                              
  },                                                                                                                                                                                                                                     
  {                                                                                                                                                                                                                                      
    "op": "add",                                                                                                                                                                                                                         
    "path": "/spec/template/spec/dnsConfig",                                                                                                                                                                                             
    "value": {                                                                                                                                                                                                                           
      "options": [                                                                                                                                                                                                                       
        {"name": "ndots", "value": "1"},                                                                                                                                                                                                 
        {"name": "timeout", "value": "2"},                                                                                                                                                                                               
        {"name": "attempts", "value": "2"},                                                                                                                                                                                              
        {"name": "rotate"}                                                                                                                                                                                                               
      ],                                                                                                                                                                                                                                 
      "searches": ["svc.cluster.local", "cluster.local"]                                                                                                                                                                                 
    }                                                                                                                                                                                                                                    
  }                                                                                                                                                                                                                                      
]'                                                                                                                                                                                                                                       

Root Cause Details

The issue stems from how Node.js's dns.lookup() interacts with the system resolver:

1. When https.get() is called, it uses dns.lookup() to resolve the hostname
2. dns.lookup() calls the system's getaddrinfo() function
3. getaddrinfo() uses /etc/resolv.conf to determine DNS behavior
4. If /etc/resolv.conf is misconfigured or has suboptimal settings, getaddrinfo() fails or hangs
5. Meanwhile, dns.resolve4() bypasses the system resolver and works fine
6. And curl (which uses different DNS libraries) also works fine

Why This is Hard to Debug

• The failure is silent - just "ENOTFOUND" with no indication of why
• It only affects HTTPS/HTTP clients, not shell commands
• DNS works (nslookup succeeds) but connections fail
• The root cause is the system resolver's behavior, not DNS itself
• Different container runtimes and Kubernetes distributions have different defaults

Impact

This bug affects:

1. OAuth2/OIDC Authentication - Users cannot log in via external OAuth2 providers
2. Storage Backups - Users cannot backup to external storage services
3. External API Integrations - Any feature requiring external HTTPS connections
4. Git Operations - Sideloading from external Git repositories may fail

Reproduction Environment

• Container Runtime: Kubernetes with various CNI plugins (Cilium, Weave, Flannel, etc.)
• Affected Versions: Wiki.js 2.x (affects any Node.js 14+)
• Kubernetes Versions: All versions (issue is container/DNS configuration, not Kubernetes-specific)
• Reproducibility: Consistent in affected environments, but not all environments affected