Skip to content

net/http: TLS handshake timeout is only 10s and not use probe timeout value #751

Open
@nurzhan86

Description

@nurzhan86

Host operating system: output of uname -a

Linux localhost 3.10.0-1160.11.1.el7.x86_64 #1 SMP Fri Dec 18 16:34:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

blackbox_exporter version: output of blackbox_exporter --version

blackbox_exporter, version 0.18.0 (branch: HEAD, revision: 60c86e6ce5a1111f7958b06ae7a08222bb6ec839)
  build user:       root@53d72328d93f
  build date:       20201012-09:46:31
  go version:       go1.15.2

What is the blackbox.yml module config.

modules:
  http_2xx:
    prober: http
    http:
      valid_status_codes: [301, 302, 200, 401, 403, 415]
      no_follow_redirects: true
      preferred_ip_protocol: ip4
      tls_config:
        insecure_skip_verify: true
  http_post_2xx:
    prober: http
    http:
      method: POST
  tcp_connect:
    prober: tcp
    timeout: 15s
  pop3s_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^+OK"
      tls: true
      tls_config:
        insecure_skip_verify: false
  ssh_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^SSH-2.0-"
  irc_banner:
    prober: tcp
    tcp:
      query_response:
      - send: "NICK prober"
      - send: "USER prober prober prober :prober"
      - expect: "PING :([^ ]+)"
        send: "PONG ${1}"
      - expect: "^:[^ ]+ 001"
  icmp:
    prober: icmp
    timeout: 20s
    icmp:
      preferred_ip_protocol: ip4

What is the prometheus.yml scrape config.

global:
  scrape_interval:     1m # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 1m # Evaluate rules every 15 seconds. The default is every 1 minute.
  scrape_timeout: 15s

scrape_configs:
  - job_name: 'service_check'
    metrics_path: /probe
    scrape_interval: 1m
    scrape_timeout: 30s
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - https://mycompany.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115
...

What logging output did you get from adding &debug=true to the probe URL?

probe_dns_lookup_time_seconds 0.000888445
probe_duration_seconds 10.036707606
probe_failed_due_to_regex 0
probe_http_content_length 0
probe_http_duration_seconds{phase="connect"} 0
probe_http_duration_seconds{phase="processing"} 0
probe_http_duration_seconds{phase="resolve"} 0.000888445
probe_http_duration_seconds{phase="tls"} 0
probe_http_duration_seconds{phase="transfer"} 0
probe_http_redirects 0
probe_http_ssl 0
probe_http_status_code 0
probe_http_uncompressed_body_length 0
probe_http_version 0
probe_ip_addr_hash 3.931567538e+09
probe_ip_protocol 4
probe_success 0

What did you do that produced an error?

curl http://localhost:9115/probe\?module\=http_2xx\&target\=https%3A%2F%2Fmycompany.com\&debug%3Dtrue

What did you expect to see?

probe_success 1 or 120s timeout

What did you see instead?

net/http: TLS handshake timeout is only 10s, instead of 120s global probe timeout

Logs for the probe:

ts=2021-02-23T06:19:22.251254881Z caller=main.go:304 module=http_2xx target="https://mycompany.com/?debug=true" level=info msg="Beginning probe" probe=http timeout_seconds=119.5
ts=2021-02-23T06:19:22.251369966Z caller=http.go:342 module=http_2xx target="https://mycompany.com/?debug=true" level=info msg="Resolving target address" ip_protocol=ip4
ts=2021-02-23T06:19:22.252372922Z caller=http.go:342 module=http_2xx target="https://mycompany.com/?debug=true" level=info msg="Resolved target address" ip=aaa.bbb.ccc.ddd
ts=2021-02-23T06:19:22.252452689Z caller=client.go:252 module=http_2xx target="https://mycompany.com/?debug=true" level=info msg="Making HTTP request" url="https://aaa.bbb.ccc.ddd/?debug=true" host=mycompany.com
ts=2021-02-23T06:19:32.288414104Z caller=main.go:119 module=http_2xx target="https://mycompany.com/?debug=true" level=error msg="Error for HTTP request" err="Get \"https://aaa.bbb.ccc.ddd/?debug=true\": net/http: TLS handshake timeout"
ts=2021-02-23T06:19:32.28846505Z caller=main.go:119 module=http_2xx target="https://mycompany.com?debug=true" level=info msg="Response timings for roundtrip" roundtrip=0 start=2021-02-23T06:19:22.25252433Z dnsDone=2021-02-23T06:19:22.25252433Z connectDone=2021-02-23T06:19:22.288073452Z gotConn=0001-01-01T00:00:00Z responseStart=0001-01-01T00:00:00Z end=0001-01-01T00:00:00Z
ts=2021-02-23T06:19:32.288508646Z caller=main.go:304 module=http_2xx target="https://mycompany.com/?debug=true" level=error msg="Probe failed" duration_seconds=10.03720212

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions