[sql_input] add TLS/SSL parameters

[mykola-elastic]

Proposed commit message

See title

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Closes https://github.com/elastic/obs-integration-team/issues/305
• Requires [metricbeat] [SQL] Add SSL support beats#44748

Screenshots

[ghost]

@mykola-elastic can we actually test the integration with SSL enabled by passing the certificates to validate that it works and we get data?

[mykola-elastic]

@gpop63 so far I tested only the metricbeat (The PR merged yesterday elastic/beats#44748). I am figuring out how to replace the metricbeat part of elastic-agent to test the integration against the version that has all that SSL code

[mykola-elastic]

Oh, the snapshot already has the SSL changes (metricbeat), I will proceed to testing

[ghost]

@mykola-elastic not sure if the elastic stack snapshot will have the changes yet 🤔 if it doesn't, in the past we have used this awesome tutorial by @zmoog zmoog/public-notes#35 to create custom agents

[mykola-elastic]

@gpop63 @ishleenk17 I have completed the testing. I tested with postgres and mysql, similar test cases to these

[ghost]

@mykola-elastic can you upload some screenshot from Discover to show the documents being ingested with SSL enabled? You could also post the agent policy to show that ssl options are being passed to metricbeat

[mykola-elastic]

@gpop63 The document looks the same as without SSL:

The SSl Configuration looks like this (for postgresql example):

I also checked that the following works for mysql driver:

verification_mode: certificate
certificate_authorities:
  - |
    -----BEGIN CERTIFICATE-----
    MIIDHzCCAgegAwIBAgIUFkOZhpmcIS2i+3A5+PEHz2RuGGYwDQYJKoZIhvcNAQEL
    BQAwFDESMBAGA1UEAwwJbG9jYWxob3N0MB4XDTI1MDYxODEzMTkzNloXDTM1MDQy
    NzEzMTkzNlowFDESMBAGA1UEAwwJbG9jYWxob3N0MIIBIjANBgkqhkiG9w0BAQEF
    AAOCAQ8AMIIBCgKCAQEA3ZBhDdV4onD6VkEtVr58nQyYLidOoTE/XE3pUg7mls+b
    Mj1dvxaeYfWtq8/DP4C1bbYU5PYjaVNGHZSLv+PVQpVfsTXzL5eeKafVK58dGKvY
    kDao/ohpJLLVLa7tClloGkhxHEkfAHI1Gq/dKtW+bBeRfgSMQSy4GTs8PWWhK0N3
    esELQhM2e3qSgEPYyI/aft7Jcxa6sf48xtUePL2+vDauuPUK7eSRkoJQ8O/U/9lL
    AJQufvaLgmeTmyCuRtauD1ADgePOIQC75GyPbtKDIN4tkRfEg/Topu6g8jxxi5Sl
    3oqpvRwsRp/2uZYdm9M3h9AhttGtHmL1MUmLj+5JPwIDAQABo2kwZzAdBgNVHQ4E
    FgQU+YweArEiYM0uxiXuQl6R5TLLQWwwHwYDVR0jBBgwFoAU+YweArEiYM0uxiXu
    Ql6R5TLLQWwwDwYDVR0TAQH/BAUwAwEB/zAUBgNVHREEDTALgglsb2NhbGhvc3Qw
    DQYJKoZIhvcNAQELBQADggEBAMZukZRqphRlJSUsxuVRG3RWyxEGGn4bPAA4mZ9P
    Nli0tyeCsP8nVRc4mv7NX/BXBZi2rFZBmGVAXl0/55M0a6bn9OrZwgwyzp4TgVXC
    ao/fSLLsPr59igDj+4HRkrFHXPH9fbdK6GWBZklkxXPg4pkY3cPYMVcD9Wn68xj1
    7wgY/oEkBMZgqsjvSzwOFojA5Nc/THuEiNMdyYtTfxyOh3+YFqtOHxsbVUrEDsMv
    47ztbXtQ2ZziS2sykXPWuo07kh8Zzq69tfSzPVJxZ0baMu9x3/GWORhWgRtC6beY
    D4SAdPOt86JsSsTtOYPeA+JrQdKesPm6l4AjDiHUDBWiCUo=
    -----END CERTIFICATE-----

Agent policy for the postgresql example:

id: elastic-agent-managed-ep
revision: 16
outputs:
  default:
    type: elasticsearch
    hosts:
      - https://elasticsearch:9200
    ssl.ca_trusted_fingerprint: 57E00CA616A85926934E861AC4731978FCA0E33BB519DCB8AEBD61CCF4F7F7AC
    preset: latency
fleet:
  hosts:
    - https://fleet-server:8220
output_permissions:
  default:
    _elastic_agent_monitoring:
      indices: []
    _elastic_agent_checks:
      cluster:
        - monitor
    8cd10de1-082a-4790-a7a3-005d31d35a22:
      indices:
        - names:
            - metrics-*-*
          privileges:
            - auto_configure
            - create_doc
agent:
  download:
    sourceURI: https://artifacts.elastic.co/downloads/
  monitoring:
    enabled: false
    logs: false
    metrics: false
    traces: false
  features: {}
  protection:
    enabled: false
    uninstall_token_hash: f1yie1SZaKhgfDdyfkZZ/VPqI3K3dliBUZqSI/Qx+88=
    signing_key: >-
      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEqmR5KY/Q4zp1cZ33MWUh6FI7BsSFkT410R2bdbnrSuYdtmOOKmNPCoP8/nBCN/Vvj/QnsvFqGOwgnjJAmE7F2Q==
inputs:
  - id: sql/metrics-sql-8cd10de1-082a-4790-a7a3-005d31d35a22
    name: sql-1
    revision: 14
    type: sql/metrics
    use_output: default
    meta:
      package:
        name: sql
        version: 0.8.0
    data_stream:
      namespace: default
    package_policy_id: 8cd10de1-082a-4790-a7a3-005d31d35a22
    streams:
      - id: sql/metrics-sql.sql-8cd10de1-082a-4790-a7a3-005d31d35a22
        data_stream:
          dataset: sql
          type: metrics
          elasticsearch:
            dynamic_dataset: true
            dynamic_namespace: true
        metricsets:
          - query
        hosts:
          - ${SECRET_0}
        driver: postgres
        sql_queries:
          - query: select * from pg_stat_database
            response_format: table
        raw_data.enabled: true
        period: 10s
        merge_results: false
        ssl:
          verification_mode: certificate
          certificate_authorities:
            - /usr/share/elastic-agent/ca.pem
signed:
  data: >-
    eyJpZCI6ImVsYXN0aWMtYWdlbnQtbWFuYWdlZC1lcCIsImFnZW50Ijp7ImZlYXR1cmVzIjp7fSwicHJvdGVjdGlvbiI6eyJlbmFibGVkIjpmYWxzZSwidW5pbnN0YWxsX3Rva2VuX2hhc2giOiJmMXlpZTFTWmFLaGdmRGR5ZmtaWi9WUHFJM0szZGxpQlVacVNJL1F4Kzg4PSIsInNpZ25pbmdfa2V5IjoiTUZrd0V3WUhLb1pJemowQ0FRWUlLb1pJemowREFRY0RRZ0FFcW1SNUtZL1E0enAxY1ozM01XVWg2Rkk3QnNTRmtUNDEwUjJiZGJuclN1WWR0bU9PS21OUENvUDgvbkJDTi9WdmovUW5zdkZxR093Z25qSkFtRTdGMlE9PSJ9fSwiaW5wdXRzIjpbeyJpZCI6InNxbC9tZXRyaWNzLXNxbC04Y2QxMGRlMS0wODJhLTQ3OTAtYTdhMy0wMDVkMzFkMzVhMjIiLCJuYW1lIjoic3FsLTEiLCJyZXZpc2lvbiI6MTQsInR5cGUiOiJzcWwvbWV0cmljcyJ9XX0=
  signature: >-
    MEQCIGQFY98Ld1GVxKGidFmKPF4pedOsCzFoapIp+v1wX7v6AiBAJ9lE7l0M86UcCYIaiC/NVPETMxbFP28FQCah4277/w==
secret_references:
  - id: M89Ap5cBF0w68bvgoYQ8
namespaces: []

[mykola-elastic]

@alaudazzi can you please review the change to the docs here? This package is going to be GA soon and this doc change is the last one before GA
CC @ishleenk17 @lalit-satapathy

[zmoog]

not sure if the elastic stack snapshot will have the changes yet 🤔 if it doesn't, in the past we have used this tutorial

Recently I found a simpler way to test an unreleased version of Beats with the Elastic Agent, I added a comment with raw notes.

I need to run some tests for the agent health status, I'll add a new public note later.

---

Update: here's a public note with a better process to build a custom agent: zmoog/public-notes#116

[alaudazzi]

I left a few editing suggestions, otherwise LGTM.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 3362fb6

History

• 💚 Build #27736 succeeded 3f46f65
• 💚 Build #27627 succeeded 9daf37c
• 💚 Build #27594 succeeded 9693e92
• 💚 Build #27408 succeeded 91a9f33
• 💚 Build #27396 succeeded a6b4e95
• 💚 Build #27339 succeeded 49a9635

cc @mykola-elastic

[elastic-vault-github-plugin-prod]

Package sql - 0.9.0 containing this change is available at https://epr.elastic.co/package/sql/0.9.0/