[vSphere] Add SSL/TLS support for tcp input in log data stream - #11061
Merged
6 commits merged intoSep 27, 2024
Merged
Conversation
andrewkroh
added
enhancement
🚀 Benchmarks reportTo see the full report comment with |
Author
|
Added SSL settings:
SSL config
enabled: true
certificate_authorities: |
-----BEGIN CERTIFICATE-----
MIIDdTCCAl2gAwIBAgIUBooX4rMwTOKEprV2sYsQKfx2DuAwDQYJKoZIhvcNAQEL
BQAwSjELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExFDASBgNVBAcM
C1NhbnRhIENsYXJhMRAwDgYDVQQDDAdmYWtlLUNBMB4XDTI0MDYxMzEzMjAzMFoX
DTM0MDQyMjEzMjAzMFowSjELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3Ju
aWExFDASBgNVBAcMC1NhbnRhIENsYXJhMRAwDgYDVQQDDAdmYWtlLUNBMIIBIjAN
BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAnWq6w/8fa/3ysjRICowV0VNQZn0v
Cw7rSBg7V84HyBxk0dPZwjDvYDjy2b7xqb8SX07HiaefxnxiSgdFch1cYGTZbiNH
OvTJfMd7kyjrPW/cAtKnI6v721S5F/qznbCTPNhIaTUyua9TGLzCtLGag4aLq+4R
0HO9BFehzgZdRKC+rtU0ndVarwzTLYT+ITp72CcWws12BtS7KJCPWRocKqJhskZg
bHlrnvLhfNpzAvWBj1hNTf+Y7fCuAMcXV8q//Nmx6GL4K+9AauMw5YzvRoJMLflN
VXXsREtppcVIeIsRc0gYMDI6CQtmf9XUA8V4j2Li1J6iXsPJdrbdJvG5ewIDAQAB
o1MwUTAdBgNVHQ4EFgQUroKnzGgzg1JWKbRwesqYEng5XDswHwYDVR0jBBgwFoAU
roKnzGgzg1JWKbRwesqYEng5XDswDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0B
AQsFAAOCAQEAN5DxIFG5QXllWftOuQN6EsTWBqVfayVOqDl53Yqd6ylU6PgKNLbF
vSFnEkun5zhLuBPzglw79qg8wX0W/xEXEBENmVDOW2QdWqO/mBniKPRzbg8bIJ9h
qF6e+m/RDQZ3zzcibKlrlVZivhVRJUvGJ3mmLMqJKUppSd4KuJfxUlVdxXze2ATY
ktaGFyYOlziyO6vFbs+j1amCilqDwE1oHjfVLCX5heu1W4e8e56nQ6dOGM/1PjWU
YMLBdqMH0mF2d6CN1HaWyKwmONHJ4r3vsY0fDAhguo9UvqB1Iz24261gnVv5wmkU
/8jy9AQpimJPX7+t0kyaLEvThRhX9dp5gA==
-----END CERTIFICATE-----
certificate: |
-----BEGIN CERTIFICATE-----
MIIDcTCCAlmgAwIBAgIBATANBgkqhkiG9w0BAQsFADBKMQswCQYDVQQGEwJVUzET
MBEGA1UECAwKQ2FsaWZvcm5pYTEUMBIGA1UEBwwLU2FudGEgQ2xhcmExEDAOBgNV
BAMMB2Zha2UtQ0EwHhcNMjQwNjEzMTMyMDMxWhcNMzQwNDIyMTMyMDMxWjBOMQsw
CQYDVQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTEUMBIGA1UEBwwLU2FudGEg
Q2xhcmExFDASBgNVBAMMC2Zha2Utc2VydmVyMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEA4QQTvAOesB2UCQDjSFwhwHCvwFAAAVbWXcRYJbygP03zhGI4
lCGdlcxUf2BZ4KId3WKh1UNu0hHj02PoWA6G7uFZwm0CaQ0pcJgjXbuBQU94PZEF
l1hgGWf9O1uwf2jt97+HmV66W4tzzoCnUpe+MCbeMk8fNcWN9NiQNQjXZgjWLC4Q
lmnSzCNzXXut8ozMuTzjoX/WSGcmavGKrIJLtdP6z3y/ONgaZmO+5HsO4lkLxivJ
5KumlTym2moPar6oxQDcDZt2SsZJeGD0oDFcy92OxK6l5UTVtgnj7ASadOGq/2AA
4/zL3XsUaPossjFQNaKfxoBEj27vfEABn3VzbQIDAQABo14wXDAaBgNVHREEEzAR
hwQAAAAAgglsb2NhbGhvc3QwHQYDVR0OBBYEFCOemoaxXKsjJRDb6+d76x50HMQH
MB8GA1UdIwQYMBaAFK6Cp8xoM4NSVim0cHrKmBJ4OVw7MA0GCSqGSIb3DQEBCwUA
A4IBAQAuPPlMB+Y4S1nrl7IyrsxPaFpqVECqtkS9iXKQtmaTns8oGHGn3iGR8q0J
b1gRN9MyBDLIaBms74O2KPm8+En3654FRL1cEyAQ2LAfFuxsJm7uaAGJB5zW23al
8nAg11vtZEypDGRgxqq5GrmGfxqQl6kEmvo+kpIg0wh3SY2p4/NyDoyNLItXyxyT
QGPAxI99SvARQ3UgmjlXZOn1MOcrulvSzYsWLwb3nS/SFigV4mu7hzlFWDO0yo7r
T7Z2Z6+OX/uXr0HMgG6trjNp44X/IVyHoiR0dL7ebMNwjP1IPFlLLkGZz6dH2RB8
YvqvSk/zFVwPmkNrhxjI2mOHlEV1
-----END CERTIFICATE-----
key: |
-----BEGIN PRIVATE KEY-----
MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQDhBBO8A56wHZQJ
AONIXCHAcK/AUAABVtZdxFglvKA/TfOEYjiUIZ2VzFR/YFngoh3dYqHVQ27SEePT
Y+hYDobu4VnCbQJpDSlwmCNdu4FBT3g9kQWXWGAZZ/07W7B/aO33v4eZXrpbi3PO
gKdSl74wJt4yTx81xY302JA1CNdmCNYsLhCWadLMI3Nde63yjMy5POOhf9ZIZyZq
8Yqsgku10/rPfL842BpmY77kew7iWQvGK8nkq6aVPKbaag9qvqjFANwNm3ZKxkl4
YPSgMVzL3Y7ErqXlRNW2CePsBJp04ar/YADj/MvdexRo+iyyMVA1op/GgESPbu98
QAGfdXNtAgMBAAECggEAAy0hjqDC5c2Io94eOsWz8Zl682mRPZALIbnwB0TJQr8P
K2MBVXQxW0Zqcu8vsgy7WWnJA+waNJgJ1UJLHKo2DdfbnvOlMxCmDcUNWPknD36C
IY/ogWhpu8BMs+tbMq2rZWOsTMm2pJ5lUop5fNkzgdy40VgYlHdKjQhArVDAooaB
vR8hma7gMRlEVpfQIT1jO+MsTk9iBVim0zYdYvGJGAnExji++MCkefgz1xcJxpBL
8XySPtA69jeIk4pFNmrkoR2jREa3uDejPTByN/0H+JT3cfukqVRu9aXt43+TAYZ0
ew8xXYN6eJwo2XCSC9bi7fp5FUAu2McBTCa9bpvAwQKBgQD4YIbmkcT0ZRuZyeXM
tsr22hiK2VERorbWc127k/hOsuuzjJ+BMNvs0Bj7fER51zfPu586G/9NHNFoIyLt
6ZPZkAYs2ShvwV2F5eF8Ck0BA76y9yV+Eca7Ossx9uvEiyyhmmCgGM2QLSSAb32u
iK+bkK/Q19M5seh76lQPrSvYBQKBgQDn7AENoORNxn8O7j9XkHHyTE3swcAcMpHy
x6jMKKQsqg49yzTZQhSufm3OXUUpqXUIoW+sDpVZ9ikK00T0bGoax+BSEKoAnt0z
I8XMtlKRNEZ0C6atzGzd012xgDjrQS2QQzRkOfKqak5tCWOrH7eL0aKx+8r2jXB6
MuckD1qSSQKBgAjM0rrEm2fVVT8DqkKwj0xjuc7hmD+5tiSu/bkxqvCO4k8AQn5/
Tqt7TAdNBdgTSgTwLtoON+Nxi/HWnNw1FGn28mEQaKBdB4VA+dY9PzE0AsOwggGP
e60SjjR8/Sl/jNtE7vt9JXtgPhTW3msaqK4oWDJcEz3OEeCiEsxjZFRJAoGADnha
JHiKZhPhb5QL3bHEekKqhgActWWuV4B/lqVegO7nTpG0LW4DDlQTE9Ib85qAE9bG
uex7Aw7pxqQh5EIzaJJZ+6TQhpNi58Lun3QsuKSTVMA5WRStIjYC3YVLh9ocNqkK
xsoayukxFXqFfnLPJ+kfho1YP4PZpbYxV72hgAkCgYAeZ5Ssh1yOSUyYBaALq3Mi
01J4kw4AwAbc77S7zPjKisPUDd2cIA5EOaEyTmzMGBxxHSldgDmSpPWu3SJK+mmJ
BYCEJWzEqD5KSRBJy8PYoY25RPgjGG9FWmTcZ6OkoeTY8aTtsR8w1yGeH/OnqAki
QDM0wzIhRG/hLQRCi1C9MQ==
-----END PRIVATE KEY-----I tried to use this utility for streaming https://github.com/elastic/stream but I don't think it supports TLS or maybe I missed it. Tested using openssl output
Update: I created a simple python script with TLS enabled that sends random logs to see if I receive them and it seems to be working.
Python script
import socket
import ssl
import time
import random
from datetime import datetime
HOST = 'localhost'
PORT = 9525
CA_CERT = '/home/gab/work/docker-compose/certs2/root-ca.pem'
CLIENT_CERT = '/home/gab/work/docker-compose/certs2/server-cert.pem'
CLIENT_KEY = '/home/gab/work/docker-compose/certs2/server-key.pem'
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
context = ssl.create_default_context(ssl.Purpose.SERVER_AUTH, cafile=CA_CERT)
context.load_cert_chain(certfile=CLIENT_CERT, keyfile=CLIENT_KEY)
conn = context.wrap_socket(sock, server_hostname=HOST)
conn.connect((HOST, PORT))
print(f"Connected to {HOST}")
log_levels = ['info', 'warn', 'error', 'debug']
def generate_log_message():
current_time = datetime.now().isoformat()
log_level = random.choice(log_levels)
event_id = random.randint(100000000, 999999999)
user_id = random.randint(1000, 9999)
log_message = f'<14>1 {current_time} host123 vpxd 24962 - - Event [{event_id}] [1-1] [{current_time}] [vim.event.UserLoginSessionEvent] [{log_level}] [root] [TEST] [{event_id}] [User root@127.0.0.1 logged in as VMware-client/6.5.0]\n'
return log_message.encode('utf-8')
try:
while True:
log_message = generate_log_message()
conn.sendall(log_message)
print("Log message sent successfully")
time.sleep(1)
except KeyboardInterrupt:
print("Interrupted by user")
conn.close() |
ghost
marked this pull request as ready for review
September 10, 2024 14:59
ghost
self-requested a review
as a code owner
September 10, 2024 14:59
andrewkroh
added
the
Team:Obs-InfraObs
ghost
self-assigned this
ishleenk17
reviewed
Sep 11, 2024
| Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details. | ||
|
|
||
| - name: ssl | ||
| type: yaml |
Member
There was a problem hiding this comment.
All the different config params will be taken care by this single SSL yaml block.
Correct?
Author
There was a problem hiding this comment.
Yes, all available SSL options can be added in that yaml block.
ishleenk17
approved these changes
Sep 27, 2024
ishleenk17
left a comment
ishleenk17
left a comment
Member
There was a problem hiding this comment.
Resolve conflicts, we are GTG!
💚 Build Succeeded
History
cc @gpop63 |
|
Contributor
|
Package vsphere - 1.16.0 containing this change is available at https://epr.elastic.co/search?package=vsphere |
harnish-crest-data
pushed a commit
to chavdaharnish/integrations
that referenced
this pull request
Feb 4, 2025
…lastic#11061) * add tls support for tcp input * bump package version * add ssl prefix * fix PR id * bump package version
harnish-crest-data
pushed a commit
to chavdaharnish/integrations
that referenced
this pull request
Feb 5, 2025
…lastic#11061) * add tls support for tcp input * bump package version * add ssl prefix * fix PR id * bump package version
2 tasks
This pull request was closed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Adds SSL/TLS support for
tcpinput inlogdata stream.Checklist
changelog.ymlfile.Author's Checklist
How to test this PR locally
Related issues
Screenshots