Skip to content

[Kafka] New custom input package - #2741

Merged
P1llus merged 10 commits into
elastic:mainfrom
P1llus:package_kafka_log
May 9, 2022
Merged

[Kafka] New custom input package#2741
P1llus merged 10 commits into
elastic:mainfrom
P1llus:package_kafka_log

Conversation

@P1llus

@P1llus P1llus commented Feb 23, 2022 •
edited
Loading

Copy link
Copy Markdown
Member

What does this PR do?

This adds custom Kafka input package, allowing users to ingest data from kafka using our input, with their own pipelines.

This has been built in the same format and principle as our other input packages.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
@elasticmachine

elasticmachine commented Feb 23, 2022

Copy link
Copy Markdown

Pinging @elastic/integrations (Team:Integrations)
@elasticmachine

elasticmachine commented Feb 23, 2022

Copy link
Copy Markdown

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)
@P1llus

P1llus commented Feb 23, 2022

Copy link
Copy Markdown
Member Author

Should work just fine, but I want to see if we can add some kafka system tests, looking for some proper docker images to do this first.
@elasticmachine

elasticmachine commented Feb 23, 2022 •
edited
Loading

Copy link
Copy Markdown

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-04-27T14:39:23.458+0000

  • Duration: 15 min 39 sec

Test stats 🧪

Test Results
Failed 0
Passed 3
Skipped 0
Total 3

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.
efd6
efd6 reviewed Feb 24, 2022
View reviewed changes
Comment thread packages/kafka_log/data_stream/generic/manifest.yml Outdated
Comment thread packages/kafka_log/data_stream/generic/manifest.yml Outdated
Comment thread packages/kafka_log/data_stream/generic/manifest.yml Outdated
@P1llus
P1llus requested a review from kaiyan-sheng March 1, 2022 13:29
kaiyan-sheng
kaiyan-sheng reviewed Mar 3, 2022
View reviewed changes

@kaiyan-sheng kaiyan-sheng left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall it looks good to me. I think we still need system tests here in order to release it as GA right?
About the owner, not sure if it's appropriate to change right now but I think eventually elastic/obs-service-integrations should own this.
@P1llus P1llus added enhancement New feature or request Integration:kafka_log Kafka Log Consumer labels Mar 7, 2022
@P1llus
P1llus requested a review from a team as a code owner March 24, 2022 12:24
@jsoriano

jsoriano commented Apr 27, 2022

Copy link
Copy Markdown
Contributor

About the owner, not sure if it's appropriate to change right now but I think eventually elastic/obs-service-integrations should own this.

+1 to this, I think we shouldn't have packages owned by integrations as this is becoming confusing, see #3202
@jsoriano
jsoriano requested review from a team and removed request for a team April 27, 2022 08:38
@elasticmachine

elasticmachine commented Apr 27, 2022 •
edited
Loading

Copy link
Copy Markdown

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (0/0) 💚
Files 100.0% (0/0) 💚 2.857
Classes 100.0% (0/0) 💚 2.857
Methods 100.0% (3/3) 💚 11.905
Lines 100.0% (0/0) 💚 11.162
Conditionals 100.0% (0/0) 💚
@lalit-satapathy

lalit-satapathy commented Apr 27, 2022

Copy link
Copy Markdown
Contributor

Just trying to clarify. This new package is developed by security team but not owned by them?
@P1llus

P1llus commented Apr 27, 2022

Copy link
Copy Markdown
Member Author

/test
@lalit-satapathy

lalit-satapathy commented May 5, 2022

Copy link
Copy Markdown
Contributor

@P1llus the changes looks good to me. The name "custom kafka logs" confused me, where I thought as the logs of kafka broker. Where as this is actually just referring to all the messages in "kafka topic".
@P1llus
P1llus merged commit 588e2b7 into elastic:main May 9, 2022
@andrewkroh andrewkroh added the New Integration Issue or pull request for creating a new integration package. label Aug 13, 2024
orestisfl pushed a commit to orestisfl/integrations that referenced this pull request May 15, 2026
@P1llus
[Kafka] New custom input package (elastic#2741)
b7b93f2 
* [Kafka] New package

* update changelog

* update text from PR comments

* stashing changes

* updating system tests

* update codeowners

* update codeowners

* Update owner in manifest.yml

* correct owners would be good
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Integration:kafka_log Kafka Log Consumer New Integration Issue or pull request for creating a new integration package. Team:Integrations Label for the Integrations team

7 participants

@P1llus @elasticmachine @jsoriano @lalit-satapathy @kaiyan-sheng @efd6 @andrewkroh