Skip to content

Remove the Arcsight module and the modules framework#16794

Merged
kaisecheng merged 7 commits intoelastic:mainfrom
kaisecheng:remove_all_modules
Dec 19, 2024
Merged

Remove the Arcsight module and the modules framework#16794
kaisecheng merged 7 commits intoelastic:mainfrom
kaisecheng:remove_all_modules

Conversation

@kaisecheng
Copy link
Contributor

@kaisecheng kaisecheng commented Dec 13, 2024

Release notes

Remove Arcsight module

What does this PR do?

Remove all module related code

  • remove arcsight module
  • remove module framework
  • remove module tests
  • remove module configs

The removal of doc is addressed in #16796

Why is it important/What is the impact to the user?

All modules are removed in v9.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files (and/or docker env variables)
  • I have added tests that prove my fix is effective or that my feature works

Author's Checklist

  • passed exhaustive test
  • tested legacy monitoring
  • tested central pipeline management
  • bin/logstash --modules arcsight --setup gives ERROR: Unrecognised option '--modules'
  • Config module in logstash.yml gives Your settings are invalid. Reason: Setting "modules" doesn't exist
    modules:
    - name: arcsight
    

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

@kaisecheng kaisecheng linked an issue Dec 13, 2024 that may be closed by this pull request
@kaisecheng kaisecheng marked this pull request as ready for review December 13, 2024 15:00
"xpack.management.elasticsearch.ssl.cipher_suites",
"xpack.geoip.download.endpoint",
"xpack.geoip.downloader.enabled",
"cloud.id",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these "cloud" settings related to the module being removed?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see these are removed in i18n etc too.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is cloud.id here only relevant to modules, or is this where it is also set for central management and internal collection?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cloud.id is only used for modules
CPM uses xpack.management.elasticsearch.cloud_id. Legacy monitoring uses xpack.monitoring.elasticsearch.cloud_id

Copy link
Member

@donoghuc donoghuc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a clean removal. +28 −7,374 is pretty nice!

Copy link
Member

@robbavey robbavey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some questions about cloud.id

# var.PLUGIN_TYPE.PLUGIN_NAME.KEY
#
# modules:
#
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Cloud Settings below still refer to module specific settings var.elasticsearch.hosts and var.kibana.host, etc.
If these settings are not relevant at all, we should remove them. If they are relevant to central management and internal collection, we should move and rework them

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

deleted

"xpack.management.elasticsearch.ssl.cipher_suites",
"xpack.geoip.download.endpoint",
"xpack.geoip.downloader.enabled",
"cloud.id",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is cloud.id here only relevant to modules, or is this where it is also set for central management and internal collection?

@robbavey
Copy link
Member

I think there are some additional removals -

I don't believe we use this ElasticsearchClient outside of Modules code, despite it living outside of the main modules code area...
Once that has gone, I believe this can go too, along with
https://github.com/elastic/logstash/blob/main/logstash-core/lib/logstash/util/modules_setting_array.rb

@elastic-sonarqube
Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

@elasticmachine
Copy link

💚 Build Succeeded

History

@kaisecheng
Copy link
Contributor Author

@robbavey this is ready for a second look
cloud.id is for modules only. I have removed the mentioned dangling code.

@kaisecheng kaisecheng requested a review from robbavey December 17, 2024 15:32
Copy link
Member

@robbavey robbavey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is so awesome! I do have a couple of questions before approval:

  • Do we know why those two require statements suddenly needed to be added? Was it related to this PR, or a general issue?
  • It didn't look like it from reading through the code, but did we confirm whether or not there are dependencies that we import that we no longer need to after this PR?
require "spec_helper"
require "open-uri"
require "webmock/rspec"
require "faraday"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, do you know what changed that forced this change? Is it part of this change? Or something outside of it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is caused by the removal of elasticsearch_client.rb and the import in runner_spec.rb. The elasticsearch related classes somehow include Faraday.

The import in other spec files affect the global namespace for the entire test suite. You can add require "faraday" in runner_spec and get a green run of ./gradlew :logstash-core:rubyTests, but it will fail when running a single rspec file as webserver_spec itself doesn't link faraday.

@kaisecheng kaisecheng requested a review from robbavey December 19, 2024 12:57
Copy link
Member

@robbavey robbavey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@Sunanda533
Copy link

Sunanda533 commented May 27, 2025

Hi,

We are upgrading to logstash v9.0.1 and there is issue observed in unit-test in pipeline.
But locally, there are no issues observed during build or unit-test.
Only in pipeline, unit-test is being failed.
And the errors are mostly related to Logstash::Runner
We tried to upgrade to logstash v8.18.0 and there are no issues observed.
The main difference between these two versions(v8.18.0 and v9.0.1) is modules framework is removed in latest logstash version.
We suspect removal of modules framework cause this issue.
I have created github ticket #17673 for the unit-test failure.
Please let us know your suggestions how to proceed further or how we can adapt the modules framework.
Also, please let us know if there are any open issues related to unit-test or anyone facing the same issue with latest logstash version v9.0.1.

Thanks & regards,
Sunanda N.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

5 participants