Make gradle the root of every dependency graph#18471
Conversation
🤖 GitHub commentsJust comment with:
|
|
This pull request does not have a backport label. Could you fix it @donoghuc? 🙏
|
Do a fist pass at removing circular dependency on rake calling to gradle. Replace every "top level" task with a gradle task. This allows doing all dependency management with gradle, while still allowing us to use rake for leaf node tasks when appropriate.
6b91281 to
4fd6bbb
Compare
|
Note to self: make sure to re-evaluate https://github.com/elastic/logstash/pull/18340/files#r2488030095 |
|
run exhaustive tests |
|
Got through PR tests, moving on the exhaustive next (Still need to do DRA and ansillary). |
rubyUtils.gradle
Outdated
| } | ||
|
|
||
| // Spawn rake as external process (for CI compatibility) | ||
| def rakeExternal(File projectDir, File buildDir, String task) { |
There was a problem hiding this comment.
Seems like file descriptor inheretance is tripping up how fpm (https://github.com/jordansissel/fpm/blob/d9be49871fe040937b63d1223491fc352508ab78/lib/fpm/util.rb#L8) wants to shell out to tar. I found jruby/jruby#5249 which seems similar (but should have been resolved).
For now tasks that use fpm shell out to rake instead of using jruby via gradle.
example failure https://buildkite.com/elastic/logstash-exhaustive-tests-pipeline/builds/2998#019b0aa3-ff39-4b6b-9f37-cfd4cf1aa73b:
2025-12-10 15:45:17 PST | [artifact:deb] building deb package for x86_64
2025-12-10 15:45:17 PST | Rake task error: Errno::EBADF: Bad file descriptor - tar
2025-12-10 15:45:27 PST | Backtrace: org/jruby/RubyProcess.java:1779:in `spawn'
2025-12-10 15:45:17 PST | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1765410090515934184/elastic/logstash-exhaustive-tests-pipeline/vendor/bundle/jruby/3.1.0/gems/fpm-1.17.0/lib/fpm/util.rb:145:in `execmd'
2025-12-10 15:45:17 PST | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1765410090515934184/elastic/logstash-exhaustive-tests-pipeline/vendor/bundle/jruby/3.1.0/gems/fpm-1.17.0/lib/fpm/util.rb:186:in `safesystem'
2025-12-10 15:44:34 PST | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1765410090515934184/elastic/logstash-exhaustive-tests-pipeline/vendor/bundle/jruby/3.1.0/gems/fpm-1.17.0/lib/fpm/package/deb.rb:1043:in `block in write_control_tarball'
2025-12-10 15:45:27 PST | <internal:uri:classloader:/jruby/kernel/kernel.rb>:19:in `tap'
2025-12-10 15:44:34 PST | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1765410090515934184/elastic/logstash-exhaustive-tests-pipeline/vendor/bundle/jruby/3.1.0/gems/fpm-1.17.0/lib/fpm/package/deb.rb:1033:in `write_control_tarball'
2025-12-10 15:45:17 PST | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1765410090515934184/elastic/logstash-exhaustive-tests-pipeline/vendor/bundle/jruby/3.1.0/gems/fpm-1.17.0/lib/fpm/package/deb.rb:677:in `output'
2025-12-10 15:45:27 PST | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1765410090515934184/elastic/logstash-exhaustive-tests-pipeline/rakelib/artifacts.rake:853:in `package'
2025-12-10 15:45:27 PST | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1765410090515934184/elastic/logstash-exhaustive-tests-pipeline/rakelib/artifacts.rake:672:in `package_with_jdk'
2025-12-10 15:45:27 PST | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1765410090515934184/elastic/logstash-exhaustive-tests-pipeline/rakelib/artifacts.rake:313:in `block in <main>'
2025-12-10 15:45:27 PST | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1765410090515934184/elastic/logstash-exhaustive-tests-pipeline/vendor/bundle/jruby/3.1.0/gems/rake-13.3.1/lib/rake/task.rb:281:in `block in execute'
2025-12-10 15:45:27 PST | org/jruby/RubyArray.java:2009:in `each'
2025-12-10 15:45:27 PST | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1765410090515934184/elastic/logstash-exhaustive-tests-pipeline/vendor/bundle/jruby/3.1.0/gems/rake-13.3.1/lib/rake/task.rb:281:in `execute'
2025-12-10 15:45:27 PST | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1765410090515934184/elastic/logstash-exhaustive-tests-pipeline/vendor/bundle/jruby/3.1.0/gems/rake-13.3.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
2025-12-10 15:45:27 PST | org/jruby/ext/monitor/Monitor.java:82:in `synchronize'
2025-12-10 15:45:27 PST | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1765410090515934184/elastic/logstash-exhaustive-tests-pipeline/vendor/bundle/jruby/3.1.0/gems/rake-13.3.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
2025-12-10 15:45:27 PST | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1765410090515934184/elastic/logstash-exhaustive-tests-pipeline/vendor/bundle/jruby/3.1.0/gems/rake-13.3.1/lib/rake/task.rb:188:in `invoke'
2025-12-10 15:45:27 PST | <script>:6:in `<main>'
|
run exhaustive tests |
|
Made more progress on acceptance tests... The Also, there will be some follow on work with invokingh bunlder directly for acceptance tests. For now, instead of invoking rake i'm invoking bundler, but i have not decided if that is actually a win or not. |
|
Today i was able to figure out the fpm issues but the problem i'm playing whack-a-mole with is the management of dowloading jdk. The exhaustive test pipeline provides an example of an issue i'm facing. Currently as a prep step we run I need to figure out exactly which tasks need the copyJdk (and which explicitly need it deleted) in order to figure out how to best represent it in gradle. |
This reverts commit c5387b6.
|
run exhaustive tests |
|
run exhaustive tests |
|
run exhaustive tests |
|
I kicked off a DRA snapshot job on this to double check things would build. Obviously it fails to be published (desireable). Just wanted to make sure build is all working. Seems good! https://buildkite.com/elastic/logstash-dra-snapshot-pipeline/builds/4307 |
💛 Build succeeded, but was flaky
Failed CI StepsHistory
|
mashhurs
left a comment
There was a problem hiding this comment.
LGTM!
I don't have much context but from the "rake calling to gradle" goal point of view, looks great. Thank you for the effort!
| } | ||
|
|
||
| clean { | ||
| dependsOn deleteLocalJdk |
| def package_with_jdk(platform, jdk_arch, variant = :standard) | ||
| safe_system("./gradlew copyJdk -Pjdk_bundle_os=linux -Pjdk_arch=#{jdk_arch}") | ||
| package(platform, variant, true, jdk_arch) | ||
| safe_system('./gradlew deleteLocalJdk -Pjdk_bundle_os=linux') |
There was a problem hiding this comment.
It looks like we removed all jdk_bundle_os usages, maybe removing in seelctOsType() also makes sense?
There was a problem hiding this comment.
Yeah, i'll look in to that with the removal tasks #18476
|
|
||
| echo "--- Acceptance: Running the tests" | ||
| # TODO: figure out how to encapsulate all this in gradle or at least ensure the bundler | ||
| # used here ends up using the runtime managed with LS. |
There was a problem hiding this comment.
Are you going to follow up with next iterations?
There was a problem hiding this comment.
yep! this will fall under the ticket for using gradle for managing ruby runtime #18477
|
Now that the 9.3 branch has been cut, i'm going to get this merged so i can build the PR for removal of unused tasks on this. I'll keep an eye on CI in the main branch to make sure there are no surprises! |
|
@Mergifyio backport 8.19 9.1 9.2 9.3 |
✅ Backports have been createdDetails
Cherry-pick of ca1fe37 has failed: To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally
Cherry-pick of ca1fe37 has failed: To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally
Cherry-pick of ca1fe37 has failed: To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally
Cherry-pick of ca1fe37 has failed: To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally |
* WIP: Make gradle the root of every dependency graph Do a fist pass at removing circular dependency on rake calling to gradle. Replace every "top level" task with a gradle task. This allows doing all dependency management with gradle, while still allowing us to use rake for leaf node tasks when appropriate. * replicate rake behavior with copyjdk * jdk for observabilitySRE artifact * serialize jdk cleanup * try sub process * and rpm... * fix acceptance script * rake task from qa dir context * revert to calling bundler directly * fix copy paste * remove complex function and add explicit require * manage jdk at task level * Revert "manage jdk at task level" This reverts commit c5387b6. * stop deleting jdk * cleanup * remove unused file * more cleanup * copyjdk for all artifacts * fpm file descriptor workaround * postpone deletion work * restore rake task impl for unused script (cherry picked from commit ca1fe37) # Conflicts: # build.gradle # rakelib/artifacts.rake # rakelib/test.rake # rakelib/vendor.rake
* WIP: Make gradle the root of every dependency graph Do a fist pass at removing circular dependency on rake calling to gradle. Replace every "top level" task with a gradle task. This allows doing all dependency management with gradle, while still allowing us to use rake for leaf node tasks when appropriate. * replicate rake behavior with copyjdk * jdk for observabilitySRE artifact * serialize jdk cleanup * try sub process * and rpm... * fix acceptance script * rake task from qa dir context * revert to calling bundler directly * fix copy paste * remove complex function and add explicit require * manage jdk at task level * Revert "manage jdk at task level" This reverts commit c5387b6. * stop deleting jdk * cleanup * remove unused file * more cleanup * copyjdk for all artifacts * fpm file descriptor workaround * postpone deletion work * restore rake task impl for unused script (cherry picked from commit ca1fe37) # Conflicts: # build.gradle # rakelib/artifacts.rake # rakelib/test.rake # rakelib/vendor.rake
* WIP: Make gradle the root of every dependency graph Do a fist pass at removing circular dependency on rake calling to gradle. Replace every "top level" task with a gradle task. This allows doing all dependency management with gradle, while still allowing us to use rake for leaf node tasks when appropriate. * replicate rake behavior with copyjdk * jdk for observabilitySRE artifact * serialize jdk cleanup * try sub process * and rpm... * fix acceptance script * rake task from qa dir context * revert to calling bundler directly * fix copy paste * remove complex function and add explicit require * manage jdk at task level * Revert "manage jdk at task level" This reverts commit c5387b6. * stop deleting jdk * cleanup * remove unused file * more cleanup * copyjdk for all artifacts * fpm file descriptor workaround * postpone deletion work * restore rake task impl for unused script (cherry picked from commit ca1fe37) # Conflicts: # build.gradle # rakelib/test.rake # rakelib/vendor.rake
* WIP: Make gradle the root of every dependency graph Do a fist pass at removing circular dependency on rake calling to gradle. Replace every "top level" task with a gradle task. This allows doing all dependency management with gradle, while still allowing us to use rake for leaf node tasks when appropriate. * replicate rake behavior with copyjdk * jdk for observabilitySRE artifact * serialize jdk cleanup * try sub process * and rpm... * fix acceptance script * rake task from qa dir context * revert to calling bundler directly * fix copy paste * remove complex function and add explicit require * manage jdk at task level * Revert "manage jdk at task level" This reverts commit c5387b6. * stop deleting jdk * cleanup * remove unused file * more cleanup * copyjdk for all artifacts * fpm file descriptor workaround * postpone deletion work * restore rake task impl for unused script (cherry picked from commit ca1fe37) # Conflicts: # build.gradle # rakelib/test.rake # rakelib/vendor.rake
…ph (#18607) * Make gradle the root of every dependency graph (#18471) * WIP: Make gradle the root of every dependency graph Do a fist pass at removing circular dependency on rake calling to gradle. Replace every "top level" task with a gradle task. This allows doing all dependency management with gradle, while still allowing us to use rake for leaf node tasks when appropriate. * replicate rake behavior with copyjdk * jdk for observabilitySRE artifact * serialize jdk cleanup * try sub process * and rpm... * fix acceptance script * rake task from qa dir context * revert to calling bundler directly * fix copy paste * remove complex function and add explicit require * manage jdk at task level * Revert "manage jdk at task level" This reverts commit c5387b6. * stop deleting jdk * cleanup * remove unused file * more cleanup * copyjdk for all artifacts * fpm file descriptor workaround * postpone deletion work * restore rake task impl for unused script (cherry picked from commit ca1fe37) # Conflicts: # build.gradle # rakelib/artifacts.rake # rakelib/test.rake # rakelib/vendor.rake * fix merge conflicts --------- Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
#18608) * Make gradle the root of every dependency graph (#18471) * WIP: Make gradle the root of every dependency graph Do a fist pass at removing circular dependency on rake calling to gradle. Replace every "top level" task with a gradle task. This allows doing all dependency management with gradle, while still allowing us to use rake for leaf node tasks when appropriate. * replicate rake behavior with copyjdk * jdk for observabilitySRE artifact * serialize jdk cleanup * try sub process * and rpm... * fix acceptance script * rake task from qa dir context * revert to calling bundler directly * fix copy paste * remove complex function and add explicit require * manage jdk at task level * Revert "manage jdk at task level" This reverts commit c5387b6. * stop deleting jdk * cleanup * remove unused file * more cleanup * copyjdk for all artifacts * fpm file descriptor workaround * postpone deletion work * restore rake task impl for unused script (cherry picked from commit ca1fe37) # Conflicts: # build.gradle # rakelib/artifacts.rake # rakelib/test.rake # rakelib/vendor.rake * resolve merge conflicts --------- Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
#18609) * Make gradle the root of every dependency graph (#18471) * WIP: Make gradle the root of every dependency graph Do a fist pass at removing circular dependency on rake calling to gradle. Replace every "top level" task with a gradle task. This allows doing all dependency management with gradle, while still allowing us to use rake for leaf node tasks when appropriate. * replicate rake behavior with copyjdk * jdk for observabilitySRE artifact * serialize jdk cleanup * try sub process * and rpm... * fix acceptance script * rake task from qa dir context * revert to calling bundler directly * fix copy paste * remove complex function and add explicit require * manage jdk at task level * Revert "manage jdk at task level" This reverts commit c5387b6. * stop deleting jdk * cleanup * remove unused file * more cleanup * copyjdk for all artifacts * fpm file descriptor workaround * postpone deletion work * restore rake task impl for unused script (cherry picked from commit ca1fe37) # Conflicts: # build.gradle # rakelib/test.rake # rakelib/vendor.rake * resolve merge conflicts --------- Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
#18610) * Make gradle the root of every dependency graph (#18471) * WIP: Make gradle the root of every dependency graph Do a fist pass at removing circular dependency on rake calling to gradle. Replace every "top level" task with a gradle task. This allows doing all dependency management with gradle, while still allowing us to use rake for leaf node tasks when appropriate. * replicate rake behavior with copyjdk * jdk for observabilitySRE artifact * serialize jdk cleanup * try sub process * and rpm... * fix acceptance script * rake task from qa dir context * revert to calling bundler directly * fix copy paste * remove complex function and add explicit require * manage jdk at task level * Revert "manage jdk at task level" This reverts commit c5387b6. * stop deleting jdk * cleanup * remove unused file * more cleanup * copyjdk for all artifacts * fpm file descriptor workaround * postpone deletion work * restore rake task impl for unused script (cherry picked from commit ca1fe37) # Conflicts: # build.gradle # rakelib/test.rake # rakelib/vendor.rake * fix merge conflicts --------- Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
Release notes
[rn:skip]
What does this PR do?
Do a fist pass at removing circular dependency on rake calling to gradle. Replace every "top level" task with a gradle task. This allows doing all dependency management with gradle, while still allowing us to use rake for leaf node tasks when appropriate. By moving everything to gradle we can start leveraging the dependency management to ensure that jruby etc is stardardized across CI use cases and therefore stop managing jruby environments that match that which is shipped with logstash separately.