Prevent overhead from excessive span creation

[SylvainJuge]

What does this PR do?

Fixes #2788 by skipping span creation when the max span limit is already reached within the active transaction.

This is applied to the following instrumentations:

• trace_methods set in configuration
• @Traced annotated methods
• @CaptureSpan annotated methods

While it does not prevent users from mis-using configuration or annotations, it removes most of the agent overhead by skipping the span creation.

The global number of dropped spans is captured in the transaction, however the dropped span metrics are not updated because they rely on the span being created (and are updated when the span ends).

As a consequence the dropped spans metrics might appear inconsistent, but I think it's safe to assume that inconsistency is a small price to pay here given it helps to lower the overhead in case of an agent mis-configuration.

Checklist

• This is an enhancement of existing features, or a new feature in existing plugins
  • I have updated CHANGELOG.asciidoc
  • I have added tests that prove my fix is effective or that my feature works
  • Added an API method or config option? Document in which version this will be introduced N/A
  • I have made corresponding changes to the documentation N/A

Performance improvements

10 millions very short spans created within a single transaction:

• the bottom half transactions with current version of the agent: transaction takes about ~7seconds
• the top half is with this change applied: transaction takes about 200~400ms

[ghost]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-05-31T11:51:50.281+0000

• Duration: 15 min 30 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• run benchmark tests : Run the benchmark tests.

• run jdk compatibility tests : Run the JDK Compatibility tests.

• run integration tests : Run the Agent Integration tests.

• run end-to-end tests : Run the APM-ITs.

• run windows tests : Build & tests on windows.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[SylvainJuge]

[for reviewer] this was necessary as this test failed when run in isolation, most tests actually trigger this init through instrumentation init.

[github-actions]

/test

[JonasKunz]

In my opinion it would be cleaner to instead alter the contract of createSpan() to allow to return null values for the cases where the tracer decides that no spans should be created anymore. This would allow this optimization to also take effect on our own instrumentations.

However, this would be a lot of work as we need to revisit all createSpan() usages in plugins and make sure that they properly handle null values and that there are no side effects of not creating spans, which is a bit much for this PR.

[SylvainJuge]

That's exactly why I took this simple approach here, and it would have been quite consistent with createExitSpan that is expected to return null in case there is already an active exit span.

Another alternative would be to have a way to return a no-op span, which could avoid most of the refactoring but could still have some unexpected side-effects.

I think it could be worth investing a bit of time for a refactoring/tech-debt PR.