tracer: Drop unsampled txs when APM Server >= 8.0

[marclop]

Description

Extends the HTTP Transport with a new method that queries the remote APM
Server / endpoint to obtain its version. Allowing the tracer to drop
unsampled transactions when the APM Server is version 8.0.0 or higher.

The version will be cached in a local variable until the APM Server URL
changes which will refresh the APM Server's version. The tracer launches
a goroutine to update the version when it is instantiated and refreshes
the remote server version every 10 seconds.

Any SendStream errors will cause the version cache to be invalidated and
will be refreshed the next time the ticker runs. It may cause unsampled
to be sent to the APM Server while the cache is invalid.

Related Issues

Closes #1153

[ghost]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-02-11T10:02:27.207+0000

• Duration: 42 min 24 sec

Test stats 🧪

Test	Results
Failed	0
Passed	7342
Skipped	174
Total	7516

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• run benchmark tests : Run the benchmark test.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[axw]

There was an unwritten assumption that SetServerURLs is not called concurrently with other methods: t.intakeURLs etc. are set without a lock. Is there some new concurrency that requires the atomic call?

[marclop]

Since it wasn't stated anywhere and it seemed like a fairly small change to avoid having a potential data race when the index is increased and SetServerURL is called at the same time.

I can revert if you feel like we don't need to do it, but it seemed like a small price to pay to avoid any potential panics on consumer applications.

[axw]

I'd prefer to be precise when it comes to concurrency: either the method needs to be concurrent-safe, or it doesn't. The atomic implies it does, but the unsynchronised mutation of t.intakeURLs etc. implies it doesn't.

Unless we need it (maybe we do?), I'd prefer to revert.

[marclop]

I see, I will revert the change since it's unclear whether this is a scenario that users may encounter.

[axw]

I'd prefer to be precise when it comes to concurrency: either the method needs to be concurrent-safe, or it doesn't. The atomic implies it does, but the unsynchronised mutation of t.intakeURLs etc. implies it doesn't.

Unless we need it (maybe we do?), I'd prefer to revert.

[axw]

LGTM with a handful of small things. Thanks!