add coverage reports by sharbuz · Pull Request #8464 · elastic/integrations

sharbuz · 2023-11-10T14:21:02Z

Proposed commit message

The partial migration of the Jenkins pipelines to the Buildkite pipelines.

How to test this PR locally

The result should be seen in each PR

Related issues

elasticmachine · 2023-11-10T14:28:00Z

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-12-11T16:46:55.583+0000
Duration: 116 min 48 sec

Test stats 🧪

Test	Results
Failed	1
Passed	5103
Skipped	6
Total	5110

Test errors

Expand to view the tests failures

`Check integrations / elastic_package_registry / elastic_package_registry: check / pipeline test: test-default.json – elastic_package_registry.metrics`

Expand to view the error details

 null

Expand to view the stacktrace

 test case failed: Expected results are different from actual ones: --- want
+++ got
@@ -196,6 +196,9 @@
             "service": {
                 "address": "http://elastic-package-service_elastic_package_registry_1:9110/metrics",
                 "type": "elastic_package_registry"
+            },
+            "test": {
+                "field_version": "9.9.9"
             }
         }
     ]

Steps errors

Expand to view the steps failures

`Test integration: aws`

Took 33 min 45 sec . View more details here
Description: eval "$(../../build/elastic-package stack shellinit)" ../../build/elastic-package test -v --report-format xUnit --report-output file --test-coverage

`Test integration: elastic_package_registry`

Took 2 min 44 sec . View more details here
Description: eval "$(../../build/elastic-package stack shellinit)" ../../build/elastic-package test -v --report-format xUnit --report-output file --test-coverage

`elastic-package report benchmark`

Took 0 min 0 sec . View more details here
Description: elastic-package report benchmark --fail-on-missing=false --new="build/benchmark-results" --old="build/main/benchmark-results" --threshold=15 --report-output-path="report.md" --full=false

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

github-advanced-security · 2023-11-10T16:33:26Z

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

mrodm

In the comment written by sonarqube there is no coverage report as it was done by Jenkins #8454 (comment), is it reported somewhere else ?

About quality gates, I think currently should not be set here... These coverage do not represent the same as in other repositories. These coverage reports have a different meaning here in integrations or elastic-package repositories:
https://github.com/elastic/integrations/blob/main/docs/testing_and_validation.md#review-test-coverage

mrodm · 2023-11-10T19:05:42Z

+      - "buildkite-agent artifact download coverage.out ."
+      - "buildkite-agent artifact download test-unit.out.xml ."


These files should not be generated using that new script ".buildkite/scripts/unit_test.sh".

These coverage XML files should be taken from the files generated by elastic-package command.

Every step that checks each package creates XML files in this path:

integrations/.ci/Jenkinsfile

Line 412 in 90c6765

def wildcard = 'build/test-coverage/*.xml'

Could all those XML files generated be merged somehow to show a global coverage as it is done in Jenkins?

integrations/.ci/Jenkinsfile

Lines 428 to 432 in 90c6765

dir("${BASE_DIR}") {

def bucketUri = getCoverageBucketURI() + "*.xml"

googleStorageDownload(bucketUri: bucketUri, credentialsId: "${JOB_GCS_CREDENTIALS}", localDirectory: 'build/test-coverage', pathPrefix: getCoveragePathPrefix())

coverageReport('build/test-coverage')

}

Yes, it could be merged, and thanks for the new inputs.
Will do that.

mrodm · 2023-11-10T19:07:19Z

+go test -covermode=atomic -v -coverprofile=coverage.out ./... | tee ${testsFileName}
+
+go install github.com/jstemmer/go-junit-report@latest
+go-junit-report > "${testsFileName}.xml" < ${testsFileName}


Coverage report is generated by elastic-package since it is used with the parameter --test-coverage

@mrodm, I don't see the coverage report in the artifacts, despite generating the reports with the correct option --test-coverage.

There are 2 files generated by the unit_test script (go test -coverprofile=coverage.out.....):
https://buildkite.com/organizations/elastic/pipelines/integrations/builds/6284/jobs/018bba52-cdb2-44dc-bc43-4e67eeb08fc3/artifacts/018bba53-9986-4484-9f18-5863a365dd7f
https://buildkite.com/organizations/elastic/pipelines/integrations/builds/6284/jobs/018bba52-cdb2-44dc-bc43-4e67eeb08fc3/artifacts/018bba53-9986-4b5f-89ba-15490e50941c
and there are we can see the coverage reports.

IIRC artifact paths for coverage reports are not added in the step definitions created.

integrations/.buildkite/scripts/trigger_integrations_in_parallel.sh

Lines 60 to 62 in af38b97

artifact_paths:

- build/test-results/*.xml

- build/benchmark-results/*.xml

Just tried locally to generate those files and it works:

$ cd /path/integrations/repo $ cd packages/elastic_package_registry $ elastic-package stack up -v -d --version 8.11-SNAPSHOT ... $ elastic-package test -v --test-coverage ... $ cd ../../ $ ls -l build/test-coverage/ total 24 -rw-r--r-- 1 mariorodriguez mariorodriguez 11755 nov 13 10:50 coverage-elastic_package_registry-1699869049973807711-report.xml -rw-r--r-- 1 mariorodriguez mariorodriguez 912 nov 13 10:50 coverage-elastic_package_registry-1699869050248885422-report.xml -rw-r--r-- 1 mariorodriguez mariorodriguez 912 nov 13 10:51 coverage-elastic_package_registry-1699869115891481631-report.xml -rw-r--r-- 1 mariorodriguez mariorodriguez 911 nov 13 10:51 coverage-elastic_package_registry-1699869119250577740-report.xml

Here in this repository, it would be special. Here it is interesting the coverage of the packages, not the ones from the check sources step. @jsoriano is that correct?

These reports contain only zeros everywhere. It's OK?

These reports have another data format:
coverage-barracuda_cloudgen_firewall-1699871427106197266-report.xml: Invalid go coverage, expect 'mode:' on the first line
The SonaqQube requires the files with data-format as I was getting the in the unit_test script - coverage.out...

These reports contain only zeros everywhere. It's OK?

The contents of these files should be XML. For instance in the test that I run above:

$ head -n 20 build/test-coverage/coverage-elastic_package_registry-1699869049973807711-report.xml <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE coverage SYSTEM "http://cobertura.sourceforge.net/xml/coverage-04.dtd"> <coverage line-rate="0" branch-rate="0" version="" timestamp="1699869049973807711" lines-covered="20" lines-valid="20" branches-covered="0" branches-valid="0" complexity="0"> <sources> <source>/home/mariorodriguez/Coding/work/integrations/packages</source> </sources> <packages> <package name="elastic_package_registry.metrics" line-rate="0" branch-rate="0" complexity="0"> <classes> <class name="default" filename="elastic_package_registry/data_stream/metrics/elasticsearch/ingest_pipeline/default.yml" line-rate="0" branch-rate="0" complexity="0"> <methods> <method name="set" signature="" line-rate="0" branch-rate="0" complexity="0" hits="1"> <lines> <line number="4" hits="1"></line> <line number="5" hits="1"></line> <line number="6" hits="1"></line> </lines> </method> <method name="set" signature="" line-rate="0" branch-rate="0" complexity="0" hits="1">

@sharbuz

I saw that, almost all values = 0

elasticmachine · 2023-11-11T00:10:34Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`472/472`)	💚
Files	96.067% (`806/839`)	👍 73.845
Classes	96.067% (`806/839`)	👍 73.845
Methods	92.406% (`7848/8493`)	👍 60.587
Lines	88.623% (`185573/209396`)	👍 68.178
Conditionals	100.0% (`0/0`)	💚

rhass · 2023-12-01T22:11:14Z

 # sonar.java.coveragePlugin=cobertura
-sonar.cobertura.reportPath==build/test-coverage/coverage_merged.xml
+# sonar.cobertura.reportPath==build/test-coverage/coverage_merged.xml
+sonar.cobertura.reportPath==build/test-coverage/coverage-*.xml


This will not work. Sonar will not merge reports, you must use one which has been consolidated before uploading it.

Co-authored-by: Mario Rodriguez Molins <marrodmo@gmail.com>

elastic-sonarqube · 2023-12-11T16:59:09Z

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

mrodm · 2023-12-21T09:42:33Z

Hi @sharbuz, please update your branch with the latest contents from main branch. There was an important PR merged updating the CI pipelines. Thanks!

mrodm · 2023-12-21T16:27:26Z

+# sonar.dynamicAnalysis=reuseReports
+# sonar.java.coveragePlugin=cobertura
+# sonar.cobertura.reportPath=build/test-coverage/coverage_merged.xml
+sonar.python.coverage.reportPaths=build/test-coverage/coverage-report/coverage.xml


It looks like python test coverage tries to process the coverage XML
It shows this error:

INFO: Sensor Cobertura Sensor for Python coverage [python]
INFO: Python test coverage
INFO: Parsing report '/buildkite/builds/bk-agent-prod-k8s-1702313828449526795/elastic/integrations/build/test-coverage/coverage-report/coverage.xml'
ERROR: Cannot resolve the file path 'elastic_package_registry/metrics' of the coverage report, the file does not exist in all 'source'.

Probably, elastic-package can add data_stream as part of the filename (elastic_package_registry/data_stream/metrics) to see if that would be valid for Sonar. Not sure if it would be required also to include packages , but that would out of the context where elastic-package command runs.

Tried in these two pull requests, and now with python coverage is able to parse the files, but it still does not appear any report

Relates Pull requests

Updating elastic-package to add specific filepaths: Add generic test coverage report format for SonarQube elastic-package#1612

Updated elastic-package version in integrations test: Add coverage reports analysis of packages #8787

botelastic · 2024-01-21T14:21:00Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

Siarhei Harbuz added 3 commits November 10, 2023 16:05

add coverage reports

1451faa

fix script name

07ed911

fix script name

a252403

Siarhei Harbuz added 9 commits November 10, 2023 16:31

add mage

2eef0e1

change the logic

aed26da

test

8e1c42e

test

b94e399

test

5a97b1e

test

990591a

add junit

95d45a2

test

6e65ee7

test

5693353

Siarhei Harbuz added 5 commits November 10, 2023 18:35

test

52e4813

test

b8af8dc

test

4492174

add test-unit.out.xml to report

f51d0ec

fix pathes

52bd7c0

sharbuz mentioned this pull request Nov 10, 2023

add uploading the test-unit.out.xml artifact elastic/fleet-server#3091

Merged

sharbuz requested review from jlind23, mrodm and pazone November 10, 2023 17:34

sharbuz self-assigned this Nov 10, 2023

full test

0a1bab0

mrodm reviewed Nov 10, 2023

View reviewed changes

Siarhei Harbuz added 3 commits November 13, 2023 10:58

change the logic

e0d7ad9

add test-coverage artifacts

1b6813c

add test-coverage artifacts

9babf0e

test

10659bf

rhass reviewed Dec 1, 2023

View reviewed changes

mrodm reviewed Dec 4, 2023

View reviewed changes

Comment thread sonar-project.properties Outdated

sharbuz and others added 18 commits December 4, 2023 11:22

Update sonar-project.properties

4d1dd27

Co-authored-by: Mario Rodriguez Molins <marrodmo@gmail.com>

change the language to yaml and add packages to scan

8a07d02

exlude java

e139612

test the idea

add1487

test the idea

7a65141

test the idea

1104bef

test the idea

9fcd33e

test the idea

41f525e

test the idea

265166d

test the idea

67aebce

test

cb84a5d

test

1c8040a

test

989d6d7

test

585379b

Merge branch 'main' into add-coverage-report

f198a2b

test

aaffc47

test

ea74650

check python plugin

e44a175

mrodm reviewed Dec 21, 2023

View reviewed changes

mrodm mentioned this pull request Dec 22, 2023

Add SonarQube generic-test-data format support for code coverage report. elastic/elastic-package#1595

Closed

This was referenced Jan 17, 2024

Add generic test coverage report format for SonarQube elastic/elastic-package#1612

Merged

Add coverage reports analysis of packages #8787

Merged

botelastic Bot added the Stalled label Jan 21, 2024

mrodm closed this in #8787 Jan 31, 2024

Navigation Menu

Uh oh!

add coverage reports - #8464

add coverage reports#8464
sharbuz wants to merge 86 commits into
elastic:mainfrom
sharbuz:add-coverage-report

sharbuz commented Nov 10, 2023 •

edited

Loading

elasticmachine commented Nov 10, 2023 •

edited

Loading

Build stats

Test stats 🧪

`Check integrations / elastic_package_registry / elastic_package_registry: check / pipeline test: test-default.json – elastic_package_registry.metrics`

`Test integration: aws`

`Test integration: elastic_package_registry`

`elastic-package report benchmark`

github-advanced-security AI commented Nov 10, 2023

mrodm left a comment

mrodm Nov 10, 2023 •

edited

Loading

sharbuz Nov 13, 2023

mrodm Nov 10, 2023

sharbuz Nov 13, 2023 •

edited

Loading

mrodm Nov 13, 2023

sharbuz Nov 13, 2023

mrodm Nov 13, 2023

sharbuz Nov 13, 2023

elasticmachine commented Nov 11, 2023 •

edited

Loading

rhass Dec 1, 2023

Uh oh!

elastic-sonarqube Bot commented Dec 11, 2023

mrodm commented Dec 21, 2023

mrodm Dec 21, 2023

mrodm Dec 22, 2023

botelastic Bot commented Jan 21, 2024

Labels

5 participants

		- "buildkite-agent artifact download coverage.out ."
		- "buildkite-agent artifact download test-unit.out.xml ."

	dir("${BASE_DIR}") {
	def bucketUri = getCoverageBucketURI() + "*.xml"
	googleStorageDownload(bucketUri: bucketUri, credentialsId: "${JOB_GCS_CREDENTIALS}", localDirectory: 'build/test-coverage', pathPrefix: getCoveragePathPrefix())
	coverageReport('build/test-coverage')
	}

	artifact_paths:
	- build/test-results/*.xml
	- build/benchmark-results/*.xml

Uh oh!

Conversation

sharbuz commented Nov 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

How to test this PR locally

Related issues

elasticmachine commented Nov 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Tests Failed

Build stats

Test stats 🧪

Test errors

Check integrations / elastic_package_registry / elastic_package_registry: check / pipeline test: test-default.json – elastic_package_registry.metrics

Steps errors

Test integration: aws

Test integration: elastic_package_registry

elastic-package report benchmark

🤖 GitHub comments

github-advanced-security AI commented Nov 10, 2023

mrodm left a comment

Choose a reason for hiding this comment

mrodm Nov 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sharbuz Nov 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Nov 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Choose a reason for hiding this comment

Uh oh!

elastic-sonarqube Bot commented Dec 11, 2023

Quality Gate passed

mrodm commented Dec 21, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

botelastic Bot commented Jan 21, 2024

Labels

5 participants

sharbuz commented Nov 10, 2023 •

edited

Loading

elasticmachine commented Nov 10, 2023 •

edited

Loading

`Check integrations / elastic_package_registry / elastic_package_registry: check / pipeline test: test-default.json – elastic_package_registry.metrics`

`Test integration: aws`

`Test integration: elastic_package_registry`

`elastic-package report benchmark`

mrodm Nov 10, 2023 •

edited

Loading

sharbuz Nov 13, 2023 •

edited

Loading

elasticmachine commented Nov 11, 2023 •

edited

Loading