Skip to content

sdd_all: use triple-brace templating in complex templates - #11315

Merged
efd6 merged 1 commit into
elastic:mainfrom
efd6:7641-sdd_all_complex
Oct 8, 2024
Merged

sdd_all: use triple-brace templating in complex templates#11315
efd6 merged 1 commit into
elastic:mainfrom
efd6:7641-sdd_all_complex

Conversation

@efd6

@efd6 efd6 commented Oct 3, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor

Proposed commit message

sdd_all: use triple-brace templating in complex templates

This is a second pass at replacing double brace template marks with triple brace
marks. It addresses more complex uses that the sed rewrite did not find by using
PCRE via perl.

[git-generate]
for f in $(
	(
		for p in $(
			yq 'select(.owner.github == "elastic/sec-deployment-and-devices")|.name' packages/**/manifest.yml \
			| grep -v -- '---'
		); do
			rg -l -g '*.yml' "[^{]\{\{[^{][ .a-zA-Z0-9_]*[^}]}}[^}]" packages/$p
		done
	)|grep "elasticsearch/ingest_pipeline"|sort|uniq
); do
	perl -pi -e 's/(?<!\{)(\{\{[^{][ .a-zA-Z0-9_]*[^}]}})(?!\})/{$1}/g' $f
done
for p in $(git diff --name-only HEAD~1|cut -d/ -f1,2|sort|uniq); do
	(
		cd $p
		elastic-package test pipeline -g
		elastic-package changelog add \
			--description "Use triple-brace Mustache templating when referencing variables in ingest pipelines." \
			--type bugfix \
			--next patch \
			--link https://github.com/elastic/integrations/pull/11315
	)>/dev/null 2>&1
done

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots
@efd6 efd6 added Integration:All Applies to all integrations [Integration not found in source] bugfix Pull request that fixes a bug issue Team:Security-Deployment and Devices DEPRECATED Deployment and Devices Security team [elastic/sec-deployment-and-devices] labels Oct 3, 2024
@efd6 efd6 self-assigned this Oct 3, 2024
@efd6
efd6 force-pushed the 7641-sdd_all_complex branch from 5366217 to 43e4ce5 Compare October 3, 2024 02:43
@efd6
sdd_all: use triple-brace templating in complex templates
d83c8d8 
This is a second pass at replacing double brace template marks with triple brace
marks. It addresses more complex uses that the sed rewrite did not find by using
PCRE via perl.

[git-generate]
for f in $(
	(
		for p in $(
			yq 'select(.owner.github == "elastic/sec-deployment-and-devices")|.name' packages/**/manifest.yml \
			| grep -v -- '---'
		); do
			rg -l -g '*.yml' "[^{]\{\{[^{][ .a-zA-Z0-9_]*[^}]}}[^}]" packages/$p
		done
	)|grep "elasticsearch/ingest_pipeline"|sort|uniq
); do
	perl -pi -e 's/(?<!\{)(\{\{[^{][ .a-zA-Z0-9_]*[^}]}})(?!\})/{$1}/g' $f
done
for p in $(git diff --name-only HEAD~1|cut -d/ -f1,2|sort|uniq); do
	(
		cd $p
		elastic-package test pipeline -g
		elastic-package changelog add \
			--description "Use triple-brace Mustache templating when referencing variables in ingest pipelines." \
			--type bugfix \
			--next patch \
			--link elastic#11315
	)>/dev/null 2>&1
done
@efd6
efd6 force-pushed the 7641-sdd_all_complex branch from 43e4ce5 to d83c8d8 Compare October 3, 2024 02:44
@andrewkroh andrewkroh added Integration:goflow2 GoFlow2 logs (Community supported) Integration:cef Common Event Format (CEF) Integration:cisco_ise Cisco ISE Integration:checkpoint Check Point Integration:cisco_ftd Cisco FTD Integration:pfsense pfSense (Community supported) Integration:sophos Sophos Integration:iptables Iptables Integration:cisco_ios Cisco IOS Integration:zeek Zeek Integration:modsecurity ModSecurity Audit (Community supported) Integration:cisco_secure_email_gateway Cisco Secure Email Gateway Integration:juniper_srx Juniper SRX and removed Integration:All Applies to all integrations [Integration not found in source] labels Oct 3, 2024
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 3, 2024

Copy link
Copy Markdown
Contributor

🚀 Benchmarks report

Package cef 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 3846.15 2500 -1346.15 (-35%) 💔

Package juniper_srx 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 891.27 605.69 -285.58 (-32.04%) 💔

Package modsecurity 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
auditlog 888.89 504.03 -384.86 (-43.3%) 💔

Package sophos 👍(1) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
xg 1394.7 1003.01 -391.69 (-28.08%) 💔

Package zeek 👍(16) 💚(7) 💔(20)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
http 50000 27027.03 -22972.97 (-45.95%) 💔
irc 41666.67 32258.06 -9408.61 (-22.58%) 💔
known_hosts 100000 83333.33 -16666.67 (-16.67%) 💔
known_services 90909.09 76923.08 -13986.01 (-15.38%) 💔
modbus 34482.76 24390.24 -10092.52 (-29.27%) 💔
notice 47619.05 40000 -7619.05 (-16%) 💔
ntlm 29411.76 17543.86 -11867.9 (-40.35%) 💔
ntp 47619.05 22222.22 -25396.83 (-53.33%) 💔
ocsp 38461.54 32258.06 -6203.48 (-16.13%) 💔
rdp 21276.6 16393.44 -4883.16 (-22.95%) 💔
smb_cmd 25000 19230.77 -5769.23 (-23.08%) 💔
connection 20408.16 15151.52 -5256.64 (-25.76%) 💔
smtp 30303.03 25641.03 -4662 (-15.38%) 💔
ssh 43478.26 35714.29 -7763.97 (-17.86%) 💔
stats 25641.03 20833.33 -4807.7 (-18.75%) 💔
syslog 62500 52631.58 -9868.42 (-15.79%) 💔
dce_rpc 18867.92 11494.25 -7373.67 (-39.08%) 💔
dnp3 25641.03 14084.51 -11556.52 (-45.07%) 💔
dns 32258.06 20408.16 -11849.9 (-36.73%) 💔
dpd 18181.82 14925.37 -3256.45 (-17.91%) 💔

To see the full report comment with /test benchmark fullreport
@elasticmachine

elasticmachine commented Oct 3, 2024

Copy link
Copy Markdown

💚 Build Succeeded

cc @efd6
@elastic-sonarqube

elastic-sonarqube Bot commented Oct 3, 2024

Copy link
Copy Markdown

Quality Gate failed Quality Gate failed

Failed conditions
53.5% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube
@efd6
efd6 marked this pull request as ready for review October 3, 2024 03:29
@efd6
efd6 requested a review from a team as a code owner October 3, 2024 03:29
@elasticmachine

elasticmachine commented Oct 3, 2024

Copy link
Copy Markdown

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)
@efd6
efd6 merged commit 2621bb0 into elastic:main Oct 8, 2024
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 8, 2024

Copy link
Copy Markdown
Contributor

Package cef - 2.17.4 containing this change is available at https://epr.elastic.co/search?package=cef
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 8, 2024

Copy link
Copy Markdown
Contributor

Package checkpoint - 1.34.2 containing this change is available at https://epr.elastic.co/search?package=checkpoint
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 8, 2024

Copy link
Copy Markdown
Contributor

Package cisco_ftd - 3.4.3 containing this change is available at https://epr.elastic.co/search?package=cisco_ftd
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 8, 2024

Copy link
Copy Markdown
Contributor

Package cisco_ios - 1.27.2 containing this change is available at https://epr.elastic.co/search?package=cisco_ios
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 8, 2024

Copy link
Copy Markdown
Contributor

Package cisco_ise - 1.23.2 containing this change is available at https://epr.elastic.co/search?package=cisco_ise
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 8, 2024

Copy link
Copy Markdown
Contributor

Package cisco_secure_email_gateway - 1.24.1 containing this change is available at https://epr.elastic.co/search?package=cisco_secure_email_gateway
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 8, 2024

Copy link
Copy Markdown
Contributor

Package goflow2 - 0.1.1 containing this change is available at https://epr.elastic.co/search?package=goflow2
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 8, 2024

Copy link
Copy Markdown
Contributor

Package iptables - 1.16.2 containing this change is available at https://epr.elastic.co/search?package=iptables
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 8, 2024

Copy link
Copy Markdown
Contributor

Package juniper_srx - 1.21.2 containing this change is available at https://epr.elastic.co/search?package=juniper_srx
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 8, 2024

Copy link
Copy Markdown
Contributor

Package modsecurity - 1.18.2 containing this change is available at https://epr.elastic.co/search?package=modsecurity
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 8, 2024

Copy link
Copy Markdown
Contributor

Package pfsense - 1.20.2 containing this change is available at https://epr.elastic.co/search?package=pfsense
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 8, 2024

Copy link
Copy Markdown
Contributor

Package sophos - 3.9.2 containing this change is available at https://epr.elastic.co/search?package=sophos
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 8, 2024

Copy link
Copy Markdown
Contributor

Package zeek - 2.24.4 containing this change is available at https://epr.elastic.co/search?package=zeek
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 4, 2025
@efd6
sdd_all: use triple-brace templating in complex templates (elastic#11315
638ffbb 
)

This is a second pass at replacing double brace template marks with triple brace
marks. It addresses more complex uses that the sed rewrite did not find by using
PCRE via perl.

[git-generate]
for f in $(
	(
		for p in $(
			yq 'select(.owner.github == "elastic/sec-deployment-and-devices")|.name' packages/**/manifest.yml \
			| grep -v -- '---'
		); do
			rg -l -g '*.yml' "[^{]\{\{[^{][ .a-zA-Z0-9_]*[^}]}}[^}]" packages/$p
		done
	)|grep "elasticsearch/ingest_pipeline"|sort|uniq
); do
	perl -pi -e 's/(?<!\{)(\{\{[^{][ .a-zA-Z0-9_]*[^}]}})(?!\})/{$1}/g' $f
done
for p in $(git diff --name-only HEAD~1|cut -d/ -f1,2|sort|uniq); do
	(
		cd $p
		elastic-package test pipeline -g
		elastic-package changelog add \
			--description "Use triple-brace Mustache templating when referencing variables in ingest pipelines." \
			--type bugfix \
			--next patch \
			--link elastic#11315
	)>/dev/null 2>&1
done
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 5, 2025
@efd6
sdd_all: use triple-brace templating in complex templates (elastic#11315
7487f40 
)

This is a second pass at replacing double brace template marks with triple brace
marks. It addresses more complex uses that the sed rewrite did not find by using
PCRE via perl.

[git-generate]
for f in $(
	(
		for p in $(
			yq 'select(.owner.github == "elastic/sec-deployment-and-devices")|.name' packages/**/manifest.yml \
			| grep -v -- '---'
		); do
			rg -l -g '*.yml' "[^{]\{\{[^{][ .a-zA-Z0-9_]*[^}]}}[^}]" packages/$p
		done
	)|grep "elasticsearch/ingest_pipeline"|sort|uniq
); do
	perl -pi -e 's/(?<!\{)(\{\{[^{][ .a-zA-Z0-9_]*[^}]}})(?!\})/{$1}/g' $f
done
for p in $(git diff --name-only HEAD~1|cut -d/ -f1,2|sort|uniq); do
	(
		cd $p
		elastic-package test pipeline -g
		elastic-package changelog add \
			--description "Use triple-brace Mustache templating when referencing variables in ingest pipelines." \
			--type bugfix \
			--next patch \
			--link elastic#11315
	)>/dev/null 2>&1
done
@efd6
efd6 deleted the 7641-sdd_all_complex branch February 5, 2025 22:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix Pull request that fixes a bug issue Integration:cef Common Event Format (CEF) Integration:checkpoint Check Point Integration:cisco_ftd Cisco FTD Integration:cisco_ios Cisco IOS Integration:cisco_ise Cisco ISE Integration:cisco_secure_email_gateway Cisco Secure Email Gateway Integration:goflow2 GoFlow2 logs (Community supported) Integration:iptables Iptables Integration:juniper_srx Juniper SRX Integration:modsecurity ModSecurity Audit (Community supported) Integration:pfsense pfSense (Community supported) Integration:sophos Sophos Integration:zeek Zeek Team:Security-Deployment and Devices DEPRECATED Deployment and Devices Security team [elastic/sec-deployment-and-devices]

3 participants

@efd6 @elasticmachine @andrewkroh