Skip to content

sdd_all: use triple-brace templating - #11286

Merged
efd6 merged 1 commit into
elastic:mainfrom
efd6:7641-sdd_all
Oct 2, 2024
Merged

sdd_all: use triple-brace templating#11286
efd6 merged 1 commit into
elastic:mainfrom
efd6:7641-sdd_all

Conversation

@efd6

@efd6 efd6 commented Oct 1, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor

Proposed commit message

sdd_all: use triple-brace templating

The mustache templating system used by ingest pipelines has two levels of
escaping available, not escaped (triple stache) and HTML escaped
(double stache) — see man mustache[1] under "tag types: variables". This can
lead to data corruption, particularly in cases where an operating system has
chosen to use a character requiring escaping in its path syntax.

[1]http://mustache.github.io/mustache.5.html

[git-generate]
for f in $(
	(
		for p in $(
			yq 'select(.owner.github == "elastic/sec-deployment-and-devices")|.name' packages/**/manifest.yml \
			| grep -v -- '---'
		); do
			rg -l -g '*.yml' ": ('\{\{[^{][ .a-zA-Z0-9_]*[^}]}}'|\"\{\{[^{][ .a-zA-Z0-9_]*[^}]}}\")" packages/$p
		done
	)|grep "elasticsearch/ingest_pipeline"|sort|uniq
); do
	sed -i -r "s/: (['\"])\{\{([^{][ .a-zA-Z0-9_]*[^}])}}['\"]/: \1{{{\2}}}\1/g" $f
done
for p in $(git diff --name-only HEAD~1|cut -d/ -f1,2|sort|uniq); do
	(
		cd $p
		elastic-package test pipeline -g
		elastic-package changelog add \
			--description "Use triple-brace Mustache templating when referencing variables in ingest pipelines." \
			--type bugfix \
			--next patch \
			--link https://github.com/elastic/integrations/pull/11286
	)>/dev/null 2>&1
done

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots
@efd6 efd6 added Integration:All Applies to all integrations [Integration not found in source] bugfix Pull request that fixes a bug issue Team:Security-Deployment and Devices DEPRECATED Deployment and Devices Security team [elastic/sec-deployment-and-devices] labels Oct 1, 2024
@efd6 efd6 self-assigned this Oct 1, 2024
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 1, 2024

Copy link
Copy Markdown
Contributor

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport
@andrewkroh andrewkroh removed the Integration:All Applies to all integrations [Integration not found in source] label Oct 1, 2024
@efd6
efd6 marked this pull request as ready for review October 1, 2024 10:15
@efd6
efd6 requested a review from a team as a code owner October 1, 2024 10:15
@elasticmachine

elasticmachine commented Oct 1, 2024

Copy link
Copy Markdown

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)
@efd6
sdd_all: use triple-brace templating
58de75f 
The mustache templating system used by ingest pipelines has two levels of
escaping available, not escaped (triple stache) and HTML escaped
(double stache) — see man mustache[1] under "tag types: variables". This can
lead to data corruption, particularly in cases where an operating system has
chosen to use a character requiring escaping in its path syntax.

[1]http://mustache.github.io/mustache.5.html

[git-generate]
for f in $(
	(
		for p in $(
			yq 'select(.owner.github == "elastic/sec-deployment-and-devices")|.name' packages/**/manifest.yml \
			| grep -v -- '---'
		); do
			rg -l -g '*.yml' ": ('\{\{[^{][ .a-zA-Z0-9_]*[^}]}}'|\"\{\{[^{][ .a-zA-Z0-9_]*[^}]}}\")" packages/$p
		done
	)|grep "elasticsearch/ingest_pipeline"|sort|uniq
); do
	sed -i -r "s/: (['\"])\{\{([^{][ .a-zA-Z0-9_]*[^}])}}['\"]/: \1{{{\2}}}\1/g" $f
done
for p in $(git diff --name-only HEAD~1|cut -d/ -f1,2|sort|uniq); do
	(
		cd $p
		elastic-package test pipeline -g
		elastic-package changelog add \
			--description "Use triple-brace Mustache templating when referencing variables in ingest pipelines." \
			--type bugfix \
			--next patch \
			--link elastic#11286
	)>/dev/null 2>&1
done
@elasticmachine

elasticmachine commented Oct 1, 2024

Copy link
Copy Markdown

💚 Build Succeeded

History

  • 💚 Build #16648 succeeded 434968daf8e76ed3cec2b74980162396ced41f17

cc @efd6
@elastic-sonarqube

elastic-sonarqube Bot commented Oct 1, 2024

Copy link
Copy Markdown

Quality Gate failed Quality Gate failed

Failed conditions
76.8% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube
@andrewkroh andrewkroh added the Integration:All Applies to all integrations [Integration not found in source] label Oct 2, 2024
@andrewkroh andrewkroh added Integration:radware Radware DefensePro Logs (Deprecated) Integration:juniper_netscreen Juniper NetScreen (Deprecated) Integration:osquery Osquery Logs Integration:fortinet_forticlient Fortinet FortiClient Logs (Deprecated) Integration:juniper_junos Juniper JunOS (Deprecated) Integration:cisco_ise Cisco ISE Integration:suricata Suricata Integration:netflow NetFlow Records Integration:sophos Sophos Integration:citrix_waf Citrix Web App Firewall Integration:fortinet_fortiedr Fortinet FortiEDR Logs and removed Integration:All Applies to all integrations [Integration not found in source] labels Oct 2, 2024
@andrewkroh andrewkroh added Integration:juniper_srx Juniper SRX Integration:pfsense pfSense (Community supported) Integration:snort Snort Integration:bluecoat Blue Coat Director Logs (Deprecated) Integration:fortinet_fortigate Fortinet FortiGate Firewall Logs labels Oct 2, 2024
@efd6
efd6 merged commit 992a01e into elastic:main Oct 2, 2024
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 2, 2024

Copy link
Copy Markdown
Contributor

Package bluecoat - 0.17.3 containing this change is available at https://epr.elastic.co/search?package=bluecoat
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 2, 2024

Copy link
Copy Markdown
Contributor

Package cef - 2.17.3 containing this change is available at https://epr.elastic.co/search?package=cef
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 2, 2024

Copy link
Copy Markdown
Contributor

Package checkpoint - 1.34.1 containing this change is available at https://epr.elastic.co/search?package=checkpoint
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 2, 2024

Copy link
Copy Markdown
Contributor

Package cisco_ise - 1.23.1 containing this change is available at https://epr.elastic.co/search?package=cisco_ise
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 2, 2024

Copy link
Copy Markdown
Contributor

Package citrix_waf - 1.16.1 containing this change is available at https://epr.elastic.co/search?package=citrix_waf
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 2, 2024

Copy link
Copy Markdown
Contributor

Package fortinet_forticlient - 1.10.3 containing this change is available at https://epr.elastic.co/search?package=fortinet_forticlient
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 2, 2024

Copy link
Copy Markdown
Contributor

Package fortinet_fortiedr - 1.15.1 containing this change is available at https://epr.elastic.co/search?package=fortinet_fortiedr
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 2, 2024

Copy link
Copy Markdown
Contributor

Package fortinet_fortigate - 1.25.8 containing this change is available at https://epr.elastic.co/search?package=fortinet_fortigate
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 2, 2024

Copy link
Copy Markdown
Contributor

Package juniper_junos - 0.10.3 containing this change is available at https://epr.elastic.co/search?package=juniper_junos
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 2, 2024

Copy link
Copy Markdown
Contributor

Package juniper_netscreen - 0.10.3 containing this change is available at https://epr.elastic.co/search?package=juniper_netscreen
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 2, 2024

Copy link
Copy Markdown
Contributor

Package juniper_srx - 1.21.1 containing this change is available at https://epr.elastic.co/search?package=juniper_srx
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 2, 2024

Copy link
Copy Markdown
Contributor

Package modsecurity - 1.18.1 containing this change is available at https://epr.elastic.co/search?package=modsecurity
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 2, 2024

Copy link
Copy Markdown
Contributor

Package netflow - 2.19.1 containing this change is available at https://epr.elastic.co/search?package=netflow
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 2, 2024

Copy link
Copy Markdown
Contributor

Package netscout - 0.20.1 containing this change is available at https://epr.elastic.co/search?package=netscout
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 2, 2024

Copy link
Copy Markdown
Contributor

Package osquery - 1.19.1 containing this change is available at https://epr.elastic.co/search?package=osquery
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 2, 2024

Copy link
Copy Markdown
Contributor

Package pfsense - 1.20.1 containing this change is available at https://epr.elastic.co/search?package=pfsense
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 2, 2024

Copy link
Copy Markdown
Contributor

Package qnap_nas - 1.20.1 containing this change is available at https://epr.elastic.co/search?package=qnap_nas
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 2, 2024

Copy link
Copy Markdown
Contributor

Package radware - 0.19.1 containing this change is available at https://epr.elastic.co/search?package=radware
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 2, 2024

Copy link
Copy Markdown
Contributor

Package snort - 1.15.1 containing this change is available at https://epr.elastic.co/search?package=snort
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 2, 2024

Copy link
Copy Markdown
Contributor

Package sophos - 3.9.1 containing this change is available at https://epr.elastic.co/search?package=sophos
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 2, 2024

Copy link
Copy Markdown
Contributor

Package suricata - 2.21.3 containing this change is available at https://epr.elastic.co/search?package=suricata
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 4, 2025
@efd6
sdd_all: use triple-brace templating (elastic#11286)
1fa8dde 
The mustache templating system used by ingest pipelines has two levels of
escaping available, not escaped (triple stache) and HTML escaped
(double stache) — see man mustache[1] under "tag types: variables". This can
lead to data corruption, particularly in cases where an operating system has
chosen to use a character requiring escaping in its path syntax.

[1]http://mustache.github.io/mustache.5.html

[git-generate]
for f in $(
	(
		for p in $(
			yq 'select(.owner.github == "elastic/sec-deployment-and-devices")|.name' packages/**/manifest.yml \
			| grep -v -- '---'
		); do
			rg -l -g '*.yml' ": ('\{\{[^{][ .a-zA-Z0-9_]*[^}]}}'|\"\{\{[^{][ .a-zA-Z0-9_]*[^}]}}\")" packages/$p
		done
	)|grep "elasticsearch/ingest_pipeline"|sort|uniq
); do
	sed -i -r "s/: (['\"])\{\{([^{][ .a-zA-Z0-9_]*[^}])}}['\"]/: \1{{{\2}}}\1/g" $f
done
for p in $(git diff --name-only HEAD~1|cut -d/ -f1,2|sort|uniq); do
	(
		cd $p
		elastic-package test pipeline -g
		elastic-package changelog add \
			--description "Use triple-brace Mustache templating when referencing variables in ingest pipelines." \
			--type bugfix \
			--next patch \
			--link elastic#11286
	)>/dev/null 2>&1
done
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 5, 2025
@efd6
sdd_all: use triple-brace templating (elastic#11286)
fe6a618 
The mustache templating system used by ingest pipelines has two levels of
escaping available, not escaped (triple stache) and HTML escaped
(double stache) — see man mustache[1] under "tag types: variables". This can
lead to data corruption, particularly in cases where an operating system has
chosen to use a character requiring escaping in its path syntax.

[1]http://mustache.github.io/mustache.5.html

[git-generate]
for f in $(
	(
		for p in $(
			yq 'select(.owner.github == "elastic/sec-deployment-and-devices")|.name' packages/**/manifest.yml \
			| grep -v -- '---'
		); do
			rg -l -g '*.yml' ": ('\{\{[^{][ .a-zA-Z0-9_]*[^}]}}'|\"\{\{[^{][ .a-zA-Z0-9_]*[^}]}}\")" packages/$p
		done
	)|grep "elasticsearch/ingest_pipeline"|sort|uniq
); do
	sed -i -r "s/: (['\"])\{\{([^{][ .a-zA-Z0-9_]*[^}])}}['\"]/: \1{{{\2}}}\1/g" $f
done
for p in $(git diff --name-only HEAD~1|cut -d/ -f1,2|sort|uniq); do
	(
		cd $p
		elastic-package test pipeline -g
		elastic-package changelog add \
			--description "Use triple-brace Mustache templating when referencing variables in ingest pipelines." \
			--type bugfix \
			--next patch \
			--link elastic#11286
	)>/dev/null 2>&1
done
@efd6
efd6 deleted the 7641-sdd_all branch February 5, 2025 22:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix Pull request that fixes a bug issue Integration:bluecoat Blue Coat Director Logs (Deprecated) Integration:cef Common Event Format (CEF) Integration:checkpoint Check Point Integration:cisco_ise Cisco ISE Integration:citrix_waf Citrix Web App Firewall Integration:fortinet_forticlient Fortinet FortiClient Logs (Deprecated) Integration:fortinet_fortiedr Fortinet FortiEDR Logs Integration:fortinet_fortigate Fortinet FortiGate Firewall Logs Integration:juniper_junos Juniper JunOS (Deprecated) Integration:juniper_netscreen Juniper NetScreen (Deprecated) Integration:juniper_srx Juniper SRX Integration:modsecurity ModSecurity Audit (Community supported) Integration:netflow NetFlow Records Integration:netscout Arbor Peakflow SP Logs (Deprecated) Integration:osquery Osquery Logs Integration:pfsense pfSense (Community supported) Integration:qnap_nas QNAP NAS (Community supported) Integration:radware Radware DefensePro Logs (Deprecated) Integration:snort Snort Integration:sophos Sophos Integration:suricata Suricata Team:Security-Deployment and Devices DEPRECATED Deployment and Devices Security team [elastic/sec-deployment-and-devices]

3 participants

@efd6 @elasticmachine @andrewkroh