Hash functions by idegtiarenko · Pull Request #118938 · elastic/elasticsearch

idegtiarenko · 2024-12-18T11:00:59Z

This change adds infrastructure to add more hash functions easier as well as md5 implementation to demonstrate it.

github-actions · 2024-12-18T11:01:13Z

Documentation preview:

✨ Changed pages

elasticsearchmachine · 2024-12-18T11:01:24Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine · 2024-12-18T11:01:24Z

Hi @idegtiarenko, I've created a changelog YAML for you.

idegtiarenko · 2024-12-18T11:08:34Z

While this is open we might add couple more functions as well.

This digests are defined and used in ES:

elasticsearch/server/src/main/java/org/elasticsearch/common/hash/MessageDigests.java

Lines 41 to 44 in a59c182

    
           private static final ThreadLocal<MessageDigest> MD5_DIGEST = createThreadLocalMessageDigest("MD5"); 
        
           private static final ThreadLocal<MessageDigest> SHA_1_DIGEST = createThreadLocalMessageDigest("SHA-1"); 
        
           private static final ThreadLocal<MessageDigest> SHA_256_DIGEST = createThreadLocalMessageDigest("SHA-256"); 
        
           private static final ThreadLocal<MessageDigest> SHA_512_DIGEST = createThreadLocalMessageDigest("SHA-512");

Following algorithms seems to be available in several JVMs I checked:

SHA-1
SHA-224
SHA-256
SHA-384
SHA-512/256
SHA-512/224
SHA-512
SHA3-224
SHA3-384
SHA3-256
SHA3-512

ivancea

Looks good!

ivancea · 2024-12-31T11:22:53Z

...n/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/string/Md5.java

+
+    private static final HashFunction MD5 = HashFunction.create("MD5");
+
+    @FunctionInfo(returnType = "keyword", description = "Computes MD5 hash of the input.")


We can add the md5Hash test as an examples = @Example(...) here

ivancea · 2024-12-31T11:24:13Z

...n/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/string/Md5.java

+
+    @Override
+    public Expression replaceChildren(List<Expression> newChildren) {
+        return new Md5(source(), field);


Suggested change

return new Md5(source(), field);

return new Md5(source(), newChildren.get(0));

Edit: Oh, looks like EsqlNodeSubclassTests.testReplaceChildren failed successfully with this in CI

ivancea · 2024-12-31T11:30:37Z

...n/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/string/Md5.java

+
+    private static final HashFunction MD5 = HashFunction.create("MD5");
+
+    @FunctionInfo(returnType = "keyword", description = "Computes MD5 hash of the input.")


Suggested change

@FunctionInfo(returnType = "keyword", description = "Computes MD5 hash of the input.")

@FunctionInfo(returnType = "keyword", description = "Computes the MD5 hash of the input.")

ivancea · 2024-12-31T11:51:47Z

.../src/test/java/org/elasticsearch/xpack/esql/expression/function/scalar/string/HashTests.java

    }

+    static List<TestCaseSupplier> createHashFunctionTestCases(String algorithm) {
+        return List.of(createHashFunctionTestCase(algorithm, DataType.KEYWORD), createHashFunctionTestCase(algorithm, DataType.TEXT));


You can use TestCaseSupplier.forUnaryStrings(...) instead of manually creating the TestCaseSuppliers. It will also take care of both KEYWORD and TEXT types

ivancea · 2024-12-31T11:58:00Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/hash.csv-spec

I would add some extra test cases, to double check both the:

One with a FROM command

Something like FROM x | EVAL input = "...", hash = MD5(input)? Trying to "break" the replaceChildren() comment bug with it

MD5(123::keyword)

Maybe having MD5() as a STATS grouping

nik9000 · 2025-01-02T13:08:09Z

...n/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/string/Md5.java

+
+    private static final HashFunction MD5 = HashFunction.create("MD5");
+
+    @FunctionInfo(returnType = "keyword", description = "Computes MD5 hash of the input.")


nik9000 · 2025-01-02T13:08:41Z

...n/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/string/Md5.java

+
+    @Override
+    public Expression replaceChildren(List<Expression> newChildren) {
+        return new Md5(source(), field);


costin · 2025-01-08T06:26:56Z

Let's get this in - this will be handy for security queries!

nik9000

LGTM. Could you link the other new hash functions in? Maybe add an example for them?

nik9000 · 2025-01-08T14:16:13Z

docs/reference/esql/functions/string-functions.asciidoc

 include::layout/length.asciidoc[]
 include::layout/locate.asciidoc[]
 include::layout/ltrim.asciidoc[]
+include::layout/md5.asciidoc[]


Can you link the other new functions?

elasticsearchmachine · 2025-01-08T15:45:39Z

💚 Backport successful

Status	Branch	Result
✅	8.x

This change adds md5, sha1 and sha256 hash functions.

Hash functions

c3e18d5

idegtiarenko added >enhancement Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) auto-backport Automatically create backport pull requests when merged :Analytics/ES|QL AKA ESQL v9.0.0 v8.18.0 labels Dec 18, 2024

idegtiarenko requested a review from nik9000 December 18, 2024 11:00

Update docs/changelog/118938.yaml

3eb01a9

simplify constructor selection

8d52bc7

ivancea approved these changes Dec 31, 2024

View reviewed changes

idegtiarenko added 2 commits January 2, 2025 13:40

Merge branch 'main' into hash_functions

dfc536d

fix functions count

961eda1

nik9000 approved these changes Jan 2, 2025

View reviewed changes

idegtiarenko added 6 commits January 2, 2025 14:13

fix replace children

e2877ec

add examples

9a9bc75

simplify test case creation

00fce7f

Merge branch 'main' into hash_functions

d62705a

add missing required capability

60834e8

Merge branch 'main' into hash_functions

13fbbfa

idegtiarenko added 4 commits January 8, 2025 11:11

Merge branch 'main' into hash_functions

5efa383

rename the capability

89be7d0

sha1 and sha256

8caebb5

update function count

4cba449

nik9000 approved these changes Jan 8, 2025

View reviewed changes

idegtiarenko added 5 commits January 8, 2025 15:24

link new hash functions

500658f

Merge branch 'main' into hash_functions

016aacb

add hash functions to the list

4d7c381

Merge branch 'main' into hash_functions

1ba0d75

use new api

f0b0deb

idegtiarenko merged commit fd1be8c into elastic:main Jan 8, 2025

idegtiarenko mentioned this pull request Jan 8, 2025

[8.x] Hash functions (#118938) #119769

Merged

idegtiarenko deleted the hash_functions branch January 8, 2025 15:46

idegtiarenko added a commit to idegtiarenko/elasticsearch that referenced this pull request Jan 8, 2025

Hash functions (elastic#118938)

6236156

This change adds md5, sha1 and sha256 hash functions.

elasticsearchmachine pushed a commit that referenced this pull request Jan 9, 2025

Hash functions (#118938) (#119769)

b398448

This change adds md5, sha1 and sha256 hash functions.

alex-spies mentioned this pull request Jan 20, 2025

[ES|QL] Hash/Fingerprint Function #115719

Closed

tvernum mentioned this pull request Jun 19, 2025

Don't enable ES|QL MD5 function in FIPS mode #129689

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hash functions#118938

Hash functions#118938
idegtiarenko merged 20 commits intoelastic:mainfrom
idegtiarenko:hash_functions

idegtiarenko commented Dec 18, 2024

github-actions bot commented Dec 18, 2024

elasticsearchmachine commented Dec 18, 2024

elasticsearchmachine commented Dec 18, 2024

idegtiarenko commented Dec 18, 2024

ivancea left a comment

ivancea Dec 31, 2024

nik9000 Jan 2, 2025

ivancea Dec 31, 2024

nik9000 Jan 2, 2025

ivancea Dec 31, 2024

ivancea Dec 31, 2024

ivancea Dec 31, 2024

nik9000 Jan 2, 2025

nik9000 Jan 2, 2025

costin commented Jan 8, 2025

nik9000 left a comment

nik9000 Jan 8, 2025

elasticsearchmachine commented Jan 8, 2025

Labels

5 participants


		private static final HashFunction MD5 = HashFunction.create("MD5");

		@FunctionInfo(returnType = "keyword", description = "Computes MD5 hash of the input.")

	return new Md5(source(), field);
	return new Md5(source(), newChildren.get(0));

Conversation

idegtiarenko commented Dec 18, 2024

github-actions bot commented Dec 18, 2024

elasticsearchmachine commented Dec 18, 2024

elasticsearchmachine commented Dec 18, 2024

idegtiarenko commented Dec 18, 2024

ivancea left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

costin commented Jan 8, 2025

nik9000 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticsearchmachine commented Jan 8, 2025

💚 Backport successful

Labels

5 participants