Skip to content

Enhance error logs for invalid custom fields#235

Merged
kaisecheng merged 5 commits intologstash-plugins:mainfrom
kaisecheng:add_logs_invalid_custom_field
Nov 4, 2025
Merged

Enhance error logs for invalid custom fields#235
kaisecheng merged 5 commits intologstash-plugins:mainfrom
kaisecheng:add_logs_invalid_custom_field

Conversation

@kaisecheng
Copy link
Contributor

@kaisecheng kaisecheng commented Oct 30, 2025

add error logs when MaxMind databases fail to deserialize custom field

Fix: #233

How to test this PR

The instruction is copied from #226

  1. modify a sample test DB hosted in this project, updating a record with a custom information is_in_european_union
go build
./mmdb-from-go-blogpost
  1. run the following pipeline, setting the path to the DB file
input {
 generator {
   message => '{"ip": "216.160.83.58"}'
   codec => json
   count => 1
 }
}


filter {
 geoip {
   ecs_compatibility => disabled
   database => "/your/path/to/GeoIP2-Country-Test.mmdb"
   source => "ip"
   target => "[client][ip]"
 }
}

output {
 stdout{}
}
  1. Logstash should show error logs
[2025-10-30T15:54:30,874][ERROR][org.logstash.filters.geoip.GeoIPFilter][main][2e27f9fca6263b0b077b091c2ad8f92a639707d3727f9184c31c3ffa4de4b004] Error from GeoIP database. field=[ip], value=216.160.83.58
org.logstash.filters.geoip.GeoIPFilter$GeoIp2InvalidCustomFieldException: The database contains invalid custom field, which caused deserialization to fail.
	at org.logstash.filters.geoip.GeoIPFilter.retrieveCountryGeoData(GeoIPFilter.java:360) ~[logstash-filter-geoip-7.3.1.jar:?]
	at org.logstash.filters.geoip.GeoIPFilter.handleEvent(GeoIPFilter.java:179) ~[logstash-filter-geoip-7.3.1.jar:?]
	at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:580) ~[?:?]
	at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:285) ~[jruby.jar:?]
	at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:152) ~[jruby.jar:?]
	at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:24) ~[jruby.jar:?]
	at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:86) ~[jruby.jar:?]
	at org.jruby.ir.targets.indy.InvokeSite.performIndirectCall(InvokeSite.java:735) ~[jruby.jar:?]
	at org.jruby.ir.targets.indy.InvokeSite.invoke(InvokeSite.java:657) ~[jruby.jar:?]
	...
	at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111) ~[jruby.jar:?]
	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: java.lang.NullPointerException: Cannot invoke "Object.getClass()" because "parameters[index]" is null
	at com.maxmind.db.Decoder.decodeMapIntoObject(Decoder.java:446) ~[maxmind-db-2.1.0.jar:?]
	at com.maxmind.db.Decoder.decodeMap(Decoder.java:342) ~[maxmind-db-2.1.0.jar:?]
	at com.maxmind.db.Decoder.decodeByType(Decoder.java:162) ~[maxmind-db-2.1.0.jar:?]
	at com.maxmind.db.Decoder.decode(Decoder.java:151) ~[maxmind-db-2.1.0.jar:?]
	at com.maxmind.db.Decoder.decodeMapIntoObject(Decoder.java:429) ~[maxmind-db-2.1.0.jar:?]
	at com.maxmind.db.Decoder.decodeMap(Decoder.java:342) ~[maxmind-db-2.1.0.jar:?]
	at com.maxmind.db.Decoder.decodeByType(Decoder.java:162) ~[maxmind-db-2.1.0.jar:?]
	at com.maxmind.db.Decoder.decode(Decoder.java:151) ~[maxmind-db-2.1.0.jar:?]
	at com.maxmind.db.Decoder.decode(Decoder.java:76) ~[maxmind-db-2.1.0.jar:?]
	at com.maxmind.db.Reader.resolveDataPointer(Reader.java:275) ~[maxmind-db-2.1.0.jar:?]
	at com.maxmind.db.Reader.getRecord(Reader.java:185) ~[maxmind-db-2.1.0.jar:?]
	at com.maxmind.geoip2.DatabaseReader.get(DatabaseReader.java:263) ~[geoip2-2.17.0.jar:2.17.0]
	at com.maxmind.geoip2.DatabaseReader.getCountry(DatabaseReader.java:309) ~[geoip2-2.17.0.jar:2.17.0]
	at com.maxmind.geoip2.DatabaseReader.country(DatabaseReader.java:292) ~[geoip2-2.17.0.jar:2.17.0]
	at org.logstash.filters.geoip.GeoIPFilter.retrieveCountryGeoData(GeoIPFilter.java:358) ~[logstash-filter-geoip-7.3.1.jar:?]
	... 67 more
[2025-10-30T15:54:30,880][DEBUG][logstash.filters.geoip   ][main][2e27f9fca6263b0b077b091c2ad8f92a639707d3727f9184c31c3ffa4de4b004] IP 216.160.83.58 was not found in the database {:event=>#<LogStash::Event:0x69c6e28c>}
{
          "tags" => [
        [0] "_geoip_lookup_failure"
    ],
          "host" => {
        "name" => "your-mac-pro"
    },
    "@timestamp" => 2025-10-30T15:54:30.783252Z,
         "event" => {
        "original" => "{\"ip\": \"216.160.83.58\"}",
        "sequence" => 0
    },
      "@version" => "1",
        "source" => {},
            "ip" => "216.160.83.58"
}
@kaisecheng kaisecheng force-pushed the add_logs_invalid_custom_field branch from 1d48c77 to e4a5c39 Compare October 30, 2025 17:20
@kaisecheng kaisecheng requested a review from alexcams November 4, 2025 10:42
static class GeoIp2InvalidCustomFieldException extends GeoIp2Exception {
public GeoIp2InvalidCustomFieldException(Throwable cause) {
super("invalid custom field", cause);
super("The database contains invalid custom field, which caused deserialization to fail.", cause);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: There is double tab.

Copy link

@alexcams alexcams left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! with minor to remove double tab.

@kaisecheng kaisecheng merged commit d048685 into logstash-plugins:main Nov 4, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants