Skip to content

Nginx module: use first not private IP address as remote_ip - #4417

Merged
exekias merged 1 commit into
elastic:masterfrom
tsg:nginx_use_first_non_private_ip
Jun 2, 2017
Merged

Nginx module: use first not private IP address as remote_ip#4417
exekias merged 1 commit into
elastic:masterfrom
tsg:nginx_use_first_non_private_ip

Conversation

@tsg

@tsg tsg commented May 29, 2017

Copy link
Copy Markdown
Contributor

A common customization to the nginx logs is to add the contents
of the X-Forwarded-For header in front of the remote IPs. This
typically results in a list of remote IPs.

This adds a new field remote_ip_list which is an array, and uses
a Painless script to automatically select the first non-private
IP for the remote_ip field, which is the field on which GeoIP is
applied.

Fixes #4322.
@tsg tsg added Filebeat Filebeat needs_backport PR is waiting to be backported to other branches. review v5.5.0 v6.0.0-alpha2 labels May 29, 2017
@tsg
tsg force-pushed the nginx_use_first_non_private_ip branch from 13a3b52 to 038ae1e Compare May 29, 2017 22:30
exekias
exekias reviewed May 30, 2017
View reviewed changes
Comment thread filebeat/module/nginx/access/test/test.log-expected.json Outdated

@exekias exekias May 30, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please check this, there is a bad merge and tests are failing
@tsg
tsg force-pushed the nginx_use_first_non_private_ip branch from 038ae1e to c1eadae Compare May 30, 2017 08:15
dedemorton
dedemorton reviewed May 30, 2017
View reviewed changes
Comment thread filebeat/docs/fields.asciidoc Outdated

@dedemorton dedemorton May 30, 2017 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So if there are no public IP addresses in remote_ip_list, this field contains a private IP address, right? If that's true, we should be a bit more direct here and say something like:

The first public IP address from the remote_ip_list array. If no public IP addresses are present, this field contains the first private IP address from the remote_ip_list array.
@tsg
tsg force-pushed the nginx_use_first_non_private_ip branch from c1eadae to 5e8f61e Compare May 31, 2017 10:18
Tudor Golubenco
Nginx module: use first not private IP address as remote_ip
9011ed7 
A common customization to the nginx logs is to add the contents
of the X-Forwarded-For header in front of the remote IPs. This
typically results in a list of remote IPs.

This adds a new field `remote_ip_list` which is an array, and uses
a Painless script to automatically select the first non-private
IP for the `remote_ip` field, which is the field on which GeoIP is
applied.

Fixes elastic#4322.
@tsg
tsg force-pushed the nginx_use_first_non_private_ip branch from 5e8f61e to 9011ed7 Compare May 31, 2017 10:19
@tsg

tsg commented May 31, 2017

Copy link
Copy Markdown
Contributor Author

Comments addressed.
exekias
exekias approved these changes May 31, 2017
View reviewed changes

@exekias exekias left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
@exekias
exekias merged commit a2c162f into elastic:master Jun 2, 2017
@tsg tsg added v5.6.0 and removed v5.5.0 labels Jul 19, 2017
tsg added a commit to tsg/beats that referenced this pull request Jul 19, 2017
@tsg Tudor Golubenco
Nginx module: use first not private IP address as remote_ip (elastic#…
91dd174 
…4417)

A common customization to the nginx logs is to add the contents
of the X-Forwarded-For header in front of the remote IPs. This
typically results in a list of remote IPs.

This adds a new field `remote_ip_list` which is an array, and uses
a Painless script to automatically select the first non-private
IP for the `remote_ip` field, which is the field on which GeoIP is
applied.

Fixes elastic#4322.
(cherry picked from commit a2c162f)
@tsg tsg mentioned this pull request Jul 19, 2017
Merged
@tsg tsg removed the needs_backport PR is waiting to be backported to other branches. label Jul 19, 2017
exekias pushed a commit that referenced this pull request Jul 19, 2017
@tsg @exekias
Cherry-pick #4417 to 5.x: Nginx module: use first not private IP addr…
5a3831c 
…ess as remote_ip (#4703)

* Nginx module: use first not private IP address as remote_ip (#4417)

A common customization to the nginx logs is to add the contents
of the X-Forwarded-For header in front of the remote IPs. This
typically results in a list of remote IPs.

This adds a new field `remote_ip_list` which is an array, and uses
a Painless script to automatically select the first non-private
IP for the `remote_ip` field, which is the field on which GeoIP is
applied.

Fixes #4322.
(cherry picked from commit a2c162f)
@redcinelli redcinelli mentioned this pull request May 2, 2022
Merged
8 tasks
redcinelli added a commit to elastic/integrations that referenced this pull request Aug 29, 2022
@redcinelli @andrewkroh
Update: Add support for x_forwarded_for headers in apaches access logs (
176203b 
#3251)

* Update: Add support for x_forwarded_for headers in apaches access logs

This Pr is heavily inpired by the work done in [this PR](elastic/beats#4417)
It is adressing [this ER](https://github.com/elastic/enhancements/issues/14402).

`Grok` pattern has been updated to match logs starting with a list of IP
adresses and store all those ip in `apache.access.remote_ip`.
This pattern is heavily insipred by the one in the nginx integration.

I also decided to fill a new field `network.forwarded_ip` as it seems to
be the perfect fit.

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
orestisfl pushed a commit to orestisfl/integrations that referenced this pull request May 15, 2026
@redcinelli @andrewkroh
Update: Add support for x_forwarded_for headers in apaches access logs (
a0654a8 
elastic#3251)

* Update: Add support for x_forwarded_for headers in apaches access logs

This Pr is heavily inpired by the work done in [this PR](elastic/beats#4417)
It is adressing [this ER](https://github.com/elastic/enhancements/issues/14402).

`Grok` pattern has been updated to match logs starting with a list of IP
adresses and store all those ip in `apache.access.remote_ip`.
This pattern is heavily insipred by the one in the nginx integration.

I also decided to fill a new field `network.forwarded_ip` as it seems to
be the perfect fit.

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment