Skip to content

[vsphere] Parse space-separated log timestamps - #19712

Merged
mykola-elastic merged 8 commits into
elastic:mainfrom
mykola-elastic:vsphere-parse-datetime-bug
Jun 25, 2026
Merged

[vsphere] Parse space-separated log timestamps#19712
mykola-elastic merged 8 commits into
elastic:mainfrom
mykola-elastic:vsphere-parse-datetime-bug

Conversation

@mykola-elastic

@mykola-elastic mykola-elastic commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

The log ingest pipeline's grok captures timestamps via TIMESTAMP_ISO8601, whose pattern allows a space separator (e.g. "2026-06-03 14:19:27.782"), but the date processor only declared ISO8601 (requires a "T" separator) and "MMM dd HH:mm:ss", so space-separated timestamps failed with "unable to parse date".

Add "yyyy-MM-dd HH:mm:ss.SSS" and "yyyy-MM-dd HH:mm:ss" to the date processor formats, with a regression test case.

Proposed commit message

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices
mykola-elastic and others added 2 commits June 23, 2026 11:52
@mykola-elastic @claude
[vsphere] Parse space-separated log timestamps
5c5fb13 
The log ingest pipeline's grok captures timestamps via TIMESTAMP_ISO8601,
whose pattern allows a space separator (e.g. "2026-06-03 14:19:27.782"),
but the date processor only declared ISO8601 (requires a "T" separator)
and "MMM dd HH:mm:ss", so space-separated timestamps failed with
"unable to parse date".

Add "yyyy-MM-dd HH:mm:ss.SSS" and "yyyy-MM-dd HH:mm:ss" to the date
processor formats, with a regression test case.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mykola-elastic
add pr number
980f17a
@mykola-elastic
mykola-elastic marked this pull request as ready for review June 23, 2026 08:55
@mykola-elastic
mykola-elastic requested a review from a team as a code owner June 23, 2026 08:55
@mykola-elastic

mykola-elastic commented Jun 23, 2026

Copy link
Copy Markdown
Contributor Author

/test
@github-actions

github-actions Bot commented Jun 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport
@mykola-elastic
mykola-elastic requested a review from shmsr June 23, 2026 10:17
@andrewkroh andrewkroh added Integration:vsphere VMware vSphere Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] labels Jun 23, 2026
stefans-elastic
stefans-elastic approved these changes Jun 24, 2026
View reviewed changes

@stefans-elastic stefans-elastic left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm
shmsr
shmsr reviewed Jun 25, 2026
View reviewed changes
Comment thread packages/vsphere/changelog.yml Outdated
shmsr
shmsr reviewed Jun 25, 2026
View reviewed changes

@shmsr shmsr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix — the root cause diagnosis is correct and the format ordering is right (T-separator ISO8601 first, then millis-precision before second-precision for the space-separated variants). Left a few inline comments below.
Comment thread packages/vsphere/data_stream/log/elasticsearch/ingest_pipeline/default.yml
Comment thread packages/vsphere/changelog.yml Outdated
Comment thread packages/vsphere/data_stream/log/_dev/test/pipeline/test-format-common.log
@mykola-elastic @shmsr
Update packages/vsphere/changelog.yml
d04b6d4 
Co-authored-by: subham sarkar <sarkar.subhams2@gmail.com>
@github-actions

This comment has been minimized.

Sign in to view
@github-actions

github-actions Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

TL;DR

Check integrations vsphere failed during package lint because packages/vsphere/changelog.yml is not valid YAML. Quote the new changelog description value; the unquoted text contains for example: , and that colon-space sequence is parsed as a mapping separator.

Remediation

  • Change packages/vsphere/changelog.yml:4 to quote the full description, for example:
    - description: 'Parse space-separated vSphere log timestamps (for example: "yyyy-MM-dd HH:mm:ss.SSS") in the log ingest pipeline.'
  • Re-run the vsphere package check after updating the changelog, e.g. the same CI command: .buildkite/scripts/test_one_package.sh packages/vsphere origin/main d04b6d425c3a56a25deeecf9375cf68365062856.
Investigation details

Root Cause

The new changelog entry in packages/vsphere/changelog.yml:4 is a plain YAML scalar:

    - description: Parse space-separated vSphere log timestamps (for example: "yyyy-MM-dd HH:mm:ss.SSS") in the log ingest pipeline.

Inside a plain scalar, : is significant YAML syntax. The substring for example: "yyyy..." therefore makes the changelog invalid before package tests can run.

Evidence

Error: checking package failed: linting package failed: found 3 validation errors:

1. file ".../packages/vsphere/changelog.yml" is invalid: unmarshalling YAML file failed: yaml: line 4: mapping values are not allowed in this context
2. can't read changelog entries: unmarshalling YAML file failed (.../packages/vsphere/changelog.yml): yaml: line 4: mapping values are not allowed in this context
3. can't read changelog entries: unmarshalling YAML file failed (.../packages/vsphere/changelog.yml): yaml: line 4: mapping values are not allowed in this context

Verification

  • Reproduced the YAML parser error with the current changelog snippet: mapping values are not allowed in this context at line 4 column 77.
  • Confirmed the same snippet parses after quoting the description.

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
@mykola-elastic @claude
[vsphere] Address review: timezone-aware formats, tests, changelog
817c8f0 
- Add timezone-aware space-separated date formats so values like
  "2026-06-03 14:19:27.782+05:30", "...+0530" and "...Z" (which grok
  captures via TIMESTAMP_ISO8601's optional ISO8601_TIMEZONE) parse
  instead of failing the date processor.
- Add pipeline test lines covering every new format (no-millis,
  millis/no-millis with +HH:MM and +HHMM offsets).
- Changelog: use a real timestamp in the example, and quote the
  description (the "for example:" colon made the YAML invalid).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mykola-elastic
mykola-elastic requested a review from shmsr June 25, 2026 07:34
shmsr
shmsr reviewed Jun 25, 2026
View reviewed changes
Comment thread packages/vsphere/changelog.yml Outdated
@mykola-elastic @shmsr
Update packages/vsphere/changelog.yml
f08852f 
Co-authored-by: subham sarkar <sarkar.subhams2@gmail.com>
@mykola-elastic @claude
[vsphere] Quote changelog description to fix invalid YAML
3683277 
The "for example:" colon makes the description a plain scalar containing
": ", which YAML parses as a mapping (and the backtick is a reserved
start char), breaking elastic-package lint. Single-quote the value.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mykola-elastic
mykola-elastic enabled auto-merge (squash) June 25, 2026 08:10
@mykola-elastic

mykola-elastic commented Jun 25, 2026

Copy link
Copy Markdown
Contributor Author

/test
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

✅ All changelog entries have the correct PR link.
@infra-vault-gh-plugin-prod

infra-vault-gh-plugin-prod Bot commented Jun 25, 2026

Copy link
Copy Markdown

�� Build Succeeded

History

cc @mykola-elastic
@mykola-elastic
mykola-elastic merged commit 0c04fa8 into elastic:main Jun 25, 2026
10 checks passed
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Package vsphere - 1.24.1 containing this change is available at https://epr.elastic.co/package/vsphere/1.24.1/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Integration:vsphere VMware vSphere Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations]

5 participants

@mykola-elastic @shmsr @ishleenk17 @stefans-elastic @andrewkroh