Add deadlock event with JVMTI-captured stacktraces - #470
Draft
jbachorik wants to merge 4 commits into
Draft
Conversation
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
jbachorik
commented
Apr 14, 2026
jbachorik
commented
Apr 14, 2026
jbachorik
commented
Apr 14, 2026
- Simplify lock_name truncation with inline min() - Add JVMTI safepoint warning on captureStackTrace - Move DeadlockDetector to test-only; profiler exposes only recordDeadlock API - Remove startDeadlockDetector/stopDeadlockDetector from JavaProfiler - Use before()/after() hooks in test Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
jbachorik
marked this pull request as ready for review
April 14, 2026 15:28
Contributor
CI Test ResultsRun: #24452346678 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled Summary: Total: 32 | Passed: 32 | Failed: 0 Updated: 2026-04-15 12:05:20 UTC |
jbachorik
marked this pull request as draft
April 14, 2026 16:52
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Collaborator
Author
|
@codex review |
![chatgpt-codex-connector[bot]](https://cdn.statically.io/img/avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e1d6653224
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new native JFR event (datadog.DeadlockedThread) intended to record detected deadlocks with JVMTI-captured stack traces, plus a test that creates a real deadlock and validates events are emitted.
Changes:
- Introduces a new JFR event type (
T_DEADLOCK) and metadata schema fordatadog.DeadlockedThread. - Adds a JNI entrypoint and native implementation to capture JVMTI stack traces for the deadlocked thread and the lock owner, storing them in
CallTraceStorage. - Adds a test-only deadlock detector and a JUnit test that triggers a deadlock and verifies the event appears in the produced JFR.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| ddprof-test/src/test/java/com/datadoghq/profiler/deadlock/DeadlockDetector.java | Test-only polling detector that finds deadlocks via ThreadMXBean and calls JavaProfiler.recordDeadlock. |
| ddprof-test/src/test/java/com/datadoghq/profiler/deadlock/DeadlockDetectionTest.java | Creates a real 2-thread monitor deadlock and asserts datadog.DeadlockedThread events exist with stack traces and consistent deadlock IDs. |
| ddprof-lib/src/main/java/com/datadoghq/profiler/JavaProfiler.java | Adds public recordDeadlock(...) API and declares native recordDeadlock0(...). |
| ddprof-lib/src/main/cpp/profiler.h | Declares Profiler::recordDeadlock(...) and Profiler::recordDeadlockWithCapture(...). |
| ddprof-lib/src/main/cpp/profiler.cpp | Implements JVMTI GetStackTrace capture and deadlock recording plumbing into the flight recorder. |
| ddprof-lib/src/main/cpp/jfrMetadata.h | Adds T_DEADLOCK enum constant. |
| ddprof-lib/src/main/cpp/jfrMetadata.cpp | Registers datadog.DeadlockedThread event metadata and fields. |
| ddprof-lib/src/main/cpp/javaApi.cpp | Adds JNI bridge JavaProfiler_recordDeadlock0 to invoke native capture+record. |
| ddprof-lib/src/main/cpp/flightRecorder.h | Declares deadlock recording methods in Recording/FlightRecorder. |
| ddprof-lib/src/main/cpp/flightRecorder.cpp | Serializes the deadlock event payload and ensures involved threads are added to the recording. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+1655
to
+1657
| // Truncate to fit within the single-byte event size prefix (max 255 bytes total) | ||
| buf->putUtf8(lock_name, min((size_t)160, strlen(lock_name))); | ||
| buf->putVar64(lock_owner_tid); |
Comment on lines
+78
to
+82
| bothLocked.await(); | ||
| Thread.sleep(200); // Let the deadlock form | ||
|
|
||
| new DeadlockDetector(profiler).check(); | ||
| } |
Comment on lines
+108
to
+130
| IMemberAccessor<String, IItem> lockNameAccessor = | ||
| LOCK_NAME.getAccessor(items.getType()); | ||
| IMemberAccessor<IMCStackTrace, IItem> stackTraceAccessor = | ||
| STACK_TRACE.getAccessor(items.getType()); | ||
|
|
||
| for (IItem item : items) { | ||
| if (deadlockIdAccessor != null) { | ||
| IQuantity id = deadlockIdAccessor.getMember(item); | ||
| if (id != null) { | ||
| deadlockIds.add(id.longValue()); | ||
| } | ||
| } | ||
| if (lockNameAccessor != null) { | ||
| String lockName = lockNameAccessor.getMember(item); | ||
| assertNotNull(lockName, "Lock name should not be null"); | ||
| assertFalse(lockName.isEmpty(), "Lock name should not be empty"); | ||
| } | ||
| if (stackTraceAccessor != null) { | ||
| IMCStackTrace st = stackTraceAccessor.getMember(item); | ||
| if (st != null && st.getFrames() != null && !st.getFrames().isEmpty()) { | ||
| hasStackTraces = true; | ||
| } | ||
| } |
Comment on lines
228
to
+239
| /** | ||
| * Records a deadlock event for a pair of deadlocked threads. | ||
| * The native side captures real stacktraces via JVMTI for both threads. | ||
| * | ||
| * @param thread the deadlocked thread | ||
| * @param lockName the lock this thread is waiting on | ||
| * @param lockOwnerThread the thread holding the lock | ||
| * @param deadlockId correlation ID grouping events in the same deadlock cycle | ||
| */ | ||
| public void recordDeadlock(Thread thread, String lockName, Thread lockOwnerThread, long deadlockId) { | ||
| recordDeadlock0(thread, lockName, lockOwnerThread, deadlockId); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?:
Adds a native JFR deadlock event (
datadog.DeadlockedThread) that captures realJVMTI stacktraces for both the deadlocked thread and the lock owner. This provides
proper navigable stacktraces in JFR tools — a significant improvement over the
string-based stacks in the dd-trace-java fallback implementation.
Key components:
T_DEADLOCKJFR event type with fields: deadlockId, eventThread, stackTrace,lockName, lockOwnerThread, lockOwnerStackTrace
GetStackTrace()capture for both threads in the deadlock pair, stored inthe profiler's
CallTraceStorageas real stacktrace constant pool referencesDeadlockDetectorJava class usingThreadMXBean.findDeadlockedThreads()withperiodic polling on a daemon thread
JavaProfiler.startDeadlockDetector()/stopDeadlockDetector()lifecycle APIMotivation:
PROF-11534 — The existing deadlock event in dd-trace-java uses JDK JFR with string
stacktraces. This PR adds the improved native implementation in java-profiler that
captures real stacktraces via JVMTI, which dd-trace-java can use when the profiler
is loaded.
Additional Notes:
DeadlockEventFactoryas a fallback forenvironments where the profiler isn't loaded
deadlockIdjava.util.concurrentlock deadlocks (falls back tofindMonitorDeadlockedThreads()when synchronizer usage is not supported)How to test the change?:
./gradlew ddprof-test:testRelease --tests '*DeadlockDetectionTest*'The test creates a real deadlock between two threads, starts the detector with a
100ms interval, and verifies that
datadog.DeadlockedThreadevents appear in theprofiler's JFR output with non-empty stacktraces and matching deadlock IDs.
For Datadog employees:
credentials of any kind, I've requested a review from
@DataDog/security-design-and-guidance.🤖 Generated with Claude Code via muse implement