[Weave] signals content-DOCS-1994#2367
Conversation
|
Images automagically compressed by Calibre's image-actions ✨ Compression reduced images by 76.5%, saving 357.9 KB.
|
📚 Mintlify Preview Links✨ Added (1 total)📄 Pages (1)
📝 Changed (2 total)📄 Pages (1)
⚙️ Other (1)
🤖 Generated automatically when Mintlify deployment succeeds |
🔗 Link Checker Results✅ All links are valid! No broken links were detected. Checked against: https://wb-21fd5541-signals-docs-1994.mintlify.app |
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
NiWaRe
left a comment
NiWaRe
left a comment
There was a problem hiding this comment.
I like the phrasing as "Monitor using built-in signals" - we're still debating on the exact name but this might be a good middle ground.
Three points that we could add:
- The most natural way of "using" signals is through the dashboard under "Project" tab (still behind feature flag but can add in screenshots) and the added tags in the "Signals" column in the trace table (if you hover over a tag it shows you the reason and confidence score, for example here)
- We might want to include the specific output pattern that will make Weave render the scorer output as a tag in case people want to create their own custom signals?
- Could we add a disclaimer with potentially a form or email address to tell people that this is currently in private preview and be tested by reaching out to us?
|
Images automagically compressed by Calibre's image-actions ✨ Compression reduced images by 75.3%, saving 149.5 KB.
1 image did not require optimisation. |
anastasiaguspan
force-pushed
the
signals-docs-1994
branch
from
7a90513 to
7618dd1
Compare
|
Images automagically compressed by Calibre's image-actions ✨ Compression reduced images by 75.2%, saving 135.4 KB.
2 images did not require optimisation. |
dbrian57
left a comment
dbrian57
left a comment
There was a problem hiding this comment.
Hey, so I think this doc needs some rework. The opening section is very markety and contains three consecutive and sizable blocks of info before anything actionable is mentioned.
I would opt to:
- Rework the intro section to more plainly and concisely state what signals are and their benefits. I would also make sure to explicitly mention that they're a type of monitor.
- Give the same treatment to the intro of the "Available signals" section
- Move some wording into the intro section (I noted these in my annotations)
- Clarify the "How signals work" section. This might just need an intro adjustment, but I'm not sure. I just found it difficult to follow.
I'm happy to discuss any of this, obviously.
| Signals provide a high-level monitoring solution designed to bridge this gap by offering automated, behavioral scoring for agents in production. | ||
|
|
||
| Signals utilize a robust backend infrastructure to provide real-time performance insights: | ||
| - Automated scoring: Every incoming production trace is automatically processed and scored based on predefined metrics. |
There was a problem hiding this comment.
Should these opening terms be bolded?
There was a problem hiding this comment.
funny you mention that, because no, i don't think they actually "should" (i don't think they technically qualify as 'lead-in')- but yes, everything AI loves to bold these, so now it looks weird if we don't? style guide grey area. there was a slack chat about this a while ago. i'm opting for no here, because they aren't that juicy/useful of terms for the user.
| 7. After running the script using several different statements, open the W&B UI and navigate to the **Traces** tab. Select any **LLMAsAJudgeScorer.score** trace to see the results. | ||
|
|
||
|  | ||
| In modern agent development, standard system metrics like latency, token count, and cost are insufficient for understanding complex agent behavior. While inspecting individual traces provides deep insight, it is impossible to scale manual reviews across the millions of traces generated in a live environment. |
There was a problem hiding this comment.
I think we should plainly state what signals are at the start. Otherwise, this kinda reads like a company blog post. We could amend this quickly by moving the second paragraph to the top and fixing it up a little at the end. We should also make the connection that signals are a type of monitor.
Also, I feel like this paragraph is very opinionated. I think we should consider nixing it entirely and in lieu of something that plainly states what Signals do and how they benefit the customer.
There was a problem hiding this comment.
hmm, i kinda disagree? this is starting with the problem statement. the problem that signals are solving. I think that explains the benefit to the customer more than trying to explain the benefit first.
I tightened up the sentences in the intro however, so it is still problem statement, then how signals answers that problem. take a look and see if you dislike it less ;)
|
|
||
| ## Available signals | ||
|
|
||
| W&B Weave offers monitors with built-in signals: preset scorers that evaluate production traces for common quality issues and errors out of the box, with no custom setup. Each built-in signal uses a benchmarked LLM prompt to classify traces and saves the results as comma-delimited tags representing the detected issues. |
There was a problem hiding this comment.
I feel like this is more introductory stuff that should be at the top of the page.
There was a problem hiding this comment.
I updated the automated scoring bullet to includ the important most-basic bit, but other than that, it is the first section.i tightened up the intro so this is now closer to teh top as well
|
|
||
| To start classifying traces immediately, enable signals from the Monitors page. Signals don't require prompt engineering or scorer configuration. | ||
|
|
||
| Signals use a [W&B Inference](/inference/) model to score traces, so no external API keys are required. |
There was a problem hiding this comment.
Probably should either be in the intro area or the How signals work section. I'd opt for the introductory section and also add whether or not this costs customers additional money to use.
|
|
||
| Each signal uses an LLM-as-a-judge approach to classify traces: | ||
|
|
||
| 1. **Trace selection**: Quality signals evaluate successful root-level traces. Error signals evaluate failed traces. Child spans and intermediate calls are not scored. |
There was a problem hiding this comment.
I feel like I'm missing context here. Are each of these a step that Weave goes through to assess and apply a signal to a trace?
Also, the "Trace selection" bullet introduces new information: quality signals only evaluate root-level traces. I feel like this should be mentioned earlier in "Available signals" so users don't wonder why child spans aren't scored.
There was a problem hiding this comment.
moved section up
There was a problem hiding this comment.
and ur right, shouldn't be numbered.
Co-authored-by: Dan Brian <dbrian@coreweave.com>
3 participants
Description
Adding new Signals content (naming still TBD).
Signals is replacing the goto starter user journey, making existing 'monitors' the more advanced, custom path - as such it is moved to a subsequent page 'custom monitors'.
Testing
mint dev)mint broken-links)Related issues