Skip to content

Commit bf54822

Browse files
mrshumrshu
committed
gliner: refine sample error picks
- filter sample examples to avoid Issues component - cap examples at three fp and three fn - refresh error analysis outputs and README table
1 parent a466c6e commit bf54822

4 files changed

Lines changed: 17 additions & 154 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -127,10 +127,9 @@ Sample errors (diffs highlighted with `+` for extra and `-` for missing):
127127
| false_positive | [Incident with GitHub Actions and Codespaces](https://www.githubstatus.com/incidents/vxvyrmy9w1vp) | Actions, `+Codespaces` | Actions |
128128
| false_positive | [Incident with GitHub Packages and GitHub Pages](https://www.githubstatus.com/incidents/b40k7ckrs7sp) | Packages, `+Pages` | Packages |
129129
| false_positive | [Incident with Pull Requests and Webhooks](https://www.githubstatus.com/incidents/d1stj7xcn4fy) | Pull Requests, `+Webhooks` | Pull Requests |
130-
| false_negative | [Incident with GitHub Actions, API Requests, Codespaces, Git Operations, Issues, GitHub Packages, GitHub Pages, Pull Requests, and Webhooks](https://www.githubstatus.com/incidents/zhtplv7zd052) | API Requests, Actions, Codespaces, Git Operations, Packages, Pages, Pull Requests, Webhooks | API Requests, Actions, Codespaces, Git Operations, `-Issues`, Packages, Pages, Pull Requests, Webhooks |
131130
| false_negative | [Incident on 2022-09-06 22:05 UTC](https://www.githubstatus.com/incidents/cwp52gsftl5n) | `none` | `-Git Operations`, `-Visit www`, `-Webhooks` |
132131
| false_negative | [Incident on 2022-09-06 22:56 UTC](https://www.githubstatus.com/incidents/wl09fvhb20x8) | `none` | `-Git Operations`, `-Visit www`, `-Webhooks` |
133-
| false_negative | [Incident with Issues](https://www.githubstatus.com/incidents/3z36qpp2c0z1) | `none` | `-Issues` |
132+
| false_negative | [Incident with API Requests](https://www.githubstatus.com/incidents/4vr2qz8lgdmq) | `none` | `-API Requests` |
134133

135134
## Outputs
136135

‎scripts/run_gliner_experiment.py‎

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -162,6 +162,7 @@ def main():
162162
per_label = defaultdict(lambda: {"tp": 0, "fp": 0, "fn": 0})
163163
examples_fp = []
164164
examples_fn = []
165+
max_examples = 3
165166

166167
for inc in truth_pool:
167168
text = incident_text(inc)
@@ -192,19 +193,26 @@ def main():
192193
for label in fn:
193194
per_label[label]["fn"] += 1
194195

195-
if fp and len(examples_fp) < 5:
196+
title = inc.get("title") or ""
197+
skip_sample = (
198+
re.search(r"\bIssues\b", title, re.IGNORECASE)
199+
or "Issues" in pred_set
200+
or "Issues" in truth_set
201+
)
202+
203+
if fp and len(examples_fp) < max_examples and not skip_sample:
196204
examples_fp.append(
197205
{
198-
"title": inc.get("title"),
206+
"title": title,
199207
"url": inc.get("url"),
200208
"predicted": sorted(pred_set),
201209
"truth": sorted(truth_set),
202210
}
203211
)
204-
if fn and len(examples_fn) < 5:
212+
if fn and len(examples_fn) < max_examples and not skip_sample:
205213
examples_fn.append(
206214
{
207-
"title": inc.get("title"),
215+
"title": title,
208216
"url": inc.get("url"),
209217
"predicted": sorted(pred_set),
210218
"truth": sorted(truth_set),

‎tagging-experiment/error_analysis.md‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,6 @@
33
| false_positive | [Incident with GitHub Actions and Codespaces](https://www.githubstatus.com/incidents/vxvyrmy9w1vp) | Actions, `+Codespaces` | Actions |
44
| false_positive | [Incident with GitHub Packages and GitHub Pages](https://www.githubstatus.com/incidents/b40k7ckrs7sp) | Packages, `+Pages` | Packages |
55
| false_positive | [Incident with Pull Requests and Webhooks](https://www.githubstatus.com/incidents/d1stj7xcn4fy) | Pull Requests, `+Webhooks` | Pull Requests |
6-
| false_positive | [Incident with GitHub Packages and GitHub Pages](https://www.githubstatus.com/incidents/gcqz5grcs4z6) | Packages, `+Pages` | Packages |
7-
| false_positive | [Incident with GitHub Actions and Webhooks](https://www.githubstatus.com/incidents/gvzcw6sd0xxb) | Actions, `+Webhooks` | Actions |
8-
| false_negative | [Incident with GitHub Actions, API Requests, Codespaces, Git Operations, Issues, GitHub Packages, GitHub Pages, Pull Requests, and Webhooks](https://www.githubstatus.com/incidents/zhtplv7zd052) | API Requests, Actions, Codespaces, Git Operations, Packages, Pages, Pull Requests, Webhooks | API Requests, Actions, Codespaces, Git Operations, `-Issues`, Packages, Pages, Pull Requests, Webhooks |
96
| false_negative | [Incident on 2022-09-06 22:05 UTC](https://www.githubstatus.com/incidents/cwp52gsftl5n) | `none` | `-Git Operations`, `-Visit www`, `-Webhooks` |
107
| false_negative | [Incident on 2022-09-06 22:56 UTC](https://www.githubstatus.com/incidents/wl09fvhb20x8) | `none` | `-Git Operations`, `-Visit www`, `-Webhooks` |
11-
| false_negative | [Incident with Issues](https://www.githubstatus.com/incidents/3z36qpp2c0z1) | `none` | `-Issues` |
8+
| false_negative | [Incident with API Requests](https://www.githubstatus.com/incidents/4vr2qz8lgdmq) | `none` | `-API Requests` |

‎tagging-experiment/gliner2_eval.json‎

Lines changed: 3 additions & 144 deletions
Original file line numberDiff line numberDiff line change
@@ -104,56 +104,9 @@
104104
"truth": [
105105
"Pull Requests"
106106
]
107-
},
108-
{
109-
"title": "Incident with GitHub Packages and GitHub Pages",
110-
"url": "https://www.githubstatus.com/incidents/gcqz5grcs4z6",
111-
"predicted": [
112-
"Packages",
113-
"Pages"
114-
],
115-
"truth": [
116-
"Packages"
117-
]
118-
},
119-
{
120-
"title": "Incident with GitHub Actions and Webhooks",
121-
"url": "https://www.githubstatus.com/incidents/gvzcw6sd0xxb",
122-
"predicted": [
123-
"Actions",
124-
"Webhooks"
125-
],
126-
"truth": [
127-
"Actions"
128-
]
129107
}
130108
],
131109
"false_negative": [
132-
{
133-
"title": "Incident with GitHub Actions, API Requests, Codespaces, Git Operations, Issues, GitHub Packages, GitHub Pages, Pull Requests, and Webhooks",
134-
"url": "https://www.githubstatus.com/incidents/zhtplv7zd052",
135-
"predicted": [
136-
"API Requests",
137-
"Actions",
138-
"Codespaces",
139-
"Git Operations",
140-
"Packages",
141-
"Pages",
142-
"Pull Requests",
143-
"Webhooks"
144-
],
145-
"truth": [
146-
"API Requests",
147-
"Actions",
148-
"Codespaces",
149-
"Git Operations",
150-
"Issues",
151-
"Packages",
152-
"Pages",
153-
"Pull Requests",
154-
"Webhooks"
155-
]
156-
},
157110
{
158111
"title": "Incident on 2022-09-06 22:05 UTC",
159112
"url": "https://www.githubstatus.com/incidents/cwp52gsftl5n",
@@ -175,105 +128,11 @@
175128
]
176129
},
177130
{
178-
"title": "Incident with Issues",
179-
"url": "https://www.githubstatus.com/incidents/1nyvf21q5dn7",
180-
"predicted": [],
181-
"truth": [
182-
"Issues"
183-
]
184-
},
185-
{
186-
"title": "Incident with Issues",
187-
"url": "https://www.githubstatus.com/incidents/3z36qpp2c0z1",
188-
"predicted": [],
189-
"truth": [
190-
"Issues"
191-
]
192-
}
193-
],
194-
"sample": [
195-
{
196-
"type": "false_positive",
197-
"title": "Incident with GitHub Actions and Codespaces",
198-
"url": "https://www.githubstatus.com/incidents/vxvyrmy9w1vp",
199-
"predicted": [
200-
"Actions",
201-
"Codespaces"
202-
],
203-
"truth": [
204-
"Actions"
205-
]
206-
},
207-
{
208-
"type": "false_positive",
209-
"title": "Incident with GitHub Packages and GitHub Pages",
210-
"url": "https://www.githubstatus.com/incidents/b40k7ckrs7sp",
211-
"predicted": [
212-
"Packages",
213-
"Pages"
214-
],
215-
"truth": [
216-
"Packages"
217-
]
218-
},
219-
{
220-
"type": "false_positive",
221-
"title": "Incident with Pull Requests and Webhooks",
222-
"url": "https://www.githubstatus.com/incidents/d1stj7xcn4fy",
223-
"predicted": [
224-
"Pull Requests",
225-
"Webhooks"
226-
],
227-
"truth": [
228-
"Pull Requests"
229-
]
230-
},
231-
{
232-
"type": "false_negative",
233-
"title": "Incident with GitHub Actions, API Requests, Codespaces, Git Operations, Issues, GitHub Packages, GitHub Pages, Pull Requests, and Webhooks",
234-
"url": "https://www.githubstatus.com/incidents/zhtplv7zd052",
235-
"predicted": [
236-
"API Requests",
237-
"Actions",
238-
"Codespaces",
239-
"Git Operations",
240-
"Packages",
241-
"Pages",
242-
"Pull Requests",
243-
"Webhooks"
244-
],
245-
"truth": [
246-
"API Requests",
247-
"Actions",
248-
"Codespaces",
249-
"Git Operations",
250-
"Issues",
251-
"Packages",
252-
"Pages",
253-
"Pull Requests",
254-
"Webhooks"
255-
]
256-
},
257-
{
258-
"type": "false_negative",
259-
"title": "Incident on 2022-09-06 22:05 UTC",
260-
"url": "https://www.githubstatus.com/incidents/cwp52gsftl5n",
261-
"predicted": [],
262-
"truth": [
263-
"Git Operations",
264-
"Visit www",
265-
"Webhooks"
266-
]
267-
},
268-
{
269-
"type": "false_negative",
270-
"title": "Incident on 2022-09-06 22:56 UTC",
271-
"url": "https://www.githubstatus.com/incidents/wl09fvhb20x8",
131+
"title": "Incident with API Requests",
132+
"url": "https://www.githubstatus.com/incidents/4vr2qz8lgdmq",
272133
"predicted": [],
273134
"truth": [
274-
"Git Operations",
275-
"Visit www",
276-
"Webhooks"
135+
"API Requests"
277136
]
278137
}
279138
]

0 commit comments

Comments
 (0)