Autoresearch for Marketplace Catalogs:
From Legacy Forms to AI-Native MatchingNote: Accepted at the KDD Workshop on Evaluation and Trustworthiness of Agentic AI (KDD 2026).

The deploying organization is a major U.S. online services marketplace that connects consumers with local service professionals (pros) across home, wellness, and event categories, including plumbing, house cleaning, photography, landscaping, and roughly 130 others. Pros specify targeting preferences that determine which consumer leads they are eligible to receive and purchase. The system described in this paper is part of a 2026 pro-side marketplace redesign that replaces rigid binary targeting defined through structured Q&A with probabilistic matching driven by a structured provider-side preference taxonomy (Einav et al., 2016).

The legacy taxonomy is organized as a hierarchy of occupations, categories, and category-specific request-form Q&A schemas. The reconstructed catalog contains two tag classes.

1. (1)

   Canonical tags determine eligibility: a provider must possess the canonical tag associated with a category for the strict-match filter to consider that provider eligible for leads in that category;

2. (2)

   Specialty preference tags allow providers to refine which leads they wish to receive within categories for which they are already eligible

Section 6 reports two production findings that depend on this distinction: a catalog-hygiene gap in which deprecated canonical tags were still emitted by request-time enrichment, and a deployment-side seeding gap in which $40.66\%$ of filter evaluations failed because the canonical tag associated with the consumer request was not present on the provider profile. A filter evaluation is a single matchmaker-side check, for one candidate pro and one incoming consumer request, to determine whether the pro’s profile contains the request’s required canonical tag.

Catalog reconstruction is a cold-start problem: initially there is no per-occupation generation prompt, so every occupation $o\in O$ ($|O|=132$ in the current production deployment) is seeded with the same baseline prompt $p^{(0)}$, i.e. $p_{o}^{\star}\!\leftarrow\!p^{(0)}$. We run Algorithm 1 (illustrated in Figure 2) independently per occupation.

At iteration $0$, the generator $\mathcal{G}$ applies the baseline prompt to the occupation’s legacy data $D_{o}$ ( RF Q&A schema for each category, % share of recently-active pros who have currently enrolled a (q, a) pair as a preference) to produce a candidate tag set $T$. The six-rubric LLM-as-judge scores the set, $s=\mathcal{E}(T)\in[0,15]$, and the seven-persona critic panel adds weighted penalties $\pi_{k}$, giving the composite score $a=s-\sum_{k}w_{k}\pi_{k}(T)$. This becomes the initial best $(p_{o}^{\star},T^{\star},s^{\star},a^{\star})$. The loop runs a cross-family model stack: the generator $\mathcal{G}$ and judge $\mathcal{E}$ on one family (GPT-5-4 / GPT-5-4-mini), the seven critics, editor, and parity mapper on another (Claude Sonnet 4.6), so the models that propose and score do not share biases with the models that critique and mutate (§4.2, §7).

Each subsequent iteration proposes and tests a single prompt edit. The editor $\mathcal{R}$ makes one targeted change to the current best prompt $p_{o}^{\star}$, focused on a single weak quality dimension: it ranks the six $\mathcal{E}$ rubrics together with an injected parity-coverage dimension by mean score and selects the lowest-scoring dimension not targeted in the previous two iterations, avoiding fixation on any one rubric. The editor conditions on the weakest tag examples in that dimension, the score trends across iterations, and the log of previously accepted and rejected edits. The revised prompt is regenerated and rescored; as a per-iteration cost saver, the critic panel is invoked only when the candidate’s $\mathcal{E}$ score is not already below the best ($s\geq s^{\star}$), since a lower-$\mathcal{E}$ candidate is very unlikely to overcome the best composite score. The edit is accepted, thus becoming the new best, iff its composite score improves, $a>a^{\star}$; otherwise the prompt reverts to $p_{o}^{\star}$. The loop runs for a fixed budget of $B$ iterations (default $5$). The supplementary material traces one iteration end-to-end on the Accounting occupation.

On termination, the parity mapping $\mathcal{P}(T^{\star},Q_{o})$ links the tags of the best catalog $T^{\star}$ to the legacy RF Q&A $Q_{o}$ as a many-to-many relation $M^{\star}\subseteq T^{\star}\times Q_{o}$, leaving residual sets $U_{Q}^{\star}$ (RF Q&A covered by no tag—typically intake-specific size/range/flexibility questions) and $U_{T}^{\star}$ (tags covered by no Q&A). We emit $M^{\star}$ as a JSON mapping and a Google Sheet for human review before deployment.

The judge model $\mathcal{E}$ scores every tag ($t\in T$) on six dimensions:

• •

  Screening vs. Intake (0–3): Does the tag describe a job type the pro might want more or less of (good), or is it a customer-intake detail the pro should not screen on (bad)? E.g. “Tax return preparation” (screening) vs. “Travel range: 15 miles” (intake).

• •

  Tag Legibility (0–3): Would a working pro understand this tag from the tag text alone? E.g. “Appliance repair” (clear) vs. the bare noun “Appliances”

• •

  Preference Variance (0–3): Do real pros split on this tag, or is everyone equally for-or-against (degenerate)? E.g. Accounting pros split on “QuickBooks” proficiency, but nearly all accountants serve the “healthcare industry” (degenerate).

• •

  Cross-Category Consistency (0–2): When the same job-attribute appears in two categories within the occupation, do the tags match? E.g. the client-entity type tagged identically as “S-Corp” wherever it recurs across the occupation’s categories, not “S-Corp” in one and “S-Corporation” in another.

• •

  Canonical Coverage (0 or 2): Does every category in the occupation have a canonical tag covering it? E.g. “Payroll services” tag cleanly covers the Payroll Services category, whereas a tag backed by no category scores 0

• •

  Information Loss (0–2): Are the tag’s sources traceable to specific legacy Q&A answers, i.e. does it cite the legacy category and answers it consolidates? E.g. a tag citing legacy_category: Accounting and the “QuickBooks” answer scores 2 vs. one with only a vague rationale ($0$).

$\mathcal{E}$ scores each tag independently on the six rubrics above, with all tags in $T$ scored in parallel; the per-tag composite is the sum of the six rubric scores (on a $/15$ scale). The set-level score is the mean per-tag composite reduced by small set-level deductions for tag proliferation and category-coverage gaps, $s=\mathcal{E}(T)\in[0,15]$; the verdict mix summarizes the per-tag outcomes. The judge model is GPT-5-4-mini at temperature zero; the generation model is GPT-5-4. The seven-persona critic panel, the editor agent, and the final parity-mapping stage all run on Claude Sonnet 4.6, a different model family from the generator and the judge. The choice is deliberate: a same-family stack would risk the loop reinforcing biases shared across mutation and evaluation, and the cross-family arrangement is the strongest readily available defense at production scale. $\mathcal{E}$ was recalibrated against product-manager (PM) review: bare-noun tags like “Appliances” originally scored a legibility of $3$ but scored $\leq 1$ after recalibration, since a working appliance-repair pro reads “Appliances” as either trivially true or meaningless, not informative.

The $\mathcal{E}$ judge scores each tag in isolation, so a catalog of individually high-scoring tags can still be poor as a set. $T$ can be bloated with near-duplicate siblings, or leave real pro preferences with no corresponding tag. The set-level penalty $\rho(T)$ corrects for this by deducting from the composite two failures the per-tag rubrics cannot see: tag proliferation (an excess of sibling tags under a single parent, which fragments the catalog and the pro-facing UI) and coverage gaps (legacy preferences in $Q_{o}$ which pros both enrolled and deselected recently that no tag captures, including whole categories left untagged; $\lx@sectionsign$4.6). Subtracting $\rho(T)$ before the critic penalties ($\lx@sectionsign$4.4), $a=s-\rho(T)$, makes the loop optimize for a coherent, well-covered catalog rather than a collection of locally excellent but globally redundant or incomplete tags.

After $\mathcal{E}$ scoring, the full tag set is reviewed by seven critic-personas running in parallel, each returning a verdict that is reduced to a weighted contribution to the composite score. Most contributions are penalties; the Reasoning critic can instead award a small bonus for strong rationales. The exact penalty formulas, caps, and bonus thresholds are tabulated in the supplementary material.¹¹ 1 An earlier design used a hard veto from the Pro Critic. In early production runs, the veto dominated keep/discard decisions, and the loop stalled on occupations where one persona consistently rejected proposals that other critics rated favorably. The current design replaces the veto with a weighted penalty capped at $2.5$ on the composite-of-15 scale; low enough that a strong proposal can still be kept over Pro objections, high enough that consistent Pro disapproval pulls the adjusted score below the discard threshold. The cap and rate were set empirically; the panel module records the change as “softened to weighted penalty, capped at 2.5.” The critics are:

• •

  PM critic: Would a product manager (PM) find this tag actionable? Surfaces design-stage issues (mutual-exclusivity, awkward dimensionality); penalized only when rejections exceed a small free allowance.

• •

  Pro critic (penalty capped at $2.5$): Would a working pro in this occupation find this useful or confusing? The cap prevents a single skeptical persona from dominating the verdict.

• •

  Taxonomy critic: Are tags at consistent levels of abstraction, are unions broken into components, are subtypes grouped? Penalized as structural consistency falls below the threshold.

• •

  Adversarial critic: Where could the tag set be gamed or misinterpreted to extract leads that the pro cannot deliver? Penalty scales with the assessed risk level and the number of critical failures.

• •

  Reasoning-Quality critic (bonus-eligible): Does the generation explain why each tag exists? Strong rationales (score $\geq 8$) earn a small bonus; weak ones incur a penalty.

• •

  Occupation-Coherence critic: Does the tag set, taken as a whole, describe this occupation rather than a generic union of services?

• •

  Parity critic: Do the generated tags map to a given occupation’s top-30 most-deselected legacy answers (i.e., answers active pros have explicitly opted out of), ranked by distinct-pro count over a 90-day window? Its penalty is small by design (at most $\sim\!1$ point); the quantitative coverage enforcement lives in the set-level penalty $\rho$ (§4.3). When an occupation has no deselection signal, the critic returns a neutral score and contributes nothing.

The editor $\mathcal{R}$ is the loop’s mutation operator: at each iteration, it proposes exactly one targeted edit to the current best prompt $p_{o}^{\star}$ and returns the revised prompt together with a one-line description of the change. A key design point is that the editor is driven by the judge $\mathcal{E}$, not the critics: the critic penalties gate which prompts survive (through the composite $a$), whereas the editor decides what to change from the $\mathcal{E}$ signal and the coverage signal alone.

Once the loop terminates with the best catalog $T^{\star}$, a final LLM pass migrates the occupation’s legacy RF Q&A $Q_{o}$ onto the new tags, producing the many-to-many relation $M^{\star}\subseteq T^{\star}\times Q_{o}$ returned by Algorithm 1. The autoresearch design ensures that the generated catalog covers a good fraction of legacy taxonomies (explicitly in $\lx@sectionsign$4.3, parity critic in $\lx@sectionsign$4.4, and set level coverage in $\lx@sectionsign$4.5). $M^{\star}$, on the other hand, is explicit and auditable: it records exactly which legacy preferences map to which tags and which remain uncovered, producing the human-reviewable artifact used to sign off the catalog before deployment. For each legacy answer, the mapper finds all tags that cover the same real-world concept; a single answer may map to several tags (for example, a specific tag and the broader tag above it), which is why $M^{\star}$ is a many-to-many relation rather than a function. Aggregated across the 132-occupation production catalog, $M^{\star}$ is also what yields the reconstruction-fidelity numbers quoted in §1: $73.3\%$ of legacy answers map directly to a tag in the regenerated catalog; $20.8\%$ are intake-only answers (sizes, ranges, “flexible”) excluded as acceptable non-tags; the remaining $6.0\%$ are regrettable misses, real preferences left uncovered. Measured against the answers that should map, pooled coverage is $92.5\%$, with a per-occupation median of $100\%$ (mean $97.8\%$).

The autoresearch loop produces a catalog whose quality is measured in one space (the $\mathcal{E}$ composite plus the seven critic penalties ), and the deployment is measured in another (filter-evaluation outcomes, occupation concentration). Conflating the two muddles attribution: a missing-canonical-tag filter rate of $40.66\%$ in production is not a verdict on the catalog, it is a verdict on the deployment-side seeding step that was supposed to attach canonical tags to existing pro profiles. Likewise, the loop-internal critic penalties say nothing about whether request-time enrichment actually emits the right tags in production. We separate the two strands explicitly in what follows.

The 14-day production post-launch cohort study ( 1,840 MVP-enrolled pros, 9,324,798 filter evaluations over 2026-04-28 to 2026-05-12) characterizes the catalog as deployed, distinct from the catalog as generated.

To characterize how often the loop converges within budget, we aggregate the best-iteration summary row for each of the $132$ production occupations from the autoresearch tracking logs. Table 3 summarizes the distribution.

Three observations hold across the full $132$-occupation cohort. (i) The best kept tag set is reached by iteration $3$ for every occupation—$36$ at iteration $1$, $53$ at iteration $2$, $38$ at iteration $3$, and $5$ that never beat their baseline (index $0$)—with none reaching its best after iteration $3$, so the kept-iteration count saturates well before the $5$-iteration budget. (ii) The $\mathcal{E}$ composite is tightly clustered (median $13.84/15$, mean $13.53$, interquartile range $13.32$–$14.20$), but the adjusted score spreads much wider (median $10.17$, IQR $8.63$–$11.41$): $19$ of $132$ occupations fall in the $<8$ band that the production verdict thresholds (§6.2) route to review or flag, almost entirely on critic penalty rather than on a weak judge score. (iii) The critic penalty (range $0.50$–$9.70$, median $3.62$) is therefore the dominant source of variance in the adjusted score, consistent with the asymmetric panel design (supplementary material) in which the Adversarial critic alone can contribute up to $2.0$ on the critical risk tier.