A Reusable Semantic Web Framework for Evidence-Grounded Fundamental Rights Impact Assessments under the EU AI Act

The corpus combines three complementary evidence sources: approximately 100 records from the AIAAIC incident repository (Pownall, 2024), 30 entries from the U.S. Federal AI Use Case Inventory (Office of Management and Budget, 2024), and 20 European Court of Human Rights cases from HUDOC (Council of Europe, 2024). Together these provide 150 records covering LLM-related and analogous AI deployments in employment and essential public services. The three sources triangulate across evidence types: AIAAIC captures real-world failures; the U.S. Federal Inventory documents active government deployments; ECtHR cases provide authoritative legal analysis of rights violations.

Each record is classified along four axes: (1) Annex III high-risk AI domain (employment or essential services); (2) implicated EU Charter rights (multi-label); (3) risk pattern (bias/discrimination, privacy breach, procedural unfairness, lack of transparency, or other); and (4) causal factor (data quality, model design, deployment context, or oversight failure).

The schema is grounded in Semantic Web standards and aligned with DPV (W3C Data Privacy Vocabulary Community Group, 2022; Pandit et al., 2024), VAIR (Golpayegani et al., 2024b), AIRO (Golpayegani et al., 2022), and the FRIA ontology (Rintamäki and Pandit, 2025). Reusing established vocabularies reduces the learning burden for adopters and maximises interoperability with existing compliance tooling. The annotated corpus is serialised in two formats: a Turtle RDF knowledge graph of 1,351 triples (193 nodes, 965 edges) enabling information retrieval using the SPARQL query language¹¹ 1 https://www.w3.org/TR/sparql11-query/, and 150 JSON-LD records for web-compatible consumption.²² 2 All schema artefacts, the knowledge graph, and the SPARQL queries are available at https://github.com/faitholopade/Dissertation.

Three annotation methods are implemented and compared. A keyword baseline uses synonym-expanded dictionary matching against controlled vocabularies. An LLM classifier applies Claude Sonnet (Anthropic, 2024) with zero-temperature sampling, few-shot prompting, and structured output constraints. A hybrid method resolves conflicts between keyword and LLM outputs using priority logic, retaining keyword classifications for domain and LLM classifications for rights where each method performs better.

A regulatory crosswalk maps the obligations arising from Annex III(4) and Annex III(5)(a) to specific articles of the EU Charter of Fundamental Rights (European Parliament, Council of the European Union, and European Commission, 2000). This mapping is not enumerated in the AI Act itself; the crosswalk makes it explicit and machine-readable, reducing interpretive uncertainty for deployers and providing the structured link between regulatory requirements and rights evidence that the schema’s multi-label rights axis builds on.

A Flask web application loads the knowledge graph on startup and allows users to filter records by Annex III domain, Charter rights, and risk pattern through a form-based interface, moving the framework toward the interactive compliance tooling anticipated by Art. 27(5) (European Parliament and Council of the European Union, 2024).

Of the 150 corpus records, 69 (46%) were manually annotated to form a gold standard, stratified across sources and both Annex III domains. Each record was annotated by reviewing original source material rather than summary fields alone, following written guidelines with explicit decision criteria per axis. Cohen’s $\kappa$ (Cohen, 1960) and percentage agreement are reported for each axis and method, interpreted using the Landis and Koch scale (Landis and Koch, 1977).

The 69 manually annotated records (46%) are a deliberately stratified gold standard rather than a partial annotation effort. The framework’s purpose is to test whether automated annotation can scale beyond what manual labelling feasibly covers, so the manual subset is sized to give reliable per-axis agreement estimates across both Annex III domains and all three sources, not to annotate the corpus exhaustively. Manually labelling all 150 records would defeat the evaluation, since the pipeline exists precisely to avoid that cost at scale.

For the essential services domain, the best-performing method achieves $\kappa=0.525$ (moderate agreement). For employment, the best-performing method achieves $\kappa=0.045$—near-chance. This divergence is attributable to spurious vocabulary correlations: a disproportionate share of records discuss employment incidentally rather than as the primary domain of harm, and both keyword and LLM classifiers over-predict employment for records containing general labour market language.

The implication for algorithmic fairness research is direct. Employment discrimination is among the most extensively documented harms of automated decision-making, and Annex III(4) covers the AI Act’s highest-risk employment applications. Near-chance agreement means that automated annotation of employment-domain risk evidence cannot currently be trusted without human review. Deployers relying on such tools to surface employment risk evidence for a FRIA risk generating an evidence base that is both incomplete and misleading. The finding holds across models: a comparison with GPT-4o-mini yields $\kappa=0.196$ for domain classification, confirming the difficulty lies in the task rather than in a single model.

Five FRIA demonstration scenarios query the populated knowledge graph for records relevant to realistic deployer contexts: welfare eligibility AI, public sector recruitment screening, surveillance in public housing, LLM decision-support for caseworkers, and a cross-domain thematic review by a national regulator. Across all five, 103 of the 150 records are surfaced (68.7% coverage). The 31.3% not surfaced are predominantly records with unknown classifications produced by pipeline underclassification—the direct consequence of the domain agreement failures described above. Figure 1 reports per-scenario retrieval: the welfare-eligibility and cross-domain profiling scenarios surface the largest evidence pools (57 and 50 records), while the employment-focused recruitment scenario surfaces the fewest (17), consistent with the near-chance employment agreement reported above.

The hybrid method reduces unknown risk pattern classifications from 61.3% (keyword baseline) to 14.0%, substantially enlarging the evidence base available for compliance queries. Agreement on risk patterns remains in the slight-to-fair range, with procedural unfairness and lack of transparency the categories of greatest disagreement, driven primarily by multi-label harm structures where incidents involve co-occurring failure modes.

A public body preparing to deploy an AI system that supports social-welfare eligibility decisions (Annex III(5)(a)) must document, before deployment, how the system may bear on the rights to social security (Art. 34) and non-discrimination (Art. 21). Using the query interface, the deployer filters the knowledge graph for essential-services records implicating these rights; the query returns 57 candidate records (Scenario A in Figure 1). Each record carries its risk-pattern and causal-factor classifications together with full provenance back to the originating incident, so a flagged non-discrimination risk can be traced to specific precedents—such as the Dutch childcare-benefits scandal (Amnesty International, 2021) and the SyRI profiling case (District Court of The Hague, 2020; Wieringa and others, 2023)—rather than to a general assertion that such risks exist. This is what separates an evidence-grounded FRIA from a narrative one: the deployer can judge whether documented failures are analogous to its own context, and the machine-readable output lets a supervisory authority later verify that comparable deployers considered the same rights, as anticipated by Art. 27(5).