Mintaka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answering

The first task was to elicit complex questions. To do this, we created tasks for each topic/complexity pair (e.g., Superlative Movie questions, Ordinal Sports questions, etc.). In each task, a worker was asked to write 5 questions and answers about the topic using the given complexity type. The questions and answers were written in free text fields. We had no restrictions on what sources workers could use to write their questions, so workers were not limited to writing questions based on a given article or facts. Workers were given explanations of the complexity type and examples in the instructions. The topics were left general, so within History, workers could write about Ancient Egypt as well as World War II.

For Count and Superlative answers, we additionally asked workers to provide a numerical value as part of the answer. For example, in Count questions, workers would both provide the answer as a number (e.g., 3) as well as the entities that make up that answer (e.g., Best Picture, Best Adapted Screenplay, and Best Film Editing). In Superlative questions, workers provided the answer (e.g., Missouri River) as well as the numerical value that makes the entity the maximum or minimum (e.g., 2,341 miles). Additionally in Count questions, if a question had multiple answers, we asked workers to list a minimum of five. For example, for the question "How many cities have hosted a Summer Olympics?", a worker could give the numerical answer 23 but provide only five of the cities. For this reason, answers to questions with more than five entities are not guaranteed to be complete but instead provide a sample of the correct answer.

We paid $1.25 per task to write five questions. Workers were limited to completing one task per topic-complexity pair. After collection, we also surveyed the MTurk workers who completed our Question Elicitation task about their demographics. The results of this survey are discussed in §5.3.

Answers were collected in the previous task in natural language. In order to link the answers to a knowledge graph, we built an Answer Entity Linking task. We chose to link the answers to Wikidata, since it is a large and up-to-date public knowledge graph. Although we link to Wikidata, we don’t guarantee that every question can be answered by Wikidata at the time of writing. It is possible that there are missing or incomplete facts that would prevent a KGQA system from reaching the answer entity in Wikidata given the question.

In this task, workers were shown a question-answer pair and asked to 1) highlight the entities in the answer, and 2) search for the entities on Wikidata and provide the correct URLs. We built a UI for MTurk workers where they could easily highlight entities and the highlighted entities would automatically generate links to search Wikidata.

Each answer was annotated by two MTurk workers. For agreement, we required two workers to identify the same entities and the same Wikidata URLs for all entities. If there was disagreement, we sent the question-answer pair to a third annotator. Question-answer pairs where the answer was a number or yes or no were excluded from answer entity linking. Overall, we annotated 20,996 answer entities and achieved 82% agreement after two annotators and 97% agreement after three annotators. The remaining 3% were verified by the authors.

We paid a base rate of $0.10 per task, which consisted of a single question-answer pair. If the answer had multiple entities, we paid a $0.05 bonus for every additional entity identified that was agreed upon by another annotator.

An end-to-end question answering model can be trained using the question and answer alone Oliya et al. (2021). However to better evaluate end-to-end methods and train models requiring entities, we also created an MTurk task to link entities in the question text.

Linking entities in questions is more challenging than answers. While answer texts are often short and contain a clear entity (e.g., "Joe Biden"), question texts can contain multiple possible entities. In the question "Who is the president of the United States?", a worker could select "United States", or "president" and "United States", or even "president of the United States". Since early test runs showed it would be difficult to get agreement on question entities, we modified the task so workers only verified a span and linked the entity in Wikidata.

To identify spans in questions, we used spaCy’s Honnibal et al. (2020) en_core_web_trf model to identify named entities and noun chunks with capitalized words in the English sentences. We then gave workers the question with a predicted entity highlighted. Workers were shown one entity at a time and asked to first verify or modify the highlighted entity and then link to Wikidata.

For question entities that were not seen before, we had the entity annotated by two annotators, followed by a third in case of no agreement. For some question entities, we were able to exact string match them against entities that were already annotated in the Answer Entity Linking task, for example United States → Q30. In cases where we had a match, the question entity was annotated by one annotator and only went to a second annotator if there was no agreement.

We annotated 12,819 new entities, for which we had 68% agreement after two annotators and 78% agreement after three annotators, and 15,075 seen entities, for which we had 80% agreement after one annotator, and 98% agreement after two annotators. The remaining entities were verified by the authors. We paid $0.10 per entity. The spans of the question entities are only annotated in English, and so English questions in Mintaka come with both the entity ID and the span, while all translated questions have only the entity ID.

We translated the 20,000 questions in Mintaka to the following languages and locales: Arabic (Saudi Arabia), French (France), German (Germany), Hindi (India), Italian (Italy), Japanese (Japan), Portuguese (Brazil), and Spanish (Mexico). Translation is our only dataset collection step where we do not use MTurk. Early experiments with MTurk on translation tasks and editing automatic translations tasks had poor results and a lack of workers in some languages, such as Japanese. For this reason, we use professional translators.

Statistics about the Mintaka dataset are shown in Table 3. Question length is based on white space splitting in all languages except Japanese, where the question length is in characters. In total, 13,232 unique Wikidata entities appear across all questions. The most common question entities are the United States (Q30; 1,495 questions), President of the United States (Q11696; 565 questions), and Super Bowl (Q32096, 345 questions). The most common answer entities are California (Q99, 102 answers), Alaska (Q797, 88 answers), and the United States (Q30, 80 answers).

Mintaka was built with only questions and answers, so we do not know the correct query path. However we can calculate an upper bound of answerable questions using Wikidata by identifying the percentage of questions that have a path connecting the question entity to the answer entity. We find that there is a path connecting the question and answer entity 62% of the time within 1 hop and 97% of the time within 2 hops for questions linked to Wikidata entities.

A majority (72%) of the questions in Mintaka can be answered using an entity. 14% can be answered using a boolean, in yes/no or comparative questions. 7% can be answered using a number, such as someone’s age. 6% can be answered using a date, such as a date of birth. And finally, 0.1% have answers in the form of a string, for example, someone’s nickname. Examples of QA pairs are in Table 3 with more examples in Appendix C.

By naturally eliciting complex questions from MTurk workers, we aimed to collect questions that were closer to what users may ask in real-world settings. In order to evaluate how Mintaka compares to previous complex QA datasets, we ran a naturalness evaluation task on Mturk with four comparison datasets. We compared datasets that collected questions in different ways: KQA Pro automatically generated questions, ComplexWebQuestions (CWQ) automatically generated questions built off WebQuestions, DROP naturally elicited questions about a given Wikipedia passage, and ComplexQuestions (CQ) collected natural questions from user logs. We compared these datasets to Mintaka in a task where workers were shown 5 questions, one from each dataset, and asked to rank them from 1 (least natural) to 5 (most natural). We uniformly sampled 500 questions from each dataset and grouped them into quartiles by length for each dataset (i.e., the longest questions in Mintaka are grouped with the longest questions in other datasets).

The results are in Figure 1 and show that Mintaka is on average ranked higher in naturalness than all other datasets. We also find that Mintaka ranks significantly higher than the other datasets using a two-sample Kolmogorov-Smirnov test (with p-values < 0.001). This shows that Mintaka questions are perceived as more natural than automatically generated or passage-constrained questions. Although ComplexQuestions contains real user questions, the questions are collected from search logs and can be phrased ungrammatically (e.g., "when did miami dolphins win super bowl?"), leading to a wider range of ranks. These results confirm that Mintaka is both a complex and natural dataset.

In total, 516 MTurk workers completed 3,503 Question Elicitation tasks to collect complex questions (some questions from tasks were removed as duplicates or under-sampled for a balanced dataset size). In order to better understand and measure who our dataset best represents, we invited all workers who completed a Question Elicitation task to participate in a demographic survey. We paid workers $1.25 to complete the survey. We received 400 responses (78% response rate). Worker IDs were only used to invite MTurk workers to take part in the survey. All demographic data is anonymous with no way to link the data back to the Worker IDs, and the data is only analyzed in aggregate.

Table 4 can be used as an indicator of who this dataset is best for modeling and for what populations it may be less representative. For example, we had more workers identify as male than female (58% vs. 42%). Only 2% of workers were between the ages of 18-24 (and workers below 18 cannot register on MTurk), while 72% of workers were between the ages of 25 and 44. We also had fewer workers who identified as Black (7%) or Hispanic (6%) than the U.S. Census U.S. Census Bureau (2021) estimates of the general population (13% and 19%, respectively), while seeing a slightly higher percentage of workers identifying as Asian (10% of workers vs. 6% in the U.S. Census).

Our workers also tend to be more educated with 61% reporting that they hold a Bachelor’s degree or higher, while the U.S. Census estimates 32% of the general population holds a Bachelor’s degree or higher U.S. Census Bureau (2021). Our workers are almost all employed either full or part-time (91%), and largely live in urban or suburban areas (85%). Geographic distribution across the U.S. shows more (35%) workers in the South.

We evaluate eight baselines on Mintaka. Since Mintaka contains only question and answer pairs, we only use models that can be trained end-to-end. We evaluate 3 language models, 3 knowledge graph-based models, and 2 retriever-reader models. For language and retriever-reader models, we use the answers written by the crowd workers in the Question Elicitation task as the label. For our knowledge graph, we use a Wikidata snapshot from October 18, 2021. We evaluate all of our baselines in English (Table 5) and three of our baselines that could easily be set up multilingually in all languages (Table 6). Details on training data size can be found in Appendix B.

T5 and MT5 Raffel et al. (2020); Xue et al. (2021) are baselines that only use a language model to predict answers to questions. We use the XL versions of T5 for English and MT5 for all other languages. We fine-tune both for 10,000 steps.

T5 for Closed Book QA (CBQA) Roberts et al. (2020) is an extension of T5 that is fine-tuned as a QA model that can implicitly store and retrieve knowledge without an external source. We use Roberts et al. (2020)’s T5-XL model and evaluate on Mintaka both as zero-shot with a model fine-tuned on Natural Questions and with a model fine-tuned on Mintaka for 10,000 steps. We run an additional translation baseline where we automatically translate non-English questions to English using the M2M_100 model Fan et al. (2020) and use our English model to return answers.

KVMemNet: Key-Value Memory Networks Miller et al. (2016) work by first storing knowledge graph triples in a key-value structured memory. Then given a question, the model learns which keys are relevant to the question, and uses the values of those keys to return an answer. We follow the implementation by Shi et al. (2020).

EmbedKGQA Saxena et al. (2020) is a method that incorporates pre-trained knowledge graph embeddings into a KGQA model. EmbedKGQA consists of 1) a KG embedding module, 2) a question embedding module, and 3) an answer scoring module, which combines the question and KG embeddings to score and select answer entities. Since EmbedKGQA makes predictions over answer entities, we exclude questions where the answer is not an entity during training time, and count these as failures during test time. Scores for the subset of the test set that is answerable by the model can be found in Appendix B.

Rigel Oliya et al. (2021); Sen et al. (2021) is an end-to-end question answering model based on ReifiedKB Cohen et al. (2020). Rigel uses an encoder to encode the question and a decoder to return a probability distribution over all relations in the knowledge graph. The relations are followed in the knowledge graph to return predicted answers. For the encoder, we use RoBERTa Liu et al. (2019) for English and XLM-RoBERTa Conneau et al. (2020) for all other languages. Again, since Rigel predicts over answer entities, we exclude questions where the answer is not an entity during training time and count them as failures during test time.

Dense Passage Retrieval (DPR) Karpukhin et al. (2020) is a retriever-reader method that uses a dense retriever model to identify relevant Wikipedia passages given a question, followed by a reader model to score answer spans from the retrieved passages. For the retriever, we use Karpukhin et al. (2020)’s model trained on Natural Questions, and for the reader, we evaluate both zero-shot with a model trained on Natural Questions and with a model trained on Mintaka. The reader sees the top 50 retrieved passages, and we take the highest-scoring span as the answer.

The results of the baselines in English in terms of hits@1 can be seen in Table 5. Hits@1 is calculated based on the number of samples where the top prediction from the model matches the labeled answer, either as exact string match for text answers or as entity IDs for entity answers. A further breakdown per complexity type is in Appendix C. The best-performing model is the fine-tuned T5 for Closed Book QA with 38% hits@1. An analysis of the outputs shows that even though the model does not have access to an external knowledge source, it does recall factual information, such as the capital of Iraq is Baghdad. For more complex questions, the model can usually predict in the correct neighborhood of the answer. For example, for "What is the second Marvel Cinematic Universe movie chronologically?", the model predicts "Thor: The Dark World", which is a Marvel movie, however it lacks the complex reasoning functionality to calculate the second movie chronologically.

The trained DPR model has the second highest score with 31% hits@1. We find that DPR can handle complex questions, as long as the complex reasoning is already done in the passage. For example, the model can answer "When did Roger Federer win his first Grand Slam?" with "2003" from a passage that includes, "Roger Federer won his first Grand Slam title in the 2003 Wimbledon Championships." However in cases where the reasoning is not included in the passage, the model struggles. For example given the same question about the second Marvel movie, the model predicts Iron Man and fails to find a passage explicitly mentioning the second movie chronologically.

Finally, the best-performing KGQA model is Rigel with 20% hits@1. Our KGQA baselines handle only entity answers and can only traverse the knowledge graph by following relations, so they are limited. Although the KGQA models score lower than the other models, they do have advantages. Language models at the scale of T5-XL are computationally expensive and the knowledge stored in the parameters is static. Knowledge graph based methods like Rigel can be updated easily by updating the external knowledge graph and can also return more interpretable answers. Given the Marvel question again, Rigel predicts a path from the Marvel Cinematic Universe to all the Marvel films. Rigel can’t perform sorting or filtering, but it’s possible to see what the next steps should be: identifying the chronological order, ordering by chronological order, and finding the second in the ordered list. Understanding how a model has arrived at an answer and what steps should be added to arrive at the correct answer are useful features for debugging and improving KGQA models.

Table 6 shows results on Mintaka evaluated both multilingually on all languages and in each language individually. For almost all models, the results are slightly lower than in English. The MT5 language model has worse performance than the English T5 model, which may be because unlike T5, MT5 is not pre-trained on any supervised tasks. Our T5 for CBQA model using English translations outperforms MT5, however the scores are still lower than on the original English questions, so the automatic translations do degrade performance. For Rigel, the main gap to English is on the encoding side, where we use XLM-RoBERTa instead of RoBERTa, showing a gap in the performance between the multilingual encoder and English encoder. All models show that there is still work needed for parity across all languages.

Overall, the baselines show that Mintaka is a challenging dataset. None of our baselines explicitly handle all of the complexity types available in Mintaka. The language model-only models especially struggle to handle questions that require numerical operations such as counts. The knowledge graph-based models rely on relation following to traverse the knowledge graph to an answer. This prevents models from correctly predicting answers that require more complex operations, even if the facts required are available in the knowledge graph. Adding additional operations and learning to select the correct operation for each question could lead to significant improvement. A combination of powerful language models, potentially to encode questions or identify question entities, with the interpretable facts and operations available in a knowledge graph is a promising direction to create better models on Mintaka.