Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions

Large Language Models (LLMs) have demonstrated impressive performance on various tasks, surpassing that of supervised models and, in some cases, even outperforming humans (Chowdhery et al., 2022; Touvron et al., 2023; OpenAI, 2023). However, despite their impressive capabilities, previous research has highlighted certain limitations. For instance, LLMs have shown significant sensitivity to small changes in the prompt (Zhao et al., 2021; Wang et al., 2023a; Zhu et al., 2023). Therefore, a more comprehensive and conclusive analysis of different aspects that can affect/limit LLMs’ performance is crucial for a fair assessment and their successful real-world adoption.

One significant limitation lies in the robustness of LLMs concerning the arrangement of various components in a prompt, as it directly impacts the assessment of their capability in understanding and reasoning for specific tasks. Prior research has demonstrated that LLMs exhibit sensitivity to the arrangement of few-shot demonstrations (Zhao et al., 2021) and the order of appearance for responses generated by candidate models when LLMs are used as referees to evaluate quality (Wang et al., 2023b). Given these findings, it becomes pertinent to inquire whether LLMs are also sensitive to the order of elements of the prompts in different tasks. For example, how much does the order of options in multiple-choice question (MCQ) answering tasks can impact the LLMs performance.

In this paper, we investigating the sensitivity of LLMs to the order of options in multiple-choice questions; using it as a proxy to understand LLMs sensitivity to the order of prompt elements in instruction- or demonstration-based paradigm. We demonstrate an example of GPT-4’s sensitivity to options order in Figure 1, using a sample from the CSQA benchmark (Talmor et al., 2018). Notably, by merely rearranging the placement of options among choices A, C, and E, GPT-4 incorrectly predicts the answer to be “outside bedroom window”. Within this context, we aim to address the following research questions: (1) To what extent do LLMs exhibit sensitivity to the order of options in multiple-choice questions? (2) What factors contribute to LLMs’ sensitivity to the order of options? (3) How can we improve LLMs’ robustness to the order of options?

To answer the first question, we conducted experiments using GPT-4 (OpenAI, 2023) and InstructGPT (text-davinci-003) (Ouyang et al., 2022) on five different multiple-choice question benchmarks. Surprisingly, we discovered a substantial sensitivity gap of up to $75\%$ in the zero-shot setting. Additionally, in the few-shot setting, we observed that introducing demonstrations to the prompt only led to marginal improvements in LLMs’ robustness if their performance increased. Moving on to the second question, we put forth a conjecture that the sensitivity of LLMs stems from their positional bias, wherein they tend to favor certain placements when uncertain about the answer among the top choices. To validate our conjecture, we analyzed instances where the models’ predictions changed upon reordering the options. Furthermore, we demonstrated that the complexity of the number of choices, while retaining the top possible answers, had only a gradual impact (if any) on performance improvement.

Additionally, we discerned patterns in the occurrence of top-2 possible choices that influence the model’s probability of selecting a particular option or somewhat mitigate LLMs’ positional bias. For amplifying bias, we found that the optimal strategy involves positioning the top two choices as the first and last options. Conversely, to mitigate bias, we recommend placing these choices among the adjacent options. To validate our findings, we conducted qualitative evaluations. Addressing the last question, we demonstrated that employing two different calibrating approaches led to a notable improvement in LLMs’ performance, up to $8$ percentage points. Through these investigations, we contribute to a deeper understanding of how the order of options affects LLMs’ decision-making in multiple-choice questions (MCQ) and offer practical solutions, which go beyond simple bootstrapping, to increase their robustness and accuracy in such scenarios.

This paper focuses on the task of multiple-choice question answering. In multiple-choice questions, the objective is to identify the correct answer to a given question from a set of possible options (an illustration is presented in Figure 1). To address this task using in-context learning models, we present a prompt in the following format: “Choose the answer to the question only from A, B, C, D, and E choices. Question: {question}. Choices: {options}. Answer:” to the models. Also, for few-shot demonstrations we randomly choose 100 samples and draw demonstrations from those samples.

In this section, we first investigation the sensitivity of InstructGPT and GPT-4 to the order of options in the zero-shot setting. Then, we set out to determine whether introducing demonstrations to the prompt in the few-shot setting can enhance the models’ robustness. To quantify sensitivity, we calculate the sensitivity gap, which is the difference between the maximum and minimum LLMs’ performance when using an oracle ordering. In other words, we examine how specific reordering of options affects the models’ predictions when the ground truth is known.

After analyzing instances in which reordering the options resulted in a change in LLMs prediction, we arrive at the following conjecture:

In the subsequent sections, we begin by empirically validating the conjecture. Furthermore, we identify specific patterns in the options that either amplify or mitigate the model’s bias towards their placement.

We conduct an in-depth investigation into how large language models react to changes in the order of options, and investigate the reasons behind their sensitivity to such changes. Through our exploration, we have clearly observed that LLMs are highly responsive to the sequence in which options are presented. This has led us to a critical juncture where we need to focus on methods to improve the models’ resilience to variations in options order, ensuring more dependable and trustworthy evaluations.

One potential solution we have considered is the calibration of LLMs predictions. The outcomes of calibrating LLMs predictions to mitigate order sensitivity by taking majority vote over models prediction in 10 random reorders in a simple bootstrapping approach (Stickland and Murray, 2020; Hou et al., 2023), are provided in Table 5. Our analysis has unveiled a significant observation: employing a majority vote approach for evaluating LLMs across a range of benchmarks results in a substantial performance improvement of up to 8 percentage points. Notably, we find that GPT-4 showcases a higher level of stability against changes in options order compared to InstructGPT, a trait that persists even after implementing prediction calibration. Furthermore, while LLMs’ performance on benchmarks featuring four options might be somewhat inferior to those with three or five options, GPT-4 displays a greater resilience following prediction calibration. In contrast, InstructGPT demonstrates heightened robustness in specific contexts like CSQA and high school chemistry, where the effects of calibration are more pronounced.

We have also incorporated the approach of Multiple Evidence Calibration (MEC) introduced by Wang et al. (2023b). In their work, they propose to counteract LLMs’ sensitivity by prompting the model to generate an explanation before providing its prediction. We adopt their provided prompt for solving MCQ tasks. The impact of applying MEC calibration and its implications for order sensitivity are outlined in Table 5.

The results from InstructGPT performance reveal that the introduction of MEC calibration results in a consistent decrease in model performance. This behavior contradicts the outcomes achieved through majority voting and underscores the unsuitability of MEC calibration for multiple-choice question tasks. In the case of GPT-4, the integration of MEC calibration also yields contrasting outcomes with respect to majority voting, particularly evident in benchmarks such as CSQA, abstract algebra, and high school chemistry. For logical deduction and professional law benchmarks, while both majority voting and MEC calibration result in improving the model performance, the amount of improvement differs considerably, thus casting doubt on the reliability of the MEC approach in GPT-4 as well. Subsequent analysis of LLMs’ mispredictions following MEC calibration unveils a noteworthy trend: the act of asking the models to explain their reasoning amplifies their uncertainty (partly due to hallucination), particularly in instances where the model’s initial confidence was not very high. This, in turn, either frequently guides the models to opt for answer among the top choices, which is positioned earlier in the available options or causing the model to not picking any final answer.

Large language models (LLMs) show remarkable accomplishments and capabilities on various NLP tasks, including answering multiple-choice questions. In order to ascertain the dependability of LLMs’ proficiency, it becomes imperative to delve into the robustness of their performance when subjected to subtle changes in the input.

We investigate the inherent sensitivity of language models to the arrangement of options in multiple-choice questions. Upon measuring the intensity of LLMs sensitivity, our aim was twofold: to pinpoint the underlying source of this sensitivity and propose potential solutions to enhance the models’ robustness. Our evaluations unequivocally reveal that LLMs not only exhibit pronounced sensitivity to options order, but also that this sensitivity diminishes only slightly when demonstrations are integrated into the few-shot setting under specific circumstances.

While our primary focus has been on multiple-choice questions, we have also detected a parallel phenomenon—albeit with varying degrees of sensitivity—in other tasks involving multiple fragments (e.g. the options in MCQ) within inputs. This encompasses tasks like odd word detection, sorting lists of items, and ranking documents. While these observations have been noted, further exploration into these tasks is reserved for future efforts.

In seeking to uncover the root cause of order sensitivity, we conjecture that the issue arises from LLMs’ positional bias, particularly manifesting in uncertain instances. We verify our conjecture by conducting diverse experiments that highlight impactful patterns that either magnify or mitigate this positional bias. Despite the validation provided through detailed experimentation, a deeper comprehension of the issue’s origin necessitates a thorough exploration of the training data.

To improve the robustness of LLMs’ sensitivity against options order, we consider two calibration techniques: majority vote and multiple evidence calibration (MEC). While both methods display promising outcomes, contributing to the improvement of model performance, they are not without their respective limitations. Majority voting is computationally expensive, while MEC diverges significantly from majority voting, casting doubts on its applicability to MCQ tasks. As a result, in order to establish a reliable and accurate evaluation framework for LLMs in the context of multiple-choice questions, it is imperative to develop more efficient calibration strategies. Moreover, refining the evaluation metrics holds the potential to improve LLMs’ ability to withstand the challenges posed by options order sensitivity. These avenues present opportunities for in-depth exploration in future works.

We would like to thank Sameer Singh for his valuable comments.