Learning New Tasks from a Few Examples with Soft-Label Prototypes

Avyav Kumar Singh Affiliation: Department of Informatics, King’s College London, United Kingdom Email: avyavkumar.singh@kcl.ac.uk Ekaterina Shutova Affiliation: ILLC, University of Amsterdam, the Netherlands Email: e.shutova@uva.nl Helen Yannakoudakis Affiliation: Department of Informatics, King’s College London, United Kingdom Email: helen.yannakoudakis@kcl.ac.uk

Abstract

Existing approaches to few-shot learning in NLP rely on large language models (LLMs) and/or fine-tuning of these to generalise on out-of-distribution data. In this work, we propose a novel few-shot learning approach based on soft-label prototypes (SLPs) designed to collectively capture the distribution of different classes across the input domain space. We focus on learning previously unseen NLP tasks from very few examples ( $4$ , $8$ , $16$ ) per class and experimentally demonstrate that our approach achieves superior performance on the majority of tested tasks in this data-lean setting while being highly parameter efficient. We also show that our few-shot adaptation method can be integrated into more generalised learning settings, primarily meta-learning, to yield superior performance against strong baselines.

1 Introduction

Humans have a remarkable ability to adapt knowledge gained in one domain and apply it in another setting, and to identify or disambiguate objects after observing only a handful of examples Lake et al. (2015). This has inspired research in few-shot learning that aims to build models that can learn a new task using only a small number of examples per class. Early few-shot learning in NLP relied on interventions at the data level, such as dataset augmentation Clark et al. (2018) or generation of adversarial examples from few-shot datasets Miyato et al. (2016), while more recent approaches van der Heijden et al. (2021); Langedijk et al. (2022) utilise meta-learning Finn et al. (2017); Snell et al. (2017) to optimise model parameters such that models adapt quickly to new tasks using past experience Dou et al. (2019); Holla et al. (2020); van der Heijden et al. (2021). The advent of large language models (LLMs) has led to a plethora of further methods, including fine-tuning on different target tasks Sun et al. (2020); Zhou and Srikumar (2022), creating prompt-enhanced few-shot datasets for training Gao et al. (2020); Schick and Schütze (2020); Lester et al. (2021) as well as parameter-efficient fine-tuning methods for very large language models, with parameters running into billions Hu et al. (2022); Dettmers et al. (2023).

In this paper, we propose a simple and effective approach to few-shot learning based on soft-label prototypes (SLPs) that capture the distribution of different classes across the input domain space, inspired by previous work on generating compact representations of input training data Sucholutsky et al. (2021). Our contributions are summarised as follows: 1) We develop a novel neural framework for few-shot learning via soft-label prototypes that has a very small computational and memory footprint, and achieves state-of-the-art results in limited-resource settings. Our approach (DeepSLP) does not rely on (expensive) LLM parameter updates or auxiliary training data. 2) We focus on few-shot learning of new, unseen NLP tasks using as little as 4 examples per class, and demonstrate that we outperform strong baselines on the majority of test tasks. 3) We demonstrate that our approach can also be effectively adapted (MetaSLP) in high-resource settings when auxiliary training data is available for few-shot learning, and performs competitively when compared against strong baselines. 4) We release our code and data to facilitate further research in the field.¹¹ 1 https://github.com/avyavkumar/meta-learned-lines

2 Related work

Early few-shot learning approaches in NLP include data augmentation and semi-supervised learning; e.g., augmentation with adversarial examples Miyato et al. (2016), interpolation of training data into a learnable higher dimensional embedding space Chen et al. (2020), and consistency training to make models more resistant to noise Xie et al. (2019). Recent research efforts on large-scale pre-training of language models Devlin et al. (2019); Radford et al. (2019); Brown et al. (2020); Touvron et al. (2023); BigScience Workshop (2023); OpenAI (2024) reduce the amount of data required for their subsequent fine-tuning or utilisation in a given task. Instruction tuning and in-context learning Brown et al. (2020); Gao et al. (2020); Sanh et al. (2021); Liu et al. (2021); Min et al. (2022); Sun et al. (2024); Zhou et al. (2024) show that natural language instructions or prompts can enhance a model’s few-shot learning abilities by leveraging the language (instruction) understanding abilities of the given pretrained LLM Zhao et al. (2021); Liu et al. (2022). To fine-tune extremely large language models (with billions of parameters) efficiently, a host of parameter-efficient fine-tuning techniques have also been developed Hu et al. (2022); Dettmers et al. (2023), which leverage pre-trained LLMs and produce superior results on tasks such as question answering, reasoning, text summarisation, coding, etc. Kotitsas et al. (2024); Jiaramaneepinit et al. (2024); Yang et al. (2024); Ding et al. (2023).

However, the search space over LLMs, prompt templates and few-shot learning is so great that there is yet to be an established standard. Different models require different styles of (few-shot) prompting, and certain prompt templates may work better with specific LLMs and datasets rather than universally across the board (e.g., Davis et al. (2024)). Furthermore, evaluating robustness of state-of-the-art / generative LLMs on new, unseen tasks (OOD generalisation) presents a significant challenge due to their vast and unknown training data, resulting in artificially inflated performance as a result of data leakage Yang et al. (2023).

Previous work has also tackled few-shot learning within the meta-learning paradigm of “learning to learn” Schmidhuber (1987); Bengio et al. (1990); Thrun and Pratt (1998), utilising methods that are trained to adapt quickly (in a few gradient steps) to new tasks and from a small number of examples, using past experience. Meta-learning has emerged as a promising technique for a range of tasks Finn et al. (2017); Koch et al. (2015); Ravi and Larochelle (2017), including NLP such as natural language inference, text classification, etc. Obamuyide and Vlachos (2019a); Obamuyide and Vlachos (2019b); Holla et al. (2020); Bansal et al. (2020b); Nooralahzadeh et al. (2020); Wang et al. (2020); Langedijk et al. (2022); Mueller et al. (2023).

In a similar spirit to parameter-efficient fine-tuning Hu et al. (2022); Dettmers et al. (2023), our work shifts away from the aforementioned paradigms that suffer from lack of standardisation (LLM few-shot prompting) and increased computational complexity for fine-tuning (e.g., fine-tuning extremely large language models such as GPT OpenAI (2024). We present a novel, parameter-efficient few-shot learning framework (DeepSLP) based on soft-label prototypes (SLPs), which we show to be effective on a range of tasks in limited and high-resource settings, while having a substantially smaller computational and memory footprint. While DeepSLP does not rely on LLM fine-tuning or auxiliary training data, we present a variant (MetaSLP) that can be used for few-shot learning via auxiliary data and fine-tuned encoders.

We target few-shot learning of new, unseen tasks (i.e., tasks and classes not previously trained on) (a) in limited-resource settings, without access to auxiliary training data, and with frozen model parameters, and (b) in high-resource settings, with access to auxiliary training data, which are used to update model parameters. For the latter, our work is similar to Bansal et al. (2020a). The authors target few-shot learning of unseen tasks via meta-learning, utilising auxiliary training data.

3 Approach: few-shot learning with Soft-Label Prototypes (SLPs)

A soft label is defined as a vector $Y$ representing a data point’s simultaneous membership to several classes Sucholutsky and Schonlau (2021), essentially denoting a point’s partial association to different classes. Using this definition, a soft-label prototype (SLP) is defined as $(\vec{X},Y)$ , where $\vec{X}$ is a point in input space (e.g., an input feature vector) and $Y$ is its corresponding soft label. The underlying idea in Sucholutsky and Schonlau (2021)’s work is that a small set of soft-label prototypes can be used to accurately represent a training set. We build on this idea and reframe SLPs for the task of few-shot learning of new, unseen tasks where very small amounts of data are available per class.

3.1 Generating soft-label prototypes

Soft-label prototypes assign soft labels to every point in the input domain; therefore, a soft-label prototype at point $\vec{X}$ represents the class distribution (determined from the training data) at $\vec{X}$ . The fundamental idea behind a “soft-label” is that, unlike hard labels, which are one-hot encoded labels, soft-labels contain a distribution of probabilistic label values at a particular point in a high-dimensional embedding space.

The process of generating soft-label prototypes from training data can be split into a two step process Sucholutsky et al. (2021): (1) finding lines that connect the class centroids in the training data, where each line connects some of the centroids, and every centroid belongs to one line; and (2) using linear constraints to derive soft-label prototypes capturing the class distribution at the ends of each line. The two steps are presented in detail below.

Refer to caption — Figure 1: Learning soft-label prototypes using two trainable linear layers (yellow): example for a 3-class prototype. Dotted lines indicate backpropagation.

3.1.1 Finding lines connecting all centroids

Here we seek to find classes that lie on the same manifold. First, we compute the centroid of each class in the input training data. Then, we find and fit class centroids on the minimum number of lines using recursive regression Sucholutsky et al. (2021). This method clusters centroids hierarchically to group similar (interval) centroids together, and fits a regression line on the centroids. The similarity of centroids within a single cluster is judged by how well all the centroids fit on a regression line. If the Euclidean distance of a particular centroid is beyond a pre-defined tolerance threshold $\epsilon$ from a line, it is removed from that cluster and assigned to another one. We use this method for all our experiments, as we experimentally find (on our dev data) that it performs well on high-dimensional data spread across many classes such as the ones we test here. In Appendix A.1, Figure 3(a), we present an example set of lines connecting all centroids.

3.1.2 Learning soft-label prototypes

Once we find the lines, we use the endpoints of each line as the location of soft-label prototypes. Therefore, for $l$ lines fitted on $n$ centroids we have $2l$ prototypes. Then, we need to find the class distribution at each end point / soft-label prototype. We develop and experiment with two different approaches to finding the class distributions, one based on constraint optimisation (constraintSLP), and another based on gradient descent (DeepSLP).

Learning via linear constraints (constraintSLP)

One way in which we can approach this is via constraint optimisation and, specifically, an optimisation problem that consists of two main sets of constraints Sucholutsky et al. (2021): (i) the target class at each centroid has the maximum influence amongst all classes at certain points along the line (endpoint of the line and midpoints between classes); and (ii) the difference between the influence of the target class and the sum of the influences of all other classes along the line is maximised. In order to make this approach powerful enough for large, high-dimensional NLP data, we require an optimiser that scales on such complex data. To this end, we use the MOSEK solver for linear programming MOSEK ApS (2019) in the CVXPY library Diamond and Boyd (2016) to perform the required computations to generate the soft-label prototype class distributions. The output of this is then a set of soft-label prototypes which “sit” at the ends of each line (i.e., $\vec{X}$ ) as shown in Appendix A.1, Figure 3(b).

Learning via gradient descent (DeepSLP)

Rather than use linear constraints to generate soft-label prototypes, we develop a novel gradient-based approach to generate soft labels as a function of an input $x$ by minimising training loss on a few-shot dataset. After generating lines connecting all class centroids (Section 3.1.1), we set two soft-label prototypes at the ends of the lines. Each soft-label prototype $p_{i}$ is denoted by $g_{i}(f(x))$ where $g$ is a neural network parameterised by $\theta_{i}$ and $f(x)$ is a point in the input space. The neural network consists of a fixed BERT Devlin et al. (2019) encoder²² 2 We use BERT as our encoder given its comparatively higher computational efficiency, and do not include LLMs such as Llama and the GPT family as they have already been pre-trained on our test tasks (found here) and hence suffer from data contamination. given by $f(x)$ , and a trainable linear layer which returns the soft-label probability distribution at any point $x$ given by $g_{i}(x)$ . Figure 1 presents a visual representation of our model. Compared to constraintSLP where we find soft-label probability distributions via linear constraint optimisation, we now parameterise our soft-label probability distribution with a neural network.

Algorithm 1 DeepSLP

\lambda\leftarrow

set of lines connecting all centroids

f_{\theta_{l}}

is the network parameterised by

\theta_{l}

for the left-end prototype on a line

f_{\theta_{r}}

is the network parameterised by

\theta_{r}

for the right-end prototype on a line

\mathcal{J}\leftarrow

loss function

D\leftarrow

training data

5 Require

\;\lambda\neq\emptyset

6 for $i\in\lambda$ do

7 for $epoch\;1.....N$ do

p_{il}\leftarrow

location of left prototype

p_{ir}\leftarrow

location of right prototype

10 for $x\in minibatch(D)$ do

d_{1}\leftarrow

||p_{il}-x||

d_{2}\leftarrow

||p_{ir}-x||

pred.append\left(\frac{f_{\theta_{il}}(x)}{d_{1}}+\frac{f_{\theta_{ir}}(x)}{d_{2}}\right)

14 end for

d\_loss\leftarrow\mathcal{J}(pred,D)

loss_{1}\leftarrow\frac{d_{2}}{d_{1}+d_{2}}*d\_loss

loss_{2}\leftarrow\frac{d_{1}}{d_{1}+d_{2}}*d\_loss

\theta_{il}\leftarrow\theta_{il}-\eta\nabla_{\theta_{il}}loss_{1}

\theta_{ir}\leftarrow\theta_{ir}-\eta\nabla_{\theta_{ir}}loss_{2}

20 end for

21 end for

Crucially, the encoder parameters are frozen as we need our input data points to have an unchanged location in the input space – changing their position might result in class centroids that were previously lying on a straight line to no longer lie on the line. The model is optimised using both soft-label prototypes along a line (see Algorithm 1 below and Section 3.2, Equation 1). Specifically, a higher distance between a data point $x$ and a prototype leads to a correspondingly smaller effect of the prototype on the final classification; therefore, we want to penalise the prototype that is closer to $x$ more if there is an incorrect classification. Each prototype is therefore assigned a fraction of the total loss that is proportional to the other prototype’s Euclidean distance from $x$ . This way, the closer prototype’s weights are corrected more in case of a misclassification. The complete algorithm can be seen in Algorithm 1, while an example optimisation is presented in Figure 2.

We use cross entropy loss which gives a measure of the difference between the true and predicted labels. We initialise the weights of $g_{1}(x)$ and $g_{2}(x)$ using a uniform Xavier initialisation Glorot and Bengio (2010) and use warmup steps to adjust the learning rate. Epochs vary based on the number of classes in the classifier head (between 15 and 25; see datasets used in Section 4): preliminary experiments on the development data show that more epochs are needed when a higher number of classes lie along a line.

3.2 Classification with soft-label prototypes

Given $M$ soft-label prototypes representing the input distribution of $N$ classes, we define $S=(\vec{X_{1}},Y_{1}),...,(\vec{X_{M}},Y_{M})$ to be our set of prototypes, where $\vec{X_{i}}$ is the location of the $i^{th}$ prototype in the input feature space and $Y_{i}$ is a matrix of dimension $[N\times 1]$ denoting the soft labels. Given a test datapoint $\vec{x}$ , we calculate the Euclidean distances ${D}={(\vec{X_{i}},\vec{x})}_{i=1,2...M}$ from each prototype to $\vec{x}$ . We then sort $S$ in ascending order of distances using ${D}$ , weigh the probability distribution of the $i^{th}$ nearest prototype inversely by its distance to $\vec{x}$ , and select the line containing the closest prototype to get $Y^{*}$ Sucholutsky et al. (2021):

Y^{*}=\sum_{i=1}^{k}\frac{Y_{i}}{d(\vec{X_{i}},\vec{x})}

(1)

As we consider the two nearest neighbours / prototypes, we set $k$ to $2$ . $\vec{x}$ is then assigned the class $C^{SLP}(\vec{x})=\underset{j}{\mathrm{argmax}}Y^{*}_{j}$ where $Y^{*}_{j}$ is the j^th element of $Y^{*}$ . In other words, we sum over the k-nearest soft-label prototypes (i.e., vectors) to $\vec{x}$ , and weigh each prototype inversely proportional to its distance from $\vec{x}$ . $\vec{x}$ is then assigned the class with the largest value in the resulting vector (see Appendix A.1 for a toy classification example).

3.3 Meta-training DeepSLP (MetaSLP)

We further test the suitability of soft-labels in high-resource settings, tuning our text encoder using auxiliary training data. This is similar to the work of Bansal et al. (2020a) that develop a meta-learning approach for few-shot learning of new, unseen tasks while utilising auxiliary training data. We employ a similar approach for rapid generalisation by utilising first-order meta-learning algorithms (which we describe in detail in Appendix A.2). Our model architecture is similar to DeepSLP – it comprises a BERT encoder with two linear layers on top. We train only the last $v$ layers of our encoder to reduce computational overhead, where $v$ is a hyperparameter (See Appendix A.6). We denote the encoder by $f_{\theta}(x)$ , and each soft-label prototype at the end of a line by $g_{1}$ and $g_{2}$ , parameterised by $\theta_{1}$ and $\theta_{2}$ respectively. The difference between DeepSLP and MetaSLP is that MetaSLP is trained using meta-learning (using auxiliary data described in Section 4), and the encoder $f_{\theta}(x)$ in MetaSLP is fine-tuned (as opposed to being fixed in DeepSLP), following previous work Bansal et al. (2020a).

Inner-loop training

We optimise the soft-label prototypes in the same manner as DeepSLP; i.e., we use Algorithmn 1 to few-shot train the linear layers $g_{1}(x)$ and $g_{2}(x)$ parameterised by $\theta_{1}$ and $\theta_{2}$ respectively. However, meta-learning requires a large set of diverse and balanced meta-learning tasks for effective learning Holla et al. (2020). To ameliorate this, we split the auxiliary datasets (Section 4) used for meta-learning into multiple pairwise tasks to meta-train MetaSLP Bansal et al. (2020a). This means that, during training, we now consider a large number of two-class problems, as opposed to a small number of multi-class problems where the number of classes $n\geq 2$ . Such a setting also enables fine-tuning of our encoder $f_{\theta}(x)$ . In general, allowing the physical location of encodings to change (in this case via meta-learning’s inner-loop training process), may result in centroids originally connected by a line to no longer be connected by that line (i.e., in the next inner-loop optimisation step). However, if we only meta-train on tasks that focus on two classes at a time, this can trivially ensure that the same line is utilised each time. Our inner-loop optimisation process is given in Algorithm 3 in Appendix A.4.

Outer-loop training

We perform meta-learning using the updated parameters in the inner-loop training process. We experiment with both Reptile Nichol et al. (2018) and FOMAML Finn et al. (2017) as our meta-learning algorithms. Reptile can be considered a simpler variant to MAML-based meta-learning algorithms. We present our outer-loop process in Algorithm 4, Appendix A.4.

Meta-testing

We construct lines for the test sets using the trained MetaSLP model, and fine-tune them on the few-shot adaptation training data for each test task. We then use these lines for classification as described in Section 3.2.

4 Experimental settings and datasets

Experimental settings

We experiment with two settings in terms of amounts of available data. The first is a limited-resource setting where we only train / fine-tune our models in a few-shot manner on small amounts of training data (i.e., in the absence of auxiliary training data). The other setting is a high-resource setting where we assume that auxiliary training data is available for additional training / fine-tuning.

Datasets

We tackle few-shot learning of previously unseen tasks (i.e., not seen during training/fine-tuning), and so our work is similar to Bansal et al. (2020a). For our high-resource setting, we train and test our models on the same data as Bansal et al. (2020a) to ensure direct comparability. For our limited-resource setting, we test in the same way but do not utilise any auxiliary training data; i.e., we only utilise few-shot fine-tuning data for unseen tasks (i.e., only using a very small set of training/fine-tuning examples for the test tasks).

High-resource setting auxiliary training data

Similar to Bansal et al. (2020a), we use GLUE (Wang et al., 2018) to train our models in the high-resource setting. This dataset consists of a range of natural language tasks such as entailment, classification and textual similarity, which are used for model training and evaluation. We use only the training split for meta-learning. Similar to Bansal et al. (2020a), the MNLI Williams et al. (2018) and SNLI Bowman et al. (2015) entailment tasks, which are three-label classification problems, are split in a pairwise manner such that they are included as multiple two-label datasets during training. Following Bansal et al. (2020a), we also train for detecting the sentiment contained within phrases of a sentence by using the phrase-level annotations in SST2 Wang et al. (2018). We utilise the same validation sets – labelled Amazon review data from music, toys and videos for sentiment classification Blitzer et al. (2007). We provide dataset and training details in Appendix A.5.

Evaluation data

We use the same test datasets and evaluation setting as Bansal et al. (2020a) for both the high-resource and low-resource settings. These cover a variety of text classification tasks: (a) Entity typing – the CoNLL-2003 Tjong Kim Sang and De Meulder (2003) and MIT-Restaurant Liu et al. (2013) datasets; (b) Review rating classification – review ratings from Amazon Reviews Blitzer et al. (2007) with a three-way classification; (c) Text classification – scraped social media data from crowdflower comprising sentiment and emotion classification in a range of domains, as well as political bias detection; and (d) Natural language inference in the scientific domain – the SciTail dataset Khot et al. (2018). We use the same data splits, which are publicly available. During evaluation, Bansal et al. (2020a) fine-tune their models using a small few-shot (support) training set per test task (using a $k$ -shot setting of $4,8,16$ examples per class), and then evaluate performance on each task’s dedicated, unseen test set. As model performance can be affected by the $k$ examples chosen for training/fine-tuning, for each task and for every k, they sample $10$ few-shot training sets and report the mean and standard deviation, which we also adopt in our experiments.

5 Baselines

Our aim is to determine how well our models – DeepSLP and MetaSLP – perform in the low and high-resource setting respectively, compared to strong baselines when given the same few-shot adaptation sets. Our focus is on (a) evaluating in an extreme few-shot learning scenario where no auxiliary data is available (using DeepSLP), and (b) evaluating in a high-resource setting when additional (auxiliary) training data is available (MetaSLP). We use BERT Devlin et al. (2019) as our text encoder throughout to facilitate model comparisons. We report DeepSLP and MetaSLP_REPTILE (i.e., using Reptile as our meta-learning algorithm) in our main table of results (given their effectiveness), and present additional baselines as well as hyperparameters and training details in Appendix A.6.

We use BERT Devlin et al. (2019) as our encoder as it is a text encoder that allows us to get passage-level encodings, it is computationally light-weight compared to decoder-based LLMs such as Llama Touvron et al. (2023) and GPT OpenAI (2024) (and we can carry out full fine-tuning) and, crucially, it does not suffer from data contamination as, in contrast to the more recent LLMs, it has not been pre-trained on our test data³³ 3 https://github.com/iesl/leopard/tree/master/data/json.

5.1 Low-resource setting baselines

BERT_fine-tuned

We use BERT_fine-tuned reported in Bansal et al. (2020a), which is fine-tuned (all layers) on the few-shot training set of each test task.

LORA_BERT

LORA Hu et al. (2022) decomposes a fine-tuned weight matrix to two low-rank matrices, which – when multiplied and added to the original weights – reproduce the fine-tuned weights. This is advantageous as, instead of fine-tuning all parameters, we fine-tune these two matrices with a low computational cost, as they are much smaller individually compared to fully fine-tuned weights.

constraintSLP

We use constraintSLP as a baseline to evaluate the effectiveness of soft-label prototypes that are based on linear constraints.

5.2 High-resource setting baselines

Reptile

We train a meta-learning Reptile Nichol et al. (2018) model on our auxiliary data and use it as another baseline. This allows us to directly assess the added advantage of utilising SLPs in MetaSLP_REPTILE.

Prototypical Networks

We use ProtoNet Snell et al. (2017) as another baseline for both the high and low-resource settings. ProtoNets use Euclidean distance as a measure of similarity between points and clusters, which is similar to DeepSLP and MetaSLP that assign test points to the closest line.

LEOPARD

Bansal et al. (2020a) present LEOPARD, a meta-learning algorithm that achieves the best performance across most test tasks for entity typing, ratings classification and text classification, and which we also use.

We do not include HSMLMT Bansal et al. (2020b) as a baseline as it is pretrained on semi-supervised meta-training tasks in addition to supervised learning and therefore it is not directly comparable.

Category (Classes)	Shot	LORA_BERT	BERT_fine-tuned*	DeepSLP	LEOPARD*	Reptile	MetaSLP_REPTILE
Political Bias (2)	4	52.75 ± 4.33	54.57 ± 5.02	53.251 ± 4.042	60.49 ± 6.66	58.82 ± 4.31	60.96 ± 6.13
	8	53.66 ± 4.25	56.15 ± 3.75	58.209 ± 5.198	61.74 ± 6.73	59.43 ± 3.79	63.65 ± 4.57
	16	59.21 ± 2.27	60.96 ± 4.25	61.479 ± 2.974	65.08 ± 2.14	62.21 ± 0.72	66.05 ± 1.57
Emotion (13)	4	7.56 ± 2.93	09.20 ± 3.22	9.076 ± 1.108	11.71 ± 2.16	11.65 ± 3.21	11.94 ± 1.95
	8	9.02 ± 2.36	08.21 ± 2.12	8.041 ± 2.797	12.90 ± 1.63	10.56 ± 2.85	13.42 ± 1.46
	16	10.29 ± 1.67	13.43 ± 2.51	10.919 ± 1.615	13.38 ± 2.20	11.62 ± 3.11	14.03 ± 2.35
Sentiment Books (2)	4	51.27 ± 2.75	54.81 ± 3.75	58.67 ± 4.753	82.54 ± 1.33	76.95 ± 1.03	83.22 ± 0.95
	8	58.16 ± 3.3	53.54 ± 5.17	64.78 ± 2.615	83.03 ± 1.28	77.49 ± 1.08	83.8 ± 0.8
	16	59.16 ± 2.59	65.56 ± 4.12	67.453 ± 3.085	83.33 ± 0.79	77.88 ± 0.56	83.8 ± 1.59
Rating DVD (3)	4	31.65 ± 4.91	32.22 ± 08.72	39.566 ± 5.086	49.76 ± 9.80	45.91 ± 9.85	45.2 ± 8.91
	8	37.69 ± 3.16	36.35 ± 12.50	38.788 ± 4.449	53.28 ± 4.66	47.23 ± 9.22	58.38 ± 2.9
	16	38.63 ± 5.52	42.79 ± 10.18	40.53 ± 4.375	53.52 ± 4.77	48.49 ± 8.88	57.41 ± 4.71
Rating Electronics (3)	4	31.66 ± 2.94	39.27 ± 10.15	39.977 ± 5.959	51.71 ± 7.20	44.47 ± 8.25	45.34 ± 7.22
	8	38.72 ± 5.95	28.74 ± 08.22	41.926 ± 3.985	54.78 ± 6.48	49.1 ± 6.81	55.10 ± 5.12
	16	39.15 ± 6.6	45.48 ± 06.13	44.917 ± 3.164	58.69 ± 2.41	50.68 ± 6.8	59.47 ± 2.29
Rating Kitchen (3)	4	36.63 ± 4.68	34.76 ± 11.20	39.624 ± 6.787	50.21 ± 09.63	45.38 ± 10.96	45.20 ± 8.78
	8	39.69 ± 6.22	34.49 ± 08.72	41.081 ± 6.777	53.72 ± 10.31	46.71 ± 9.84	54.53 ± 9.9
	16	38.17 ± 7.14	47.94 ± 08.28	45.801 ± 4.562	57.00 ± 08.69	52.87 ± 9.52	58.94 ± 7.58
Political Audience (2)	4	49.75 ± 1.03	51.02 ± 1.72	51.741 ± 2.827	52.60 ± 3.51	52.45 ± 4.26	54.1 ± 3.66
	8	54.05 ± 2.54	52.80 ± 2.72	54.506 ± 3.274	54.31 ± 3.95	52.87 ± 4.31	56.01 ± 3.65
	16	55.39 ± 3.66	58.45 ± 4.98	56.956 ± 3.045	57.71 ± 3.52	55.6 ± 1.85	58.57 ± 2.04
Sentiment Kitchen (2)	4	53.02 ± 1.54	56.93 ± 7.10	60.76 ± 4.426	78.35 ± 18.36	69.81 ± 14.58	81.96 ± 3.73
	8	55.54 ± 3.47	57.13 ± 6.60	65.733 ± 3.198	84.88 ± 1.12	75.76 ± 1.13	83.33 ± 1.99
	16	58.59 ± 4.83	68.88 ± 3.39	69.18 ± 2.589	85.27 ± 1.31	76.41 ± 0.66	84.33 ± 1.81
Disaster (2)	4	56.02 ± 6.35	55.73 ± 10.29	54.252 ± 9.843	51.45 ± 4.25	49.76 ± 4.73	55.03 ± 8.73
	8	57.46 ± 6.9	56.31 ± 09.57	61.3 ± 7.961	55.96 ± 3.58	52.17 ± 5.17	57.77 ± 6.40
	16	65.79 ± 2.03	64.52 ± 08.93	69.28 ± 2.358	61.32 ± 2.83	55.37 ± 4.53	65.18 ± 4.41
Airline (3)	4	24.36 ± 5.42	42.76 ± 13.50	50.987 ± 4.936	54.95 ± 11.81	57.11 ± 14.16	57.39 ± 7.83
	8	52.31 ± 7.89	38.00 ± 17.06	55.209 ± 6.049	61.44 ± 03.90	64.37 ± 3.49	65.67 ± 4.82
	16	54.1 ± 8.57	58.01 ± 08.23	60.247 ± 4.577	62.15 ± 05.56	66.31 ± 2.55	69.48 ± 2.06
Rating Books (3)	4	34.69 ± 2.12	39.42 ± 07.22	42.116 ± 4.725	54.92 ± 6.18	56.57 ± 8.17	55.79 ± 5.61
	8	39.36 ± 6.33	39.55 ± 10.01	42.156 ± 4.608	59.16 ± 4.13	57.33 ± 7.63	65.74 ± 5.58
	16	41.23 ± 5.32	43.08 ± 11.78	46.513 ± 3.036	61.02 ± 4.19	63.26 ± 3.59	67.87 ± 3.45
Political Message (9)	4	12.16 ± 1.46	15.64 ± 2.73	14.421 ± 1.095	15.69 ± 1.57	14.58 ± 1.78	18.84 ± 1.82
	8	15.71 ± 2.04	13.38 ± 1.74	16.919 ± 1.756	18.02 ± 2.32	15.13 ± 2.16	20.09 ± 2.71
	16	15.53 ± 2.55	20.67 ± 3.89	18.319 ± 1.74	18.07 ± 2.41	16.38 ± 2.15	23.22 ± 1.17
Sentiment DVD (2)	4	50.77 ± 0.78	54.98 ± 3.96	55.003 ± 2.936	80.32 ± 1.02	72.03 ± 11.61	80.97 ± 1.21
	8	52.24 ± 1.54	55.63 ± 4.34	57.527 ± 3.562	80.85 ± 1.23	75.79 ± 1.62	81.85 ± 1.79
	16	52.6 ± 2.09	58.69 ± 6.08	60.76 ± 2.944	81.25 ± 1.41	76.69 ± 0.8	83.48 ± 1.01
Scitail (2)	4	43.36 ± 4.74	58.53 ± 09.74	54.101 ± 3.759	69.50 ± 9.56	59.13 ± 10.58	53.48 ± 5.59
	8	54.29 ± 5.25	57.93 ± 10.70	56.341 ± 5.786	75.00 ± 2.42	62.63 ± 10.85	60.79 ± 4.6
	16	52.68 ± 3.0	65.66 ± 06.82	59.692 ± 4.227	77.03 ± 1.82	68.03 ± 1.57	61.67 ± 3.61
Restaurant (8)	4	10.56 ± 1.36	49.37 ± 4.28	47.634 ± 5.237	49.84 ± 3.31	13.37 ± 2.25	27.00 ± 2.61
	8	20.92 ± 2.4	49.38 ± 7.76	55.912 ± 4.494	62.99 ± 3.28	16.83 ± 3.42	35.66 ± 2.39
	16	29.37 ± 4.05	69.24 ± 3.68	61.716 ± 2.208	70.44 ± 2.89	16.0 ± 3.44	37.20 ± 2.68
CoNLL (4)	4	21.48 ± 2.71	50.44 ± 08.57	52.724 ± 5.84	54.16 ± 6.32	31.31 ± 5.32	40.79 ± 3.40
	8	29.84 ± 3.28	50.06 ± 11.30	60.374 ± 3.731	67.38 ± 4.33	33.17 ± 5.1	41.25 ± 5.21
	16	37.18 ± 3.32	74.47 ± 03.10	67.496 ± 4.551	76.37 ± 3.08	34.04 ± 3.59	45.96 ± 4.75

Table 1: Classification performance (accuracy) of our methods (DeepSLP and MetaSLP) and baselines. * refers to the baselines as reported in Bansal et al. (2020a). The best performing models for each setting (without and with auxiliary data) are highlighted in gray and green respectively. Double lines group similar tasks together: the first set contains intent classification tasks, the second focuses on natural language inference, and the last contains entity typing tasks.

6 Results and Discussion

Due to space restrictions, we present and discuss our best models in Table 1. All other baselines and results are discussed in Appendix A.7.

Low-resource setting

In the low-resource setting with no auxiliary data (left side of Table 1), DeepSLP outperforms all baselines, including BERT_fine-tuned in $31/48$ tasks and LORA_BERT in $45/48$ tasks, achieving a new state-of-the-art result. Our results demonstrate the usefulness of soft-label prototypes and their superiority over strong baselines (i.e., LLM fine-tuning and LORA / low-rank adaptation) in the low-resource setting.

Unlike BERT_fine-tuned, DeepSLP and LORA_BERT do not fine-tune the encoder. Specifically, we only need to fine-tune $1500-10K$ parameters (based on the number of classes) for each line with two soft-label prototypes for DeepSLP, compared to $50K-100K$ parameters for LORA_BERT with $rank=2$ , and $>10^{8}$ parameters for BERT_fine-tuned. We also note that DeepSLP is lightweight and does not require a GPU. LORA_BERT mostly achieves accuracies within $90\%$ of BERT_fine-tuned, in line with previous work Hu et al. (2022); Dettmers et al. (2023) (even outperforming BERT_fine-tuned in $15/48$ tasks), except for entity-typing tasks where LORA_BERT struggles to generalise and achieves substantially lower performance compared to BERT_fine-tuned.

On the other hand, constraintSLP, a simpler variant of DeepSLP (see Appendix A.7, Table 5 for results) is one of the lower performing baselines, together with ProtoNet. We find that constraintSLP exhibits a substantial weakness (see further details in Theorem A.2, Appendix A.3): given Euclidean distance, constraintSLP does not always select the nearest class centroid to a test point. This violates our inductive bias that points located closest to a class centroid are assigned to that class. If we consider the case where $N=2$ , constraintSLP essentially acts as a 1-NN with soft labels trivially at $[1,0]$ and $[0,1]$ , with class centroids acting as the nearest neighbour. However, when generalising beyond this setting, the model’s stability is affected. constraintSLP optimises soft labels using the geometric properties of a line and does not consider each (training) data point individually – the soft labels produced by SLP are constants. DeepSLP, on the other hand, learns from training data and produces soft labels as a function of the input; therefore, it has the ability to output soft labels based on the location of an input (test) point (with the location of the prototypes being fixed).

High-resource setting

MetaSLP_REPTILE has the highest performance overall in text classification and entailment tasks (Tasks 1-14), with the best accuracy in $33/42$ tasks/settings. LEOPARD, on the other hand, achieves the highest score in only $8/42$ . Interestingly, we find that all models in the high-resource setting have lower performance for Disaster compared to the models in the low-resource setting. We surmise this to be due to the auxiliary data and the fact that the meta-training distribution differs substantially from the test distribution.

For entity typing tasks (CoNLL and Restaurant), LEOPARD outperforms all models, with MetaSLP_REPTILE and Reptile performing comparatively poorly, even outperformed by the low-resource methods (DeepSLP and BERT_fine-tuned). It should be noted that there seems to be little benefit of meta-learning with auxiliary data when tackling entity typing tasks, even for LEOPARD, as the difference between LEOPARD and BERT_fine-tuned is not substantial. We surmise this to be due to the fact that the meta-training distribution (i.e., GLUE tasks) is different from the test distribution for entity typing tasks which degrades performance for the test tasks. Note that we do not meta-train the entire model (only top 4 layers for Reptile and MetaSLP_REPTILE), unlike LEOPARD.

Overall, MetaSLP_REPTILE outperforms all models and baselines, including Reptile in $42/48$ tasks and LEOPARD in $34/48$ tasks. Specifically, MetaSLP_REPTILE consistently outperforms Reptile, demonstrating the effectiveness of our approach over its meta-learning variant (i.e., Reptile) that does not use SLPs. In Appendix A.8 we present detailed analyses of DeepSLP and show that it displays several desirable properties of ensemble methods which drive its performance, in addition to it being a computationally efficient approach that only utilises a small number of parameters.

7 Conclusion and future work

We presented a novel few-shot learning paradigm that is based on soft-label prototypes capturing the simultaneous membership of data points over several classes, and demonstrated its effectiveness in low and high-resource settings. We evaluated our approach on $48$ different tasks / settings and showed that it outperforms a range of strong baselines. In the future, we plan to use meta-learning algorithms such as PACMAML Ding et al. (2021) and Bayesian MAML Kim et al. (2018) that relax assumptions with respect to train–test set distributions and thus alleviate this current limitation in our work.

8 Ethics

To the best of our knowledge, there are no ethical concerns involved in this research. We conduct our work using publicly available English datasets and tasks, and models pre-trained on English text. Our results may not generalise to other languages. To facilitate further research in the field, we release our source code and models.

References

Abe et al. (2022) Taiga Abe, Estefany Kelly Buchanan, Geoff Pleiss, Richard Zemel, and John P Cunningham. 2022. Deep ensembles work, but are they necessary? In Advances in Neural Information Processing Systems, volume 35, pages 33646–33660. Curran Associates, Inc.
Antoniou et al. (2019) Antreas Antoniou, Harrison Edwards, and Amos Storkey. 2019. How to train your MAML. In International Conference on Learning Representations.
Ashukha et al. (2020) Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. 2020. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. arXiv preprint arXiv:2002.06470.
Bansal et al. (2020a) Trapit Bansal, Rishikesh Jha, and Andrew McCallum. 2020a. Learning to few-shot learn across diverse natural language classification tasks. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5108–5123, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Bansal et al. (2020b) Trapit Bansal, Rishikesh Jha, Tsendsuren Munkhdalai, and Andrew McCallum. 2020b. Self-supervised meta-learning for few-shot natural language classification tasks.
Bengio et al. (1990) Yoshua Bengio, Samy Bengio, and Jocelyn Cloutier. 1990. Learning a synaptic learning rule. Citeseer.
BigScience Workshop (2023) BigScience Workshop. 2023. Bloom: A 176b-parameter open-access multilingual language model.
Blitzer et al. (2007) John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 440–447, Prague, Czech Republic. Association for Computational Linguistics.
Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference.
Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Chen et al. (2020) Jiaao Chen, Zichao Yang, and Diyi Yang. 2020. Mixtext: Linguistically-informed interpolation of hidden space for semi-supervised text classification.
Clark et al. (2018) Kevin Clark, Minh-Thang Luong, Christopher D. Manning, and Quoc V. Le. 2018. Semi-supervised sequence modeling with cross-view training.
Davis et al. (2024) Christopher Davis, Andrew Caines, Øistein Andersen, Shiva Taslimipoor, Helen Yannakoudakis, Zheng Yuan, Christopher Bryant, Marek Rei, and Paula Buttery. 2024. Prompting open-source and commercial language models for grammatical error correction of english learner text.
Dettmers et al. (2023) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. In Advances in Neural Information Processing Systems, volume 36, pages 10088–10115. Curran Associates, Inc.
Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Diamond and Boyd (2016) Steven Diamond and Stephen Boyd. 2016. CVXPY: A Python-embedded modeling language for convex optimization. Journal of Machine Learning Research, 17(83):1–5.
Dietterich (2000) Thomas G. Dietterich. 2000. Ensemble methods in machine learning. In Multiple Classifier Systems, pages 1–15, Berlin, Heidelberg. Springer Berlin Heidelberg.
Ding et al. (2021) Nan Ding, Xi Chen, Tomer Levinboim, Sebastian Goodman, and Radu Soricut. 2021. Bridging the gap between practice and pac-bayes theory in few-shot meta-learning. In Advances in Neural Information Processing Systems, volume 34, pages 29506–29516. Curran Associates, Inc.
Ding et al. (2023) Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. 2023. Sparse low-rank adaptation of pre-trained language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4133–4145, Singapore. Association for Computational Linguistics.
Dolan and Brockett (2005) William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
Dou et al. (2019) Zi-Yi Dou, Keyi Yu, and Antonios Anastasopoulos. 2019. Investigating meta-learning algorithms for low-resource natural language understanding tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1192–1197, Hong Kong, China. Association for Computational Linguistics.
Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1126–1135. PMLR.
Gao et al. (2020) Tianyu Gao, Adam Fisch, and Danqi Chen. 2020. Making pre-trained language models better few-shot learners.
Giampiccolo et al. (2008) Danilo Giampiccolo, Hoa Trang Dang, Bernardo Magnini, Ido Dagan, Elena Cabrio, and William B. Dolan. 2008. The fourth pascal recognizing textual entailment challenge. In Text Analysis Conference.
Glorot and Bengio (2010) Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pages 249–256, Chia Laguna Resort, Sardinia, Italy. PMLR.
Holla et al. (2020) Nithin Holla, Pushkar Mishra, Helen Yannakoudakis, and Ekaterina Shutova. 2020. Learning to learn to disambiguate: Meta-learning for few-shot word sense disambiguation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4517–4533, Online. Association for Computational Linguistics.
Hu et al. (2022) Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Jiaramaneepinit et al. (2024) Boonnithi Jiaramaneepinit, Thodsaporn Chay-intr, Kotaro Funakoshi, and Manabu Okumura. 2024. Extreme fine-tuning: A novel and fast fine-tuning approach for text classification. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pages 368–379, St. Julian’s, Malta. Association for Computational Linguistics.
Khot et al. (2018) Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. In AAAI.
Kim et al. (2018) Taesup Kim, Jaesik Yoon, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. 2018. Bayesian model-agnostic meta-learning.
Koch et al. (2015) Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. 2015. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2. Lille.
Kotitsas et al. (2024) Sotiris Kotitsas, Panagiotis Kounoudis, Eleni Koutli, and Haris Papageorgiou. 2024. Leveraging fine-tuned large language models with LoRA for effective claim, claimer, and claim object detection. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2540–2554, St. Julian’s, Malta. Association for Computational Linguistics.
Lake et al. (2015) Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. 2015. Human-level concept learning through probabilistic program induction. Science, 350(6266):1332–1338.
Lakshminarayanan et al. (2017) Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Langedijk et al. (2022) Anna Langedijk, Verna Dankers, Phillip Lippe, Sander Bos, Bryan Cardenas Guevara, Helen Yannakoudakis, and Ekaterina Shutova. 2022. Meta-learning for fast cross-lingual adaptation in dependency parsing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8503–8520, Dublin, Ireland. Association for Computational Linguistics.
Lee et al. (2015) Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. 2015. Why m heads are better than one: Training a diverse ensemble of deep networks.
Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning.
Liu et al. (2022) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 100–114, Dublin, Ireland and Online. Association for Computational Linguistics.
Liu et al. (2013) Jingjing Liu, Panupong Pasupat, Scott Cyphers, and Jim Glass. 2013. Asgard: A portable architecture for multilingual dialogue systems. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 8386–8390.
Liu et al. (2021) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55:1 – 35.
Loshchilov and Hutter (2017a) Ilya Loshchilov and Frank Hutter. 2017a. Decoupled weight decay regularization.
Loshchilov and Hutter (2017b) Ilya Loshchilov and Frank Hutter. 2017b. SGDR: stochastic gradient descent with warm restarts. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048–11064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Miyato et al. (2016) Takeru Miyato, Andrew M. Dai, and Ian Goodfellow. 2016. Adversarial training methods for semi-supervised text classification.
MOSEK ApS (2019) MOSEK ApS. 2019. The MOSEK optimization toolbox for MATLAB manual. Version 9.3.
Mueller et al. (2023) Aaron Mueller, Kanika Narang, Lambert Mathias, Qifan Wang, and Hamed Firooz. 2023. Meta-training with demonstration retrieval for efficient few-shot learning. In Findings of the Association for Computational Linguistics: ACL 2023, pages 6049–6064, Toronto, Canada. Association for Computational Linguistics.
Nichol et al. (2018) Alex Nichol, Joshua Achiam, and John Schulman. 2018. On first-order meta-learning algorithms.
Nooralahzadeh et al. (2020) Farhad Nooralahzadeh, Giannis Bekoulis, Johannes Bjerva, and Isabelle Augenstein. 2020. Zero-shot cross-lingual transfer with meta learning.
Obamuyide and Vlachos (2019a) Abiola Obamuyide and Andreas Vlachos. 2019a. Meta-learning improves lifelong relation extraction. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 224–229, Florence, Italy. Association for Computational Linguistics.
Obamuyide and Vlachos (2019b) Abiola Obamuyide and Andreas Vlachos. 2019b. Model-agnostic meta-learning for relation classification with limited supervision. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5873–5879, Florence, Italy. Association for Computational Linguistics.
OpenAI (2024) OpenAI. 2024. Gpt-4 technical report.
Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Ravi and Larochelle (2017) Sachin Ravi and Hugo Larochelle. 2017. Optimization as a model for few-shot learning. In International Conference on Learning Representations.
Reddi et al. (2018) Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. 2018. On the convergence of adam and beyond. In International Conference on Learning Representations.
Sanh et al. (2021) Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2021. Multitask prompted training enables zero-shot task generalization.
Schick and Schütze (2020) Timo Schick and Hinrich Schütze. 2020. It’s not just size that matters: Small language models are also few-shot learners.
Schmidhuber (1987) Jurgen Schmidhuber. 1987. Evolutionary principles in self-referential learning. On learning how to learn: The meta-meta-… hook.) Diploma thesis, Institut f. Informatik, Tech. Univ. Munich, 1(2).
Snell et al. (2017) Jake Snell, Kevin Swersky, and Richard S. Zemel. 2017. Prototypical networks for few-shot learning.
Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
Sucholutsky et al. (2021) Ilia Sucholutsky, Nam-Hwui Kim, Ryan P. Browne, and Matthias Schonlau. 2021. One line to rule them all: Generating LO-shot soft-label prototypes. In 2021 International Joint Conference on Neural Networks (IJCNN). IEEE.
Sucholutsky and Schonlau (2021) Ilia Sucholutsky and Matthias Schonlau. 2021. ‘less than one’-shot learning: Learning n classes from m < n samples. Proceedings of the AAAI Conference on Artificial Intelligence, 35(11):9739–9746.
Sun et al. (2020) Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2020. How to fine-tune bert for text classification?
Sun et al. (2024) Simeng Sun, Yang Liu, Shuohang Wang, Dan Iter, Chenguang Zhu, and Mohit Iyyer. 2024. PEARL: Prompting large language models to plan and execute actions over long documents. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 469–486, St. Julian’s, Malta. Association for Computational Linguistics.
Thrun and Pratt (1998) Sebastian Thrun and Lorien Pratt. 1998. Learning to learn: Introduction and overview. In Learning to learn, pages 3–17. Springer.
Tjong Kim Sang and De Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142–147.
Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
van der Heijden et al. (2021) Niels van der Heijden, Helen Yannakoudakis, Pushkar Mishra, and Ekaterina Shutova. 2021. Multilingual and cross-lingual document classification: A meta-learning approach. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1966–1976, Online. Association for Computational Linguistics.
Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
Wang et al. (2017) Zhiguo Wang, Wael Hamza, and Radu Florian. 2017. Bilateral multi-perspective matching for natural language sentences.
Wang et al. (2020) Zirui Wang, Zachary C. Lipton, and Yulia Tsvetkov. 2020. On negative interference in multilingual models: Findings and a meta-learning treatment. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4438–4450, Online. Association for Computational Linguistics.
Warstadt et al. (2018) Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2018. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471.
Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
Wilson and Izmailov (2020) Andrew G Wilson and Pavel Izmailov. 2020. Bayesian deep learning and a probabilistic perspective of generalization. In Advances in Neural Information Processing Systems, volume 33, pages 4697–4708. Curran Associates, Inc.
Xie et al. (2019) Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V. Le. 2019. Unsupervised data augmentation for consistency training.
Yang et al. (2024) Adam Yang, Maxime Robeyns, Xi Wang, and Laurence Aitchison. 2024. Bayesian low-rank adaptation for large language models. In Socially Responsible Language Modelling Research.
Yang et al. (2023) Linyi Yang, Yaoxian Song, Xuan Ren, Chenyang Lyu, Yidong Wang, Jingming Zhuo, Lingqiao Liu, Jindong Wang, Jennifer Foster, and Yue Zhang. 2023. Out-of-distribution generalization in natural language processing: Past, present, and future. In The 2023 Conference on Empirical Methods in Natural Language Processing.
Zhao et al. (2021) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12697–12706. PMLR.
Zhou et al. (2024) Kun Zhou, Yifan Li, Xin Zhao, and Ji-Rong Wen. 2024. Diffusion-NAT: Self-prompting discrete diffusion for non-autoregressive text generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1438–1451, St. Julian’s, Malta. Association for Computational Linguistics.
Zhou and Srikumar (2022) Yichu Zhou and Vivek Srikumar. 2022. A closer look at how fine-tuning changes BERT. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1046–1061, Dublin, Ireland. Association for Computational Linguistics.

Appendix A Appendix

A.1 Deriving soft-label prototypes using constraintSLP

Finding lines connecting all centroids

In Figure 3(a), we present an example set of lines connecting all class centroids. For further details on recursive regression, we refer the reader to Sucholutsky et al. (2021).

Deriving soft-label prototypes by optimising for linear constraints

Example soft-label prototypes which are “set” at the ends of each line are shown in Figure 3(b).

Classification with constraintSLP: A toy example

Figure 4 presents an example classification with soft-label prototypes. Given the class centroids for blue, green and yellow are located at $(0,0)$ , $(1.5,0)$ and $(3,0)$ respectively, two soft-label prototypes are defined by a line connecting yellow and blue, and are thus located at $(3,0)$ and $(0,0)$ respectively. The soft labels in Figure 4(a) contain the per-class probability distribution derived by the constraintSLP method; for example, $p(x=blue)=0.6$ and $p(x=green)=0.4$ for the left prototype, and $p(x=green)=0.4$ and $p(x=yellow)=0.6$ for the right prototype. When a new test instance $x$ located at $(1.5,0.8)$ is presented, we make predictions as follows: we find the nearest line to $x$ and consider its distance from the two prototypes at the ends of the line and multiply the class distribution of each prototype by the inverse distance as per Eq. 1.

Since $x$ is equidistant from both prototypes, the distance between the $x$ and each prototype is $1.5$ . Therefore, the values for blue and yellow (given both soft-label prototypes) become $soft\_label(x=blue)=\frac{0.6}{1.5}+\frac{0}{1.5}=0.4$ and $soft\_label(x=yellow)=\frac{0}{1.5}+\frac{0.6}{1.5}=0.4$ respectively. In contrast, for green, which is directly informed by both prototypes (i.e., no zero values in the numerator), the probability distribution becomes $soft\_label(x=green)=\frac{0.4}{1.5}+\frac{0.4}{1.5}=0.53$ . Therefore $x$ is classified as green. This decision boundary can be seen in Figure 4(b).

A.2 Meta-learning

For encoder-based models, meta-learning has emerged as a viable methodology for few-shot learning. In the meta-learning paradigm, the training and test sets, referred to as $\mathcal{D}$ _meta-train and $\mathcal{D}$ _meta-test, are split into episodes. Each episode encompasses a task $\mathcal{T}$ _i and consists of a support set $\mathcal{D}^{(i)}$ _support and a query set $\mathcal{D}^{(i)}$ _query. Meta-learning algorithms initially fit the model on the support set of the episode (inner-loop optimisation) and then achieve generalisation across episodes by optimising performance on the query sets of the episodes (outer-loop optimisation). For evaluation, the model is first fine-tuned on the support set and then evaluated on the query set for each task $\mathcal{T}$ _i $\in$ $\mathcal{D}$ _meta-test. We describe the process algorithmically in Algorithm 2 and describe the MetaUpdate process for different algorithms subsequently.

Algorithm 2 Meta-learning

\alpha,\beta\leftarrow

learning rates

1 Sample batch of tasks

\{T_{i}\}\sim p(T)

2 Initialise

\mathbf{\theta^{\prime}_{i}}\leftarrow\theta

3 for $T_{i}\sim p(T)$ do

4 Partition

T_{i}

into

D^{s}_{i}

and

D^{q}_{i}

\mathbf{\theta^{\prime}_{i}}\leftarrow\mathbf{\theta}-\alpha\nabla_{\theta}\mathcal{L}^{s}_{\mathcal{D}_{i}}(f_{\theta})

for

k

steps

6 end for

\theta\leftarrow

MetaUpdate

(\theta_{i},D_{i}^{q},\beta)

Model Agnostic Meta-Learning

MAML Finn et al. (2017) is an optimisation-based meta-learning approach which incorporates generalisability across tasks in its cost function. The task loss $\mathcal{L}^{q}_{\mathcal{T}_{i}}$ is computed on the query examples in each episode, using this task-specific model. The initial model parameters $\theta$ are then updated so as to minimize the sum of the losses of all tasks in a batch, leading to improved generalisation across tasks. The MetaUpdate step is thus defined as

\mathbf{\theta}\leftarrow\mathbf{\theta}-\beta\>\nabla_{\theta}\sum_{\mathcal{T}_{i}\sim p(\mathcal{T})}\mathcal{L}^{q}_{\mathcal{D}_{i}}(f_{\theta_{i}^{\prime}})

Note that the MetaUpdate expression calculates the gradients of each $\theta_{i}$ with respect to $\theta$ , thus necessitating the computation of second-order gradients. To ease computation, we use a first-order approximation of MAML (FOMAML) wherein the gradients of each $\theta_{i}$ are calculated with respect to $\theta_{i}$ and reduce the MetaUpdate term to

\mathbf{\theta}\leftarrow\mathbf{\theta}-\beta\>\nabla_{\theta_{i}}\sum_{\mathcal{T}_{i}\sim p(\mathcal{T})}\mathcal{L}^{q}_{\mathcal{D}_{i}}(f_{\theta_{i}^{\prime}})

LEOPARD

Bansal et al. (2020a) employ meta-learning for diverse NLP tasks in an approach inspired from MAML which integrates a text encoder model with a meta-learned parameter generator to tailor task-specific initialisations for the classification head. Their inner-loop update learns the parameter generator for the task, adapts task-specific model parameters and the MetaUpdate step adapts model parameters as done in MAML. They show that their method, LEOPARD, outperforms multi-task trained models as well as a range of other meta-learning methods.

Reptile

This meta-learning algorithm, introduced by Nichol et al. (2018), is computationally simple compared to MAML and LEOPARD - the MetaUpdate step simply moves the model parameters towards inner-loop fine-tuned model parameters, thus assuming the form:

\theta\leftarrow\theta+\beta\frac{1}{|\{T_{i}\}|}\sum_{T_{i}\sim p(T)}(\theta_{i}-\theta)

Despite it’s simplicity, it reports strong performance on a variety of few-shot learning tasks Dou et al. (2019).

Prototypical Networks

Unlike optimisation-driven meta-learning methods, Prototypical Networks Snell et al. (2017) is a metric-based meta-learning method that uses an embedding function $f_{\theta}$ to encode training support samples and compute a high-dimensional vector $\mu_{c}$ that is the arithmetic mean of the training data points of class $c$ . It then uses a distance function $d$ to compute the similarity between a query instance $x$ and the mean vector of each class to get the class distribution as:

	$\displaystyle p(y=c\|x)=softmax(-d(f_{\theta}(x),\mu_{c}))$
	$\displaystyle=\frac{exp(-d(f_{\theta}(x),\mu_{c}))}{\sum_{c^{\prime}\in C}exp(-d(f_{\theta}(x),\mu_{c}^{\prime}))}$

Model optimisation is done using the loss function $J(\mathbf{\theta})=-log(p(y=c^{*}|x,\theta))$ .

A.3 Analysis of soft-labels derived from linear constraints: constraintSLP

Theorem A.1.

The soft-label value of each class within a single soft-label prototype generated using constraintSLP is inversely proportional to its distance from the soft-label prototype along the line connecting all classes captured by it.

Proof of Theorem A.1 (Informal)

Consider a line $l$ connecting three class centroids (while we focus on a three class system, the conclusions generalise to $n>3$ classes too). The class centroids are represented by A, B and C. The soft-label prototypes at ends A and C contain the values $[a_{1},a_{2},a_{3}]$ and $[c_{1},c_{2},c_{3}]$ respectively. Consider a support example $x\in A$ at a distance $d_{a}$ and $d_{b}$ from A and C respectively. Directly using the constraints in Algorithm 4 of Sucholutsky et al. (2021), we state that the influence of A (i.e., the distance-weighted sum of the soft-labels at $x$ ) should be more than the sum of the influence of the other two classes. Thus, we need to maximise:

\frac{a_{1}}{d_{a}}+\frac{c_{1}}{d_{b}}>\left(\frac{a_{2}}{d_{a}}+\frac{c_{2}}{d_{b}}\right)+\left(\frac{a_{3}}{d_{a}}+\frac{c_{3}}{d_{b}}\right)

(2)

As we move $x$ further towards A, $d_{a}\rightarrow 0$ and the influence of $[a_{1}/d_{1},a_{2}/d_{2},a_{3}/d_{3}]$ increases thus $\sum^{i=3}_{i=2}c_{i}/d_{b}<<<\sum^{i=3}_{i=2}a_{i}/d_{a}$ . Therefore we have the approximation:

\centering\frac{a_{1}}{d_{a}}>\frac{a_{2}}{d_{a}}+\frac{a_{3}}{d_{a}}\implies a_{1}>a_{2}+a_{3}\@add@centering

(3)

If we take a support example $x\in C$ , by symmetry as $x$ is moved towards C, $d_{b}\rightarrow 0$ , we can also write:

\centering\frac{c_{3}}{d_{b}}>\frac{c_{2}}{d_{b}}+\frac{c_{1}}{d_{b}}\implies c_{3}>c_{2}+c_{1}\@add@centering

(4)

Furthermore, consider a point $x$ in the middle of $l$ equidistant from A and C (by a distance $d$ ) – such a point will always be classified as $x\in B$ . Thus, the influence of B should be higher than both A and C. Thus we have:

	$\displaystyle\frac{a_{2}}{d}+\frac{c_{2}}{d}>\frac{a_{1}}{d}+\frac{c_{1}}{d}\;\;\;\&\;\;\;\frac{a_{2}}{d}+\frac{c_{2}}{d}>\frac{a_{3}}{d}+\frac{c_{3}}{d}$
	$\displaystyle\implies a_{2}+c_{2}>a_{1}+c_{1}\;\;\;\&\;\;\;a_{2}+c_{2}>a_{3}+c_{3}$

From Equation 3 and Equation 4 we can replace $a_{1}$ and $c_{3}$ and get:

	$\displaystyle a_{2}+c_{2}>a_{2}+a_{3}+c_{1}\implies c_{2}>c_{1}$
	$\displaystyle a_{2}+c_{2}>a_{3}+c_{2}+c_{1}\implies a_{2}>a_{3}$

Therefore, we have:

\displaystyle a_{1}>a_{2}>a_{3}\;\;\;\&\;\;\;c_{3}>c_{2}>c_{1}

This is an intuitive result as the soft-label value of each class decreases as the distance of the class centroid increases from the prototype location – the class nearest to the prototype has the highest soft-label value and the class furthest away has the lowest soft-label value.

Recall that $\sum_{i=1}^{3}a_{i}=1$ and $\sum_{i=1}^{3}c_{i}=1$ and $a_{i},c_{i}\geq 0$ $\forall i=\{1,2,3\}$ otherwise the optimisation problem becomes unbounded. Therefore, the ranges of values for $[a_{1},a_{2},a_{3}]$ and $[c_{1},c_{2},c_{3}]$ are:

	$\displaystyle a_{3}\in[0,a_{2}),\;a_{2}\in(a_{3},a_{1}),\;a_{1}\in(a_{2},1]$
	$\displaystyle c_{3}\in(c_{2},1],\;c_{2}\in(c_{1},c_{3}),\;c_{1}\in[0,c_{2})$

Using Algorithm 4 in Sucholutsky et al. (2021), we see that there are multiple constraints for $a_{1}$ and $a_{2}$ which require them to be maximised, but there are none for $a_{3}$ . Thus, to maximise Equation 3, $a_{3}$ adjusts to the minimum value it can get:

a_{3}=min(0,a_{2})=\epsilon\simeq 0

By symmetry, we can also conclude that:

c_{1}=min(0,c_{2})=\epsilon\simeq 0

These approximations are also generalisable to multiple classes connected by a line, for example, if a line connects only two centroids, the soft-labels at each end are derived as $[0,1]$ and $[1,0]$ - the same as a “hard" label. These findings are substantiated experimentally in Table 2 where we examine the soft-labels generated by DeepSLP and constraintSLP using a few-shot training support set of the task airline with $8$ examples per class – the constraintSLP soft label corresponding to the furthest class from the prototype location drops to almost zero compared to other soft label values. On the other hand, DeepSLP prevents overfitting on the nearest classes and produces a more generalised distribution of soft-labels. This is a trend generally observed in other tasks and classes as well. We further use this theorem to prove the main theorem given by Theorem A.2.

#	constraintSLP	DeepSLP( $x$ )
1	$5.6422e-01$	$9.7887e-01$
	$4.3577e-01$	$2.0995e-02$
	$9.7973e-16$	$1.3500e-04$
2	$5.3090e-13$	$2.5240e-02$
	$4.3212e-01$	$9.7475e-01$
	$5.6787e-01$	$1.0000e-05$

Table 2: Soft labels derived using constraintSLP and DeepSLP. # denotes the index of the soft-label prototype lying on the line. Soft labels are constant for constraintSLP, however, they are a function of input point

x

for DeepSLP, thus allowing more flexibility.

Theorem A.2.

The constant soft-labels in constraintSLP do not always select the closest class centroid to a test point.

Proof of Theorem A.2 (Informal)

Furthermore, consider a line $b$ perpendicular to $l$ – it intersects $l$ between A and B. We select $\theta$ such that $\phi=\pi/2$ . We denote the complete setup diagrammatically in Figure 5.

Consider the distance weighted influences at $x$ for class A. We have the influence as $\frac{a_{1}sin\theta}{e}+\frac{c_{1}cos\theta}{e}=\frac{a_{1}sin\theta}{e}$ as $c_{1}\simeq 0$ . Similarly, for class B, we have the weighted influence as $\frac{a_{2}sin\theta}{e}+\frac{c_{2}cos\theta}{e}$ . To calculate values of $\theta$ where the weighted influence of B is more than the weighted influence of A, we get:

	$\displaystyle\frac{a_{2}sin\theta}{e}+\frac{c_{2}cos\theta}{e}>\frac{a_{1}sin\theta}{e}$
	$\displaystyle\implies a_{2}sin\theta+c_{2}cos\theta>a_{1}sin\theta$
	$\displaystyle\implies c_{2}cos\theta-(a_{1}-a_{2})sin\theta>0$
	$\displaystyle\implies\frac{c_{2}cos\theta-(a_{1}-a_{2})sin\theta}{\sqrt{c_{2}^{2}+(a_{1}-a_{2})^{2}}}>0$
	$\displaystyle\implies cos(\theta+\alpha)>0$

where $\alpha=tan^{-1}\left(\frac{a_{1}-a_{2}}{c_{2}}\right)$ . Since $cos(\theta+\alpha)>0$ we have $(\theta+\alpha)\in(-\pi/2,\pi/2)$ and since $\theta>0$ , thus for $\theta\in[0,\pi/2-\alpha)$ , the weighted influence of B is more than the weighted influence of A.

However, it is worth observing the result for $\theta$ derived above can contain points closer to A (using Euclidean distance) which are actually classified as B. We can easily demonstrate this with a counter example explained in Figure 6.

Therefore, for points closer to A compared to B using an Euclidean measure, constraintSLP can still return a higher value for the influence at B compared to A. This adversely affects performance in classifiers where we rely on selection of the closest class centroid for classification – such as 1-NN, Prototypical Networks, and constraintSLP – and we believe this is the reason behind the poor performance of constraintSLP for cases where the total classes is greater than two.