Assessing Suicide Risk in Arabic Crisis Helpline Calls: A Comparison of Arabic and English Large Language Models

Most NLP work on suicide risk uses social media posts or clinical notes Arowosegbe and Oyelade (2023); Castillo-Sánchez et al. (2020); Fodeh et al. (2017). Helpline calls are different in ways that matter. They are conversations rather than written statements, and the risk assessment happens during the call instead of being pieced together afterwards. The label also comes from a trained operator at the time, not from later coding. Several groups have worked on helpline contacts directly. One study classified risk levels in text-based crisis counselling. It found that a neural model missed fewer at-risk clients than a term-frequency model Broadbent et al. (2023). Two other studies classified low against high risk from call audio rather than from words. One used voice biomarkers on an Australian telehealth service, with counsellor ratings re-checked against the C-SSRS Iyer et al. (2022). The other used acoustic features on a Chinese hotline Su et al. (2025). A further study on a Chinese hotline combined pitch features with deep learning features to analyse the emotion expressed during calls Wang et al. (2025). The Sahar helpline answers in both Hebrew and Arabic, but its published analysis used only the Hebrew chats Grimland et al. (2025). That gap shows how little is known about Arabic crisis calls. The helpline studied here has been analysed before. One study evaluated whether calls reduced caller distress Zeinoun et al. (2021), and another modelled which caller characteristics went with suicidal ideation, intent and self-harm Farran et al. (2025). Both used the structured fields the operator fills in, not the speech. As far as we can tell, two things here are new. 1) This is the first suicide-risk classifier we know of for Levantine Arabic-language crisis helpline calls. 2) It is also the first report of a full pipeline where speech recognition and de-identification run inside the service, so the recordings are never moved. Reviews of this literature have asked for work outside English and inside real services Castillo-Sánchez et al. (2020). Our results agree with the general finding that transformer models pick up suicidal content well above chance. But we also show that performance varied substantially across both outcome definitions and model backbones.

Data came from calls to the Embrace National Lifeline (1564) in Lebanon. Trained operators answer the calls, supervised by clinical psychologists. They complete a standard C-SSRS assessment and record the result in a service database. The helpline is anonymous, so call records cannot be linked to national identifiers. This was a secondary analysis of routine service data. We had no contact with callers. Ethical approval was in place before the data were accessed. A call was eligible if the recording lasted between five and ten minutes. This window drops very short contacts, which are mostly wrong numbers, silent calls and immediate hang-ups. Section 4.3 discusses what this leaves out.

Recordings did not leave the helpline. All speech processing ran on site, and only redacted Arabic text went to the analysis team. Recordings were transcribed with a Whisper-family speech recognition model Radford et al. (2023). A call is longer than the model’s input window, so we split each recording into overlapping segments and removed the duplicated text when joining the segments back together. The calls are in Levantine Arabic rather than Modern Standard Arabic, so we used a decoding setup adapted to the dialect. The transcripts then went through an Arabic named-entity recognition model, which removed names of people, places and organisations along with other direct identifiers. Phone number blocks were stripped from the filenames before release. The whole code for the transcription and de-identification is available online (see the Section Code availability).

Operators record the five ideation items of the C-SSRS ladder: wish to be dead; non-specific active suicidal thoughts; active ideation with any method; active ideation with some intent to act; and active ideation with a specific plan and intent Posner et al. (2011). Each item is marked present or absent. Other codes cover items that were not asked, items skipped by the logic of the questionnaire, and items the caller did not answer. We grouped the items into two categories by severity, following the order of the ladder. The two lower rungs, a wish to be dead and non-specific active thoughts, make up the at-risk category. The three items that involve a method, an intent or a plan make up high risk:

A call is at-risk if either of the two lower items is present, and high risk if any of the three higher items is present. The two categories cover overlapping but different sets of calls, and we modelled them as two separate yes-or-no tasks.

Not every call has all five items recorded. A missing item does not always leave the category unclear. If any recorded item is present, the category is positive whatever the missing items would have said. The category is unclear only when every recorded item is absent and at least one item was never recorded, because then the missing item alone decides the answer. We kept the calls whose category was clear under this rule and dropped the rest. This resulted in the exclusion of $32$ calls from the at-risk category and $16$ from the high-risk category. Treating unrecorded items as absent would have allowed these calls to remain in the analysis, but would risk introducing false negatives into a corpus designed to detect suicide risk. This could bias the models toward missing positive cases, which is the more consequential error in this setting.

To compare modelling the source language against translate then classify, we translated each de-identified Arabic transcript into English with Qwen/Qwen2.5-72B-Instruct model under greedy decoding. We then checked that each English output covered the whole call. $29$ of the $438$ transcripts were ill-formed or incomplete, so we dropped them. We removed these calls from both corpora rather than from the English one alone. Dropping them only on the English side would leave the two languages covering different sets of calls, which would have affected the comparison between languages.

Table 1 lists every model we fine-tuned. We selected five instruction-tuned large language model (LLM) backbone decoders covering a range of size and pretraining language, and six transformer encoders (BERT-based Models) for comparison. To fine-tune these models, we turned each call into an instruction-tuning triple. The instruction states the yes or no question, the input is the full transcript, and the target output is a single word. Every model used the same 4-bit QLoRA recipe with no per-model hyperparameter search, so that differences between models reflect the backbone rather than tuning effort Hu et al. (2022); Dettmers et al. (2023). We attached low-rank adapters ($r=16$, $\alpha=32$, dropout 0.05) to the attention and feed-forward projections. Base weights were frozen and quantised. Training minimised causal language-model cross-entropy on the answer token only, with prompt tokens masked, using a cosine schedule (peak learning rate $2\times 10^{-4}$, effective batch size 16). We trained separate adapters per model, task and language, and ran both a three-epoch and a ten-epoch schedule. At inference we decoded the answer token greedily and mapped it to a binary label. Generations that could not be parsed were counted as negative. Zero-shot baselines used the same prompt on the 0-Shot base model. Every transcript is several times longer than a 512-token encoder window, so we read documents in overlapping windows and pooled the window representations, following standard practice for long documents Pappagari et al. (2019).

We split at the call level, stratified on the category label, with an 80:20 training to test ratio. The split was defined over transcript identifiers and applied the same way to the Arabic corpus and to its English translation. The two datasets therefore hold the same calls, in the same splits, with the same labels, and differ only in language. No call appears on both sides of the split. For evaluation, we used the area under the receiver operating characteristic curve (ROC-AUC), the area under the precision and recall curve (PR-AUC), macro-averaged F1, positive-class recall and overall accuracy. Positive-class recall is the same thing as sensitivity to the positive class. Macro-F1 here is the mean of the two per-class F1 scores, so it need not fall between macro-averaged precision and recall. All values are percentages. Table 5 reports sensitivity, specificity and the two predictive values, together with the two-by-two table of correct and incorrect decisions they derive from. Each decoder answers yes or no, so the threshold is whatever the model itself applies. ROC-AUC and PR-AUC need a continuous score rather than a decision, so for those two measures we used the probability of the positive answer token, normalised over the yes and no tokens. Proportions in Table 5 are given with a 95% Wilson score interval, which behaves better than a normal approximation when the counts are small. We report accuracy for completeness, but we did not use it to pick or compare models. With 30% to 52% of calls positive, accuracy is partly inflated by guessing the majority class, which is the behaviour this analysis is meant to catch. The study used one stratified split. With $75$ and $60$ test calls, moving one call changes macro-F1 by about one to two points. We report the rankings descriptively and draw no statistical comparison between models. The intervals in Table 5 show how wide the uncertainty is. We report ROC-AUC and PR-AUC without intervals and treat them as descriptive.

Figures 1 and 2 show every fine-tuned model on all five measures, one figure per corpus, at ten epochs. Both figures use the same scale, so the two risk categories can be read against each other. Most models sit between 65 and 90 on macro-F1, ROC-AUC and accuracy, while positive-class recall spreads much more widely, from 27.78 to 100.00 on the Arabic corpus. The best encoder falls below the best decoder in both languages and on both categories, and two Arabic encoders sit near the floor on the at-risk category. The spread across decoders is narrower on the English corpus than on the Arabic one, so the choice of backbone matters less once the text has been translated. The rest of this section reads out of these two figures.

Table 4 reports one 70B decoder per corpus against its 0-Shot baseline, the Arabic-centric model on Arabic and the general-purpose model on English. For high risk, the Arabic model reached a macro-F1 of $81.19$ and a ROC-AUC of $90.61$, against $41.18$ and $58.93$ 0-Shot. The 0-Shot model found none of the high-risk calls in the test split. After fine-tuning it found $66.7$% of them. The English model reached a macro-F1 of $85.00$ and a ROC-AUC of $92.59$ and found $88.9$% of high-risk calls, against $73.78$ and $85.45$ 0-Shot. For the at-risk category, the Arabic model reached a macro-F1 of $73.29$ and a ROC-AUC of $82.19$. The English model reached $75.96$ and $87.25$. Both improved on their 0-Shot baselines by a wide margin. Those baselines found $12.8$% and $38.5$% of at-risk calls.

Table 5 gives sensitivity, specificity and the predictive values for the same eight conditions. PPV is positive predictive value and NPV is the negative predictive value. The two fine-tuned models behaved differently on the high-risk category. The English model found 16 of the 18 high-risk calls, giving a sensitivity of $88.9$% and a negative predictive value of $94.7$%. It also raised 6 false alarms out of 42 negative calls. The Arabic model missed 6 of the 18, so its sensitivity was $66.7$%, but it raised only 3 false alarms. On the at-risk category the two models were close, and both sat near $75$% on all four measures. The intervals are wide throughout. On the high-risk category they are wide enough that the two languages cannot be told apart. With only 18 positive calls in that test split, one call moves sensitivity by more than five percentage points.

The best model in each language scored higher on the high-risk category than on the at-risk category, even though high risk is the rarer outcome. The best Arabic figures were $81.19$ macro-F1 for high risk against $78.63$ for the at-risk category. The best English figures were $85.00$ against $78.66$ (Table 6). ROC-AUC followed the same order. This is the opposite of what class prevalence alone would predict. Smaller backbones did not follow it: on the Arabic corpus the 1.5B, 14B and 8B decoders all scored higher on the at-risk category. Positive-class recall did not follow. Every Arabic decoder found a smaller share of high-risk calls than of at-risk calls, and so did three of the five English ones. On the Arabic corpus the gain on high risk is therefore in separation rather than in how many positive calls are caught at the model’s own threshold. The separation itself suggests the main limit is not how rare the label is but how the construct is expressed. Talk of a method, an intent or a plan is direct and easy to quote. Lower-severity ideation is indirect, and its wording overlaps heavily with the wording of general distress, which the helpline also receives in volume.

Table 6 reports all ten decoder conditions at ten epochs. Parameter count did not set the order: the 8B Arabic-centric model reached $78.63$ macro-F1 on the Arabic at-risk category against $73.29$ for its 70B counterpart.

The best fine-tuned decoder (LLM) beat the best encoder baseline (BERT) on every task and language (Table 7). The best Arabic encoder score was $69.25$ macro-F1 on the at-risk category and $72.82$ on high risk, against $78.63$ and $81.19$ for decoders. The best English encoder score was $64.57$ and $71.29$, against $78.66$ and $85.00$. No single encoder was best on both categories in either language.

Several conditions produced trivial classifiers, and they failed in opposite directions (Table 8). The smallest decoder on Arabic high risk, trained for three epochs, answered negative to every call, giving $0.00$ positive-class recall at $41.18$ macro-F1. Three Arabic encoders on the at-risk category did the reverse. They answered positive to almost every call, with positive-class recall of $97.4$ to $100.0$ at macro-F1 between $33.63$ and $37.22$. Both kinds of failure therefore land in a similar macro-F1 range of roughly $33$ to $41$. Macro-F1 on its own cannot tell a model that detects no positive case from one that flags every caller. Gaps between ROC-AUC and PR-AUC picked out the same conditions. One Arabic encoder trained for three epochs reached $75.13$ ROC-AUC with $47.24$ PR-AUC on high risk, which describes a model that ranks calls acceptably while deciding poorly. Positive-class recall and PR-AUC should therefore be reported alongside macro-averaged measures whenever risk labels are imbalanced.

Our analysis have shown that sorting de-identified Levantine Arabic helpline transcripts into two risk categories is feasible. For the high-risk category, the models achieved performance sufficient to support further evaluation as a screening aid for helpline operators. Importantly, this performance was achieved within a privacy-preserving pipeline in which the audio remains within the helpline environment. Additionally, we found that lower-severity ideation was more difficult to distinguish than high-risk ideation. This contrasts with the field’s predominant focus on detecting rare, severe outcomes and suggests that distinguishing less severe but clinically meaningful ideation may require greater attention. We have also shown that translating the transcripts into English before classification did not reduce performance. This is notable because the English input undergoes two transformations—automatic speech recognition followed by translation—whereas the Arabic input undergoes only speech recognition. One possible explanation is that the larger volume of English-language pretraining data helps compensate for information lost during these transformations. However, this finding is based on a single sample and translation model and should not be interpreted as evidence that translation will generally improve or preserve performance across settings. In addition, macro-F1 and accuracy alone can mask clinically important failure modes. A model that predicts high risk for nearly every caller and one that predicts no callers as high risk may achieve seemingly moderate aggregate scores while being clinically unusable. Positive-class recall and PR-AUC should therefore be reported alongside macro-averaged metrics to provide a clearer picture of performance on the risk-positive class.

The role of the model is not to determine suicide risk on its own, but to support the people doing this work. For example, the model could flag calls for a supervisor to review, help prioritize callers for follow-up, or remind an operator to complete a structured risk assessment. In this role, detecting positive cases is more important than avoiding false alarms. An unnecessary review may cost a supervisor a few minutes, whereas missing a high-risk caller could have far more serious consequences. Two steps should precede any use in practice. First, performance should be evaluated on consecutive calls as they arrive, rather than on a retrospective sample that may be limited by factors such as call duration. Second, helpline staff should be given estimates of the expected number of additional reviews per week so they can understand the workload implications before the tool is deployed, rather than after implementation.

All the data come from one helpline in one country over a limited period. We do not know how the model would do in another service, another dialect region or another time. Only $545$ of the $987$ de-identified transcripts matched a service record, and a further $17$ were dropped because duplicate records disagreed. The $442$ unmatched calls carry no recorded assessment, so we cannot check whether they differ from the calls we kept. Calls longer than ten minutes were outside the eligibility window, so the sample does not cover the longest calls. We did not measure word error rate against transcripts written by hand. Levantine crisis speech is hard to transcribe, with dialect, broken speech, crying and overlapping voices. Both languages inherit those errors, and the English side adds translation error on top. Part of what we are comparing is therefore the quality of the text going in. A small hand-transcribed sample would settle this. The labels are the operator’s recorded C-SSRS assessment, not an independent review. A category is positive if any of its items is positive, so one wrong item makes the whole label positive. Errors in the labels therefore push in one direction only. At best the model reproduces operator judgement, including its mistakes. It does not predict suicidal behaviour, which was never observed here.

Sorting privacy-preserved Arabic helpline transcripts into two risk categories is feasible. Performance for the high-risk category supports further evaluation of the model as a potential decision-support tool for helpline operators. Lower-severity ideation was more difficult to distinguish than high-risk ideation. Translating the transcripts into English before classification also produced useful results. Positive-class recall and PR-AUC should be reported alongside macro-averaged metrics because macro-averaged scores alone can obscure poor performance on the positive class.