Showing 1–50 of 60 results for author: Saifullah

EcoBoat: Design and Experimental Validation of an Autonomous Body-Board Boat For Cleaning Water Bodies

Authors: M. Aman Ansari, Saifullah Khan, Rahul Kulkarni, PB Sujit

Abstract: Cleaning water bodies such as swimming pools and lakes typically demands significant manual effort or reliance on costly, sensor-intensive robotic systems. This paper presents EcoBoat, a low-cost autonomous surface vehicle built on a modified hull, designed to collect floating debris in both indoor and outdoor water bodies. For indoor environments, EcoBoat uses ultrasonic sensors to detect boundar… ▽ More Cleaning water bodies such as swimming pools and lakes typically demands significant manual effort or reliance on costly, sensor-intensive robotic systems. This paper presents EcoBoat, a low-cost autonomous surface vehicle built on a modified hull, designed to collect floating debris in both indoor and outdoor water bodies. For indoor environments, EcoBoat uses ultrasonic sensors to detect boundaries and obstacles, combining random-walk motion with boundary-following behavior. For outdoor environments, it relies on GPS-based geofencing paired with a random-walk strategy for area coverage. A key design feature is a Tesla-valve-inspired collection basket that allows debris intake during forward motion while preventing its escape during turning or braking. Field experiments in pools and lakes validated the design through iterative refinement, demonstrating that a minimalist sensing and computation approach can achieve effective, versatile debris collection across diverse water bodies. △ Less

Submitted 13 September, 2026; originally announced September 2026.

HyperWorld: Hypergraph-Structured State Serialization Improves Learned Textual World Models

Authors: Yun-Jian Zhang, Chen-Wei Liang, Tian-Yi Zhang, Jian Ding, Yi-Lun Wu, Ao-Bo Li, Wei-Cong Su, Saifullah, Hong-Yu An, Mu-Jiang-Shan Wang

Abstract: World models enable language-model agents to predict environment dynamics and plan before acting. In text environments, the model must learn symbolic action effects from serialized state descriptions, but the role of serialization structure remains underexplored. We present HyperWorld, a controlled study of state serialization for learned textual world models. We compare raw observations with thre… ▽ More World models enable language-model agents to predict environment dynamics and plan before acting. In text environments, the model must learn symbolic action effects from serialized state descriptions, but the role of serialization structure remains underexplored. We present HyperWorld, a controlled study of state serialization for learned textual world models. We compare raw observations with three symbolic serializations of the same ground-truth state: independent sentences, pairwise triples, and entity-centered hyperedge units that group multiple related facts around entities and relations. All variants use the same training objective: given a state and an action, predict symbolic effects or judge the action infeasible. Across model scales, data budgets, and in-distribution and out-of-distribution test worlds, hyperedge serialization gives the clearest gains for 0.5B--1.5B models and under distribution shift. Larger models reduce the gap, and pairwise triples can match or slightly exceed hyperedges on in-distribution exact match, but hyperedges achieve the strongest out-of-distribution fact F1 and the best small-to-medium scale trade-off between feasibility detection and effect prediction. In downstream greedy planning, the hyperedge world model also attains the highest success rate among the tested representations. These results show that higher-order state organization is a simple but effective inductive bias for learned symbolic world models, especially when model capacity is limited or test environments differ from training. △ Less

Submitted 12 June, 2026; originally announced September 2026.

Comments: 10 pages, 4 figures, 3 tables

PAUSE: Editable Strategy Artifacts for Long-Form Cultural Story Adaptation

Authors: Taaha Kazi, Vasu Sharma, Mohammad Saifullah, Abdur Rahman

Abstract: Generative AI systems increasingly mediate cultural adaptation, but their cultural decisions are often hidden inside prompts, transient model plans, or final prose. We study PAUSE (Pause-And-Update Strategy Editing), an intervention that exposes an editable adaptation strategy as a human control surface for cultural decisions in long-form story adaptation. The strategy is a structured artifact tha… ▽ More Generative AI systems increasingly mediate cultural adaptation, but their cultural decisions are often hidden inside prompts, transient model plans, or final prose. We study PAUSE (Pause-And-Update Strategy Editing), an intervention that exposes an editable adaptation strategy as a human control surface for cultural decisions in long-form story adaptation. The strategy is a structured artifact that can be inspected, edited, and then projected through downstream character, entity, and chapter-localization stages. In two Chinese-source serialized novels, we test whether human edits to this strategy propagate into chapter-level prose. Across 9 edited-vs-control chapter comparisons, judges select the edited-strategy output in all 9; a marker audit shows target markers in 8/9 edited outputs and 0/9 controls, with forbidden markers absent from edited outputs and present in all controls. We frame these results as a smoke-scale edit-adherence study, not a claim that the outputs are culturally authoritative or literary-quality improvements. PAUSE offers one practical way to make AI-mediated cultural adaptation more inspectable and contestable before decisions propagate through long-form generation. △ Less

Submitted 7 August, 2026; originally announced August 2026.

Comments: 6 pages, 1 figure, 3 tables. Accepted at the 1st Workshop on Culture x AI: Evaluating AI as a Cultural Technology, ICML 2026. Project page: https://abdur75648.github.io/pause/

Hierarchical Channel Stacking: A Structured Decision Framework for AI-Generated Image Detection

Authors: Saifullah Shoaib, Akash Borigi, Rupendra Lekkala, Amaury Lendasse, Edward Ratner, Sai Sowjanya Bhamidipati, Alexander Schlager, Peggy Lindner

Abstract: Many synthetic-image detectors produce accurate predictions but offer limited insight into how those decisions are formed. This paper introduces Hierarchical Channel Stacking (HCS), a compact framework for AI-generated image detection that converts intermediate CNN activations into a structured 60-dimensional representation organized across three progressively deeper backbone stages. HCS uses per-… ▽ More Many synthetic-image detectors produce accurate predictions but offer limited insight into how those decisions are formed. This paper introduces Hierarchical Channel Stacking (HCS), a compact framework for AI-generated image detection that converts intermediate CNN activations into a structured 60-dimensional representation organized across three progressively deeper backbone stages. HCS uses per-channel Level-1 classifiers and a Level-2 aggregator to produce image-level predictions while preserving explicit hierarchical structure for analysis. On a benchmark spanning GAN and diffusion generators, HCS achieves 86.7% accuracy and 86.7% macro-F1 on the held-out test set. Stage ablation shows that the full three-stage system outperforms reduced single-stage and two-stage variants, indicating that the hierarchy carries complementary predictive information. Stage-level contribution analysis further shows that, in the analyzed detector setting, fake GAN and fake diffusion images exhibit distinct stage-level contribution profiles. These results position HCS not simply as a compact detector, but as a structured framework for studying how synthetic-image detectors assemble evidence across representation levels. △ Less

Submitted 2 September, 2026; v1 submitted 27 August, 2026; originally announced August 2026.

Comments: Withdrawn due to insufficient consent approval

HyCE-RAG: Hypergraph Chain-of-Evidence Retrieval-Augmented Generation for Explainable Multi-hop Question Answering

Authors: Hong-Yu An, Yun-Jian Zhang, Chen-Wei Liang, Tian-Yi Zhang, Jian Ding, Yi-Lun Wu, Ao-Bo Li, Wei-Cong Su, Saifullah, Mujiangshan Wang

Abstract: Multi-hop question answering requires systems to retrieve evidence from multiple documents and connect scattered facts into a coherent reasoning process. Standard retrieval-augmented generation (RAG) mainly relies on semantic similarity between a query and text chunks, and therefore often fails to model structural relations among entities, facts, and evidence units. Graph-based RAG improves this b… ▽ More Multi-hop question answering requires systems to retrieve evidence from multiple documents and connect scattered facts into a coherent reasoning process. Standard retrieval-augmented generation (RAG) mainly relies on semantic similarity between a query and text chunks, and therefore often fails to model structural relations among entities, facts, and evidence units. Graph-based RAG improves this by introducing graph-structured knowledge, but pairwise edges are still limited in representing higher-order associations involving multiple entities and contexts. We propose HyCE-RAG, a Hypergraph Chain-of-Evidence Retrieval-Augmented Generation framework for explainable multi-hop question answering. HyCE-RAG organizes entities, relations, and contextual evidence into hyperedges, builds a query-aware evidence hypergraph, and performs confidence propagation over entity--hyperedge incidence structures. It then uses confidence-guided evidence assembly to select, connect, and rank evidence paths before answer generation. The scoring process jointly considers semantic relevance, entity connectivity, evidence coverage, relation reliability, extraction confidence, and propagated confidence. By providing the language model with structured evidence chains rather than flat retrieved passages, HyCE-RAG supports more faithful and interpretable reasoning. Experiments on HotpotQA, 2WikiMultihopQA, MuSiQue, and two GraphRAG-Bench subsets show that HyCE-RAG consistently outperforms standard RAG and graph-based RAG baselines in answer accuracy, context relevance, and faithfulness. These results suggest that hypergraph-based evidence organization is a promising direction for post-retrieval reasoning in complex question answering. △ Less

Submitted 12 June, 2026; originally announced July 2026.

Comments: 15 pages, 3 figures, 4 tables

MSC Class: 68T50; 68T30; 68P20 ACM Class: I.2.7; H.3.3; I.2.4

Narrative World Model: Narratology-Grounded Writer Memory for Long-Form Fiction

Authors: Mohammad Saifullah, Thomas Kornmaier, Taaha Kazi, Vasu Sharma, Aditya Sanjiv Kanade, Aanand Kumar Yadav

Abstract: Long-form fiction writers need memory that answers multi-hop questions about evolving story state: who knows a secret and when they learned it, whether an event preceded the narration that revealed it, whether a setup paid off, and how a relationship shifted. General-purpose retrieval and agent-memory systems represent entities and facts but not the narratological structure these questions turn on… ▽ More Long-form fiction writers need memory that answers multi-hop questions about evolving story state: who knows a secret and when they learned it, whether an event preceded the narration that revealed it, whether a setup paid off, and how a relationship shifted. General-purpose retrieval and agent-memory systems represent entities and facts but not the narratological structure these questions turn on, so they surface the wrong evidence or none at all. We introduce the Narrative World Model (NWM), a writer-memory system that pairs a narratology-grounded typed temporal-state graph with query-conditioned hybrid retrieval. To measure memory rather than the answerer, we read every system through a single held-constant Opus 4.8 reader over only that system's chapter-safe evidence, on a reproducible public corpus and a validated multi-hop benchmark, and we compare against the strongest existing temporal-knowledge-graph agent-memory framework, Graphiti/Zep (Rasmussen et al., 2025). NWM substantially and significantly outperforms this baseline on multi-hop narratological QA across both corpora, and far exceeds GraphRAG and flat retrieval. The advantage is representational rather than an artifact of extraction: it survives rebuilding the baseline with NWM's own extractor, and traces to its narratology-grounded structure and query-conditioned retrieval, not to graph size or extractor quality. △ Less

Submitted 6 July, 2026; originally announced July 2026.

Comments: 23 pages, 4 figures; 9-page main text plus appendix. Preprint

NarrativeWorldBench: A Frontier-Saturated Benchmark and a Latent World Model for Long-Horizon Co-Creative Audio Drama

Authors: Logan Mann, Abdur Rahman, Mohammad Saifullah, Taaha Kazi, Vasu Sharma

Abstract: Long-form serialized audio drama, with arcs that run for 200 to 800 episodes, is a major creative medium and a setting where frontier large language models (LLMs) fail. We benchmark 21 models, spanning classical, fine-tuned, open-frontier, closed-frontier, and reasoning tiers, on a uniform set of structural narrative metrics. All closed-frontier systems saturate at a plot-beat F1 in the band [0.78… ▽ More Long-form serialized audio drama, with arcs that run for 200 to 800 episodes, is a major creative medium and a setting where frontier large language models (LLMs) fail. We benchmark 21 models, spanning classical, fine-tuned, open-frontier, closed-frontier, and reasoning tiers, on a uniform set of structural narrative metrics. All closed-frontier systems saturate at a plot-beat F1 in the band [0.78, 0.81] and collapse by about -0.20 F1 at horizon h=200. We introduce NarrativeWorldBench, an open benchmark of nine narrative-structure metrics evaluated across horizons h in {10, 20, 50, 100, 200}, with cross-lingual evaluation across four Indic languages (Hindi, Tamil, Telugu, Marathi). We introduce N-VSSM, a Narrative Variational State-Space Model that maintains a structured 256-dimensional latent world state over more than 200 episodes via a Mamba-2 backbone with an event-conditioned posterior and an 8B decoder. N-VSSM holds plot-beat F1 >= 0.84 across all horizons at 4x lower compute than the closed-frontier band. A learned Cultural Transfer Function lifts cross-language fidelity by +0.20 to +0.23 Likert points. In a within-subjects writer study (n = 12 professional authors, 240 trials), N-VSSM is preferred over Claude Opus 4.5 on long-arc consistency 71% of the time and rated +1.3 Likert points higher on controllability. △ Less

Submitted 15 June, 2026; originally announced June 2026.

Comments: 10 pages. Accepted to the ICML 2026 Workshops on High-dimensional Learning Dynamics (HiLD) and Culture x AI

SiNFluD: Creating and Evaluating Figurative Language Dataset for Sindhi

Authors: Wazir Ali, Adeeb Noor, Saifullah Tumrani

Abstract: In this article, we introduce SiNFluD, a novel benchmark dataset for Sindhi figurative language classification. We first collect raw text from various blogs, social media platforms, and literary sources, and subsequently prepare the corpus for annotation. Two native annotators label the data using the Doccano text annotation tool, achieving an inter-annotator agreement of 0.81. We then establish b… ▽ More In this article, we introduce SiNFluD, a novel benchmark dataset for Sindhi figurative language classification. We first collect raw text from various blogs, social media platforms, and literary sources, and subsequently prepare the corpus for annotation. Two native annotators label the data using the Doccano text annotation tool, achieving an inter-annotator agreement of 0.81. We then establish baseline results using 5-fold and 10-fold cross-validation. Finally, we evaluate mBERT, XLM-RoBERTa, and XLM-RoBERTa-XL models, along with SetFit for few-shot fine-tuning of sentence transformers. Among these, the pretrained XLM-RoBERTa-XL achieves the best performance. △ Less

Submitted 9 May, 2026; v1 submitted 2 May, 2026; originally announced May 2026.

DocDjinn: Controllable Synthetic Document Generation with VLMs and Handwriting Diffusion

Authors: Marcel Lamott, Saifullah Saifullah, Nauman Riaz, Yves-Noel Weweler, Tobias Alt-Veit, Ahmad Sarmad Ali, Muhammad Armaghan Shakir, Adrian Kalwa, Momina Moetesum, Andreas Dengel, Sheraz Ahmed, Faisal Shafait, Ulrich Schwanecke, Adrian Ulges

Abstract: Effective document intelligence models rely on large amounts of annotated training data. However, procuring sufficient and high-quality data poses significant challenges due to the labor-intensive and costly nature of data acquisition. Additionally, leveraging language models to annotate real documents raises concerns about data privacy. Synthetic document generation has emerged as a promising, pr… ▽ More Effective document intelligence models rely on large amounts of annotated training data. However, procuring sufficient and high-quality data poses significant challenges due to the labor-intensive and costly nature of data acquisition. Additionally, leveraging language models to annotate real documents raises concerns about data privacy. Synthetic document generation has emerged as a promising, privacy-preserving alternative. We propose DocDjinn, a novel framework for controllable synthetic document generation using Vision-Language Models (VLMs) that produces annotated documents from unlabeled seed samples. Our approach generates visually plausible and semantically consistent synthetic documents that follow the distribution of an existing source dataset through clustering-based seed selection with parametrized sampling. By enriching documents with realistic diffusion-based handwriting and contextual visual elements via semantic-visual decoupling, we generate diverse, high-quality annotated synthetic documents. We evaluate across eleven benchmarks spanning key information extraction, question answering, document classification, and document layout analysis. To our knowledge, this is the first work demonstrating that VLMs can generate faithful annotated document datasets at scale from unlabeled seeds that can effectively enrich or approximate real, manually annotated data for diverse document understanding tasks. We show that with only 100 real training samples, our framework achieves on average $87\%$ of the performance of the full real-world dataset. We publicly release our code and 140k+ synthetic document samples. △ Less

Submitted 25 February, 2026; originally announced February 2026.

ZeroDVFS: Zero-Shot LLM-Guided Core and Frequency Allocation for Embedded Platforms

Authors: Mohammad Pivezhandi, Mahdi Banisharif, Abusayeed Saifullah, Ali Jannesari

Abstract: Dynamic voltage and frequency scaling (DVFS) and task-to-core allocation are critical for thermal management and balancing energy and performance in embedded systems. Existing approaches either rely on utilization-based heuristics that overlook stall times, or require extensive offline profiling for table generation, preventing runtime adaptation. Building upon hierarchical multi-agent scheduling,… ▽ More Dynamic voltage and frequency scaling (DVFS) and task-to-core allocation are critical for thermal management and balancing energy and performance in embedded systems. Existing approaches either rely on utilization-based heuristics that overlook stall times, or require extensive offline profiling for table generation, preventing runtime adaptation. Building upon hierarchical multi-agent scheduling, we contribute model-based reinforcement learning with accurate environment models that predict thermal dynamics and performance states, enabling synthetic training data generation and converging 20 times faster than model-free methods. We introduce Large Language Model (LLM)-based semantic feature extraction that characterizes OpenMP programs through code-level features without execution, enabling zero-shot deployment for new workloads in under 5 seconds without workload-specific profiling. Two collaborative agents decompose the exponential action space, achieving 358ms latency for subsequent decisions. Experiments on Barcelona OpenMP Tasks Suite (BOTS) and PolybenchC benchmarks across NVIDIA Jetson TX2, Jetson Orin NX, RubikPi, and Intel Core i7 demonstrate 7.09 times better energy efficiency, 4.0 times better makespan, and 358ms decision latency compared to existing power management techniques. △ Less

Submitted 28 February, 2026; v1 submitted 12 January, 2026; originally announced January 2026.

Comments: 56 pages, 14 figures, 18 tables (including appendix)

HiDVFS: Hierarchical Multi-Agent DVFS for Real-Time OpenMP DAG Workloads

Authors: Mohammad Pivezhandi, Abusayeed Saifullah, Ali Jannesari

Abstract: Leakage power in multicore embedded systems now rivals dynamic power, so DVFS schedulers must respect deadlines and thermal limits, not just average makespan. Existing heuristics lack per-core, temperature-aware control and overlook the irregular execution of OpenMP DAGs. We propose HiDVFS, a general, extensible hierarchical multi-agent DVFS scheduler: a profiler agent selects cores and frequencie… ▽ More Leakage power in multicore embedded systems now rivals dynamic power, so DVFS schedulers must respect deadlines and thermal limits, not just average makespan. Existing heuristics lack per-core, temperature-aware control and overlook the irregular execution of OpenMP DAGs. We propose HiDVFS, a general, extensible hierarchical multi-agent DVFS scheduler: a profiler agent selects cores and frequencies, a thermal agent groups cores by temperature, and a priority agent orders tasks under contention, all trained with a makespan-focused reward using short-horizon future-state shaping for sample efficiency. Deadlines are soft, derived from a measured reference cost; a federated schedulability gate keeps operating points feasible, and a calibrated split-conformal shield bounds each action's predicted response time. On Jetson TX2 with multi-seed validation, HiDVFS attains a 4.16+/-0.58 s L10 makespan, a 2.83x speedup and 32.9% energy reduction over a fairness-corrected GearDVFS port, and a 4.62x average speedup with 55.7% energy reduction across all 12 BOTS benchmarks. Cross-platform results on TX2, Orin NX, and RubikPi show deadline-aware DVFS cuts energy 15 to 18% versus pinning the maximum frequency, and a measured mixed-criticality study shows cluster-aware reservation is required to keep a high-criticality task's deadline-miss ratio at zero. △ Less

Submitted 8 July, 2026; v1 submitted 9 January, 2026; originally announced January 2026.

Comments: 52 pages, 24 figures, 32 tables (supplement included as appendices). Under review at IEEE TPDS. v2: fairness-corrected GearDVFS baseline with fair-port study, multi-seed same-window re-measurement and full number audit, 12-benchmark BOTS evaluation, real-time evaluation (feasibility gate, conformal shield, mixed-criticality). Data: https://doi.org/10.5281/zenodo.21212162

ACM Class: I.2.6; C.3; D.4.1

GraphPerf-RT: A Graph-Driven Performance Model for Hardware-Aware Scheduling of OpenMP Codes

Authors: Mohammad Pivezhandi, Mahdi Banisharif, Saeed Bakhshan, Abusayeed Saifullah, Ali Jannesari

Abstract: Autonomous AI agents on embedded platforms require real-time, risk-aware scheduling under resource and thermal constraints. Classical heuristics struggle with workload irregularity, tabular regressors discard structural information, and model-free reinforcement learning (RL) risks overheating. We introduce GraphPerf-RT, a graph neural network surrogate achieving deep learning accuracy at heuristic… ▽ More Autonomous AI agents on embedded platforms require real-time, risk-aware scheduling under resource and thermal constraints. Classical heuristics struggle with workload irregularity, tabular regressors discard structural information, and model-free reinforcement learning (RL) risks overheating. We introduce GraphPerf-RT, a graph neural network surrogate achieving deep learning accuracy at heuristic speeds (2-7ms). GraphPerf-RT is, to our knowledge, the first to unify task DAG topology, CFG-derived code semantics, and runtime context (per-core DVFS, thermal state, utilization) in a heterogeneous graph with typed edges encoding precedence, placement, and contention. Evidential regression with Normal-Inverse-Gamma priors provides calibrated uncertainty; we validate on makespan prediction for risk-aware scheduling. Experiments on three ARM platforms (Jetson TX2, Orin NX, RUBIK Pi) achieve R^2 = 0.81 on log-transformed makespan with Spearman rho = 0.95 and conservative uncertainty calibration (PICP = 99.9% at 95% confidence). Integration with four RL methods demonstrates that multi-agent model-based RL with GraphPerf-RT as the world model achieves 66% makespan reduction and 82% energy reduction versus model-free baselines, with zero thermal violations. △ Less

Submitted 21 January, 2026; v1 submitted 12 December, 2025; originally announced December 2025.

Comments: 49 pages, 4 figures, 19 tables

ACM Class: C.4; D.1.3; I.2.6

Multi-Agent Coordination in Autonomous Vehicle Routing: A Simulation-Based Study of Communication, Memory, and Routing Loops

Authors: KM Khalid Saifullah, Daniel Palmer

Abstract: Multi-agent coordination is critical for next-generation autonomous vehicle (AV) systems, yet naive implementations of communication-based rerouting can lead to catastrophic performance degradation. This study investigates a fundamental problem in decentralized multi-agent navigation: routing loops, where vehicles without persistent obstacle memory become trapped in cycles of inefficient path reca… ▽ More Multi-agent coordination is critical for next-generation autonomous vehicle (AV) systems, yet naive implementations of communication-based rerouting can lead to catastrophic performance degradation. This study investigates a fundamental problem in decentralized multi-agent navigation: routing loops, where vehicles without persistent obstacle memory become trapped in cycles of inefficient path recalculation. Through systematic simulation experiments involving 72 unique configurations across varying vehicle densities (15, 35, 55 vehicles) and obstacle frequencies (6, 20 obstacles), we demonstrate that memory-less reactive rerouting increases average travel time by up to 682% compared to baseline conditions. To address this, we introduce Object Memory Management (OMM), a lightweight mechanism enabling agents to retain and share knowledge of previously encountered obstacles. OMM operates by maintaining a distributed blacklist of blocked nodes, which each agent consults during Dijkstra-based path recalculation, effectively preventing redundant routing attempts. Our results show that OMM-enabled coordination reduces average travel time by 75.7% and wait time by 88% compared to memory-less systems, while requiring only 1.67 route recalculations per vehicle versus 9.83 in memory-less scenarios. This work provides empirical evidence that persistent, shared memory is not merely beneficial but essential for robust multi-agent coordination in dynamic environments. The findings have implications beyond autonomous vehicles, informing the design of decentralized systems in robotics, network routing, and distributed AI. We provide a comprehensive experimental analysis, including detailed scenario breakdowns, scalability assessments, and visual documentation of the routing loop phenomenon, demonstrating OMM's critical role in preventing detrimental feedback cycles in cooperative multi-agent systems. △ Less

Submitted 20 November, 2025; originally announced November 2025.

DynaGuard: A Dynamic Guardian Model With User-Defined Policies

Authors: Monte Hoover, Vatsal Baherwani, Neel Jain, Khalid Saifullah, Joseph Vincent, Chirag Jain, Melissa Kazemi Rad, C. Bayan Bruss, Ashwinee Panda, Tom Goldstein

Abstract: Guardian models play a crucial role in ensuring the safety and ethical behavior of user-facing AI applications by enforcing guardrails and detecting harmful content. While standard guardian models are limited to predefined, static harm categories, we introduce DynaGuard, a suite of dynamic guardian models offering novel flexibility by evaluating text based on user-defined policies, and DynaBench,… ▽ More Guardian models play a crucial role in ensuring the safety and ethical behavior of user-facing AI applications by enforcing guardrails and detecting harmful content. While standard guardian models are limited to predefined, static harm categories, we introduce DynaGuard, a suite of dynamic guardian models offering novel flexibility by evaluating text based on user-defined policies, and DynaBench, a dataset for training and evaluating dynamic guardian models. Our models provide both rapid detection of policy violations and a chain-of-thought reasoning option that articulate and justify model outputs. Critically, DynaGuard not only surpasses static models in detection accuracy on traditional safety categories, but is competitive with frontier reasoning models on free-form policy violations, all in a fraction of the time. This makes DynaGuard an critical tool for language model guardrails. △ Less

Submitted 6 October, 2025; v1 submitted 2 September, 2025; originally announced September 2025.

Comments: 22 Pages

DocVCE: Diffusion-based Visual Counterfactual Explanations for Document Image Classification

Authors: Saifullah Saifullah, Stefan Agne, Andreas Dengel, Sheraz Ahmed

Abstract: As black-box AI-driven decision-making systems become increasingly widespread in modern document processing workflows, improving their transparency and reliability has become critical, especially in high-stakes applications where biases or spurious correlations in decision-making could lead to serious consequences. One vital component often found in such document processing workflows is document i… ▽ More As black-box AI-driven decision-making systems become increasingly widespread in modern document processing workflows, improving their transparency and reliability has become critical, especially in high-stakes applications where biases or spurious correlations in decision-making could lead to serious consequences. One vital component often found in such document processing workflows is document image classification, which, despite its widespread use, remains difficult to explain. While some recent works have attempted to explain the decisions of document image classification models through feature-importance maps, these maps are often difficult to interpret and fail to provide insights into the global features learned by the model. In this paper, we aim to bridge this research gap by introducing generative document counterfactuals that provide meaningful insights into the model's decision-making through actionable explanations. In particular, we propose DocVCE, a novel approach that leverages latent diffusion models in combination with classifier guidance to first generate plausible in-distribution visual counterfactual explanations, and then performs hierarchical patch-wise refinement to search for a refined counterfactual that is closest to the target factual image. We demonstrate the effectiveness of our approach through a rigorous qualitative and quantitative assessment on 3 different document classification datasets -- RVL-CDIP, Tobacco3482, and DocLayNet -- and 3 different models -- ResNet, ConvNeXt, and DiT -- using well-established evaluation criteria such as validity, closeness, and realism. To the best of the authors' knowledge, this is the first work to explore generative counterfactual explanations in document image analysis. △ Less

Submitted 6 August, 2025; originally announced August 2025.

DP-DocLDM: Differentially Private Document Image Generation using Latent Diffusion Models

Authors: Saifullah Saifullah, Stefan Agne, Andreas Dengel, Sheraz Ahmed

Abstract: As deep learning-based, data-driven information extraction systems become increasingly integrated into modern document processing workflows, one primary concern is the risk of malicious leakage of sensitive private data from these systems. While some recent works have explored Differential Privacy (DP) to mitigate these privacy risks, DP-based training is known to cause significant performance deg… ▽ More As deep learning-based, data-driven information extraction systems become increasingly integrated into modern document processing workflows, one primary concern is the risk of malicious leakage of sensitive private data from these systems. While some recent works have explored Differential Privacy (DP) to mitigate these privacy risks, DP-based training is known to cause significant performance degradation and impose several limitations on standard training procedures, making its direct application to downstream tasks both difficult and costly. In this work, we aim to address the above challenges within the context of document image classification by substituting real private data with a synthetic counterpart. In particular, we propose to use conditional latent diffusion models (LDMs) in combination with differential privacy (DP) to generate class-specific synthetic document images under strict privacy constraints, which can then be utilized to train a downstream classifier following standard training procedures. We investigate our approach under various pretraining setups, including unconditional, class-conditional, and layout-conditional pretraining, in combination with multiple private training strategies such as class-conditional and per-label private fine-tuning with DPDM and DP-Promise algorithms. Additionally, we evaluate it on two well-known document benchmark datasets, RVL-CDIP and Tobacco3482, and show that it can generate useful and realistic document samples across various document types and privacy levels ($\varepsilon \in \{1, 5, 10\}$). Lastly, we show that our approach achieves substantial performance improvements in downstream evaluations on small-scale datasets, compared to the direct application of DP-Adam. △ Less

Submitted 6 August, 2025; originally announced August 2025.

Comments: Accepted in ICDAR 2025

LoRaIN: A Constructive Interference-Assisted Reliable and Energy-Efficient LoRa Indoor Network

Authors: Mahbubur Rahman, Abusayeed Saifullah

Abstract: LoRa is a promising communication technology for enabling the next-generation indoor Internet of Things applications. Very few studies, however, have analyzed its performance indoors. Besides, these indoor studies investigate mostly the RSSI and SNR of the received packets at the gateway, which, as we show, may not unfold the poor performance of LoRa and its MAC protocol, LoRaWAN, indoors in terms… ▽ More LoRa is a promising communication technology for enabling the next-generation indoor Internet of Things applications. Very few studies, however, have analyzed its performance indoors. Besides, these indoor studies investigate mostly the RSSI and SNR of the received packets at the gateway, which, as we show, may not unfold the poor performance of LoRa and its MAC protocol, LoRaWAN, indoors in terms of reliability and energy-efficiency. In this paper, we extensively evaluate the performance of LoRaWAN indoors and then use the key insights to boost its reliability and energy-efficiency by proposing LoRaIN, LoRa Indoor Network, a new link-layer protocol that can be effectively used for indoor deployments. The approach to boosting the reliability and energy efficiency in LoRaIN is underpinned by enabling constructive interference with specific timing requirements analyzed both empirically and mathematically for different pairs of channel bandwidth and spreading factor and relaying precious acknowledgments to the end-devices with the assistance of several booster nodes. The booster nodes do not need any special capability and can be a subset of the LoRa end-devices. To our knowledge, LoRaIN is the first protocol for boosting reliability and energy-efficiency in indoor LoRa networks. We evaluate its performance in an indoor testbed consisting of one LoRaWAN gateway and 20 end-devices. Our extensive evaluation shows that when 15% of the end-devices operate as booster nodes, the reliability at the gateway increases from 62% to 95%, and the end-devices are approximately 2.5x energy-efficient. △ Less

Submitted 19 June, 2025; originally announced June 2025.

Sentiment Analysis in Software Engineering: Evaluating Generative Pre-trained Transformers

Authors: KM Khalid Saifullah, Faiaz Azmain, Habiba Hye

Abstract: Sentiment analysis plays a crucial role in understanding developer interactions, issue resolutions, and project dynamics within software engineering (SE). While traditional SE-specific sentiment analysis tools have made significant strides, they often fail to account for the nuanced and context-dependent language inherent to the domain. This study systematically evaluates the performance of bidire… ▽ More Sentiment analysis plays a crucial role in understanding developer interactions, issue resolutions, and project dynamics within software engineering (SE). While traditional SE-specific sentiment analysis tools have made significant strides, they often fail to account for the nuanced and context-dependent language inherent to the domain. This study systematically evaluates the performance of bidirectional transformers, such as BERT, against generative pre-trained transformers, specifically GPT-4o-mini, in SE sentiment analysis. Using datasets from GitHub, Stack Overflow, and Jira, we benchmark the models' capabilities with fine-tuned and default configurations. The results reveal that fine-tuned GPT-4o-mini performs comparable to BERT and other bidirectional models on structured and balanced datasets like GitHub and Jira, achieving macro-averaged F1-scores of 0.93 and 0.98, respectively. However, on linguistically complex datasets with imbalanced sentiment distributions, such as Stack Overflow, the default GPT-4o-mini model exhibits superior generalization, achieving an accuracy of 85.3\% compared to the fine-tuned model's 13.1\%. These findings highlight the trade-offs between fine-tuning and leveraging pre-trained models for SE tasks. The study underscores the importance of aligning model architectures with dataset characteristics to optimize performance and proposes directions for future research in refining sentiment analysis tools tailored to the SE domain. △ Less

Submitted 22 April, 2025; originally announced May 2025.

ForestVO: Enhancing Visual Odometry in Forest Environments through ForestGlue

Authors: Thomas Pritchard, Saifullah Ijaz, Ronald Clark, Basaran Bahadir Kocer

Abstract: Recent advancements in visual odometry systems have improved autonomous navigation; however, challenges persist in complex environments like forests, where dense foliage, variable lighting, and repetitive textures compromise feature correspondence accuracy. To address these challenges, we introduce ForestGlue, enhancing the SuperPoint feature detector through four configurations - grayscale, RGB,… ▽ More Recent advancements in visual odometry systems have improved autonomous navigation; however, challenges persist in complex environments like forests, where dense foliage, variable lighting, and repetitive textures compromise feature correspondence accuracy. To address these challenges, we introduce ForestGlue, enhancing the SuperPoint feature detector through four configurations - grayscale, RGB, RGB-D, and stereo-vision - optimised for various sensing modalities. For feature matching, we employ LightGlue or SuperGlue, retrained with synthetic forest data. ForestGlue achieves comparable pose estimation accuracy to baseline models but requires only 512 keypoints - just 25% of the baseline's 2048 - to reach an LO-RANSAC AUC score of 0.745 at a 10° threshold. With only a quarter of keypoints needed, ForestGlue significantly reduces computational overhead, demonstrating effectiveness in dynamic forest environments, and making it suitable for real-time deployment on resource-constrained platforms. By combining ForestGlue with a transformer-based pose estimation model, we propose ForestVO, which estimates relative camera poses using matched 2D pixel coordinates between frames. On challenging TartanAir forest sequences, ForestVO achieves an average relative pose error (RPE) of 1.09 m and a kitti_score of 2.33%, outperforming direct-based methods like DSO by 40% in dynamic scenes. Despite using only 10% of the dataset for training, ForestVO maintains competitive performance with TartanVO while being a significantly lighter model. This work establishes an end-to-end deep learning pipeline specifically tailored for visual odometry in forested environments, leveraging forest-specific training data to optimise feature correspondence and pose estimation, thereby enhancing the accuracy and robustness of autonomous navigation systems. △ Less

Submitted 1 April, 2025; originally announced April 2025.

Comments: Accepted to the IEEE Robotics and Automation Letters

PSO-UNet: Particle Swarm-Optimized U-Net Framework for Precise Multimodal Brain Tumor Segmentation

Authors: Shoffan Saifullah, Rafał Dreżewski

Abstract: Medical image segmentation, particularly for brain tumor analysis, demands precise and computationally efficient models due to the complexity of multimodal MRI datasets and diverse tumor morphologies. This study introduces PSO-UNet, which integrates Particle Swarm Optimization (PSO) with the U-Net architecture for dynamic hyperparameter optimization. Unlike traditional manual tuning or alternative… ▽ More Medical image segmentation, particularly for brain tumor analysis, demands precise and computationally efficient models due to the complexity of multimodal MRI datasets and diverse tumor morphologies. This study introduces PSO-UNet, which integrates Particle Swarm Optimization (PSO) with the U-Net architecture for dynamic hyperparameter optimization. Unlike traditional manual tuning or alternative optimization approaches, PSO effectively navigates complex hyperparameter search spaces, explicitly optimizing the number of filters, kernel size, and learning rate. PSO-UNet substantially enhances segmentation performance, achieving Dice Similarity Coefficients (DSC) of 0.9578 and 0.9523 and Intersection over Union (IoU) scores of 0.9194 and 0.9097 on the BraTS 2021 and Figshare datasets, respectively. Moreover, the method reduces computational complexity significantly, utilizing only 7.8 million parameters and executing in approximately 906 seconds, markedly faster than comparable U-Net-based frameworks. These outcomes underscore PSO-UNet's robust generalization capabilities across diverse MRI modalities and tumor classifications, emphasizing its clinical potential and clear advantages over conventional hyperparameter tuning methods. Future research will explore hybrid optimization strategies and validate the framework against other bio-inspired algorithms to enhance its robustness and scalability. △ Less

Submitted 24 March, 2025; originally announced March 2025.

Comments: 9 pages, 6 figures, 4 tables, Gecco 2025 Conference

MSC Class: 68Q07 ACM Class: I.4.6; I.2

Journal ref: GECCO '25 Companion: Proceedings of the Genetic and Evolutionary Computation Conference Companion, 2025

Feature-Aware Task-to-Core Allocation in Embedded Multi-core Platforms via Statistical Learning

Authors: Mohammad Pivezhandi, Abusayeed Saifullah, Prashant Modekurthy

Abstract: Optimizing task-to-core allocation can substantially reduce power consumption in multi-core platforms without degrading user experience. However, existing approaches overlook critical factors such as parallelism, compute intensity, and heterogeneous core types. In this paper, we introduce a statistical learning approach for feature selection that identifies the most influential features-such as co… ▽ More Optimizing task-to-core allocation can substantially reduce power consumption in multi-core platforms without degrading user experience. However, existing approaches overlook critical factors such as parallelism, compute intensity, and heterogeneous core types. In this paper, we introduce a statistical learning approach for feature selection that identifies the most influential features-such as core type, speed, temperature, and application-level parallelism or memory intensity-for accurate environment modeling and efficient energy minimization, a critical consideration for embedded systems. Our experiments, conducted with state-of-the-art Linux governors and thermal modeling techniques, show that correlation-aware task-to-core allocation lowers energy consumption by up to 10% and reduces core temperature by up to 5C compared to random core selection. Furthermore, our compressed, bootstrapped regression model improves thermal prediction accuracy by 6% while cutting model parameters by 16%, yielding an overall mean square error reduction of 61.6% relative to existing approaches. We provided results based on superscalar Intel Core i7 12th Gen processors with 14 cores, and validated our method across a diverse set of hardware platforms and effectively balanced performance, power, and thermal demands through statistical feature evaluation. △ Less

Submitted 10 January, 2026; v1 submitted 26 January, 2025; originally announced February 2025.

Comments: 15 pages, 9 figures. Published in IEEE RTCSA 2025

ACM Class: C.3; D.4.1; I.2.6

A light-weight model to generate NDWI from Sentinel-1

Authors: Saleh Sakib Ahmed, Saifur Rahman Jony, Md. Toufikuzzaman, Saifullah Sayed, Rashed Uz Zzaman, Sara Nowreen, M. Sohel Rahman

Abstract: The use of Sentinel-2 images to compute Normalized Difference Water Index (NDWI) has many applications, including water body area detection. However, cloud cover poses significant challenges in this regard, which hampers the effectiveness of Sentinel-2 images in this context. In this paper, we present a deep learning model that can generate NDWI given Sentinel-1 images, thereby overcoming this clo… ▽ More The use of Sentinel-2 images to compute Normalized Difference Water Index (NDWI) has many applications, including water body area detection. However, cloud cover poses significant challenges in this regard, which hampers the effectiveness of Sentinel-2 images in this context. In this paper, we present a deep learning model that can generate NDWI given Sentinel-1 images, thereby overcoming this cloud barrier. We show the effectiveness of our model, where it demonstrates a high accuracy of 0.9134 and an AUC of 0.8656 to predict the NDWI. Additionally, we observe promising results with an R2 score of 0.4984 (for regressing the NDWI values) and a Mean IoU of 0.4139 (for the underlying segmentation task). In conclusion, our model offers a first and robust solution for generating NDWI images directly from Sentinel-1 images and subsequent use for various applications even under challenging conditions such as cloud cover and nighttime. △ Less

Submitted 22 January, 2025; originally announced January 2025.

WordVIS: A Color Worth A Thousand Words

Authors: Umar Khan, Saifullah, Stefan Agne, Andreas Dengel, Sheraz Ahmed

Abstract: Document classification is considered a critical element in automated document processing systems. In recent years multi-modal approaches have become increasingly popular for document classification. Despite their improvements, these approaches are underutilized in the industry due to their requirement for a tremendous volume of training data and extensive computational power. In this paper, we at… ▽ More Document classification is considered a critical element in automated document processing systems. In recent years multi-modal approaches have become increasingly popular for document classification. Despite their improvements, these approaches are underutilized in the industry due to their requirement for a tremendous volume of training data and extensive computational power. In this paper, we attempt to address these issues by embedding textual features directly into the visual space, allowing lightweight image-based classifiers to achieve state-of-the-art results using small-scale datasets in document classification. To evaluate the efficacy of the visual features generated from our approach on limited data, we tested on the standard dataset Tobacco-3482. Our experiments show a tremendous improvement in image-based classifiers, achieving an improvement of 4.64% using ResNet50 with no document pre-training. It also sets a new record for the best accuracy of the Tobacco-3482 dataset with a score of 91.14% using the image-based DocXClassifier with no document pre-training. The simplicity of the approach, its resource requirements, and subsequent results provide a good prospect for its use in industrial use cases. △ Less

Submitted 13 December, 2024; originally announced December 2024.

FlowRL: Flow-Augmented Few-Shot Reinforcement Learning for Semi-Structured Sensor Data

Authors: Mohammad Pivezhandi, Abusayeed Saifullah

Abstract: Reinforcement learning (RL) in few-shot scenarios with limited sensor data is challenging due to insufficient training samples, particularly in applications like Dynamic Voltage and Frequency Scaling (DVFS) where sensor readings are semi-structured with inherent correlations. We propose Flow-Augmented Reinforcement Learning (FlowRL), a novel method that leverages continuous normalizing flows to ge… ▽ More Reinforcement learning (RL) in few-shot scenarios with limited sensor data is challenging due to insufficient training samples, particularly in applications like Dynamic Voltage and Frequency Scaling (DVFS) where sensor readings are semi-structured with inherent correlations. We propose Flow-Augmented Reinforcement Learning (FlowRL), a novel method that leverages continuous normalizing flows to generate high-quality synthetic data for few-shot RL. By integrating latent space bootstrapping for diversity and feature-weighted flow matching to preserve critical data correlations, FlowRL enhances sample efficiency and policy robustness. Evaluated on a DVFS case study using the NVIDIA Jetson TX2, our approach achieves up to 35\% higher frame rates and faster Q-value convergence compared to baselines, demonstrating its effectiveness in resource-constrained environments. FlowRL generalizes to other semi-structured domains, such as robotics and smart grids, offering a scalable solution for data-scarce RL settings. △ Less

Submitted 10 January, 2026; v1 submitted 21 September, 2024; originally announced September 2024.

Comments: 13 pages, 5 figures, 2 tables

ACM Class: I.2.6; I.2.8

An Evaluation of Sindhi Word Embedding in Semantic Analogies and Downstream Tasks

Authors: Wazir Ali, Saifullah Tumrani, Jay Kumar, Tariq Rahim Soomro

Abstract: In this paper, we propose a new word embedding based corpus consisting of more than 61 million words crawled from multiple web resources. We design a preprocessing pipeline for the filtration of unwanted text from crawled data. Afterwards, the cleaned vocabulary is fed to state-of-the-art continuous-bag-of-words, skip-gram, and GloVe word embedding algorithms. For the evaluation of pretrained embe… ▽ More In this paper, we propose a new word embedding based corpus consisting of more than 61 million words crawled from multiple web resources. We design a preprocessing pipeline for the filtration of unwanted text from crawled data. Afterwards, the cleaned vocabulary is fed to state-of-the-art continuous-bag-of-words, skip-gram, and GloVe word embedding algorithms. For the evaluation of pretrained embeddings, we use popular intrinsic and extrinsic evaluation approaches. The evaluation results reveal that continuous-bag-of-words and skip-gram perform better than GloVe and existing Sindhi fastText word embedding on both intrinsic and extrinsic evaluation approaches △ Less

Submitted 28 August, 2024; originally announced August 2024.

Comments: arXiv admin note: substantial text overlap with arXiv:1911.12579

Latent Diffusion for Guided Document Table Generation

Authors: Syed Jawwad Haider Hamdani, Saifullah Saifullah, Stefan Agne, Andreas Dengel, Sheraz Ahmed

Abstract: Obtaining annotated table structure data for complex tables is a challenging task due to the inherent diversity and complexity of real-world document layouts. The scarcity of publicly available datasets with comprehensive annotations for intricate table structures hinders the development and evaluation of models designed for such scenarios. This research paper introduces a novel approach for gener… ▽ More Obtaining annotated table structure data for complex tables is a challenging task due to the inherent diversity and complexity of real-world document layouts. The scarcity of publicly available datasets with comprehensive annotations for intricate table structures hinders the development and evaluation of models designed for such scenarios. This research paper introduces a novel approach for generating annotated images for table structure by leveraging conditioned mask images of rows and columns through the application of latent diffusion models. The proposed method aims to enhance the quality of synthetic data used for training object detection models. Specifically, the study employs a conditioning mechanism to guide the generation of complex document table images, ensuring a realistic representation of table layouts. To evaluate the effectiveness of the generated data, we employ the popular YOLOv5 object detection model for training. The generated table images serve as valuable training samples, enriching the dataset with diverse table structures. The model is subsequently tested on the challenging pubtables-1m testset, a benchmark for table structure recognition in complex document layouts. Experimental results demonstrate that the introduced approach significantly improves the quality of synthetic data for training, leading to YOLOv5 models with enhanced performance. The mean Average Precision (mAP) values obtained on the pubtables-1m testset showcase results closely aligned with state-of-the-art methods. Furthermore, low FID results obtained on the synthetic data further validate the efficacy of the proposed methodology in generating annotated images for table structure. △ Less

Submitted 19 August, 2024; originally announced August 2024.

Comments: Accepted in ICDAR 2024

StylusAI: Stylistic Adaptation for Robust German Handwritten Text Generation

Authors: Nauman Riaz, Saifullah Saifullah, Stefan Agne, Andreas Dengel, Sheraz Ahmed

Abstract: In this study, we introduce StylusAI, a novel architecture leveraging diffusion models in the domain of handwriting style generation. StylusAI is specifically designed to adapt and integrate the stylistic nuances of one language's handwriting into another, particularly focusing on blending English handwriting styles into the context of the German writing system. This approach enables the generatio… ▽ More In this study, we introduce StylusAI, a novel architecture leveraging diffusion models in the domain of handwriting style generation. StylusAI is specifically designed to adapt and integrate the stylistic nuances of one language's handwriting into another, particularly focusing on blending English handwriting styles into the context of the German writing system. This approach enables the generation of German text in English handwriting styles and German handwriting styles into English, enriching machine-generated handwriting diversity while ensuring that the generated text remains legible across both languages. To support the development and evaluation of StylusAI, we present the \lq{Deutscher Handschriften-Datensatz}\rq~(DHSD), a comprehensive dataset encompassing 37 distinct handwriting styles within the German language. This dataset provides a fundamental resource for training and benchmarking in the realm of handwritten text generation. Our results demonstrate that StylusAI not only introduces a new method for style adaptation in handwritten text generation but also surpasses existing models in generating handwriting samples that improve both text quality and stylistic fidelity, evidenced by its performance on the IAM database and our newly proposed DHSD. Thus, StylusAI represents a significant advancement in the field of handwriting style generation, offering promising avenues for future research and applications in cross-linguistic style adaptation for languages with similar scripts. △ Less

Submitted 22 July, 2024; originally announced July 2024.

Comments: Accepted in ICDAR 2024

DocXplain: A Novel Model-Agnostic Explainability Method for Document Image Classification

Authors: Saifullah Saifullah, Stefan Agne, Andreas Dengel, Sheraz Ahmed

Abstract: Deep learning (DL) has revolutionized the field of document image analysis, showcasing superhuman performance across a diverse set of tasks. However, the inherent black-box nature of deep learning models still presents a significant challenge to their safe and robust deployment in industry. Regrettably, while a plethora of research has been dedicated in recent years to the development of DL-powere… ▽ More Deep learning (DL) has revolutionized the field of document image analysis, showcasing superhuman performance across a diverse set of tasks. However, the inherent black-box nature of deep learning models still presents a significant challenge to their safe and robust deployment in industry. Regrettably, while a plethora of research has been dedicated in recent years to the development of DL-powered document analysis systems, research addressing their transparency aspects has been relatively scarce. In this paper, we aim to bridge this research gap by introducing DocXplain, a novel model-agnostic explainability method specifically designed for generating high interpretability feature attribution maps for the task of document image classification. In particular, our approach involves independently segmenting the foreground and background features of the documents into different document elements and then ablating these elements to assign feature importance. We extensively evaluate our proposed approach in the context of document image classification, utilizing 4 different evaluation metrics, 2 widely recognized document benchmark datasets, and 10 state-of-the-art document image classification models. By conducting a thorough quantitative and qualitative analysis against 9 existing state-of-the-art attribution methods, we demonstrate the superiority of our approach in terms of both faithfulness and interpretability. To the best of the authors' knowledge, this work presents the first model-agnostic attribution-based explainability method specifically tailored for document images. We anticipate that our work will significantly contribute to advancing research on transparency, fairness, and robustness of document image classification models. △ Less

Submitted 4 July, 2024; originally announced July 2024.

Comments: Accepted in ICDAR 2024

LiveBench: A Challenging, Contamination-Limited LLM Benchmark

Authors: Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh-Agrawal, Sandeep Singh Sandha, Siddartha Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, Micah Goldblum

Abstract: Test set contamination, wherein test data from a benchmark ends up in a newer model's training set, is a well-documented obstacle for fair LLM evaluation and can quickly render benchmarks obsolete. To mitigate this, many recent benchmarks crowdsource new prompts and evaluations from human or LLM judges; however, these can introduce significant biases, and break down when scoring hard questions. In… ▽ More Test set contamination, wherein test data from a benchmark ends up in a newer model's training set, is a well-documented obstacle for fair LLM evaluation and can quickly render benchmarks obsolete. To mitigate this, many recent benchmarks crowdsource new prompts and evaluations from human or LLM judges; however, these can introduce significant biases, and break down when scoring hard questions. In this work, we introduce a new benchmark for LLMs designed to be resistant to both test set contamination and the pitfalls of LLM judging and human crowdsourcing. We release LiveBench, the first benchmark that (1) contains frequently-updated questions from recent information sources, (2) scores answers automatically according to objective ground-truth values, and (3) contains a wide variety of challenging tasks, spanning math, coding, reasoning, language, instruction following, and data analysis. To achieve this, LiveBench contains questions that are based on recently-released math competitions, arXiv papers, news articles, and datasets, and it contains harder, contamination-limited versions of tasks from previous benchmarks such as Big-Bench Hard, AMPS, and IFEval. We evaluate many prominent closed-source models, as well as dozens of open-source models ranging from 0.5B to 405B in size. LiveBench is difficult, with top models achieving below 70% accuracy. We release all questions, code, and model answers. Questions are added and updated on a monthly basis, and we release new tasks and harder versions of tasks over time so that LiveBench can distinguish between the capabilities of LLMs as they improve in the future. We welcome community engagement and collaboration for expanding the benchmark tasks and models. △ Less

Submitted 18 April, 2025; v1 submitted 27 June, 2024; originally announced June 2024.

Comments: ICLR 2025 Spotlight

CinePile: A Long Video Question Answering Dataset and Benchmark

Authors: Ruchit Rawal, Khalid Saifullah, Miquel Farré, Ronen Basri, David Jacobs, Gowthami Somepalli, Tom Goldstein

Abstract: Current datasets for long-form video understanding often fall short of providing genuine long-form comprehension challenges, as many tasks derived from these datasets can be successfully tackled by analyzing just one or a few random frames from a video. To address this issue, we present a novel dataset and benchmark, CinePile, specifically designed for authentic long-form video understanding. This… ▽ More Current datasets for long-form video understanding often fall short of providing genuine long-form comprehension challenges, as many tasks derived from these datasets can be successfully tackled by analyzing just one or a few random frames from a video. To address this issue, we present a novel dataset and benchmark, CinePile, specifically designed for authentic long-form video understanding. This paper details our innovative approach for creating a question-answer dataset, utilizing advanced LLMs with human-in-the-loop and building upon human-generated raw data. Our comprehensive dataset comprises 305,000 multiple-choice questions (MCQs), covering various visual and multimodal aspects, including temporal comprehension, understanding human-object interactions, and reasoning about events or actions within a scene. Additionally, we fine-tuned open-source Video-LLMs on the training split and evaluated both open-source and proprietary video-centric LLMs on the test split of our dataset. The findings indicate that although current models underperform compared to humans, fine-tuning these models can lead to significant improvements in their performance. △ Less

Submitted 20 October, 2024; v1 submitted 14 May, 2024; originally announced May 2024.

Comments: Project page with all the artifacts - https://ruchitrawal.github.io/cinepile/. Updated version with adversarial refinement pipeline and more model evaluations

Comparative Analysis of Image Enhancement Techniques for Brain Tumor Segmentation: Contrast, Histogram, and Hybrid Approaches

Authors: Shoffan Saifullah, Andri Pranolo, Rafał Dreżewski

Abstract: This study systematically investigates the impact of image enhancement techniques on Convolutional Neural Network (CNN)-based Brain Tumor Segmentation, focusing on Histogram Equalization (HE), Contrast Limited Adaptive Histogram Equalization (CLAHE), and their hybrid variations. Employing the U-Net architecture on a dataset of 3064 Brain MRI images, the research delves into preprocessing steps, in… ▽ More This study systematically investigates the impact of image enhancement techniques on Convolutional Neural Network (CNN)-based Brain Tumor Segmentation, focusing on Histogram Equalization (HE), Contrast Limited Adaptive Histogram Equalization (CLAHE), and their hybrid variations. Employing the U-Net architecture on a dataset of 3064 Brain MRI images, the research delves into preprocessing steps, including resizing and enhancement, to optimize segmentation accuracy. A detailed analysis of the CNN-based U-Net architecture, training, and validation processes is provided. The comparative analysis, utilizing metrics such as Accuracy, Loss, MSE, IoU, and DSC, reveals that the hybrid approach CLAHE-HE consistently outperforms others. Results highlight its superior accuracy (0.9982, 0.9939, 0.9936 for training, testing, and validation, respectively) and robust segmentation overlap, with Jaccard values of 0.9862, 0.9847, and 0.9864, and Dice values of 0.993, 0.9923, and 0.9932 for the same phases, emphasizing its potential in neuro-oncological applications. The study concludes with a call for refinement in segmentation methodologies to further enhance diagnostic precision and treatment planning in neuro-oncology. △ Less

Submitted 8 April, 2024; originally announced April 2024.

Comments: 9 Pages, & Figures, 2 Tables, International Conference on Computer Science Electronics and Information (ICCSEI 2023)

ACM Class: I.4.3; I.4.6

Journal ref: E3S Web Conf. E3S Web Conf., Volume 501, 2024

Coercing LLMs to do and reveal (almost) anything

Authors: Jonas Geiping, Alex Stein, Manli Shu, Khalid Saifullah, Yuxin Wen, Tom Goldstein

Abstract: It has recently been shown that adversarial attacks on large language models (LLMs) can "jailbreak" the model into making harmful statements. In this work, we argue that the spectrum of adversarial attacks on LLMs is much larger than merely jailbreaking. We provide a broad overview of possible attack surfaces and attack goals. Based on a series of concrete examples, we discuss, categorize and syst… ▽ More It has recently been shown that adversarial attacks on large language models (LLMs) can "jailbreak" the model into making harmful statements. In this work, we argue that the spectrum of adversarial attacks on LLMs is much larger than merely jailbreaking. We provide a broad overview of possible attack surfaces and attack goals. Based on a series of concrete examples, we discuss, categorize and systematize attacks that coerce varied unintended behaviors, such as misdirection, model control, denial-of-service, or data extraction. We analyze these attacks in controlled experiments, and find that many of them stem from the practice of pre-training LLMs with coding capabilities, as well as the continued existence of strange "glitch" tokens in common LLM vocabularies that should be removed for security reasons. △ Less

Submitted 21 February, 2024; originally announced February 2024.

Comments: 32 pages. Implementation available at https://github.com/JonasGeiping/carving

ML-ASPA: A Contemplation of Machine Learning-based Acoustic Signal Processing Analysis for Sounds, & Strains Emerging Technology

Authors: Ratul Ali, Aktarul Islam, Md. Shohel Rana, Saila Nasrin, Sohel Afzal Shajol, A. H. M. Saifullah Sadi

Abstract: Acoustic data serves as a fundamental cornerstone in advancing scientific and engineering understanding across diverse disciplines, spanning biology, communications, and ocean and Earth science. This inquiry meticulously explores recent advancements and transformative potential within the domain of acoustics, specifically focusing on machine learning (ML) and deep learning. ML, comprising an exten… ▽ More Acoustic data serves as a fundamental cornerstone in advancing scientific and engineering understanding across diverse disciplines, spanning biology, communications, and ocean and Earth science. This inquiry meticulously explores recent advancements and transformative potential within the domain of acoustics, specifically focusing on machine learning (ML) and deep learning. ML, comprising an extensive array of statistical techniques, proves indispensable for autonomously discerning and leveraging patterns within data. In contrast to traditional acoustics and signal processing, ML adopts a data-driven approach, unveiling intricate relationships between features and desired labels or actions, as well as among features themselves, given ample training data. The application of ML to expansive sets of training data facilitates the discovery of models elucidating complex acoustic phenomena such as human speech and reverberation. The dynamic evolution of ML in acoustics yields compelling results and holds substantial promise for the future. The advent of electronic stethoscopes and analogous recording and data logging devices has expanded the application of acoustic signal processing concepts to the analysis of bowel sounds. This paper critically reviews existing literature on acoustic signal processing for bowel sound analysis, outlining fundamental approaches and applicable machine learning principles. It chronicles historical progress in signal processing techniques that have facilitated the extraction of valuable information from bowel sounds, emphasizing advancements in noise reduction, segmentation, signal enhancement, feature extraction, sound localization, and machine learning techniques... △ Less

Submitted 17 December, 2023; originally announced February 2024.

Comments: 7 pages, 5 figures, Article

MSC Class: 68Qxx; 68Uxx; 68Vxx; 68Wxx; 68Txx; 68-XX ACM Class: J.7; D.2; G.4

Multi-agent deep reinforcement learning with centralized training and decentralized execution for transportation infrastructure management

Authors: M. Saifullah, K. G. Papakonstantinou, A. Bhattacharya, S. M. Stoffels, C. P. Andriotis

Abstract: Life-cycle management of large-scale transportation systems requires determining a sequence of inspection and maintenance decisions to minimize long-term risks and costs while dealing with multiple uncertainties and constraints that lie in high-dimensional spaces. Traditional approaches have been widely applied but often suffer from limitations related to optimality, scalability, and the ability t… ▽ More Life-cycle management of large-scale transportation systems requires determining a sequence of inspection and maintenance decisions to minimize long-term risks and costs while dealing with multiple uncertainties and constraints that lie in high-dimensional spaces. Traditional approaches have been widely applied but often suffer from limitations related to optimality, scalability, and the ability to properly handle uncertainty. Moreover, many existing methods rely on unconstrained formulations that overlook critical operational constraints. We address these issues in this work by casting the optimization problem within the framework of constrained Partially Observable Markov Decision Processes (POMDPs), which provide a robust mathematical foundation for stochastic sequential decision-making under observation uncertainties, in the presence of risk and resource limitations. To tackle the high dimensionality of state and action spaces, we propose DDMAC-CTDE, a Deep Decentralized Multi-Agent Actor-Critic (DDMAC) reinforcement learning architecture with Centralized Training and Decentralized Execution (CTDE). To demonstrate the utility of the proposed framework, we also develop a new comprehensive benchmark environment representing an existing transportation network in Virginia, U.S., with heterogeneous pavement and bridge assets undergoing nonstationary degradation. This environment incorporates multiple practical constraints related to budget limits, performance guidelines, traffic delays, and risk considerations. On this benchmark, DDMAC-CTDE consistently outperforms standard transportation management baselines, producing better policies. Together, the proposed framework and benchmark provide (i) a scalable, constraint-aware methodology, and (ii) a realistic, rigorous testbed for comprehensive evaluation of Deep Reinforcement Learning (DRL) for transportation infrastructure management. △ Less

Submitted 24 February, 2026; v1 submitted 22 January, 2024; originally announced January 2024.

PrIeD-KIE: Towards Privacy Preserved Document Key Information Extraction

Authors: Saifullah Saifullah, Stefan Agne, Andreas Dengel, Sheraz Ahmed

Abstract: In this paper, we introduce strategies for developing private Key Information Extraction (KIE) systems by leveraging large pretrained document foundation models in conjunction with differential privacy (DP), federated learning (FL), and Differentially Private Federated Learning (DP-FL). Through extensive experimentation on six benchmark datasets (FUNSD, CORD, SROIE, WildReceipts, XFUND, and DOCILE… ▽ More In this paper, we introduce strategies for developing private Key Information Extraction (KIE) systems by leveraging large pretrained document foundation models in conjunction with differential privacy (DP), federated learning (FL), and Differentially Private Federated Learning (DP-FL). Through extensive experimentation on six benchmark datasets (FUNSD, CORD, SROIE, WildReceipts, XFUND, and DOCILE), we demonstrate that large document foundation models can be effectively fine-tuned for the KIE task under private settings to achieve adequate performance while maintaining strong privacy guarantees. Moreover, by thoroughly analyzing the impact of various training and model parameters on model performance, we propose simple yet effective guidelines for achieving an optimal privacy-utility trade-off for the KIE task under global DP. Finally, we introduce FeAm-DP, a novel DP-FL algorithm that enables efficiently upscaling global DP from a standalone context to a multi-client federated environment. We conduct a comprehensive evaluation of the algorithm across various client and privacy settings, and demonstrate its capability to achieve comparable performance and privacy guarantees to standalone DP, even when accommodating an increasing number of participating clients. Overall, our study offers valuable insights into the development of private KIE systems, and highlights the potential of document foundation models for privacy-preserved Document AI applications. To the best of authors' knowledge, this is the first work that explores privacy preserved document KIE using document foundation models. △ Less

Submitted 5 October, 2023; originally announced October 2023.

Nondestructive chicken egg fertility detection using CNN-transfer learning algorithms

Authors: Shoffan Saifullah, Rafal Drezewski, Anton Yudhana, Andri Pranolo, Wilis Kaswijanti, Andiko Putro Suryotomo, Seno Aji Putra, Alin Khaliduzzaman, Anton Satria Prabuwono, Nathalie Japkowicz

Abstract: This study explored the application of CNN-Transfer Learning for nondestructive chicken egg fertility detection for precision poultry hatchery practices. Four models, VGG16, ResNet50, InceptionNet, and MobileNet, were trained and evaluated on a dataset (200 single egg images) using augmented images (rotation, flip, scale, translation, and reflection). Although the training results demonstrated tha… ▽ More This study explored the application of CNN-Transfer Learning for nondestructive chicken egg fertility detection for precision poultry hatchery practices. Four models, VGG16, ResNet50, InceptionNet, and MobileNet, were trained and evaluated on a dataset (200 single egg images) using augmented images (rotation, flip, scale, translation, and reflection). Although the training results demonstrated that all models achieved high accuracy, indicating their ability to accurately learn and classify chicken eggs' fertility state, when evaluated on the testing set, variations in accuracy and performance were observed. InceptionNet exhibited the best overall performance, accurately classifying fertile and non-fertile eggs. It demonstrated excellent performance in both training and testing sets in all parameters of the evaluation metrics. In testing set, it achieved an accuracy of 0.98, a sensitivity of 1 for detecting fertile eggs, and a specificity of 0.96 for identifying non-fertile eggs. The higher performance is attributed to its unique architecture efficiently capturing features at different scales leading to improved accuracy and robustness. Further optimization and fine-tuning of the models might necessary to address the limitations in accurately detecting fertile and non-fertile eggs in case of other models. This study highlighted the potential of CNN-Transfer Learning for nondestructive fertility detection and emphasizes the need for further research to enhance the models' capabilities and ensure accurate classification. △ Less

Submitted 28 September, 2023; originally announced September 2023.

Comments: 18 pages, 9 figures, 1 table, journal article published

MSC Class: CS-Class: 68T07; 68T45; 68U10; ICT-Class: 94A08 ACM Class: I.2; I.4; I.5

Journal ref: Jurnal Ilmiah Teknik Elektro Komputer dan Informatika (JITEKI), Vol 9, No 3 (2023)

ReviewRanker: A Semi-Supervised Learning Based Approach for Code Review Quality Estimation

Authors: Saifullah Mahbub, Md. Easin Arafat, Chowdhury Rafeed Rahman, Zannatul Ferdows, Masum Hasan

Abstract: Code review is considered a key process in the software industry for minimizing bugs and improving code quality. Inspection of review process effectiveness and continuous improvement can boost development productivity. Such inspection is a time-consuming and human-bias-prone task. We propose a semi-supervised learning based system ReviewRanker which is aimed at assigning each code review a confide… ▽ More Code review is considered a key process in the software industry for minimizing bugs and improving code quality. Inspection of review process effectiveness and continuous improvement can boost development productivity. Such inspection is a time-consuming and human-bias-prone task. We propose a semi-supervised learning based system ReviewRanker which is aimed at assigning each code review a confidence score which is expected to resonate with the quality of the review. Our proposed method is trained based on simple and and well defined labels provided by developers. The labeling task requires little to no effort from the developers and has an indirect relation to the end goal (assignment of review confidence score). ReviewRanker is expected to improve industry-wide code review quality inspection through reducing human bias and effort required for such task. The system has the potential of minimizing the back-and-forth cycle existing in the development and review process. Usable code and dataset for this research can be found at: https://github.com/saifarnab/code_review △ Less

Submitted 8 July, 2023; originally announced July 2023.

Seeing in Words: Learning to Classify through Language Bottlenecks

Authors: Khalid Saifullah, Yuxin Wen, Jonas Geiping, Micah Goldblum, Tom Goldstein

Abstract: Neural networks for computer vision extract uninterpretable features despite achieving high accuracy on benchmarks. In contrast, humans can explain their predictions using succinct and intuitive descriptions. To incorporate explainability into neural networks, we train a vision model whose feature representations are text. We show that such a model can effectively classify ImageNet images, and we… ▽ More Neural networks for computer vision extract uninterpretable features despite achieving high accuracy on benchmarks. In contrast, humans can explain their predictions using succinct and intuitive descriptions. To incorporate explainability into neural networks, we train a vision model whose feature representations are text. We show that such a model can effectively classify ImageNet images, and we discuss the challenges we encountered when training it. △ Less

Submitted 28 June, 2023; originally announced July 2023.

Comments: 5 pages, 2 figures, Published as a Tiny Paper at ICLR 2023

Bring Your Own Data! Self-Supervised Evaluation for Large Language Models

Authors: Neel Jain, Khalid Saifullah, Yuxin Wen, John Kirchenbauer, Manli Shu, Aniruddha Saha, Micah Goldblum, Jonas Geiping, Tom Goldstein

Abstract: With the rise of Large Language Models (LLMs) and their ubiquitous deployment in diverse domains, measuring language model behavior on realistic data is imperative. For example, a company deploying a client-facing chatbot must ensure that the model will not respond to client requests with profanity. Current evaluations approach this problem using small, domain-specific datasets with human-curated… ▽ More With the rise of Large Language Models (LLMs) and their ubiquitous deployment in diverse domains, measuring language model behavior on realistic data is imperative. For example, a company deploying a client-facing chatbot must ensure that the model will not respond to client requests with profanity. Current evaluations approach this problem using small, domain-specific datasets with human-curated labels. These evaluation sets are often sampled from a narrow and simplified distribution, and data sources can unknowingly be leaked into the training set which can lead to misleading evaluations. To bypass these drawbacks, we propose a framework for self-supervised evaluation of LLMs by analyzing their sensitivity or invariance to transformations on the input text. Self-supervised evaluation can directly monitor LLM behavior on datasets collected in the wild or streamed during live model deployment. We demonstrate self-supervised evaluation strategies for measuring closed-book knowledge, toxicity, and long-range context dependence, in addition to sensitivity to grammatical structure and tokenization errors. When comparisons to similar human-labeled benchmarks are available, we find strong correlations between self-supervised and human-supervised evaluations. The self-supervised paradigm complements current evaluation strategies that rely on labeled data. △ Less

Submitted 29 June, 2023; v1 submitted 23 June, 2023; originally announced June 2023.

Comments: Code is available at https://github.com/neelsjain/BYOD. First two authors contributed equally. 21 pages, 22 figures

On the Reliability of Watermarks for Large Language Models

Authors: John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, Tom Goldstein

Abstract: As LLMs become commonplace, machine-generated text has the potential to flood the internet with spam, social media bots, and valueless content. Watermarking is a simple and effective strategy for mitigating such harms by enabling the detection and documentation of LLM-generated text. Yet a crucial question remains: How reliable is watermarking in realistic settings in the wild? There, watermarked… ▽ More As LLMs become commonplace, machine-generated text has the potential to flood the internet with spam, social media bots, and valueless content. Watermarking is a simple and effective strategy for mitigating such harms by enabling the detection and documentation of LLM-generated text. Yet a crucial question remains: How reliable is watermarking in realistic settings in the wild? There, watermarked text may be modified to suit a user's needs, or entirely rewritten to avoid detection. We study the robustness of watermarked text after it is re-written by humans, paraphrased by a non-watermarked LLM, or mixed into a longer hand-written document. We find that watermarks remain detectable even after human and machine paraphrasing. While these attacks dilute the strength of the watermark, paraphrases are statistically likely to leak n-grams or even longer fragments of the original text, resulting in high-confidence detections when enough tokens are observed. For example, after strong human paraphrasing the watermark is detectable after observing 800 tokens on average, when setting a 1e-5 false positive rate. We also consider a range of new detection schemes that are sensitive to short spans of watermarked text embedded inside a large document, and we compare the robustness of watermarking to other kinds of detectors. △ Less

Submitted 1 May, 2024; v1 submitted 7 June, 2023; originally announced June 2023.

Comments: 9 pages in the main body. Published at ICLR 2024. Code is available at https://github.com/jwkirchenbauer/lm-watermarking

Learning UI-to-Code Reverse Generator Using Visual Critic Without Rendering

Authors: Davit Soselia, Khalid Saifullah, Tianyi Zhou

Abstract: Automated reverse engineering of HTML/CSS code from UI screenshots is an important yet challenging problem with broad applications in website development and design. In this paper, we propose a novel vision-code transformer (ViCT) composed of a vision encoder processing the screenshots and a language decoder to generate the code. They are initialized by pre-trained models such as ViT/DiT and GPT-2… ▽ More Automated reverse engineering of HTML/CSS code from UI screenshots is an important yet challenging problem with broad applications in website development and design. In this paper, we propose a novel vision-code transformer (ViCT) composed of a vision encoder processing the screenshots and a language decoder to generate the code. They are initialized by pre-trained models such as ViT/DiT and GPT-2/LLaMA but aligning the two modalities requires end-to-end finetuning, which aims to minimize the visual discrepancy between the code-rendered webpage and the original screenshot. However, the rendering is non-differentiable and causes costly overhead. We address this problem by actor-critic fine-tuning where a visual critic without rendering (ViCR) is developed to predict visual discrepancy given the original and generated code. To train and evaluate our models, we created two synthetic datasets of varying complexity, with over 75,000 unique (code, screenshot) pairs. We evaluate the UI-to-Code performance using a combination of automated metrics such as MSE, BLEU, IoU, and a novel htmlBLEU score. ViCT outperforms a strong baseline model DiT-GPT2, improving IoU from 0.64 to 0.79 and lowering MSE from 12.25 to 9.02. With much lower computational cost, it can achieve comparable performance as when using a larger decoder such as LLaMA. △ Less

Submitted 3 November, 2023; v1 submitted 23 May, 2023; originally announced May 2023.

OSRE: Object-to-Spot Rotation Estimation for Bike Parking Assessment

Authors: Saghir Alfasly, Zaid Al-huda, Saifullah Bello, Ahmed Elazab, Jian Lu, Chen Xu

Abstract: Current deep models provide remarkable object detection in terms of object classification and localization. However, estimating object rotation with respect to other visual objects in the visual context of an input image still lacks deep studies due to the unavailability of object datasets with rotation annotations. This paper tackles these two challenges to solve the rotation estimation of a pa… ▽ More Current deep models provide remarkable object detection in terms of object classification and localization. However, estimating object rotation with respect to other visual objects in the visual context of an input image still lacks deep studies due to the unavailability of object datasets with rotation annotations. This paper tackles these two challenges to solve the rotation estimation of a parked bike with respect to its parking area. First, we leverage the power of 3D graphics to build a camera-agnostic well-annotated Synthetic Bike Rotation Dataset (SynthBRSet). Then, we propose an object-to-spot rotation estimator (OSRE) by extending the object detection task to further regress the bike rotations in two axes. Since our model is purely trained on synthetic data, we adopt image smoothing techniques when deploying it on real-world images. The proposed OSRE is evaluated on synthetic and real-world data providing promising results. Our data and code are available at \href{https://github.com/saghiralfasly/OSRE-Project}{https://github.com/saghiralfasly/OSRE-Project}. △ Less

Submitted 1 March, 2023; originally announced March 2023.

Privacy Meets Explainability: A Comprehensive Impact Benchmark

Authors: Saifullah Saifullah, Dominique Mercier, Adriano Lucieri, Andreas Dengel, Sheraz Ahmed

Abstract: Since the mid-10s, the era of Deep Learning (DL) has continued to this day, bringing forth new superlatives and innovations each year. Nevertheless, the speed with which these innovations translate into real applications lags behind this fast pace. Safety-critical applications, in particular, underlie strict regulatory and ethical requirements which need to be taken care of and are still active ar… ▽ More Since the mid-10s, the era of Deep Learning (DL) has continued to this day, bringing forth new superlatives and innovations each year. Nevertheless, the speed with which these innovations translate into real applications lags behind this fast pace. Safety-critical applications, in particular, underlie strict regulatory and ethical requirements which need to be taken care of and are still active areas of debate. eXplainable AI (XAI) and privacy-preserving machine learning (PPML) are both crucial research fields, aiming at mitigating some of the drawbacks of prevailing data-hungry black-box models in DL. Despite brisk research activity in the respective fields, no attention has yet been paid to their interaction. This work is the first to investigate the impact of private learning techniques on generated explanations for DL-based models. In an extensive experimental analysis covering various image and time series datasets from multiple domains, as well as varying privacy techniques, XAI methods, and model architectures, the effects of private training on generated explanations are studied. The findings suggest non-negligible changes in explanations through the introduction of privacy. Apart from reporting individual effects of PPML on XAI, the paper gives clear recommendations for the choice of techniques in real applications. By unveiling the interdependencies of these pivotal technologies, this work is a first step towards overcoming the remaining hurdles for practically applicable AI in safety-critical domains. △ Less

Submitted 8 November, 2022; originally announced November 2022.

Comments: Under Submission

Journal ref: Frontiers in Artificial Intelligence, 2024 (7), 1236947

Tourism's trend Ranking on Social Media Data Using Fuzzy-AHP vs. AHP

Authors: Shoffan Saifullah

Abstract: Tourism is an exciting thing to be visited by people in the world. Search for attractive and popular places can be done through social media. Data from social media or websites can be used as a reference to find current travel trends and get information about reviews, stories, likes, forums, blogs, and feedback from a place. However, if the search is done manually one by one, it takes a long time,… ▽ More Tourism is an exciting thing to be visited by people in the world. Search for attractive and popular places can be done through social media. Data from social media or websites can be used as a reference to find current travel trends and get information about reviews, stories, likes, forums, blogs, and feedback from a place. However, if the search is done manually one by one, it takes a long time, and it becomes interesting to do research. So, searching based on current trends will be easier and faster. For this reason, this study uses a computer base to search by ranking tourist facilities from social media data or websites using the multi-criteria decision-making (MCDM) method. The implementation of the method used in finding the trend is the Fuzzy-AHP method in comparison with the AHP. The data used is data reviews, stories, likes, forums, blogs, and feedback from the web or social media. Because with these components, tourism can be developed according to visitors\'' wishes. The research aims to rank facilities\'' tourism attractions (trends) and development priorities. The priority and ranking used the fuzzy-AHP and AHP method to determine weight criteria and the ranking process. The highest ranking is on the Parks/Picnic Spots attraction, and make it a priority to develop. The methods have an average value MSE of all data is \approx 0.0002, which can be used for this ranking. △ Less

Submitted 28 September, 2022; originally announced September 2022.

Comments: 7 pages, 5 Tables, 3 figures

Journal ref: JIKO (Jurnal Informatika dan Komputer), 6(2), 2022, 153-159

Transparent and Tamper-Proof Event Ordering in the Internet of Things Platforms

Authors: Mahbubur Rahman, Abusayeed Saifullah

Abstract: Today, the audit and diagnosis of the causal relationships between the events in a trigger-action-based event chain (e.g., why is a light turned on in a smart home?) in the Internet of Things (IoT) platforms are untrustworthy and unreliable. The current IoT platforms lack techniques for transparent and tamper-proof ordering of events due to their device-centric logging mechanism. In this paper, we… ▽ More Today, the audit and diagnosis of the causal relationships between the events in a trigger-action-based event chain (e.g., why is a light turned on in a smart home?) in the Internet of Things (IoT) platforms are untrustworthy and unreliable. The current IoT platforms lack techniques for transparent and tamper-proof ordering of events due to their device-centric logging mechanism. In this paper, we develop a framework that facilitates tamper-proof transparency and event order in an IoT platform by proposing a Blockchain protocol and adopting the vector clock system, both tailored for the resource-constrained heterogeneous IoT devices, respectively. To cope with the unsuited storage (e.g., ledger) and computing power (e.g., proof of work puzzle) requirements of the Blockchain in the commercial off-the-shelf IoT devices, we propose a partial consistent cut protocol and engineer a modular arithmetic-based lightweight proof of work puzzle, respectively. To the best of our knowledge, this is the first Blockchain designed for resource-constrained heterogeneous IoT platforms. Our event ordering protocol based on the vector clock system is also novel for the IoT platforms. We implement our framework using an IoT gateway and 30 IoT devices. We experiment with 10 concurrent trigger-action-based event chains while each chain involves 20 devices, and each device participates in 5 different chains. The results show that our framework may order these events in 2.5 seconds while consuming only 140 mJ of energy per device. The results hence demonstrate the proposed platform as a practical choice for many IoT applications such as smart home, traffic monitoring, and crime investigation. △ Less

Submitted 11 August, 2022; originally announced August 2022.

Comments: 12 pages, 13 eps figures

ACM Class: C.2.0; C.2.1; C.2.2; C.2.3; C.2.4

Identification of chicken egg fertility using SVM classifier based on first-order statistical feature extraction

Authors: Shoffan Saifullah, Andiko Putro Suryotomo

Abstract: This study aims to identify chicken eggs fertility using the support vector machine (SVM) classifier method. The classification basis used the first-order statistical (FOS) parameters as feature extraction in the identification process. This research was developed based on the process's identification process, which is still manual (conventional). Although currently there are many technologies in… ▽ More This study aims to identify chicken eggs fertility using the support vector machine (SVM) classifier method. The classification basis used the first-order statistical (FOS) parameters as feature extraction in the identification process. This research was developed based on the process's identification process, which is still manual (conventional). Although currently there are many technologies in the identification process, they still need development. Thus, this research is one of the developments in the field of image processing technology. The sample data uses datasets from previous studies with a total of 100 egg images. The egg object in the image is a single object. From these data, the classification of each fertile and infertile egg is 50 image data. Chicken egg image data became input in image processing, with the initial process is segmentation. This initial segmentation aims to get the cropped image according to the object. The cropped image is repaired using image preprocessing with grayscaling and image enhancement methods. This method (image enhancement) used two combination methods: contrast limited adaptive histogram equalization (CLAHE) and histogram equalization (HE). The improved image becomes the input for feature extraction using the FOS method. The FOS uses five parameters, namely mean, entropy, variance, skewness, and kurtosis. The five parameters entered into the SVM classifier method to identify the fertility of chicken eggs. The results of these experiments, the method proposed in the identification process has a success percentage of 84.57%. Thus, the implementation of this method can be used as a reference for future research improvements. In addition, it may be possible to use a second-order feature extraction method to improve its accuracy and improve supervised learning for classification. △ Less

Submitted 9 January, 2022; originally announced January 2022.

Comments: 9 Pages, 5 Figures, 2 Tables

MSC Class: 94A08 ACM Class: I.5.1; I.4.m

Journal ref: ILKOM Jurnal Ilmiah, 13(3), (2021), 285-293

K-means segmentation based-on lab color space for embryo detection in incubated egg

Authors: Shoffan Saifullah, Rafal Drezewski, Alin Khaliduzzaman, Lean Karlo Tolentino, Rabbimov Ilyos

Abstract: The quality of the hatching process influences the success of the hatch rate besides the inherent egg factors. Eliminating infertile or dead eggs and monitoring embryonic growth are very important factors in efficient hatchery practices. This process aims to sort eggs that only have embryos to remain in the incubator until the end of the hatching process. This process aims to sort eggs with embryo… ▽ More The quality of the hatching process influences the success of the hatch rate besides the inherent egg factors. Eliminating infertile or dead eggs and monitoring embryonic growth are very important factors in efficient hatchery practices. This process aims to sort eggs that only have embryos to remain in the incubator until the end of the hatching process. This process aims to sort eggs with embryos to remain hatched until the end. Maximum checking is done the first week in the hatching period. This study aims to detect the presence of embryos in eggs and processed by segmentation. Egg images are segmented using the K-means algorithm based on Lab color images. The results of the image acquisition are converted into Lab color space images. The results of Lab color space images are processed using K-means for each color. The K-means process uses cluster k=3 and divides into three parts: background, eggs, and yolk. Egg yolks are part of eggs that have embryonic characteristics. This study applies the concept of color in the initial segmentation and grayscale in the final stages. The initial phase results show that the image segmentation results using k-means clustering based on Lab color space provide a grouping of three parts. At the grayscale image processing stage, the results of color image segmentation are processed with grayscaling, image enhancement, and morphology. Thus, it seems clear that the yolk segmented shows the presence of egg embryos. Based on this results, the initial stages of the embryo detection process used K-means segmentation based on Lab color space. The evaluation uses MSE and MSSIM, with values of 0.0486 and 0.9979; this can be used to reference that the results obtained can detect embryos in egg yolk. This protocol could be used in a non-destructive quantitative study on embryos and their morphology in a precision poultry production system in the future. △ Less

Submitted 1 August, 2022; v1 submitted 3 March, 2021; originally announced March 2021.

Comments: 11 pages, 6 figures, ICoSiET Conference 2020, Jurnal Ilmiah Teknik Elektro Komputer dan Informatika (JITEKI)

Journal ref: J. Ilm. Tek. Elektro Komput. dan Inform., 2022, Vol. 7, No. 2, p. 175-185

Fuzzy-AHP approach using Normalized Decision Matrix on Tourism Trend Ranking based-on Social Media

Authors: Shoffan Saifullah

Abstract: This research discusses multi-criteria decision making (MCDM) using Fuzzy-AHP methods of tourism. The fuzzy-AHP process will rank tourism trends based on data from social media. Social media is one of the channels with the largest source of data input in determining tourism development. The development uses social media interactions based on the facilities visited, including reviews, stories, like… ▽ More This research discusses multi-criteria decision making (MCDM) using Fuzzy-AHP methods of tourism. The fuzzy-AHP process will rank tourism trends based on data from social media. Social media is one of the channels with the largest source of data input in determining tourism development. The development uses social media interactions based on the facilities visited, including reviews, stories, likes, forums, blogs, and feedback. This experiment aims to prioritize facilities that are the trend of tourism. The priority ranking uses weight criteria and the ranking process. The highest rank is in the attractions of the Park/Picnic Area, with the final weight calculation value of 0.6361. Fuzzy-AHP can rank optimally with an MSE value of \approx 0.0002. △ Less

Submitted 8 February, 2021; originally announced February 2021.

Comments: 8 pages, 4 figures

Journal ref: Jurnal Informatika, 13(2), (2019), 16-23

Segmentasi Citra Menggunakan Metode Watershed Transform Berdasarkan Image Enhancement Dalam Mendeteksi Embrio Telur

Authors: Shoffan Saifullah

Abstract: Image processing can be applied in the detection of egg embryos. The egg embryos detection is processed using a segmentation process. The segmentation divides the image according to the area that is divided. This process requires improvement of the image that is processed to obtain optimal results. This study will analyze the detection of egg embryos based on image processing with image enhancemen… ▽ More Image processing can be applied in the detection of egg embryos. The egg embryos detection is processed using a segmentation process. The segmentation divides the image according to the area that is divided. This process requires improvement of the image that is processed to obtain optimal results. This study will analyze the detection of egg embryos based on image processing with image enhancement and the concept of segmentation using the watershed method. Image enhancement in preprocessing in image improvement uses a combination of Contrast Limited Adaptive Histogram Equalization (CLAHE) and Histogram Equalization (HE) methods. The grayscale egg image is corrected using the CLAHE method, and the results are reprocessed using HE. The image improvement results show that the CLAHE-HE combination method gives a clear picture of the object area of the egg image that has an embryo. The segmentation process using image conversion to black and white image and watershed segmentation can clearly show the object of a chicken egg that has an embryo. The results of segmentation can divide the area of the egg having embryos in a real and accurate way with a percentage \approx 98\%. △ Less

Submitted 8 February, 2021; originally announced February 2021.

Comments: 8 pages, in Indonesian language, 6 figures

ACM Class: I.4.6

Journal ref: Systemic: Information System and Informatics Journal, 5(2), (2019), 53-60

LPWAN in the TV White Spaces: A Practical Implementation and Deployment Experiences

Authors: Mahbubur Rahman, Dali Ismail, Venkata P Modekurthy, Abusayeed Saifullah

Abstract: Low-Power Wide-Area Network (LPWAN) is an enabling Internet-of-Things (IoT) technology that supports long-range, low-power, and low-cost connectivity to numerous devices. To avoid the crowd in the limited ISM band (where most LPWANs operate) and cost of licensed band, the recently proposed SNOW (Sensor Network over White Spaces) is a promising LPWAN platform that operates over the TV white spaces.… ▽ More Low-Power Wide-Area Network (LPWAN) is an enabling Internet-of-Things (IoT) technology that supports long-range, low-power, and low-cost connectivity to numerous devices. To avoid the crowd in the limited ISM band (where most LPWANs operate) and cost of licensed band, the recently proposed SNOW (Sensor Network over White Spaces) is a promising LPWAN platform that operates over the TV white spaces. As it is a very recent technology and is still in its infancy, the current SNOW implementation uses the USRP devices as LPWAN nodes, which has high costs (~$750 USD per device) and large form-factors, hindering its applicability in practical deployment. In this paper, we implement SNOW using low-cost, low form-factor, low-power, and widely available commercial off-the-shelf (COTS) devices to enable its practical and large-scale deployment. Our choice of the COTS device (TI CC13x0: CC1310 or CC1350) consequently brings down the cost and form-factor of a SNOW node by 25x and 10x, respectively. Such implementation of SNOW on the CC13x0 devices, however, faces a number of challenges to enable link reliability and communication range. Our implementation addresses these challenges by handling peak-to-average power ratio problem, channel state information estimation, carrier frequency offset estimation, and near-far power problem. Our deployment in the city of Detroit, Michigan demonstrates that CC13x0-based SNOW can achieve uplink and downlink throughputs of 11.2kbps and 4.8kbps per node, respectively, over a distance of 1km. Also, the overall throughput in the uplink increases linearly with the increase in the number of SNOW nodes. △ Less

Submitted 30 January, 2021; originally announced February 2021.

Comments: ACM Transactions on Embedded Computing Systems (Accepted), 2021