Hierarchical Self-Supervised Representation Learning Framework for Multivariate Time Series Grounded in ECG Analysis

Introduced by Vaswani et al. [32], self-attention is a token-wise operation that computes mutual correlations among tokens within an input sequence to generate context-aware outputs. Although initially designed for natural language processing, attention mechanisms are now ubiquitous across various domains, including computer vision. The Vision Transformer (ViT) [8] adapts this architecture for image data by processing non-overlapping image patches as an input sequence. To retain spatial information, it incorporates two-dimensional positional embeddings, which are vectors added to the tokens to encode their relative positions. Because the attention operation inherently requires discrete tokens, continuous input data must be discretized prior to processing.

While self-attention can be integrated into broader, more complex structures, its core mechanism relies on the following fundamental operation. Given $N$ input tokens $\{X_{i}\mid 1\leq i\leq N\}$ with an embedding dimension of $d_{\text{emb}}$, we define the query ($Q$), key ($K$), and value ($V$) matrices in $\mathbb{R}^{N\times d_{\text{emb}}}$ as learnable linear transformations of the input tokens. Written in matrix form, the output is:

where each output row can be interpreted as a linear combination of value vectors weighted by their mutual correlations:

Here, $q_{i},k_{i},v_{i}$ are the rows of the $Q,K,V$ matrices, respectively, and the attention weight $\alpha_{i,j}$ is derived from the scaled dot-product such that $\alpha_{i,j}\propto\exp(\langle q_{i},k_{j}\rangle/\sqrt{d_{\text{emb}}})$.

Energy-Based Models (EBMs) [18] provide an SSL framework that captures dependencies between variables by learning to assign lower energy to compatible configurations. The Joint-Embedding Predictive Architecture (JEPA) [19] extends this concept using two encoders to generate representations, followed by a predictor network joining two representations. Given a pair of intrinsically linked inputs, JEPA learns their representations by predicting the output embedding of one encoder from the output embedding of the other encoder, guided by a latent variable (see Figure 1(a)).

The Image-based Joint-Embedding Predictive Architecture (I-JEPA) [1] adapts this framework for image data utilizing a context and target encoder pair. It learns the representation of image patches by predicting the embeddings of target patches sampled in a block from a sparse collection of context patches. Unlike generative architectures such as MAE [12], which reconstruct targets in the raw pixel space, I-JEPA predicts targets exclusively in the abstract representation space. This design choice is widely cited as a key factor in its enhanced semantic performance. Consequently, I-JEPA has been successfully adapted for various modalities beyond vision, serving as a core building block for the proposed architecture.

Each encoder in a JEPA is constrained to designated input specifications, such as a fixed scope and resolution. For instance, two identical encoders can have distinct input resolutions (e.g., varying sampling frequencies) or scopes (e.g., different patch sizes) depending on an associated tokenization. Furthermore, for multimodal inputs such as an image-text pair, separate encoders can be assigned to process each distinct modality.

To capture complex data structures, representation learning literature frequently emphasizes the importance of hierarchical processing. In this context, a Hierarchical JEPA (H-JEPA) [19] comprises multiple interconnected JEPAs, where the hierarchy is induced by the varying scopes and resolutions of the sub-encoders. For instance, a JEPA processing fine representations corresponding to short time intervals can be chained to another JEPA processing coarser representations corresponding to longer time intervals (see Figure 1(b)). This aligns with the theoretical premise that complex tasks require multiple levels of abstraction, ranging from fine-grained details to coarse semantic concepts, to effectively interpret the input data.

Motivated by the diagnostic approach of cardiologists, ER-JEPA is divided into two modules: (1) the event reconstruction module, and (2) the event analysis module. A 12-lead ECG records the electrical activity of the heart projected across 12 spatial axes [30]. While these projections yield diverse signals, crucially, every projection originates from the same underlying cardiac event. Therefore, the diagnostic essence does not lie in the impulses of a specific channel, but rather in the holistic inference of the heart’s activity derived from all concurrent projections. This observation suggests that, instead of analyzing patches from every channel, analyzing a unified representation of the reconstructed cardiac event is a highly efficient strategy in a situation where multivariate processing can be cumbersome. Throughout this section, we refer to Figure 3 as a visual guide to facilitate the understanding of this two-stage token processing.

Because the channel JEPA and temporal JEPA are both derived from I-JEPA, their core mechanisms are identical: modules with a ViT backbone; a context encoder, a target encoder, and a predictor for learning joint-embeddings; and updates of the target encoder via an exponential moving average. For detailed information on I-JEPA, we refer to [1].

The paradigm of two-stage training, which first learns the underlying structure of the data using a large unlabeled dataset, is the essence of SSL. Following the methodology of I-JEPA [1], ER-JEPA learns by predicting the encoding of target patches from context patches during the pretraining process. Learning robust semantic representations from the inherent structure of the data fundamentally requires a large-scale dataset.

Because the learning process of JEPA operates in the representation space, the architecture is prone to representation collapse, a phenomenon where the model trivially solves the prediction task by collapsing all encodings into a constant embedding. This can be largely mitigated by design choices, such as utilizing asymmetric encoders [2, 5, 10], as seen in many Joint-Embedding Architecture (JEA) models and I-JEPA; however, an inherent susceptibility remains prevalent across joint-embedding architectures.

Representation collapse is an especially serious issue for ER-JEPA, since the input to the temporal JEPA is not a token derived directly from the raw data, but rather a representation constructed by the preceding channel JEPA, which is itself inherently prone to representation collapse. Indeed, as depicted in Figure 5(a), the loss of each JEPA drastically drops in the early stages, then recovers within a few epochs, a recovery pattern typical for models utilizing the JEPA framework.

To further investigate this behavior, the evolution of the temporal loss can be compared against baseline models that lack a concatenated JEPA structure, as shown in Figure 5(b). Under an identical pretraining environment, both an ablation model omitting the channel JEPA (detailed below) and a standard 1D I-JEPA adaptation for ECG (see Section 5.3.1) exhibit a significantly shallower initial drop in loss during the early epochs. This distinction highlights that the compound hierarchical stacking in the complete ER-JEPA model amplifies early-stage vulnerability prior to its recovery.

While the complete model ultimately stabilizes, an unrecovered loss during pretraining serves as typical evidence of representation collapse. Building on this observation, the following paragraphs detail specific phenomena regarding the sensitive nature of hierarchical representation learning within ER-JEPA.

The evaluation of representation learning assesses the encoder’s performance on specific tasks, both with and without further altering the model’s weights. To evaluate the static embeddings produced by the pretrained encoder, linear probing measures the performance of a basic linear classifier using the frozen encoder’s output as features. To evaluate the full capacity of the model, fine-tuning trains the entire model end-to-end after appending a classification layer.

Following I-JEPA [1], the target encoder of the JEPA framework, which is updated via an exponential moving average, is the standard choice for the representing encoder in downstream applications. For ER-JEPA, the concatenation of the channel and temporal JEPA encoders, joined by an average pooling layer, serves as the primary encoder for these tasks. For the classification of an entire $10$-second $12$-lead ECG, an additional average pooling layer is appended to summarize the sequence of interval representations into a single global representation (see Figure 2).

Most ViT-based SSL models in the benchmark, such as the encoders proposed by Weimann and Conrad [34], ST-MEM [25], and ECG-JEPA, share common hyperparameter standards. Following the configuration of the original ViT [8] or the foundational Transformer [32], an embedding dimension equal to a multiple of $768$ and an encoder depth of $12$ are prevalent baseline choices. ER-JEPA also follows this general convention, utilizing an embedding dimension of $d_{\text{emb}}=$384$$, but splits the network depth, equally distributing six attention layers to the channel encoder and six to the temporal encoder.

While sections 5.3.2 and 5.3.3 prove that this configuration, $d_{\text{emb}}=$384$$ with depths $(6,6)$, is highly effective, the ER-JEPA encoder also gains a significant advantage with respect to resource usage, as half of its attention layers (the temporal encoder) process purely temporal tokens without requiring expensive multichannel operations. The multichannel processing is dedicated entirely to the channel encoder, which itself achieves efficiency through a strictly focused scope. Compared to the temporal encoder, the channel encoder processes a much smaller sequence length of $N=N_{\text{ch}}$, at the expense of increasing the effective batch size by a factor of $N_{\text{t}}$. However, in the current implementation ($N_{\text{ch}}=8$ and $N_{\text{t}}=50$), this increase in effective batch size dominates the computational cost over the reduced sequence length, despite the $\mathcal{O}(N^{2})$ reduction in attention complexity. Furthermore, computational overhead caused by tensor handling in the PyTorch implementation widens the execution time gap between the temporal and channel encoders, regardless of the configuration.

Figure 7 presents an efficiency comparison among the ViT encoders of various SSL models. The ER-JEPA encoder is compared in two settings: (a) under the native settings used in the downstream classification benchmark, and (b) under a unified embedding dimension of $d_{\text{emb}}=768$. The compared models include a 1D ViT-S; a 2D ViT-B, representing the encoder of ST-MEM; and the Cross-Pattern ViT from ECG-JEPA [17]. Furthermore, ViT-1D-S-F, a closer surrogate for the encoder in Weimann and Conrad, is included in the native setting benchmark to account for their specific data resolution ($500\text{\,}\mathrm{H}\mathrm{z}$ data with a smaller patch size of $p=25$).

With the hyperparameters used in the classification benchmark, the ER-JEPA encoder is the fastest among the architectures that explicitly process multichannel tokens, achieving speeds relatively close to the baseline ViT-1D encoder (which entirely bypasses multichannel attention). The ER-JEPA encoder achieves up to a $7.8$$\times$ speedup under benchmark conditions, and maintains a maximum $2.6$$\times$ speedup even under the unified embedding dimension of $d_{\text{emb}}=768$.

The memory allocation exhibits a trend similar to the speed efficiency. Compared to the other multichannel ViT encoders in the benchmark, ER-JEPA requires less than one-third of the memory for small batch sizes, and less than one-fourth for larger batch sizes. When all models are scaled to an embedding dimension of $d_{\text{emb}}=768$, memory usage is similar for a unit batch size, but the baseline multichannel encoders scale to consume between $1.5$ and $2$ times as much memory as ER-JEPA for larger batch sizes.

While the basic 1D ViT achieves the lightest computational footprint by delegating multichannel analysis entirely to the tokenization step, the finer-resolution surrogate (ViT-1D-S-F) demands significantly heavier processing. As a result, the efficiency gap between ViT-1D-S-F and the ER-JEPA encoder narrows, with ER-JEPA requiring only a $25\text{\,}\mathrm{\char 37\relax}$ increase in operation time and $42\text{\,}\mathrm{\char 37\relax}$ more memory despite multichannel processing.

In this section, we qualitatively assess the characteristics of the representations produced by the channel encoder. Because the representation from the channel encoder corresponds to a specific time interval, as with the output tokens of the temporal encoder, both representations can be utilized in downstream applications involving time-interval analysis. To this end, we compare the efficacy of each representation in an ECG segmentation task. We restrict this analysis strictly to a qualitative proof-of-concept, reserving comprehensive quantitative evaluation for future studies.

In contrast to the linear evaluation, the performance across ablation studies on the fine-tuning downstream task did not exhibit significant deviation, although the default hyperparameters remained optimal by a minimal margin. Table 9 presents the average and standard deviation of the benchmark results grouped by broader categories. While most of the categories showed heavily saturated performance, the encoders trained without the channel JEPA noticeably and consistently deviated from the majority.