Non-Autoregressive Typed Decisions: How Does Laya Compare to BERT for On-Device Classification?

Text classification with a fixed set of labels, such as sentiment (binary) or topic (multi-class), is traditionally solved with encoder-only architectures including BERT, RoBERTa, and DistilBERT [1]. These models achieve strong predictive accuracy while maintaining low inference latency and modest computational footprints [1]. On-device deployment adds constraints that server-side deployment does not: limited memory, limited compute, and a hard requirement on latency per inference. This is why any claim of a cheaper or faster alternative to BERT deserves scrutiny before it is adopted for on-device use, and why a study that actually measures latency and cost side by side with accuracy is more useful to a builder than one that reports accuracy alone.

Large language models such as GPT-4o and Claude Sonnet 4.5 have demonstrated strong capabilities in open-ended reasoning and generative language tasks [1]. But LLMs generate output token by token even when producing a single class label, introducing inherent latency and usage-based costs [1]. A study that evaluates zero- and few-shot prompt-based LLMs against fully fine-tuned encoder-only architectures across four benchmarks, IMDB, SST-2, AG News, and DBPedia, measuring macro F1, inference latency, and monetary cost, found that fine-tuned encoder-based models from the BERT family achieve competitive, and often superior, classification performance while operating at one to two orders of magnitude lower cost and latency compared to zero- and few-shot LLM prompting [1]. This establishes BERT-family models as the practical baseline for on-device or cost-sensitive classification, not the LLM, and it is the baseline this note uses when discussing what a candidate replacement for BERT would need to beat.

Separately, a different line of work asks whether the standard practice of taking a prediction only from the last hidden layer is the right design choice for a neural classifier. Most architectures ultimately derive their predictions exclusively from the representation produced by the last hidden layer [12]. Empirical and theoretical studies suggest that different layers capture distinct and complementary aspects of the input: early layers encode local patterns, middle layers capture structural relations, and deeper layers represent increasingly abstract semantics [12]. LAYA (Layer-wise Attention Aggregator) is a novel output head that dynamically aggregates internal representations through attention, built to make use of this observation instead of discarding it [12].

The practical question a builder faces is whether replacing a standard classifier head with something like LAYA, or replacing a full LLM with a fine-tuned BERT, gives a better accuracy-per-millisecond trade-off on a phone or an embedded chip. The evidence collected here answers the second question directly and does not answer the first for text classification, because LAYA has only been tested on image datasets [12]. The rest of this note treats these as two separate, well-supported bodies of evidence, describes the mechanism and reported numbers for each, and is explicit throughout about where they do not meet, so that a builder who wants the missing comparison knows exactly what experiment to run and what it would need to control for. A study that frames model evaluation as a multi-objective decision problem and analyzes trade-offs using Pareto frontier projections and a parameterized utility function reflecting different deployment regimes offers the right lens for this kind of comparison once the missing measurement is filled in [1].

BERT is based on a multi-layer bidirectional Transformer and is trained on plain text for masked word prediction and next sentence prediction tasks [6]. It is the first fine-tuning based representation model that achieves state-of-the-art results for a range of NLP tasks [6]. This means BERT is pre-trained once on large unlabeled text using two self-supervised objectives, and the same pre-trained weights are then adapted, or fine-tuned, to many different downstream tasks with comparatively little labeled data. That two-stage recipe, pre-train then fine-tune, is the reason BERT and its descendants are the default choice for fixed-label text classification rather than a model trained from scratch for each task.

Mechanically, BERT generates an embedding vector contextualized through a stack of Transformer blocks for each input token, and prepends a special [CLS] token to the input sentence for classification tasks [5]. For sentence-level classification, an added linear classifier projects the [CLS] embedding to an unnormalized probability vector over the output classes [5]. This is the standard classification head: a single learned linear layer sitting on top of the final Transformer block's output for one designated token, followed by a normalization step that turns the unnormalized scores into class probabilities. The entire stack, encoder plus linear head, is trained end to end during fine-tuning, which is why fine-tuning BERT for a new task requires updating all of its parameters unless a parameter-efficient variant is used.

LAYA takes a different view of what the head should look at. LAY A (Layer-wise Attention Aggregator) is a lightweight output module that learns to weight and combine all hidden representations in an input-dependent way [12]. LAYA learns input-conditioned attention weights over layer-wise features [12]. Instead of the head reading only the last hidden layer, as BERT's [CLS]-based classifier does, LAYA's head reads every hidden layer produced by the backbone network and lets an attention mechanism decide, for each individual input, how much weight each layer's representation should receive. LAYA provides intrinsic layer-attribution scores that explicitly quantify the contribution of each representation to the final decision without external post-hoc explanation methods [12], which means the attention weights themselves double as an explanation of which depths of the network mattered for a given prediction, with no separate interpretability tool required.

The two designs therefore differ at exactly the point where a classification decision is made. BERT's classifier head is fixed and reads a single, final representation, the [CLS] embedding after the last Transformer block [5]. LAYA's head is itself a small trainable attention module that reads all the intermediate representations and produces a per-input, per-layer weighting before making the final prediction [12]. Both approaches sit on top of some feature-extracting backbone, an encoder in BERT's case, and the description of LAYA in the source does not specify a text-encoder backbone; the reported experiments use image classification datasets [12]. This distinction, backbone versus head, is the reason a fair comparison of Laya and BERT for text classification would need to specify exactly what backbone LAYA's aggregator sits on top of before any accuracy or latency numbers could be compared. It also means that describing LAYA as a competitor to BERT is only partly accurate: LAYA replaces the classification head, while BERT is usually described and benchmarked as a full encoder plus head, so a fair test would need to hold the backbone fixed and swap only the head, or state clearly that the whole architecture, backbone included, is being swapped.

The internal computation of LAYA is described precisely enough in the source to reproduce. LAY A replaces the standard classifier head with an attention-based aggregator: h_agg = sum_{i=1}^{L} alpha_i(x) g_i(h_i), where alpha_i(x) are learnable, input-conditioned coefficients [12]. In plain terms, the backbone produces L hidden representations, one per layer, and instead of keeping only the last one, LAYA computes a weighted sum of all of them, where the weight given to each layer depends on the specific input x currently being processed, not on a fixed, input-independent schedule. This is the core structural claim of the method: the weighting is a function of the input, so two different inputs passing through the same trained network can end up relying on different depths of representation.

Before the weighted sum is computed, each layer's representation is put on a common footing. Each hidden state in LAY A is mapped into a shared latent space through a lightweight adapter g_i(.), ensuring comparability across layers [12]. This adapter step matters because hidden states from different depths of a network can differ in scale, dimensionality, or semantic granularity, and summing them directly without first mapping them into a shared space could let a layer dominate the aggregate for reasons unrelated to how useful its representation actually is. The lightweight adapter is described as small, which keeps LAYA's total parameter and compute overhead low relative to the backbone it sits on, though the source does not give an exact parameter count or a formula for the adapter's size relative to the backbone's hidden dimension.

The output stage is a simple linear-plus-nonlinearity step applied to the aggregated representation. The final prediction in LAY A is computed as y_hat = phi(W h_agg + b) [12]. This is structurally similar to BERT's own classification head, a linear projection followed by a nonlinearity, with the key difference being what is fed into that linear layer: BERT feeds in the single final-layer [CLS] embedding [5], while LAYA feeds in the attention-aggregated combination of all layers' adapted representations [12]. The extra machinery in LAYA, relative to a standard head, is therefore the per-layer adapter g_i and the input-conditioned attention weights alpha_i(x); everything else about how the final class score is produced is a conventional linear classifier with a nonlinearity phi applied at the end.

Because the attention weights alpha_i(x) are learned and depend on the input, they can be inspected after training to see which layers the model relied on for which kinds of inputs, and the source treats this as a first-class feature rather than a side effect. Quantitative and qualitative interpretability analyses demonstrate that LAY A's attention scores closely reflect the actual contribution of individual layers [12]. LAY A reveals structured, task-dependent patterns of depth utilization while providing intuitive explanations of how different abstraction levels contribute to each prediction [12]. For a builder, this means that adopting LAYA as a classifier head is not purely an accuracy or latency decision; it also buys a built-in, per-prediction explanation of which network depths drove the decision, at the cost of the extra adapter and attention parameters described above. None of this mechanism has been described or measured for a text-encoder backbone in the evidence gathered here, so applying it to a BERT-style stack of Transformer blocks would be a direct, reasonable extension to test, not something already reported.

The clearest quantitative comparison available in the evidence is between fine-tuned BERT-family encoders and LLM prompting, not between BERT and LAYA. The relevant study measured macro F1, inference latency, and monetary cost across four benchmarks, IMDB, SST-2, AG News, and DBPedia, comparing zero- and few-shot prompt-based LLMs such as GPT-4o and Claude Sonnet 4.5 against fully fine-tuned encoder-only architectures [1]. The result was that fine-tuned encoder-based models from the BERT family achieve competitive, and often superior, classification performance while operating at one to two orders of magnitude lower cost and latency compared to zero- and few-shot LLM prompting [1]. The same paper frames model evaluation as a multi-objective decision problem and analyzes trade-offs using Pareto frontier projections and a parameterized utility function reflecting different deployment regimes [1], which is a methodological point a builder can reuse: accuracy alone is not the right lens when latency and cost differ by orders of magnitude between candidates. This methodology, not the specific numbers, is the part that transfers cleanly to a Laya-versus-BERT study, because it gives a template for reporting three axes together rather than accuracy in isolation.

Other sources add further, separate numbers about BERT variants without comparing them to LAYA. TinyBERT 4 with 4 layers achieves more than 96.8% the performance of its teacher BERT BASE on GLUE benchmark, while being 7.5x smaller and 9.4x faster on inference [7]. TinyBERT 6 with 6 layers performs on-par with its teacher BERT BASE [7]. These numbers describe distillation, compressing BERT BASE into a smaller student model, and they establish that BERT's accuracy can be preserved at a fraction of its size and latency through architectural compression, which is directly relevant to on-device deployment even though it says nothing about LAYA. A builder who wants a size-latency budget for the BERT side of a comparison has, in these numbers, two concrete reference points already measured on GLUE: a 4-layer student at 7.5x smaller and 9.4x faster with over 96.8% of teacher performance, and a 6-layer student at on-par performance.

Fine-tuning BERT for specific NLP tasks requires high computational resources and expensive inference time [2], a cost that motivated a hybrid classical-quantum approach: the proposed hybrid classical-quantum BERT model integrates an n-qubit quantum circuit with a classical BERT model for text classification [2], using the quantum circuit to fine-tune the classical model [2]. The proposed hybrid classical-quantum BERT model achieves performance that is competitive with, and in some cases better than, the classical baselines on standard benchmark datasets [2]. However, current quantum hardware is not yet fault-tolerant enough to achieve actual computational speedup, so experiments are performed with quantum circuit simulations [2], meaning the reported competitiveness is on accuracy, not on any demonstrated latency or cost advantage over classical BERT fine-tuning; the simulation runs on classical hardware, so no on-device latency number is available from this comparison either.

LAYA's own reported measurement is on a different domain entirely. Experiments on image classification datasets show that LAYA achieves competitive predictive performance while producing meaningful depth-aware explanations [12]. There is no benchmark named, no accuracy number given, and no text classification task involved in this measurement; the claim states only that performance is competitive and that the explanations are meaningful, on image data. Because none of the sources report LAYA's macro F1, latency, or cost on IMDB, SST-2, AG News, DBPedia, or any other text classification benchmark, and because none of the sources run BERT and LAYA on the same task with the same measurement protocol, there is no shared number in the evidence that lets a reader place LAYA and BERT on the same Pareto frontier for text classification. This gap is the central finding of this note: not that LAYA loses or wins, but that the comparison has not been measured, and any statement to the contrary would be going beyond what the sources say.

A builder choosing between BERT and any alternative head or architecture also needs to know about failure modes that are specific to BERT fine-tuning, independent of the comparison to LAYA. Fine-tuning BERT remains unstable, especially when using BERTLarge on small datasets, where identical learning processes with different random seeds often result in significantly different models [5]. This means that reported accuracy numbers for a single fine-tuning run of BERT can be misleading unless the run was repeated across multiple seeds, because two runs from the same setup can land at meaningfully different accuracy levels purely due to randomness in initialization or data ordering. This is a direct methodological warning for anyone designing a comparison study: a single-seed accuracy number for a BERT baseline is not enough to draw a conclusion against a competing method such as LAYA.

The same source identifies concrete causes of this instability and a documented mitigation. Factors causing instability in few-sample BERT fine-tuning include: the use of a non-standard optimization method with biased gradient estimation, limited applicability of parts of the BERT network, and the use of a pre-determined small number of training iterations [5]. Fine-tuning the pre-trained BERT model on a large intermediate task stabilizes later fine-tuning on small datasets [5]. For a builder working with a small on-device classification dataset, this gives a specific, actionable recipe: fine-tune first on a larger, related intermediate task before fine-tuning on the small target dataset, rather than fine-tuning directly on the small dataset from the base pre-trained checkpoint.

Cost is a recurring theme across multiple independent sources, not only the LLM-versus-encoder comparison. Fine-tuning BERT produces similar performance to classical models at significant additional cost in automated essay scoring [9]. This is a domain-specific result, automated essay scoring, but it reinforces the general pattern seen in the LLM-prompting comparison [1] and the quantum-hybrid motivation [2]: BERT's accuracy is often matched by cheaper alternatives, and the decision to use BERT should weigh that additional cost against the accuracy gained, task by task, rather than assuming BERT is always worth its cost.

Finally, there is a structural argument about why encoder architectures such as BERT might or might not be the right computational shape for a given task, independent of accuracy. Transformer layers are linear in the input dimensionality and quadratic in the input length [10]. Recurrent networks scale linearly in the input length and quadratically in the input dimensionality [10]. This trade-off is documented in a different application, sequence modeling for control, where Decision Transformer is an offline RL algorithm that auto-regressively models trajectories using the GPT-2 architecture [10], and struggles with continuous control problems such as inverted pendulum and Furuta pendulum stabilization [10], while Decision LSTM is able to achieve expert-level performance on inverted pendulum and Furuta pendulum stabilization tasks [10]. This result does not transfer directly to text classification or to LAYA, but it establishes, as a general and reproducible pattern, that the best architecture for a task depends on the shape of the input, length versus dimensionality, and that Transformer-based designs are not universally superior to alternatives even when the alternative is a much simpler recurrent network. A builder should treat this as a caution against assuming a Transformer-based encoder is automatically the right backbone for every classification setting, on-device or otherwise, without measuring it directly against alternatives on the specific input shape at hand.

Beyond raw accuracy and latency numbers, deployment context matters for an on-device decision, and one source speaks directly to this. Fine-tuned encoders' open-source nature enables on-premise deployment, offering advantages in privacy, data governance, and reproducibility [1]. For on-device classification specifically, this property is close to a requirement: the model weights must be available locally, inference must not depend on a network call to an external API, and the organization deploying the model needs to be able to reproduce and audit its behavior. Fine-tuned BERT-family models satisfy this because they are open-source and can be exported and run locally [1], whereas prompt-based LLM APIs, by contrast, typically require a network round trip and are billed per use, since LLMs generate output token by token even when producing a single class label, introducing inherent latency and usage-based costs [1].

This same property, open weights that can be exported and run locally, is not addressed for LAYA in the evidence. The claims describe LAYA as a lightweight output module [12], and lightweight generally implies a small number of additional parameters and low additional compute relative to whatever backbone it sits on, but no claim states LAYA's parameter count, memory footprint, or whether the reference implementation is openly released. A builder evaluating LAYA for on-device use would need to establish this directly from the LAYA implementation rather than from the claims summarized here, since none of the verified claims give a number for it.

Encoder footprint has already been reduced substantially within the BERT family through distillation, which is directly relevant to on-device budgets. TinyBERT 4 achieving more than 96.8% of BERT BASE's GLUE performance at 7.5x smaller size and 9.4x faster inference [7], and TinyBERT 6 performing on par with BERT BASE [7], both describe deployable, smaller checkpoints that keep BERT's fine-tuning recipe and its [CLS]-based classification head [5] while shrinking the compute and memory needed at inference time. This is the most concrete, reproducible path in the evidence to a low-latency, low-memory BERT-family classifier suitable for on-device use, and it is the natural baseline against which any new head design, including LAYA, should be measured if the goal is on-device deployment.

Taken together, the deployment-relevant claims describe BERT-family models as open-source, exportable, privacy-preserving when self-hosted [1], compressible without much accuracy loss through distillation [7], but unstable across seeds on small datasets unless an intermediate fine-tuning step is used [5]. None of these deployment properties, open-source status, parameter count, exportability, are documented for LAYA in the evidence gathered here; LAYA's documented properties are architectural, its aggregation mechanism [12], and evaluative, competitive accuracy and interpretability on image datasets [12], not operational. This asymmetry in what has been measured is itself an important finding for a builder: the two candidates are not yet described on the same axes, so an on-device deployment decision cannot be made from these claims alone, and closing that gap is exactly the missing experiment this note points toward.

The central limit of this evidence base is stated plainly: no claim reports a head-to-head experiment between LAYA and a fine-tuned BERT-base model on any text classification task, binary or multi-class, on-device or otherwise. LAYA's only reported experiments are on image classification datasets, where it achieves competitive predictive performance while producing meaningful depth-aware explanations [12]. BERT's reported comparisons are against zero- and few-shot LLM prompting [1], against a quantum-hybrid fine-tuning variant of itself [2], against a distilled student version of itself [7], and against classical, non-BERT models in essay scoring [9]. None of these comparisons involve LAYA, and none of LAYA's comparisons involve BERT or text data.

A second limit concerns latency specifically, which is the term named in the research question. The only latency numbers in the evidence are: fine-tuned BERT-family models operating at one to two orders of magnitude lower latency than zero- and few-shot LLM prompting on IMDB, SST-2, AG News, and DBPedia [1], and TinyBERT 4 being 9.4x faster on inference than teacher BERT BASE on GLUE [7]. No claim reports an inference latency number for LAYA on any dataset, image or text, and no claim reports an on-device (as opposed to server or simulated) latency number for either BERT or LAYA. A builder cannot therefore quote a latency figure for LAYA at all, only for BERT-family models against LLMs or against a distilled version of themselves.

A third limit concerns architecture mismatch. LAYA is described and evaluated as an output head attached to some unspecified backbone on image classification datasets [12], whereas the BERT-family results summarized here describe a full encoder architecture, pre-training objective, and fine-tuning recipe together [6] [5]. Swapping only the head, keeping a BERT encoder as the backbone and attaching a LAYA-style aggregator in place of the [CLS] linear classifier, is a natural way to close this gap, but it has not been done in any of the verified claims, so its outcome, whether it helps, hurts, or makes no measurable difference to accuracy or latency, is unknown from this evidence.

A fourth limit concerns reproducibility of any future comparison. Given that fine-tuning BERT remains unstable across random seeds on small datasets [5], any future study comparing BERT and LAYA on a small on-device-scale dataset would need to report results across multiple seeds, not a single run, to avoid confusing seed variance with a genuine architectural difference. Given that current quantum-hybrid BERT results rely on quantum circuit simulations rather than real hardware [2], any claim of a computational advantage from non-standard components should be checked for whether it was measured on the actual target hardware or only in simulation. Given that the LLM-versus-encoder study used a multi-objective, Pareto-frontier framing rather than a single accuracy number [1], a future Laya-versus-BERT study should adopt the same multi-axis reporting, accuracy, latency, and resource cost together, rather than reporting accuracy alone and calling the comparison complete.

1. Define the task and pick benchmarks that match the research question. Choose one binary task and one multi-class task so the comparison covers both regimes named in the question; the evidence names IMDB and SST-2 as binary-style sentiment benchmarks and AG News and DBPedia as multi-class topic benchmarks [1]. Fix the label set, train/validation/test split, and evaluation metric, macro F1, to match the metric already used for the BERT-versus-LLM comparison so results are comparable to that prior work [1].

2. Choose a shared backbone for both heads. Because LAYA is documented as an output head that reads all hidden layers of a backbone [12], and BERT's own classifier reads only the final [CLS] embedding [5], the fairest test holds the backbone fixed, for example a standard BERT-base encoder pre-trained on masked word prediction and next sentence prediction [6], and attaches two different heads to it: the standard [CLS]-plus-linear head [5], and a LAYA-style aggregator [12].

3. Implement the baseline head exactly as described. Prepend the [CLS] token to the input, run it through the Transformer stack, take the final-layer [CLS] embedding, and project it through a linear classifier to unnormalized class scores [5]. This is the reference configuration; do not modify it, since any change would make the comparison unfair to BERT.

4. Implement the LAYA head following its published computation. Collect the hidden state h_i from every one of the L layers of the shared backbone. Map each h_i into a shared latent space with a lightweight adapter g_i(.) [12]. Compute input-conditioned attention coefficients alpha_i(x) [12], and form the aggregate h_agg = sum_i alpha_i(x) g_i(h_i) [12]. Compute the final prediction as y_hat = phi(W h_agg + b) [12].

for each input x:
  h_1 ... h_L = backbone(x)          # one hidden state per layer
  for i in 1..L:
    z_i = adapter_i(h_i)            # g_i(.), shared latent space
  alpha = attention(x, z_1..z_L)     # input-conditioned weights
  h_agg = sum_i alpha_i * z_i
  y_hat = phi(W * h_agg + b)

5. Fine-tune both configurations on the same data with the same optimizer and the same number of epochs, changing only the head. Because fine-tuning BERT-based models is known to be unstable on small datasets across random seeds [5], run each configuration, baseline head and LAYA head, across at least three or four different random seeds and report the spread, not a single number.

6. If any target dataset is small, apply the documented stabilization step before the final fine-tuning run: first fine-tune the pre-trained backbone on a larger, related intermediate task, then fine-tune on the small target dataset [5]. Apply this identically to both head configurations so it does not become a confound.

7. Measure accuracy with macro F1 on held-out test data for both configurations, matching the metric used in the BERT-versus-LLM study so results sit in the same frame of reference [1]. Report mean and spread across seeds from step 5.

8. Measure inference latency for both configurations on the actual target on-device hardware, not a server or a simulator, since the evidence base warns that simulated results, as in the quantum-hybrid case, do not establish a real hardware speedup [2]. Record wall-clock time per example at a fixed batch size of one, matching the on-device deployment scenario in the research question.

9. Measure memory footprint and parameter count for both head configurations attached to the shared backbone. Use the TinyBERT-style compression numbers, 7.5x smaller and 9.4x faster for a 4-layer student versus BERT BASE [7], as a reference scale for what a meaningful size or speed difference looks like, so the LAYA-versus-baseline-head difference can be judged against a known magnitude.

10. Report results as a multi-objective comparison, not a single accuracy table, following the Pareto-frontier and utility-function framing used for the BERT-versus-LLM study [1]. Plot accuracy against latency for both head configurations and state clearly whether one dominates the other or whether there is a genuine trade-off.

11. Common failure points to check before trusting the result: confirm the adapter g_i is applied identically to every layer so no layer is given an unfair scale advantage [12]; confirm the attention coefficients alpha_i(x) are actually input-dependent and not collapsing to a fixed, input-independent weighting, since the interpretability value of LAYA depends on this being true [12]; and confirm that seed variance in the baseline BERT head, documented as a known instability [5], is not being mistaken for a genuine effect of the LAYA head.

12. Report the interpretability outputs alongside accuracy and latency if LAYA is used, since its attention weights are meant to double as a layer-attribution explanation without a separate post-hoc method [12]. Check qualitatively whether the pattern of layer usage looks structured and task-dependent, as reported for image classification [12], or whether it looks uniform or noisy on the text task, since this was not previously tested and is a new observation this experiment would produce.