We present a compact transformer architecture that achieves competitive accuracy on low-resource text-classification tasks while reducing parameter count by 60%. Across five benchmark datasets the model matches larger baselines and trains in under an hour on a single GPU, making it practical for resource-constrained settings.
Read full text
1. Introduction Transformer architectures have become the backbone of modern natural language processing, but their computational and memory footprint limits their use in resource-constrained settings such as classrooms and small research groups. In this paper we ask a simple question: how small can a transformer be while remaining competitive on low-resource text-classification tasks? 2. Method We introduce a compact encoder with weight-tying across layers and a reduced attention head count. The model is trained with a distillation objective from a larger teacher, and we apply aggressive but structured pruning after convergence. All experiments use a single consumer GPU. 3. Results Across five benchmark datasets the compact model matches larger baselines within one accuracy point while using 60% fewer parameters and training in under an hour. Ablations show that layer weight-tying contributes most of the parameter savings with negligible accuracy loss. 4. Conclusion Compact transformers are a practical choice for low-resource settings. We release our configuration and training recipe to support reproducibility.