Systolic Array-based Architecture for Low-Bit Integerized Vision Transformers

We've built a low-bit, model-specialized hardware accelerator for vision transformer inference, using multiple systolic arrays with deep, fine-grained pipelines tailored to the multi-head self-attention operations that dominate transformer compute. Rather than relying on general-purpose GPU flexibility, the architecture pipelines each attention head within a single accelerator to maximize data reuse and minimize off-chip bandwidth, the two levers that matter most for inference cost and energy at scale. Inference workloads are characterized by high query-per-model ratios and fixed model weights, conditions under which GPUs exhibit low operational intensity due to their general-purpose design. Our approach combines hardware and software optimization: on the hardware side, we specialize the datapath to the MSA operation structure and pipeline SA heads within a single unit to increase OP reuse and reduce bandwidth requirements; on the software side, we use a 3-bit integerized model that reduces precision while retaining strong accuracy, 96.83% on CIFAR-10 and 77.81% top-1 on ImageNet, so the hardware specialization has a compact, low-bit workload to exploit. We've validated the combined design on a 16nm FPGA (Alveo U250), achieving 13,568 GOPs/s at 219.4 GOPs/s/W. Against a same-generation GPU (GTX 1080), that's 1.50x higher throughput and 4.47x better power efficiency. Even against a current state-of-the-art GPU (RTX 5090), we hold a 20% power-efficiency edge despite 87% lower raw throughput, a favorable trade for power or cost constrained inference deployments. A provisional patent has been filed covering the architecture. We are looking to partner for other machine learning workloads.