Bytes Speak All Languages: Cross-Script Name Retrieval via Contrastive Learning

Standard name-matching systems often fail when comparing names across different writing systems, such as "Vladimir Putin" in Latin script versus "Владимир Путин" in Cyrillic, as they share no common characters. A novel approach tackles this by training a compact transformer model to operate directly on raw UTF-8 bytes, creating a universal encoder that bypasses the need for script detection or transliteration. This model was trained from scratch using contrastive learning on a massive dataset of over 4.6 million phonetic name pairs, which were synthetically generated at scale using a multi-stage LLM pipeline. By incorporating advanced techniques like hard negative mining, the byte-level model achieved high performance, drastically reducing the accuracy gap between Latin and non-Latin name retrieval compared to traditional baselines.