Generative Representational Instruction Tuning

Meta info.

• Authors: Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, Douwe Kiela
• Paper: https://arxiv.org/pdf/2402.09906.pdf
• Affiliation: Contextual AI, Hong Kong Univ., Microsoft Corporation

TL; DR

text embedding과 generation 통합하는 Generative Representational Instruction Tuning 제안. 단일모델인 GritLM은 embedding(MTEB) 및 generation task(BBH...)에서 모두 SoTA를 달성.

Effects

• for embedding: bidirectional attention with mean pooling
• for generation: causal attention with a multi-turn format ideal for chat
• RAG 속도도 최대 60% 향상 가능 (!)