Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

NIPS’2020

摘要

该论文介绍了一种检索增强生成（Retrieval-Augmented Generation, RAG）模型，该模型结合了预训练的参数记忆和非参数记忆用于语言生成。参数记忆是一个预训练的序列到序列模型，非参数记忆是一个密集的维基百科向量索引，通过预训练的神经检索器访问。**论文探讨了两种RAG的形式：一种是在整个生成序列中条件相同的检索段落，另一种可以在每个令牌上使用不同的段落。**模型在一系列知识密集型NLP任务上进行了微调和评估，在三个开放域问答任务上设定了新的最先进结果。RAG模型被证明可以生成比仅参数序列到序列模型更具体、多样和事实性的语言。

Introduction

Pre-trained neural language models:

• learn a substantial amount of in-depth knowledge from data without any access to an external memory.
• a parameterized implicit knowledge base
• they cannot easily expand or revise their memory.
• can’t straightforwardly provide insight into their predictions.
• may produce “hallucinations”.

Hybrid models that combine parametric memory with non-parametric (i.e., retrieval-based) memories can address some of these issues because knowledge can be directly revised and expanded, and accessed knowledge can be inspected and interpreted.

• REALM and ORQA = masked language model + differentiable retriever, but only explored open-domain extractive question answering.

RAG

• pre-trained, parametric-memory generation models + a non-parametric memory
• a general-purpose fine-tuning approach (RAG)
• pre-trained model ⇒ pre-trained seq2seq transformer