KILM (short for Knowledgeable Injection into Language Model) is an approach for injecting knowledge into pre-trained language models. This repository provides the necessary code for KILM. KILM propose a new (second-step) pre-training method to inject information about entities (such as entity descriptions) into pre-trained language models.
This repository contains code for
- Creating the dataset needed for model training. The code goes through a dump of Wikipedia and extracts the necessary data (short descriptions, etc.).
- Language model continued pre-training (knowledge injection)
- Evaluation of the final model on some downstream-tasks.
In the following sections, we describe the dependencies, the steps to reproduce our pre-training checkpoint, the scripts for downstream task evaluation, and the code structure illustration.
