Reinforcement Learning Generator-Evaluator Architecture for Question Generation.
We present in this work a fully Transformer-based Generator-Discriminator architecture for question generation. Transformer has been identified as a universal approximator of any sequence-to-sequence function. That is, a Transformer model which is deep enough can theoritically address the question generation task. For the sake of diversity –on the semantic sense– at the inference, we experimented the Generator-discriminator architecture to address the question generation problem. T5 was used as the generator and the mixture of BLEU and the cosine similarity of the representation output by ELECTRA-discriminator was used as the discriminator.
Description
Model Architecture
Dependencies
torch
transformers
sentencepiece
pytorch_lightning
nltk
collections
bert-score
Dataset
SQUAD v.2
Evaluation
We used NUBIA to evaluate our model, we got the following results
| Model | BLEU | BERTScore | Semantic relation | Logical Agreement |
|---|---|---|---|---|
| T5-RL | 22.05 | 52.62 | 61.31 | 42.52 |
Some Results
| Context | When Hitler launched World War II by invading Poland, British and French officials in Cameroon seized the German-owned plantations (1939-40). After the fall of France (June 1940), the country's colonial dependencies proclaimed loyalty to Marshal Petain' new Vichy regime. Central Africa was an exception | answer |
|---|---|---|
| Question predicted | Who launched World War II by invading Poland? | Hitler |
| What war did Hitler begin by invading Poland? | World War II | |
| What country did Hitler invade in World War II? | Poland | |
| Who did Cameroon's colonial dependencies proclaim loyalty to? | Marshal Petain | |
| What region was an exception to the Vichy regime? | Central Africa | |
| What regime did the colonial dependencies of Cameroon resign to? | Vichy |
| Context | The Douala-Yaoundé railway line, begun under the German regime, had been completed. Thousands of workers were forcibly deported to this site to work fifty-four hours a week. Workers also suffered from lack of food and the massive presence of mosquitoes. In 1925, the mortality rate on the site was 61.7%. However, the other sites were not as deadly, although working conditions were generally very harsh | answer |
|---|---|---|
| Question predicted | What was the mortality rate in 1925? | 61.7% |
| Which regime began the Douala-Yaoundé railway? | German | |
| Which railroad line in the German era was completed in 1925? | Douala | |
| How many workers were deported to this railroad site? | Thousands |