In a previous blog post, I discussed how multi-task learning (MTL) can be used to improve the performance of a model by leveraging a related task. Multi-task learning consists of two main components: a) The architecture used for learning and b) the auxiliary task(s) that are trained jointly. Both facets still have a lot of room for improvement. In addition, multi-task learning has the potential to be a key technique on the path to more robust models that learn from limited data: Training a model to acquire proficiency in performing a wide range of NLP tasks would allow us to induce representations, which should be useful for transferring knowledge to many other tasks, as outlined in this blog post.

On the way to this goal, we first need to learn more about the relationships between our tasks, what we can learn from each, and how to combine them most effectively. Most of the existing theory in MTL has focused on homogeneous tasks, i.e. tasks that are variations of the same classification or regression problem, such as classifying individual MNIST digits. These guarantees, however, do not hold for the heterogeneous tasks to which MTL is most often applied in Natural Language Processing (NLP) and Computer Vision.

There have been some recent studies looking into when multi-task learning between different NLP tasks works but we still do not understand very well which tasks are useful. To this end, as inspiration, I will give an overview in the following of different approaches for multi-task learning for NLP. I will focus on the second component of multi-task learning; instead of discussing howa model is trained, as most architectures only differ in which layers they share, I will concentrate on the auxiliary tasks and objectives that are used for learning.

This post has two main parts: In the first part, I will talk about artificial tasks that can be used as auxiliary objectives for MTL. In the second part, I will focus on common NLP tasks and discuss which other NLP tasks have benefited them.

Artificial auxiliary objectives

Multi-task learning is all about coming up with ways to add a suitable bias to your model. Incorporating artificial auxiliary tasks that cleverly complement your target task is arguably one of the most ingenious and fun ways to do MTL. It is a feature-engineering of sorts: instead of engineering the features, you are engineering the auxiliary task you optimize. Similarly to feature engineering, domain expertise is therefore required as we will see in the following:

Language modelling Language modelling has been shown to be beneficial for many NLP tasks and can be incorporated in various ways. Word embeddings pre-trained by word2vec have been shown to beneficial — as is known, word2vec approximates the language modelling objective; languages models have been used to pre-train MT and sequence-to-sequence models [³]; contextual language model embeddings have also been found useful for many tasks [⁴]. In this context, we can also treat language modelling as an auxiliary task that is learned together with the main task. Rei (2017) [²] shows that this improves performance on several sequence labelling tasks.

Conditioning the initial state   The initial state of a recurrent neural network is typically initialized to a 00 vector. According to a lecture by Hinton in 2013, it is beneficial to learn the initial state just like any other sets of weights. While a learned state will be more helpful than a 00vector it will be independent of the sequence and thus unable to adapt. Weng et al. (2017) [¹] propose to add a suitable bias to the initial encoder and decoder states for NMT by training it to predict the words in the sentence. In this sense, this objective can essentially be seen as a language modelling objective for the initial state and might thus be helpful for other tasks. Similarly, we can think of other task-specific biases that could be encoded in the initial state to aid learning: A sentiment model might benefit from knowing about the general audience response to a movie or whether a user is more likely to be sarcastic while a parser might be able to leverage prior knowledge of the domain’s tree depth or complexity.

Adversarial loss   An auxiliary adversarial loss was first found to be useful for domain adaptation [⁵, ⁶], where it is used to learn domain-invariant representations by rendering the model unable to distinguish between different domains. This is typically done by adding a gradient reversal layer that reverses the sign of the gradient during back-propagation, which in turn leads to a maximization rather than a minimization of the adversarial loss. It is not to be confused with adversarial examples [⁷], which significantly increase the model’s loss typically via small perturbations to its input; adversarial training [⁸], which trains a model to correctly classify such examples; or Generative Adversarial Networks, which are trained to generate some output representation. An adversarial loss can be added to many tasks in order to learn task-independent representations [⁹]. It can also be used to ignore certain features of the input that have been found to be detrimental to generalization, such as data-specific properties that are unlikely to generalize. Finally, an adversarial auxiliary task might also help to combat bias and ensure more privacy by encouraging the model to learn representations, which do not contain information that would allow the reconstruction of sensitive user attributes.

Predicting data statistics   An auxiliary loss can also be to predict certain underlying statistics of the training data. In contrast to the adversarial loss, which tries to make the model oblivious to certain features, this auxiliary task explicitly encourages the model to predict certain data statistics. Plank et al. (2016) [¹⁰] predict the log frequency of a word as an auxiliary task for language modelling. Intuitively, this makes the representation predictive of frequency, which encourages the model to not share representations between common and rare words, which benefits the handling of rare tokens. Another facet of this auxiliary task is to predict attributes of the user, such as their gender, which has been shown to be beneficial for predicting mental health conditions [⁴⁹] or other demographic information [⁵¹]. We can think of other statistics that might be beneficial for a model to encode, such as the frequency of POS tags, parsing structures, or entities, the preferences of users, a sentence’s coverage for summarization, or even a user’s website usage patterns.

Learning the inverse   Another auxiliary task that might be useful in many circumstances is to learn the inverse of the task together with the main task. A popular example of this framework is CycleGAN [⁴³], which can generate photos from paintings. An inverse auxiliary loss, however, is applicable to many other tasks: MT might be the most intuitive, as every translation direction such as English->French directly provides data for the inverse direction, as Xia et al. (2016) [⁴⁴] demonstrate. Xia et al. (2017) [⁴⁵] show that this has applications not only to MT, but also to image classification (with image generation as its inverse) and sentiment classification (paired with sentence generation). For multimodal translation, Elliott and Kádár (2017) [¹⁹] jointly learn an inverse task by predicting image representations. It is not difficult to think of inverse complements for many other tasks: Entailment has hypothesis generation; video captioning has video generation; speech recognition has speech synthesis, etc.

Predicting what should be there   For many tasks, where a model has to pick up on certain features of the training data, we can focus the model’s attention on these characteristics by encouraging it explicitly to predict them. For sentiment analysis, for instance, Yu and Jiang (2016) [²⁰] predict whether the sentence contains a positive or negative domain-independent sentiment word, which sensitizes the model towards the sentiment of the words in the sentence. For name error detection, Cheng et al. (2015) [⁵⁰] predict if a sentence contains a name. We can envision similar auxiliary tasks that might be useful for other tasks: Predicting whether certain entities occur in a sentence might be useful for relation extraction; predicting whether a headline contains certain lurid terms might help for clickbait detection, while predicting whether an emotion word occurs in the sentence might benefit emotion detection. In summary, this auxiliary task should be helpful whenever a task includes certain highly predictive terms or features.

Joint training of existing NLP tasks

In this second section, we will now look at existing NLP tasks, which have been used to improve the performance of a main task. While certain tasks such as chunking and semantic tagging have been found to be useful for many tasks [⁶⁰], the choice whether to use a particular auxiliary task largely depends on characteristics of the main task. In the following, I will thus highlight different strategies and rationals that were used to select auxiliary tasks for many common tasks in NLP:

Speech recognition   Recent multi-task learning approaches for automatic speech recognition (ASR) typically use additional supervision signals that are available in the speech recognition pipeline as auxiliary tasks to train an ASR model end-to-end. Phonetic recognition and frame-level state classification can be used as auxiliary tasks to induce helpful intermediate representations. Toshniwal et al. (2017) [¹¹] find that positioning the auxiliary loss at an intermediate layer improves performance. Similarly, Arık et al. (2017) [¹²] predict the phoneme duration and frequency profile as auxiliary tasks for speech synthesis.

Machine translation   The main benefit MTL has brought to machine translation (MT) is by jointly training translation models from and to different languages: Dong et al. (2015) [¹³] jointly train the decoders; Zoph and Knight (2016) [¹⁴] jointly train the encoders, while Johnson et al. (2016) [¹⁵] jointly train both encoders and decoders; Malaviya et al. (2017) [¹⁶] train one model to translate from 1017 languages into English.

Other tasks have also shown to be useful for MT: Luong et al. (2015) [¹⁷] show gains using parsing and image captioning as auxiliary tasks; Niehues and Cho (2017) [¹⁸] combine NMT with POS tagging and NER; Wu et al. (2017) [⁵⁵] jointly model the target word sequence and its dependency tree structure.

Multilingual tasks   Similarly to MT, it can often be beneficial to jointly train models for different languages: Gains have been shown for dependency parsing [²², ²⁸], named entity recognition [²³], part-of-speech tagging [²⁴], document classification [²⁵], discourse segmentation [²⁶], and sequence tagging [²⁷].

Language grounding   For grounding language in images or videos, it is often useful to enable the model to learn causal relationships in the data. For video captioning, Pasunuru and Bansal (2017) [³⁰] jointly learn to predict the next frame in the video and to predict entailment, while Hermann et al. (2017) [⁴⁶] also predict the next frame in a video and the words that represent the visual state for language learning in a simulated environment.

Semantic parsing   For a task where multiple label sets or formalisms are available such as for semantic parsing, an interesting MTL strategy is to learn these formalisms together: To this end, Guo et al. (2016) [³¹] jointly train on multi-typed treebanks; Peng et al. (2017) [³²] learn three semantic dependency graph formalisms simultaneously; Fan et al. (2017) [³³] jointly learn different Alexa-based semantic parsing formalisms; and Zhao and Huang (2017) [⁵⁷] jointly train a syntactic and a discourse parser.

Representation learning   For learning general-purpose representations, the challenge often is in defining the objective. Most existing representation learning models have been based on a single loss function, such as predicting the next word [³⁴] or sentence [³⁵] or training on a certain task such as entailment [³⁶] or MT [³⁷]. Rather than learning representations based on a single loss, intuitively, representations should become more general as more tasks are used to learn them. As an example of this strategy, Hashimoto et al. (2017) [⁵⁹] jointly train a model on multiple NLP tasks, while Jernite et al. (2017) [³⁸] propose several discourse-based artificial auxiliary tasks for sentence representation learning.

Question answering   For question answering (QA) and reading comprehension, it is beneficial to learn the different parts of a more complex end-to-end model together: Choi et al. (2017) [⁵²] jointly learn a sentence selection and answer generation model, while Wang et al. (2017) [⁵⁶] jointly train a ranking and reader model for open-domain QA.

Information retrieval   For relation extraction, information related to different relations or roles can often be shared. To this end, Jiang (2009) [²⁹] jointly learn linear models between different relation types; Yang and Mitchell (2017) [⁵³] jointly predict semantic role labels and relations; Katiyar and Cardie (2017) [⁵⁸] jointly extract entities and relations; and Liu et al. (2015) [³⁹] jointly train domain classification and web search ranking.

Chunking   Chunking has been shown to benefit from being jointly trained with low-level tasks such as POS tagging [⁴⁰, ⁴¹, ⁴²].

Miscellaneous   Besides the tasks mentioned above, various other tasks have been shown to benefit from MTL: Balikas and Moura (2017) [²¹] jointly train coarse-grained and fine-grained sentiment analysis; Luo et al. (2017) [⁴⁷] jointly predict charges and extract articles; Augenstein and Søgaard (2017) [⁴⁸] use several auxiliary tasks for keyphrase boundary detection; and Isonuma et al. (2017) [⁵⁴] pair sentence extraction with document classification.

Conclusion

I hope this blog post was able to provide you with some insight with regard to which strategies are employed to select auxiliary tasks and objectives for multi-task learning in NLP. As I mentioned before, multi-task learning can be very broadly defined. I have tried to provide as broad of an overview as possible but I still likely have omitted many relevant approaches. If you are aware of an approach that provides a valuable perspective that is not represented here, please let me know in the comments below.

References