Compositional Task Representations for Large Language Models
ICLR 2023 (Poster) · PDF · Code · Poster
Represent a new task by recombining learned task codes. Compositional Task Representations (CTR) approaches cross-task generalization through a discrete codebook learned during multitask training. A task is represented by a composition of code embeddings that conditions a language model.
For unseen tasks, CTR searches over new combinations of the learned codes. Code Ensemble targets the zero-label setting using pseudo-labeled task inputs, while Bitwise Search uses a small number of labeled examples to improve a candidate code. The released implementation includes the training code, processed task data and model checkpoints.
The paper reports stronger average zero-label performance than its evaluated prompt-based methods and analyzes whether individual learned codes have interpretable, controllable effects. These comparisons concern the paper’s multitask training and unseen-task evaluation setup.
Resources
Datasets: Processed · No prompt
Checkpoints: Checkpoints
Poster
