Humans and animals can quickly adapt to new task demands while retaining capabilities developed previously. Such flexible sensory-guided behavior requires reliable encoding of stimulus information in neural populations, and task-specific readout through selective combination of these responses. The former has been the topic of intensive study, but the latter remains largely a mystery. Here we propose that targeted stochastic gain modulation could support flexible readout of task-information from an encoding population. In experiments, we find that responses of neurons in area V1 of monkeys performing a visual orientation discrimination task exhibit low-dimensional comodulation. This modulation fluctuates rapidly, and is stronger in those neurons that are most informative for the behavioral task. We propose a theoretical framework in which this modulation serves as a label to facilitate downstream readout. We demonstrate that the shared modulatory fluctuations found in V1 can be used to decode from the recorded neural activity within a small number of training trials, consistent with observed behavior. Simulations of visual information processing in a hierarchical neural network demonstrate that learned, modulator-induced labels can accompany task-information across several stages to guide readout at a decision stage and thereby fine-tune the network without reorganization of the feedforward weights. This allows the circuit to reach high levels of performance in novel tasks with minimal training, outperforming previously proposed attentional mechanisms based on gain increases, while also being able to instantly revert back to the initial operating regime once task demands change. The theory predicts that the modulator label should be maintained across processing stages and indeed we find that the trial-by-trial modulatory signal estimated from V1 populations is also present in the activity of simultaneously recorded MT units, preferentially so if they are task-informative. Overall, these results provide a new framework for how intelligent systems can flexibly and robustly adapt to changes in task structure by adjusting information routing via a shared modulator label.