Semi-supervised learning is important for practical applications of deep models because it alleviates the dependence on labeled data. It is particularly interesting in the context of dense prediction, where labeling at the pixel level requires a lot of effort. This paper considers forms of semi-supervised learning that, together with supervised loss, promote the consistency of predictions in perturbations of the input data. We investigated consistency formulations with respect to the consistency direction and input perturbations. We designed an efficient algorithm for semi-supervised learning for semantic segmentation. Finally, we proposed a new perturbation model that is particularly suitable for semi-supervised learning of dense prediction models. Theoretical analysis and empirical insights show that the best generalization is achieved by one-way consistency with a clean teacher. This form of semi-supervised consistency learning requires one instance of the model, the student, to be consistent with another, the teacher, and only the student's input is (strongly) perturbed. One-way consistency also has the advantage of memory efficiency compared to two-way consistency because it does not have to keep the teacher's intermediate results in memory for computing the gradient. Experiments focus on efficient models that are important for resource-efficient and real-time applications. Some other research findings include: (1) photometric perturbations are much more useful than geometric ones for semantic segmentation, (2) using a moving average of the student's parameters in the teacher is advantageous over simple parameter sharing when there is very little labeled data available, (3) semi-supervised learning can be a better alternative to additional coarse labels. The generalization performance of the proposed method surpasses all previous and competitive approaches for semi-supervised semantic segmentation.