Figure 2. The pipeline of our proposed ZegFormer for zero-shot semantic segmentation. We first feed N queries and feature maps to a
transformer decoder to generate N segment embeddings. We then feed each segment embedding to a mask projection layer and a semantic
projection layer to obtain a mask embedding and a semantic embedding. Mask embedding is multiplied with the output of pixel decoder
to generate a class-agnostic binary mask, while The semantic embedding is classified by the text embeddings. The text embeddings are
generated by putting the class names into a prompt template and then feeding them to a text encoder of a vision-language model. During
training, only the seen classes are used to train the segment-level classification head. During inference, both the text embeddings of seen
and unseen classes are used for segment-level classification. We can obtain two segment-level classification scores with semantic segment
embeddings and image embeddings. Finally, we fuse the these two classification scores as our final class prediction of segments.