Masks on Models
Specular and Glow
A masks.jpg is an image that combines a glowmap and a specular map. To fit both into one image, they are essentially greyscale images in the red (specular) and green (glow) channel. The blue channel is unused, but might be used in the future for another type of map.
Easiest way to grasp the concept would be to open an existing mask (for example /packages/models/boost/masks.jpg). Open your "channels" window in either Photoshop or the GIMP, and have a look at the individual channels. Blue is black (unused) and the red and green channels will hold the specular and glow information. The brighter the values, the stronger the effect will be. With the specular map, you can control the shininess of the materials (and for example make the metal more reflective than the wood parts of a model) and the glow map will allow you to have certain parts of the model appear as if they emit light (for example lamps, LEDs, lava, flames). Note that the color, since the green channel only holds brightness values, will have to be controlled via the color map (usually referred to as the "skin"). If your model doesn't need any glow, just keep the green channel empty (i.e. black).
To use your mask, you have basically two options. On an MD2, you simply drop a file called "masks.jpg" into your model´s directory, and it will be used on the model. On an md3, you either use it without configuration, and add a "masks.jpg" like for an MD2, or you can assign different mask textures to the parts of your model, which then can have individual names. (see "md3 how-to" below for further info).
You can also control the overall strength of the specular part with the "mdlspec" command in your model's *.cfg file. Be sure to keep the blue channel empty, for one day it might be used for a third type of map, and you don't want any unwanted results to appear on your model that day.