Over the years I have made banjo, guitar, and mandolin nuts from all six of the materials Peter Roehling listed (plastic, bone, ivory, fossil ivory, brass, and oyster mother of pearl), plus many from ebony (Mossman guitars used ebony as a production nut material from 1975 to their primary demise in 1977, and ebony is the standard violin family nut material) and even a couple from lignum vitae wood, just about the hardest and densest of all woods. I have not made any from graphite composite, a current fad among some electric players. In my own instruments I have found little to tell in tone among the various materials, but plenty of adherents to each, based on sometimes mystical and almost always scientifically unmeasurable qualities. In workability most of the biological materials (bone, ivory, mother-of-pearl, and ebony) are about the same, and micarta is similar to those. All take a high polish and all are equally elegant (in my opinion) except for micarta, which seems to me a bit clunky. Brass is difficult only from a machinability standpoint, but is easily cut with a medium-slow bandsaw and shaped with a single-cut mill file. It is more difficult to polish because minute scratches are difficult to remove, but it does respond to heavy buffing. Much the most difficult material I have used is lignum vitae, although graphite composite is reputed to be very difficult as well. Lignum vitae will dull or ruin most steel woodworking tools, and shaping and polishing are tedious processes at best. The result on the several instruments I tried was a nut with a highly unusual appearance and no perceptible tonal difference from bone. Luthiers equipped with lapidary tools (as was recommended during the 1950s by the Lignum Vitae Corporation, the primary importer) might explore this material, but quality lignum vitae is rather difficult to obtain these days. The other materials are easily cut with machine and hand saws, shaped with files and sanding belts/wheels, and polished with fine sandpaper and fine compound on a buffing wheel.
Micarta and other hard plastics are used by instrument companies primarily because they are inexpensive, generally consistent among batches, and in steady supply. I question the assertion that these plastics are more "environmentally friendly" than such natural materials as ivory, mother of pearl, and wood, because the plastics are petroleum products, the extraction, refinement, and derivative synthesis of which are anything but "environmentally friendly.". By that measure, only bone from domestic cattle and mother-of-pearl from farmed pearl oysters are acceptable nut materials, which is good because they are two of the very best materials for this purpose. Bone must be cleaned and degreased to be usable for instruments. Most of the material currently in use is imported from Asia, already cleaned, degreased, and bleached, ready to use, and it is so cheap that it is difficult to justify the effort required to prepare homegrown bone. The process is easy but involved: obtain a cow "knuckle" or leg bone from the butcher, saw off and discard the ends, extract and discard the marrow, scrape as much soft tissue from the bone as possible, simmer the bone in hot (200 degree F) water with a little detergent added for 60-80 minutes, air-dry the bone, bandsaw into rough nut and saddle blanks, degrease for a couple of weeks or longer in household ammonia or white gas (heavy duty fire hazard), rinse in detergent-water, air-dry, bleach in 3% hydrogen peroxide for 5-10 minutes. Whew. Degreasing is absolutely critical because grease will otherwise seep eventually from the bone into the nearby wood, and cause finish and wood to part company, glue joints to fail, and in general make a permanent, indelible mess.
Ivory is probably the best nut material from the standpoints of workability, hardness, tradition, and elegance, but because of its origins from large endangered mammals it has become unacceptable to use for virtually any purpose. Ivory from walrus, narwhal, and Indian/African elephants has been proscribed from importation into the US and other CITES signatory countries for years or decades, although in most states there is no law against using existing domestic stocks in instruments. However, interstate sales of such new instruments many be problematic. Also, if you export an instrument with an ivory nut/saddle (such as to a foreign gig), it may be difficult to import it again unless you can prove prior ownership--I have also heard that in reality this is not a problem, and I know of no one who has had to surrender a valuable instrument to the Fish and Wildife Service because of the ivory nut. Hippopotamus ivory is sometimes available and presumably "legal," and I leave it to the reader to decide whether it is therefore more acceptable than the proscribed ivories. So-called "fossil" ivory is not really fossilized at all, rather it is ancient or at least very old ivory from walruses and extinct elephant species. THE source for this ivory is the arctic, where temperatures generally remain low enough to allow the ivory to survive hundreds and thousands of years of weathering and immersion. Such ivory is generally not as hard as new material, and many luthiers feel that its best use is for inlay. The most unique source of this ivory is large deposits of 300--500 year old walrus tusks found buried at Inuit habitations in the Little Diomede Islands in the Bering Straits. During the 1970s tons of these tusks found their way into the US fossil ivory trade. They were easily differentiated from new ivory by their brown-stained color and by their relative softness.
Mother-of-pearl is relatively easy to cut and polish and it is hard enough to make a good, elegant nut. Unless you want to cut your own shell, you are limited to (rather expensive) blanks offered by lutherie suppliers. Also, fine pearl dust is quite toxic, primarily because it is difficult to eject from the lungs. Chronic exposure can cause silicosis, which can severely impair pulmonary function and can even be fatal. Actually, the fine dust from any of the nut materials discussed here can cause similar lung problems, so always wear a dust mask (or stronger protection) when working with these materials.
Ebony is very straightforward to cut and polish, and its only drawbacks as nut material are that it is softer than the plastic and other materials (and therefore does not last as long under string wear) and it does not contrast with a dark fingerboard as do the lighter colored materials. Again, tonally it may not differ substantially from bone, mother-of-pearl, or plastic, but some will say otherwise, unfortunately without objective supportive data.
Brass was reputed to increase sustain in electric guitars, yet there is no evidence that this material achieves that objective in all electric instruments, let alone in acoustics. Sustain is one of the few sound/tonal characters that can be measured in a scientifically meaningful fashion, and the chart traces that I have seen indicate little if any additional sustain (=retarded sound decay) in at least some electric instruments from a brass nut. Still, the legend lives on....
Based on the above, I think that most better instruments are best fitted with a bone or farmed oyster mother-of-pearl nut, for the best combination of appearance, workability, durability, and impact on resources. Bone is preferable from an economic standpoint, though few of us will make enough instruments so that the relative cost of pearl and bone will become a deciding issue. Tone may be an issue with some, but either of those materials should satisfy most players and luthiers. Other materials should be investigated, but I suspect that few if any other materials will satisfy the above considerations as well as do bone and pearl.