All buildings are made of earth, one way or another. No one has ever built anything whose materials and products didn't come out of the ground. It just takes a bit more processing to get, say, structural steel or an LED light bulb than to make an adobe brick. Low carbon or low-impact building means choosing products and materials that provide the desired roles and functions with a minimum of energy, water or chemical fuss.
"Earthen buildings", in their many forms around the world, can be thought of as clay-based concrete as opposed to cement-based concrete. Many are, yes, mere mud huts, but many are magnificent architectural gems, and very effective as shelter. And they are cheap, so that's what the poorest of the poor throughout history have had to use - thus the strong association of earthen building with squalor. But some of the most effective and lovely new structures anywhere are made with earth, signals of a worldwide renaissance of earthen architecture.
We already use plant fibers like wood and bamboo in most buildings, but of particular interest are by-products of food production that currently present disposal problems. The straw, seed husks, cane and other fibers that we used to throw away are increasingly being used as fuel to generate electricity, and as feedstock for a huge range of building materials such as cement, insulation, and sheathing panels.
See the Products page for building products currently for sale, and meeting some standard of "green". Here we look at manmade items like plastic bottles, concrete rubble, and shipping containers that can be effectively repurposed as components of new buildings, but aren't owned or promoted by commercial interests, and have no clear place in building codes.