Browncoats were soldiers who fought for the Independent Planets, who lost to the Union of Allied Planets in the Unification War. They were named after the simple, beige brown leather trench coats they wore as uniforms. They also wore red scarves around their necks, red vests, and green combat helmets. In some areas, they wore brown aviator styled uniforms and dark brown bandannas. The simplicity of their gear was because technology and resources were scarce on the Outer Planets. Conversely, the Alliance's soldiers were called "Purple Bellies", because of the purple tinted body armour they wore.
After the war's end, clothiers made good money dyeing brown coats blue or gray as folks wanted to forget the past and let the past forget them. Those that still "wear the brown" do it on purpose, as such, Captain Malcolm Reynolds and Zoë Alleyne continue to wear the signature brown coat; however, Zoe wears a slightly different beige brown coat during jobs, and Mal wears a dark brown coat. It is not known whether these coats were simply dress uniforms for the Browncoats or they are supposed to be same ones they've worn in the war.
Behind the scenes
The term "Browncoat" has also come to apply to fans of the show. By taking agency over the term Browncoat, fans align themselves with the "Independents" as they fight against the Alliance. The Alliance being the FOX industry that stopped airing Firefly after its first season.