If you look at successful lifestyle brands in today's niche sport market, they offer premium and expensive clothing. Patagonia, Arcteryx, etc... all offer expensive and functional clothing that people pay a premium for.
The problem with surfers is two fold. One, most are broke dicks or the types that don't spend money on expensive clothing items. Two, outside of wet-suits, there's not much need for expensive clothing. A $10 rash guard from Target does the same thing as a $60 dollar Rip Curl.
Compare a rash guard or board shorts to a poorly fit winter ski jacket or pants and you'll see that being cold usually makes people spend money to be comfortable. What is the benefit of buying a Quiksilver T-shirt? It says Quiksilver?