One of my buddies at UCSB had a dad who was Navy so he was able to get on base. We would drive down in two cars, then pile into one and smile at the guards who obviously hated us. This was in the late 90's, I don't know how it is to get on now.
We never got it off the charts, I only went 6 or 8 times, but every time I went it was really fun. We would typically go in the summer or early fall, it would usually be sloppy and thigh high in Ventura, flat in Santa Barbara, but on the base it would be clean and shoulder high to head high. It was fun to surf a wave with a bit of push after surfing waist high Sands windswell all summer. Looking back I wish we had taken advantage more. What an area though.
I have not been there since the late 90's, but my buddy had an ID card, or maybe a sticker on the car or something, and they would always let us right in after checking. A couple of times we saw people boat in, not sure if anyone does that anymore.