The Devils play at the Meadowlands.....the same facility where the New York Jets and New York Giants play football.......which is in NJ but just outside New York City. Which teams you follow if you live in NJ also has a lot to do with what television market you grew up in. A big chunk of New Jersey gets its TV out of Philadelphia.....especially back in the days before Cable television when you relied on your roof top antenna. Your roof top antenna determined what sports teams you followed.
Using your San Diego analogy it would be like the Devils played in Mission Beach with Downtown San Diego representing New York City and you lived up in Newport Beach and LA represents Philadelphia. Those distances are probably about right.
The drive time for me to go to a Devils game is about half an hour longer than a Flyers game (Both drives are over an hour for me) and driving to Philadelphia is a hell of a lot less stressful.