It's been a long time but I always rode large fins in my m80 because like you I found it a little too loose and lively with medium fins. I ran large AM's and would sometimes switch out the center fin for something smaller when the waves were weak. A larger, raked fin like the AM's should help draw out your turns and settle things down.A bigger anchor, who knows
Maybe I give the medium ARC sides with PC5 mid a go.
Any other fin guru's or Coil M80 riders who can share some setups?