I've only one Whitney once. Stayed at Lone Pine Lake on Friday, woke at ~3:45AM Saturday, summited (where I poo'd), hiked back to my truck at the portal, and was back in Santa Barbara, having a celebratory beer around 9pm. I don't think it's terribly difficult, just long, and easily doable in a day, but I felt the portal to LP lake was the most difficult part, so waking up with fresh legs was nice. On a side note, I stayed at LP Lake to avoid the marmot hell at base camp and upper reaches. I didn't see any where I stayed. I got in the lottery this for this Summer on opening day, and I'll probably stick around Mirror Lake which I thought was a bit prettier. Maybe one day at LP and one day at Mirror, or extend the trip and go up and over to JMW and see Guitar Lake etc.
I do a LOT of southern sierra backpacking (Sherman Pass area, Domeland Wilderness, far upper reaches of the Kern and Little Kern rivers, Golden Trout Wilderness) and I usually see very few if not zero people.