We're always or almost always 5. 3 kids, 2 adults. It's hard/impossible to make sure everyone is happy at all times. I usually take Anders and myself out of any equation and think "1/3 happy, always something".
As long as 3/3 aren't discontent (that often happens too), I can manage.