It’s a balance. You want enough signal per frame to permit accurate alignment/tracking of motion, but without excess motion within each frame.
I usually split EER frames so that dose rate is 1 e-/frame in CryoSPARC. Deviating much from that doesn’t make an appreciable difference.
Also note that RBMC can re-split the frame grouping so if you think you have excessive drift which can be corrected, you can always try later in the processing.