Baby keeps waking up with own flailing hands?

He's over 4 months now and we're phasing out the swaddle...we've got one arm out now. When he's asleep, his arm flails all over his face...hitting himself...pulling his ear...it eventually wakes him up and he cries. Will he stop doing this? Are we in for double the frustration when we let the other arm out?