We describe new boundary conditions for $AdS_2$ in Jackiw-Teitelboim gravity. The asymptotic symmetry group is enhanced to $\operatorname{Diff}(S^1) \times C^{\infty}(S^1)$, whose breaking to $\operatorname{SL}(2, \mathbb{R}) \times U(1)$ controls the near-$AdS_2$ dynamics. The action reduces to a boundary term which is a generalization of the Schwarzian theory. It can be interpreted as the coadjoint action of the warped Virasoro group. We show that this theory is holographically dual to the complex SYK model. We compute the Euclidean path integral and derive its relation to the random matrix ensemble of Saad, Shenker and Stanford. We study the flat space version of this action, and show that the corresponding path integral also gives an ensemble average, but of a much simpler nature. We explore some applications to near-extremal black holes.