We analyze the impact of a proposed tidal instability coupling p modes and g modes within neutron stars on GW170817. This nonresonant instability transfers energy from the orbit of the binary to internal modes of the stars, accelerating the gravitational-wave driven inspiral. We model the impact of this instability on the phasing of the gravitational wave signal using three parameters per star: an overall amplitude, a saturation frequency, and a spectral index. Incorporating these additional parameters, we compute the Bayes factor (lnB_!pg{\^}pg) comparing our p-g model to a standard one. We find that the observed signal is consistent with waveform models that neglect p-g effects, with lnB_!pg{\^}pg=0.03_-0.58{\^}+0.70 (maximum a posteriori and 90\% credible region). By injecting simulated signals that do not include p-g effects and recovering them with the p-g model, we show that there is a ≃50\% probability of obtaining similar lnB_!pg{\^}pg even when p-g effects are absent. We find that the p-g amplitude for 1.4  M_⊙ neutron stars is constrained to less than a few tenths of the theoretical maximum, with maxima a posteriori near one-tenth this maximum and p-g saturation frequency ∼70  Hz. This suggests that there are less than a few hundred excited modes, assuming they all saturate by wave breaking. For comparison, theoretical upper bounds suggest ≲10{\^}3 modes saturate by wave breaking. Thus, the measured constraints only rule out extreme values of the p-g parameters. They also imply that the instability dissipates ≲10{\^}51  erg over the entire inspiral, i.e., less than a few percent of the energy radiated as gravitational waves.