The Berezinskii-Kosterlitz-Thouless (BKT) phase transition has long been sought yet undiscovered directly in magnetic materials. Here, the authors identify two phase transitions with BKT fluctuations detected by NMR and critical scaling behavior in magnetic susceptibility expected for the BKT transition in a frustrated magnet TmMgGaO4.