Fiber-based single-frequency amplifier, e.g. for gravitational wave detectors, are typically limited by the nonlinear effect of the stimulated Brillouin scattering (SBS). In addition to a decreasing output power level, SBS also initiates a broadband intensity noise in a frequency regime up to several MHz. Here, we report new theoretical and experimental studies, which suggest that the conversion from phase to intensity noise by an asymmetric Brillouin gain spectrum is responsible for the excess intensity noise.