It depends. People vary in their sensitivity to mold, both as to amount and type needed to cause reactions. For some, a relatively small number of mold spores can cause health problems. For others, it may take much more.
The basic rule is, if you can see or smell it, take steps to eliminate the excess moistureand get it tested.