The normal sperm count ranges from 20 million to 100 million sperm per cubic centimeter. Counts below 20 million may be a cause for difficulty in conception. Factors such as fatigue, excessive alcohol, smoking or other drug abuse, prostate gland infection, poor diet, or occupational exposure to chemicals can cause a small, temporary decrease in the sperm count. Varicocele is a condition when there are varicose veins on the scrotum. This is another cause for infertility and can only be corrected through surgery.