Just an update...I had an A/C guy come out Saturday (company from Angie's List, not Big Bear or any of the commonly recommended guys here) and he was basically useless. He would hardly even look at what I wanted him to and was really no help. I called an air duct cleaning company yesterday morning that got great reviews on Angie's List and the owner came out yesterday. He doesn't think I have a mold problem at all and I probably did overreact. He said he sees my particular issue a lot. Part of the reason I freaked out is that I looked at one of the air returns in the house and it was black inside. I didn't realize that they are painted black, so on initial view, I though it was covered in mold/mildew. Duh! Not the case. When I took the cover off, I could then see it looked like spray paint. My ducts are extremely dusty, though, and definitely need a cleaning. This has been a good learning opportunity for me on A/C vents, returns, etc.
As for the vent that originally started the whole panic, it's in the master closet, closest to that A/C unit in the attic. That unit only cools the master bedroom, bath and closet and I like to sleep with it really cool, so I have been running that unit as low as 70 at night. It is possible all that cold air, mixing with warm air from the attic and duct, along with the closet door being closed all the time caused the issue. In reality, it was a very tiny amount of mold/mildew and the bleach/water solution seemed to take care of it.
I am planning to sell the house soon so I can buy something smaller and easier to take care of, so I just didn't want any issues that would make it hard for me to sell, hence the freak out. I will be having the dirty air ducts cleaned this week and hopefully that will help take care of the problem. Thanks for the responses/advice.