Anton:
My job involves regulation of cleanup of hazardous waste sites, so I am familiar with many types of chemicals from that perspective. "Solvents" can cover a lot of materials. I wouldn't be too worried about acetone or ethanol, for example, but I would be more concerned about other solvents, such as tetrachloroethene. Are there identifying marks on the barrels, with numeric codes (such CAS number), names, hazard warning labels, or transportation safety information related to the former contents? If yes, photograph them, I may be able to help you figure out what was in the barrel.
I wouldn't go sticking my nose in a barrel that formerly held an unidentified solvent, but if you detect a solvent odor when the barrel is open, it may not be completely decontaminated.
Many solvents are volatile, some could potentially evaporate if given enough time exposed to open air. Others (especially petroleum solvents) may be tougher to get rid of.