Something to try: Figure out when your male flowers typically first open for the day, and hand-pollinate soon after that time.
In addition to bees, ants and other insects eat the pollen. If you hand-pollinate late in the day, this might explain the lack of pollen in the male flowers. Another trick you can try if your male flowers are suffering from (ahem..) "low pollen count", if there is at least a little pollen on the anther, just snap it off and drop it into the center of rhe female flower. Insects moving around inside the flower may distribute the little bit of pollen available.
I hope some of the replies here provide an anther to your problem!