The way you have described the fruit, suggests brown rot. You need to destroy all the shrivelled fruit from the tree and any on the ground. You will need a fungicide to contain the problem this year. Check the fruit very often, you will know you have it when the fruit sticks together.