Lemon tree - lacks lemons

We have had a lemon tree since early spring - it has flowered constantly, but the baby lemons all just shrivel up and fall off. I moved it outside in summer, re-potted it and gently pruned it, as demonstrated by Monty on GW, it has been looking really healthy, lots of new foliage, lots of flowers, but as before, apart from one, every single lemon has dropped off. The one that has grown at all is now about the size of a small brazil nut, and appears to have now stopped growing as well. I have been feeding it every 10 days with citrus food, as per the instructions.
Is this normal or am I doing something wrong?