Olive Tree

I have an 8 year old olive tree growing in a large pot. For the past two years the tree has produced some quite small olives. It is now covered in small white flowers and I am wondering if I should take some of the buds off, so that I might get bigger fruits. Also a fellow allotmenteer said that olive trees best planted in the ground, not in  pots!

The size of the fruits depend on variety rather than how many set fruit.
Small olives are for oil, big fat olive varieties are bred for curing&eating.
I don't think you can change one into the other.