I thought that when the flowers die, that's a sign that the potatoes are ready to dig up. So how would you know when they are ready to dig up if you have removed the flowers?
Ideally, you should have noted when you planted the seed tubers, then leave it an appropriate length of time from then depending on the type of potato (I heard on Gardeners World last week that it's 90 days for first early varieties, 100 days for second early varieties, and 120 days for maincrops - but have heard different timescales from other sources, although this gives you an idea). Then you stick a fork or hand in and have a rummage