Hi Kathie
Check the feed bag, it should advise you how long to keep going with their particular brand, if not contact the company or maybe your supplier. I've raised my ducklings on organic chick growers pellets mainly because they were brought up with the chicks and they all have to eat the same, but they are starting to visit the goose food dish, which is fine because it is for all waterfowl, they are about 6-7 weeks old.
I wouldn't let them have the layers pellets, I called the Smallholder feed company when I had the exact same thing happen with my goslings, and it's really important to make sure they have their own food while developing because the levels of protein in particular will be different. In young ducks/geese the wrong nutrition can lead to problems like angel wing.
I've got a little routine where everyone gets fed first thing in their own respective parts of the enclosure or in the house/run. Then a bit later in the morning I pick up all the food that hasn't been eaten and open the gates between their areas so the chickens, chicks, ducklings and geese can all mix.
It's a bit of a faff but it mostly works, we purposely divided up the poultry area with permanent mesh and posts to be able to do this. Obviously there are times when the chickens fly over or the little ones run under the gate but on the whole they are all hungry enough in the morning to eat their own food first, and aren't looking for anything else til it's corn o'clock!
Hope this helps! good luck
Helen x