Sweet peas are perfectly safe to touch, but the seeds are poisonous if you eat enough of them. Unlike, say, monkshood, which can affect you just from handling it.
Yes, we are careful around kids. This is because town-bred kids (or stupid ones!) will try to eat anything, either because they think it'll make them look cool to eat something poisonous, or because they mistake it for something edible.
In my own experience, the former is represented by a kid at my primary school who ate laburnam seeds (luckily the witnesses got him straight to hospital and they were out of his stomach within half an hour; since he'd not chewed them the only ill effects were from the stomach pump and the doctors chewing his ear off).
The latter, mostly younger kids, have included (in my own acquaintance) ragwort, yew seeds, tulip bulbs and datura flowers. In the last three cases, the kids were OK because someone saw and hooked the stuff out of their mouths before they'd swallowed much (the yew was just one berry), and then made them sick.
The ragwort showed a spectacular dermatitis with some scarring and a low fever, but wasn't otherwise ill (the stuff is pretty bitter, so I think he spat it out himself; the inside of his mouth was a mess, but no further back than the hard palate). Alarmingly, his mother couldn't understand why we were getting so panicked about "pretty yellow daisies"; when I volunteered to come round her house and weed the damn garden myself she took the point.
As with almost everything else, sensible kids (and parents) need no protection, but the majority of people are both stupid and ignorant, with the particular "it won't happen to me" delusion, so they need protecting from themselves (because dealing with the consequences of accidental poisoning in a child, even if the parent is a blithering idiot, is terrifying).