there is no need to stop handling your birds!
You will often see pictures of top flyers with birds sat on there shoulders, hands etc because the birds have a bond with the owner as well as the loft etc.
Quiet, calm birds are always happier.
Handle the birds well as babys and continue to handle them to keep them calm and never afraid of you, don't take birds of eggs etc just to handle them but pick them of the floor, perches etc.
When you catch them to basket them they will be much calmer.
Another good tip is to basket train your young birds when they are baby's so that they are used to being in a basket otherwise when you do basket them they won't know whats going on and will be stressed!
Phil