I don't drive but if I did I'd always try and avoid giving one child a lift. There are ways around it such as giving a lift to another Leader or another parent and their child or even just two siblings, and dropping the lone child off first. However ultimately if you are think that the risk of being accused of acting inappropriately or even just driving dangerously or whatever is low enough for you to be ok with then it is your risk to take. Also if you do drive a lone child IIRC the advice is to put them in the back seat.
Also bear in mind the rules about having child safety seats (even if one off lifts might be arguably except) it would still be your liability if something happended and they weren't in a safety seat when they maybe should have been.