The real reason Lenore and Bryan's marriage ended
Here is my theory on why Lenore hated Bryan so much and didn't want him around Kim.
First off I think Bryan wasn't that bad of a father. Remember he always came back for Kim's birthdays even to the point of being sent to Alaska once for punishment for doing that.
My theory is that it was Lenore/Lennie who had a problem with being the wife of a CIA field operative. Bryan was gone a lot which comes with the job but he did try to be a good dad by not missing Kim's birthdays. I think that as he got more senior in the CIA, Lennie hoped that Bryan would accept a billet stateside behind a desk in Langley perhaps. Or maybe Bryan might become a station chief in a safe country like France, where the Mills would hang out with Jean Claude and his wife.
I think that Bryan kept refusing to leave the special operations side of the agency. I have a feeling that Sam brought up leaving the agency to Bryan before while he was still married but Bryan decided to stick with the agency.
So I believe that if Bryan had took an admin position at the agency or went into the private sector Lennie would have stayed.