I don’t want birthmom’s boyfriend at visits. Am I in the right?

I am an adoptive mom of a 6 year old boy and I am maintaining an open adoption with my son’s birthmom. He was removed from her home due to neglect 5 1/2 years ago and, eventually, adopted by me by the time he was two. We now live in a different state from her and have been having two weekend visits per year. Either she travels to our house, or we meet her for a weekend somewhere like an amusement park, beach, etc. She now has had another child, a two year old son and she is raising him with the child’s father. The last couple times she has asked if she could bring her boyfriend (the other “baby’s daddy”, not my son’s father) with her. I have said no for a couple reasons. 1) This visit is not a family vacation, it is a time for her to visit my son. I allow her to bring her other child because he is my son’s half brother, and it would seem unreasonable to ask her to leave him at home. 2) The boyfriend in question is an ex-con, went to prison for auto theft. She is upset at me because I will not allow her boyfriend to come on these visits, and I think he is upset too, but these visits are paid for by me, and I am doing them for my son to be able to maintain a relationship with his birthmon, and I feel her boyfriend’s presence would significantly change the dynamic. Not to mention my concerns for safety, since he is a convicted felon. What are your thoughts on this? Am I doing the right thing?