Am I wrong for thinking my son’s birthparents are being unreasonable?

My Husband and I adopted our son 3 years ago. He was pretty much just abandoned at our door step by his parents. Now his parents have moved back into town and my mother-in-law decided it would be ok to let them rent her home which is technically in our back yard! At first I was really ok with this but now it is becoming a problem due to the fact that every time I as much as call him Son they get their feelings hurt and decide to tell my husband or mother-in-law that they need to tell me to keep my mouth shut. Our son is 16 years old and is doing great and everything has been so smooth since we took him in until the past month. I never have tried to make them feel uncomfortable or upset them intentionally but apparantly I have. I feel like it should not offend them that my son and I have a good relationship. Am I wrong for feeling this way? How do I manage to live so close and have them so involved and it still be ok? Please Help Me!