Should I contact CPS?

CPS removed my son's half sister (age 6 months) and her brother(age 7) from my ex-wife's (and her boyfriend's) home because it was determined to be an unsafe environment. Our 2 year old son was with me for visitation and was not part of the removal. Now she wants him returned, as scheduled in a few days. The other children are not being returned any time soon because of the issues in the home and I do not want our son to be in the unsafe environment. Should I contact CPS to make sure they are aware of the fact the he also lives in the home? Should I simply not return him as per the visitation agreement? My ex will not give me the name of the CPS caseworker or any information as she tells me it is not my business.