Family n Boyfriend Problem HELP!!!?

Okay so ive been with my Boyfriend, fiance, Baby Daddy (whatever category) for about 5 yrs already. You can say his like my High School Sweet Heart.. We live with my parents and my dad never really seem to like him.. My dad always tells me his worthless, not GOOD ENOUGH, and his always finding things to say just negative thing about him. And my whole Family including uncles and cousins and aunts etc. get together and talk shyt about him.. i really don't know waz up with that.. to me sounds like they are just hating, and avoiding the fact that their lives are just as Fcked up as any bodyelses. They make my boyfriend seem likes his a bad, terrible, unworthy, and dishonest person. But they don't know only because he had a really fcked up life growing up and His parents were always going in and out of jail, they think his gonna be the same. my BoYFRIEND grew up with 6 sibling, he has 2 older brother and 3 younger sisters. his the middle baby. His parents were in the Drug Business and for that reason they both kept going in n out of jail until the CPS took them away from each other, when he was 13 yrs old he kept getting foster parents that really didnt give a shyt about the kids only the money and he saw that so he became rebellious and started doing drugs n getting into fights going in n out jail too.( the rest was history) after i met him in my senior year we really didnt last as friends and we started going out like 4 days after we met. then he moved again and started a long distance relationship for about 2 years, then we move in together, and he was still the same but not quite, he had settle himself and started working, but he still kept smoking weed, thats what my family dosent like aboout him, and since he still had shyt pending in other counties the police came and took him to jail, i had to move back to my parents. Then he came back and started living with me and my parents, after that happen my dad did not like him for nothing and starting talking bad about him to my family and from there they started disliking him. We now have a 6 month beautiful baby girl, and he is a wonderful father, and a great partner, he make both of happy. his not working at the moment because he one of my cousins started talking shyt to their boss about him and started picking on him like he was a red target. See my family really dosent want to admit the fact that we have family members that do worst then my boyfriend. i have a cuzin that smokes weed has no job lives with his parents his 18 and has his baby mama living there too, and she has no job smokes weed and lives her baby with her granma so she can go a party and smoke weed all the time but they dont see nothing wrong with that.. My boyfriend is the bad person.. idk wats the problem but is causing problems with our realtionship because we are always arguin about the same thing. Can ya give me some advice. im running out of options here.. we really working on getting an apartment together but i can not afford it since im the only one working...