I found the man of my dreams ...I am frightened?

About two months ago a met a guy.I'm 24 and he is 19. My family and friend have given me **** about how young he is and at some point it really mad me feel bad. I am a College graduate. Just recently actually and i moved back home with my parents. This town that i live in is a TRAP OF racists cops confederate flags, and lots of individuals who havent learned to think for themselves or to explore the world. Quite simply put, this 19 yr old has been the apple of my eye since we've met. He knows how to treat me. He says what he means and he is always honest no matter how much it may hurt. He is street and book smart. On the other end of the spectrum....he has one pending charge(drug possesion) and he lost his last job for fighting. His life plans right now are school....mine are grad school and getting the hell out of this town. When he says he wants to leave, I dont believe him. Have I fallen in love with someone who may hold me back? How hard would it be to shoulder both of us getting out of this town when I havent even gotten myself to the place i want to be? Is this all overrated ( the whole love, man of my dreams thing??) I have no support from my family simply because they just don't care what happens in my life....One of my biggest fears is to fall in love HERE, get pregnant HERE,( which i feel as though I may be already) and stay HERE. I know its up to me but HOW? How do I channel the drive, intelligence and pion that I have for life away from this 19 yr and loving him towards moving forward in MY own life?? do i even have to? Is he a red flag ?