4 years of college athletics; do years you DONT play sports in college count towards the 4 years?

just wanted to know, because i always wanted to play college basketball, but i keep hearing you have only 4 years of college sports, so does that mean that if you go to college for 2 years, but dont play any sports during those two years, you wasted 2 years out of the 4 years your allowed to play??