GH controllers can be used with Rock Band, but Rock Band controllers cannot be used with Guitar Hero.
If you are thinking about picking up just a game you can also do that, but remember that the cheapest way to get everything, by far, is to buy the bundle. The bundle only comes with one Guitar, so if you ever have a bunch of friends come over only you would have enough.
Also if you are looking for the Guitar playing aspect of Rock Band only, stay with Guitar Hero 3, because that was designed just for that. Rock Band is designed to have four friends playing at all times.
My plan is to pick up a bundle and make my friends pay for the intrusments. I would pay 60 for the game and we would split the 110 remaining three ways for the instruments.