The main thing is not to scrimp on the quality of ink and stamps. When I first started, I bought the cheap stuff thinking it couldn't be that different. Well, it is. And now all that stuff is just junk to me and I donated it so didn't really save me money.
I have all the colors of Stampin Up classic ink pads. I got lucky and bought an entire set of all colors and the storage caddy for them all on ebay for around $220 shipped. They were used but almost all were in good condition and I have just bought refills whenever my demonstrator has a promotion. (I buy them in the set since it's even cheaper that way).
I also no longer use the inexpensive clear stamps. I have noticed a HUGE difference when using quality acrylic stamps. (photopolymer ones are the best clear stamps). I also convert my rubber stamps to clear mount bases using EZ Mount foam.
I recently made the investment in the Stampin Up acrylic blocks since they just came out with clear mount systems. Their blocks are awesome and have a great wedge for holding them. I bought the whole set since it was cheaper and my demonstrator offered a discount on any one item last month. That's when I always buy my big ticket items. As soon as I tried them at a class, I knew I had to get rid of my old blocks since they were so much better.
Once you find a brand you like, try to stick to it since you can match everything up. Most of the stampers that I know and whose blogs I follow also use Stampin Up. I love that all their stuff coordinates (paper, ink, ribbon, accents, markers, etc). It's almost all I buy now.
I will eventually buy some craft ink pads too, other than the white/ivory ones I have. These can be used for colored heat embossing as well as stamping on darker papers.
Let me know if you have any other questions.
http://sweetteetle.blogspot.com/