I made a bootable USB pendrive which works just like a live CD. When I use it and make changes, for example, install an application, it does not save the changes I make on next reboot. Is it possible to do something about this?

UPDATE: I have followed the instructions in kingmilo's comment. Eventually, I ended up using "startup disk creater". Which is a great application. I decided to give 2.6 G as storage....but I faced the following problem:

Basically, I start up with the "Try Ubuntu" options...and everything gets saved (applications, wallpaper, music), but I need the extra space to work. How can I make it work in the easiest way?

UPDATE2 it is working now with kingmilo's help/comment and some other info on properly partitioning and formatting pendrives.