Open up Firefox and its preferences window. Go to the "Applications" tab and you should see "magnet" on the list. Click on "Always ask" and if Transmission does not appear as an option then click on "Use other" and navigate to /usr/bin on your file system. Look for "transmission" or "transmission-gtk" and click that.

Update: Firefox has changed so the above no longer works. Instead, when you click on a magnet link, a "Launch Application" window will appear, from which you can choose to always associate an application (make sure to check the checkbox at the bottom to always use the chosen app). To find the torrent app, choose "file system" on the left side, and then navigate to /usr/bin as described above.