If you permanently delete something, it will not create a .Trash folder and put it there. To do this in Nautilus (the default file browser in Ubuntu), simply hold the Shift key while pressing delete. This will bypass the Trash, which is similar to the Recycling Bin in Windows and allows undeleting.

You can also add a Delete right-click command which bypasses trash in Nautilus. To do this, open Nautilus and select Edit -> Preferences. Then click the Behavior tab, and select "Include a Delete command that bypasses Trash".

Also, as Uri Herrera mentioned, you can empty the trash to remove it.