The more flexible method of bypassing the recycling bin is to simply hold SHIFT while you delete your selected item(s). This will prompt you to choose whether you want to permanently delete the item(s) or not. Simply click "yes" and the action will be completed.

I find this method much more convenient as sometimes I do not want my files to bypass the recycling bin.