Stock front speakers don't always work. how do I fix them?

I own a 97 Toyota Camry xle and my front speakers work on occassion but when I hit a speed bump or the song I'm listening to has too much bass the front speakers stop working. I took off the door panel and checked for any loose connections and they were all good. I found that when I push on the soft outer black part of the speaker it works. 5 minutes (or less) after I let it go it stops working. Does anyone know how to fix this problem? Hopefully I don't have to shell out $100+ for new speakers...