Would a deaf spectator still think your act was funny?
If an act is funny, the answer should be "YES".

I wonder what is it that I'm not doing that you should be doing?

HaHaaa! Gallaudet kids on Spring Break. That was a learning experience.
A friend of mine was a deaf busker. He did really well that week.

I "pitch" or demonstrate/sell magic tricks part time and since being in a tourist destination, I get all kinds of crowds. Recently I dealt with a group of deaf high school students. I pitched the visual stuff and just had fun doing it, they got caught up in it as well and had a great time, though didn't sell anything..hey, I tried.

The elderly in audiences today require us to deal with this. Keep it visual!

Bob Sanders
Magic by Sander

I perform at nursing homes all the time where the residents are either hard of hearing or completely deaf. They always have a good time watching the show and I've been asked back several times. I didn't even realize they couldn't hear till after my second show I commented to the administrator that my jokes must have been a little weak because no one really laughs, that's when she told me, "they probably don't hear or understand a single word you say, most of them leave thier hearing aids in the drawer."