I think Vernon was turned off at Houdini's constant, relentless self-promotion. I also read that T. Nelson Downs, who was a friend of both, was also turned off by this, but then again, that's part of what made Houdini such a huge success, his marketing genius.
Anyway, this is an often told story, but since nobody has told it here, I will. Dai Vernon once fooled Houdini seven or eight times in a row with a simple version of "The Ambitious Card". Houdini always boasted that he could detect any magic trick if he saw it done three times in a row. Houdini's agent finally said, "Harry, let it go --- you're fooled!" But he never would admit it. He kept saying that Vernon was using a duplicate card, even though he had signed his initials on it. When Vernon pointed that out, he said that Vernon must have some kind of thumb-writer that used ink. Vernon was actually using a double-backed card to help with the trick. Houdini's wife Bess later wrote Vernon a letter begging him to tell Houdini how he'd done the trick because he had stayed up all night trying to puzzle it out and had nearly driven himself and her both crazy in the process.