Is it possible that your script is one you got with the routine? It's possible you don't sound or feel natural when you're saying it because it doesn't fit your style. Try writing your own script if this is the case.

I had the same problem when I started but with sleights instead of patter. I felt guilty it was like the specs knew what I was doing because I knew what I was doing. Try picturing your audience in front of you or keep reading your script not just until you have it fully memorized but when saying it becvomes second nature and you can say it without thinkin of it.

I feel patter isn't something you can write or practice. If you rehearse and stick to a script to much, you may miss opportunities during live performances that could come to you, and be better than anything you can practice.