He did use it only 3 times. I believe the third time was on Macubex. The remainder... must have been a show of his tactical genius. You see, Macubex knew that Ban can only use his evil eye power once in 24 Hrs. He didn't know it was 3 times. Hence, Macubex was easily fooled in the end.
...
At least that's what I remember of it...