You recall the Venerator had been holding the book throughout the battle and he appeared to draw his sorcery from it. Perhaps the book itself is what holds power over these villagers?