I don't know what triggers the spawning, so be prepared to accept that it's based on the structure location and not some piece of it that you can tear down. This is how it is for ocean monuments, which is the only place where guardians can spawn.
You can tear down the pillager tower and they will still spawn. Your best option to stop them spawning would be to spawn proof the area around the tower. As others have said, this is a 71x71x71 cube around the tower.