Let each one remain with his child, etc. Note how much effort they had in working, who also continued in labor at night. Thus the Apostle did, who worked night and day with his hands, so as not to burden any of the believers by seeking a livelihood (I Cor. IV). Unless perhaps they are to be believed to have succeeded each other in turns working, so that some worked by day on the wall, and others kept night watch against the attacks of enemies. For even the builders of our city do both at the same time. And those same ones who build up the Church by instructing the faithful, also drive away unbelievers and contradictors by reproving them from injuring the Church.
Source: Commentary on Ezra and Nehemiah