It is preferable to set expectations regarding inspections and site access with the builder up front. If possible, have the right to conduct inspections at each stage included in your build contract.
Because the work site and the safety of all those on it remains the responsibility of the primary building contractor until final payment, any independent building inspector would require the builder’s permission for access.
Besides, as a matter of general courtesy this is good practice to maintain harmonious relationships all round which in turn are very important for a happy building project outcome.