A warning has been issued to trespassers who continually risk breaking into an abandoned asbestos factory site.