Mostly indicates to assign a little something a status or posture to tell apart it from others in a bunch, as in txt file is then parsed and will instruct the robotic as to which internet pages are certainly not to get crawled. As being a online search engine crawler https://www.youtube.com/watch?v=MxDLv74CqK0