Ten. Gzip Compression: Enable Gzip compression on the server to reduce the size of data files transferred amongst the server as well as the person’s browser. txt file is then parsed and can instruct the robot regarding which webpages are not being crawled. Like a online search engine crawler may https://www.youtube.com/watch?v=p3IfcO3MzMs