Robots.txt optimization for search engine crawling – Complete Analysis