Web page optimization
The robots.txt creator helps you to create a robots.txt file for your website. The robots.txt file can be found in the root directory of your web server. It is a plain text file that should be as simple as possible. A robots.txt file can help you to exclude pages from search engines.
To start the robots.txt creator, click the ‘robots.txt creator’ link at the bottom of the Robots.txt page in the website audit tool in SEOprofiler. You will see the following:
Go from step 1 to step 4 in the robots.txt creator and then copy the text to your robots.txt file. If you need further information about an option, move the mouse pointer over the question mark icons on the page.