-
June 23, 2012, 12:41 AM
#1
Senior Member
Robots.txt - SEO Tips
I will recommend that everyone installed a robots.txt to block duplicate content, I am still improvising mine
=======================================
User-agent: *
Crawl-Delay: 25
Disallow: *.js
Disallow: /backup/
Disallow: /includes/
Disallow: /cron/
Disallow: /libs/
Disallow: /plugins/
Disallow: /tmp/
Disallow: /print.html
Disallow: /my-messages.html
Disallow: /my-profile.html
Disallow: /my-favorites.html
Disallow: /my-listings.html
Disallow: /my-packages.html
Disallow: /payment-history.html
Disallow: /payment.html
Disallow: /listings-by-field.html
Sitemap: http://www.xxx.com/sitemap.xml
Last edited by rosegarden; June 23, 2012 at 01:05 AM.
-
June 23, 2012, 01:05 AM
#2
Senior Member
Any feedback on whether to
Disallow: /*?sort_by=
-
December 9, 2012, 06:31 PM
#3
Master
Also wondering if i gonna add Disallow: /*?sort_by= also.
Did you add it ?
Also i have a question. What does * do in the robots.txt ?
-
January 16, 2013, 01:01 PM
#4
Member
This is a very good idea.
-
January 16, 2013, 01:39 PM
#5
Master
What page are listings-by-field.html ?
-
June 8, 2013, 10:22 AM
#6
Senior Member
-
January 14, 2014, 07:03 PM
#7
Junior Member
A rel="nofollow" should also be added to categoryFilter links.