PDA

View Full Version : Robots.txt - SEO Tips



rosegarden
June 23, 2012, 12:41 AM
I will recommend that everyone installed a robots.txt to block duplicate content, I am still improvising mine



=======================================

User-agent: *
Crawl-Delay: 25
Disallow: *.js
Disallow: /backup/
Disallow: /includes/
Disallow: /cron/
Disallow: /libs/
Disallow: /plugins/
Disallow: /tmp/
Disallow: /print.html
Disallow: /my-messages.html
Disallow: /my-profile.html
Disallow: /my-favorites.html
Disallow: /my-listings.html
Disallow: /my-packages.html
Disallow: /payment-history.html
Disallow: /payment.html
Disallow: /listings-by-field.html

Sitemap: http://www.xxx.com/sitemap.xml

rosegarden
June 23, 2012, 01:05 AM
Any feedback on whether to

Disallow: /*?sort_by=

Peter Jonsson
December 9, 2012, 06:31 PM
Also wondering if i gonna add Disallow: /*?sort_by= also.

Did you add it ?

Also i have a question. What does * do in the robots.txt ?

Milea Ionut
January 16, 2013, 01:01 PM
This is a very good idea.

Peter Jonsson
January 16, 2013, 01:39 PM
What page are listings-by-field.html ?

Nguyen Dat tai
June 8, 2013, 10:22 AM
Disallow: /admin/

Fred Moyse
January 14, 2014, 07:03 PM
A rel="nofollow" should also be added to categoryFilter links.