Robots.txt - SEO Tips

[rosegarden]

I will recommend that everyone installed a robots.txt to block duplicate content, I am still improvising mine



=======================================

User-agent: *
Crawl-Delay: 25
Disallow: *.js
Disallow: /backup/
Disallow: /includes/
Disallow: /cron/
Disallow: /libs/
Disallow: /plugins/
Disallow: /tmp/
Disallow: /print.html
Disallow: /my-messages.html
Disallow: /my-profile.html
Disallow: /my-favorites.html
Disallow: /my-listings.html
Disallow: /my-packages.html
Disallow: /payment-history.html
Disallow: /payment.html
Disallow: /listings-by-field.html

Sitemap: http://www.xxx.com/sitemap.xml

[rosegarden]

Any feedback on whether to

Disallow: /*?sort_by=

[Peter Jonsson]

Also wondering if i gonna add Disallow: /*?sort_by= also.

Did you add it ?

Also i have a question. What does * do in the robots.txt ?

[Milea Ionut]

This is a very good idea.

[Peter Jonsson]

What page are listings-by-field.html ?

[Nguyen Dat tai]

Disallow: /admin/

[Fred Moyse]

A rel="nofollow" should also be added to categoryFilter links.