User-agent: MJ12bot Disallow: / User-agent: TwengaBot Crawl-delay: 10 # maximum rate is one page every 10 seconds Visit-time: 0100-0600 # only visit between 01:00 and 06:00 UTC (GMT) User-agent: * Disallow: /shopreports/ Disallow: /admin/ Disallow: /catalog/ Disallow: /customer/ Disallow: /files/ Disallow: /include/ Disallow: /log/ Disallow: /mail/ Disallow: /modules/ Disallow: /payment/ Disallow: /promozioni/ Disallow: /provider/ Disallow: /reports/ Disallow: /schemes/ Disallow: /save/ Disallow: /shipping/ Disallow: /shop2/ Disallow: /shop_orig/ Disallow: /shopreports/ Disallow: /shopreports/ Disallow: /skin1/ Disallow: /skin1_original/ Disallow: /Smarty-2.6.3/ Disallow: /Smarty-2.6.9/ Disallow: /sql/ Disallow: /templates_c/ Disallow: /upgrade/