Indexed Internet Software

Robot-Manager

Website Management Tools

Manage and track visits by robots, spiders and crawlers to your website. Identify which spiders visited, when they visited, and what pages they indexed. Direct spiders to your most important pages, and exclude them from non-public pages. Easy to use

Keywords: robots-txt robot spider crawler log file log file analysis log file analyzer search engine optimization seo spider tracking referrer logs identify which spiders visited website pages indexed

Trellian SiteMapper

Trellian

Site Mapper will analyze the contents of a web site, and create a detailed map with an indexed listing of all resources by page and category. It will also validate all links, so that users need never come across 'File not found' errors and broken

Keywords: trellian sitemapper sitemap website validate link validate site map site map creates detailed map indexed listing resources