What is “Spider Trap”, A Simple Definition

A spider trap is an infinite loop that a spider may get caught in if it explores a dynamic site where the URLs of pages keep changing. For example, a home page may have a different URL and the search engine may not be able to ascertain that it is the home page that it has already indexed but under another URL. If search engines were to completely index dynamic web sites, they would inevitably have large amounts of redundant content and download millions of pages.

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.