In early days internet users were very less and amount of available information was relatively very small. Accessing of internet resource was restricted to very few employees and used to access scientific information. In previous days finding information on internet was not critical as it is now. Site directories are the first method used to access information on the internet. Links to these resources are grouped by topics.
Now a days such link Directories have lost much of their popularity. The reason is for example modern web directories with lots of web resources only provide information on a tiny fraction of the internet. The largest directory on the internet is currently DMOZ (OPEN DIRECTORY PROJECT). It contains information of about 5 million Resources, if we compare this database with Google search engine database it is very minimum or less as Google has more than 8 billion documents.
Currently, there are three major leading international search engines ? Google, Yahoo and MSN Search.? These 3 search engines maintain their own database and search algorithms. Many other search engines use these three major search engines for originating search results and the same seo expertise can be applied to all of them.
The Spider+Crawler+Indexer component groups are implemented as a single program that downloads web pages analyzes the web pages and then uses their links to find new resources.? How ever the above combinations are inherent to all search engines and the principles of seo are the same. However, the components listed are inherent to all search engines and the seo principles are the same.
Architecture Of search Engines
Spider – a browser-like program that downloads web pages.
Crawler ? a program that automatically follows all of the links on each web page.
Indexer – a program that analyzes web pages downloaded by the spider and the crawler.
Database? storage for downloaded and processed pages.
Results Engine ? extracts search results from the database.
Web server ? a server that is responsible for interaction between the user and other search engine components.