Search engine bots (sometimes called spiders or crawlers) are computer programs(bots) that crawl the web pages. In other words, they visit webpages, find links to further pages, and visit them. Often they map content that they find to use later for search purposes (indexing). They also help developers diagnose issues with their websites.
As we know, the ever-growing usage of JavaScript on the web is certainly beneficial to users, but rendering JS is a challenge for search engines. If your website is not properly handled by bots, or your content changes frequently, you should render your pages dynamically, and serve rendered HTML to crawlers instead of JavaScript code. Thus, in order to do so, you have to know if a request was made by a real user, or by a crawler(search engine bot).
PHP doesn’t have any built-in function to detect search engine bots. However, the following function can be used for this purpose.
Example:
<?php function is_bot($system) { // Bots list $bot_list = array( 'Googlebot', 'Baiduspider', 'ia_archiver', 'R6_FeedFetcher', 'NetcraftSurveyAgent', 'Sogou web spider', 'bingbot', 'Yahoo! Slurp', 'facebookexternalhit', 'PrintfulBot', 'msnbot', 'Twitterbot', 'UnwindFetchor', 'urlresolver' ); // If it is search engine bot // returns true, else returns false foreach($bot_list as $bl) { if( stripos( $system, $bl ) !== false ) return true; } return false; } echo is_bot('Googlebot'); ?> |
Output:
1
This function compares the PHP user agent with a list of common spiders from search engines, more than 180 bots, spiders and crawlers.
When ‘Googlebot’ is given as input, the function returns true(1) as the provided input is the name of a search engine bot.
