Report: 40% of all Internet traffic are bots Jul 12, 2019 A recent report shows that 40% of all Internet traffic are bots. In the context of websites, bots are computer programs that “visit” a website in a way similar to that of a human using a browser. The intents of bots are varied. Helpful bots are sent by search engines to index and crawl content, or to render useful previews of the website in social networks when somebody shares your content. Unfortunately not all bots are helpful, and many of them have less than noble purposes, like taking control of your site and ransoming your data or your business. In this post we examine traffic from bad bots and how ShimmerCat can help protect your website from it. What are bad bots out for? Bad bots often scrap data from sites without permission, in order to reuse it and to gain a competitive edge. Some bad bots can even undertake criminal activities such as fraud. Unfortunately bad bots are common in the ecommerce business, and some of the most frequent activitites they perform are listed in the table below. Bad Bot Categories What the Bad Bot Does Price Scraping Scraps your website to gain competitive edge. Risk of decreased sales because competitors win the SEO search on price. Declined conversion rates and dropped SEO rankings can be signs. Content Scraping Scraps you website and steal content to damage SEO rankings. Risk of having your appear on other sites. Unexplained website slowdowns and downtime can be signs. Account Takeover Visits you website to steal credentials. Risk of account lockouts, and financial fraud. An increase in failed logins, customer account lockouts and complaints are indications. Credit Card Fraud Visits your website to steal credit card information in your payment processor. An increase in payment error logs and customer support calls are indications. Can good bots be a problem? Actually yes. Good bots are often bots that ensure that ecommerces and their products can be found by prospective customers. Examples include search engine crawler that help people match their search queries with the most relevant sets of websites. It is worth noticing that good bots also can have negative effects. Good bots can for instance skew web analytics reports, making some pages appear more popular than they really are. For example, if you advertise on your website, good bots can generate an impression, but that ad click never converts in the sales funnel. This means that good bots can result in lower performance for advertisers. Bot report: 40% of all Internet traffic are bots According to a recent bot report1, almost 40% of all the Internet traffic in 2018 wasn’t human. Even though this is a remarkable number, note that still there was a decrease in bot traffic compared to the previous year. The year-over-year total decrease in bad bot traffic was -6%, and for good bot traffic -14%, while human traffic increased by +8% to a total of around 60% of all Internet traffic. Some interesting takeaways in the report: Bad bots continue to follow the trends in browser popularity, impersonating the Chrome browser in almost half of all fake browser attempts. With most bad bot traffic originating from data centers, the United States remains the “bad bot superpower” with half of the bad bot traffic coming from the country. Amazon was the source of the most global bad bot traffic at 18%. It doesn't matter so much if your site has high or small traffic, the average proportion of bad bot traffic is still around 20%. Profiling, challenging and classifying bots Being able to intelligently separate traffic generated by legitimate human users, good bots, and bad bots are essential for making informed business decisions. This is also essential for the performance of your ecommerce website, and ShimmerCat has exceptional features for intelligent bot identification and blocking. To identify and decide whether a visitor should be classified as a human, good bot, or bad bot, ShimmerCat’s machine learning engine uses many different methods for profiling, including: user agent string requests amount and timing visitor location protocol version presence of conditional requests or not cookie contents Some bots also provide indications that can be used for identification, and this is especially true for all the common crawlers including Bing, Google bot, Facebook, Yahoo, etc. By combining all this info and learning features, ShimmerCat’s helps to intelligently, and automatically, separate traffic generated by legitimate human users, good bots, and bad bots2. If ShimmerCat suspects that traffic from a specific IP is bad, that visitor will be faced with a challenge. And depending on whether the challenge is defeated or not, the IP will be classified as either good or bad. Then next time the same IP tries to access a site, ShimmerCat will recognize it and act accordingly - to protect and safeguard the ecommerce. If you want to know more about how ShimmerCat you can help to protect your e-commerce website from bad bots, and at the same time make it faster, contact us now! [1] https://resources.distilnetworks.com/white-paper-reports/bad-bot-report-2019 [2] https://www.shimmercat.com/blog/bots/wrestling-the-bad-bots.html Share on Facebook Share on Twitter
