Digital Commons offers a showcase for your scholarly collections with analytics tools to help accurately measure the impact of your research. Among our analytics tools, download filtering is complex, requiring deep knowledge of bot behavior and constant vigilance of dedicated technical staff.

While human visitors access your content, automated bots are also attempting to scrape the same material. Digital Commons has multiple layers of protection to help block abusive bots and filter out ‘good’ bots. These tools are necessary to help ensure that your site remains stable, and your download counts are as accurate as possible.

As bots become increasingly sophisticated and better at mimicking human behavior, we continuously work to improve the methods we use to identify, block, and filter bots.

How we count downloads

Digital Commons incorporates excellent search engine optimization (SEO) features which allow your content to appear in search engines like Google. Search engines use bots to find new content by crawling sites, so it is crucial that we do not interfere with these ‘good’ bots in doing their work. For this reason, there are three possible responses to visitors attempting to download content from Digital Commons:

Blocking: We block ‘bad’ bots so that they cannot access the content on Digital Commons. This prevents the bots from negatively impacting system performance through their sheer volume of traffic, as well as keeping them from appearing in your download counts.

Filtering: We allow ‘good’ bots to download your content, but we do not count their downloads in the Dashboard. We want Dashboards to reflect real human traffic, but we also want ‘good’ bots to access your content, so filtering is a good ‘middle ground.’ We also filter out unusual activity by human visitors, such as if a researcher tries to snag a spot in your Top Downloads by downloading their own work over and over!

Counted: When our analysis concludes that a download came from a human visitor, we allow the visitor to download the content, and we count it as a download in the Dashboard and on article metadata pages.

The changing environment and our approach

Although many ‘good’ bots announce themselves as bots, ‘bad’ bots use increasingly sophisticated methods to mimic human traffic to avoid detection. As AI companies become more ubiquitous, they are sending an ever-increasing number of bots to crawl the internet for high-quality text they can use to train their LLMs (Large Language Models). Training LLMs involves crawling the same content many times, which has the potential to inflate download counts and place a strain on the system several times that of the busiest human traffic.

As we refine our methods for blocking abusive bots and filtering ‘good’ bots, we have a set of principles that guide our approach:

• Minimize impact on human visitors. We believe in open access, and we want to make your research as easily accessible as possible. We allow visitors direct access to articles, often going directly from search engines to PDFs. We do not require visitors to set up accounts before downloading your content, because we believe that the easier it is to access your content, the greater the impact.
• Err on the side of caution. We only block activity if we have a very high amount of certainty that it is coming from a bot. If we find that an anti-bot method is impacting human visitors, we adjust or disable it, even at the risk of allowing in more bots. Humans always come first!
• Provide stable and reliable service. Abusive bot traffic has the potential to negatively impact site stability, so we take steps to block bots before they can impact our servers. Nipping a bot that attacks one site increases the resilience of the community overall. We also design Digital Commons in a way that mitigates the system impact that bots can cause.
• Provide accurate download counts. We understand the need for accurate reporting to demonstrate the impact of your scholarship, so we continually refine our methods to provide download counts that are as accurate as possible. 

Questions?

If you have any questions or if you see unusual activity on your Dashboard, please contact your consultant.

Explore your metrics

Use the following links to learn more about Digital Commons metrics for your repository, journal, or individual works.

• Overview of Digital Commons Reports 
• Digital Commons Dashboard
• Author Dashboard: Real-Time Usage Statistics for Authors