All posts / Analytics

How Bot Filtering Works in Link Analytics (and Why It Matters)

Email security scanners, link preview generators, and crawlers inflate your click counts. Here's how bot filtering works and what to look for in a link shortener.

Truthylink Team · · 6 min read

If you've ever looked at your link analytics and thought "these numbers seem high," you may be right. A meaningful share of recorded "clicks" on any given link are not clicks from humans — they're automated requests from email security scanners, link preview generators, search engine crawlers, and monitoring bots. If your link shortener doesn't filter them out, you're making decisions based on inflated data.

This piece explains how bot traffic enters your analytics, which categories of bots matter most, and how bot filtering works so you can evaluate whether a given tool is doing it correctly.

Where bot clicks come from

Email security gateways

This is the largest source of non-human click inflation for email marketers. When a corporate email server receives a message, security systems like Microsoft Defender for Office 365, Proofpoint, Mimecast, and Barracuda automatically follow every link in the email. They're looking for phishing sites, malware downloads, and recently-parked domains.

The problem: this shows up in your analytics as a click. It happens within seconds of delivery, often before the recipient has even opened the email. Studies on B2B email lists — which have higher rates of corporate email infrastructure — have found that email security scanners can account for 20–40% of recorded clicks. If your list is 100,000 corporate emails and you record 5,000 "clicks," a significant portion may never have involved a human at all.

Link preview generators

Messaging platforms like Slack, Telegram, iMessage, and WhatsApp generate link previews when someone pastes a URL into a chat. To do this, they fetch the destination URL to extract the title, description, and image. This appears as a request to your link shortener — which, if unfiltered, gets counted as a click.

Unlike email security scanners, preview generators usually only fire once per URL per platform session. But in a busy Slack workspace where the same link gets shared multiple times, you can accumulate meaningful artificial traffic.

Search engine crawlers

Googlebot, Bingbot, and other crawlers follow links they discover. A publicly accessible short link will eventually be crawled. Good link shorteners return a 301 redirect (which crawlers don't count as a click) rather than a 302 or JavaScript redirect, and they identify crawler user-agents correctly.

Uptime monitors and synthetic testers

If you or anyone else runs an uptime check or synthetic monitoring test on a URL that happens to go through your link shortener, those pings register as requests. This is usually low volume but worth knowing about.

How bot filtering works

There's no perfect method, but good bot filtering uses several signals in combination:

User-agent matching

Bots typically identify themselves in the HTTP User-Agent header. Email security gateways often use user-agents that include strings like SafeLinks, Proofpoint, or vendor-specific identifiers. Crawlers use Googlebot, Bingbot, etc. Maintaining and regularly updating a list of known bot user-agents is the first layer of filtering.

The limitation: some scanners use generic browser user-agents to avoid detection. User-agent matching alone misses these.

Click velocity and timing patterns

Email security scanners fire almost immediately after delivery — within seconds. They also tend to fire in rapid bursts from a small number of IP addresses. A sudden spike of clicks within 2–3 seconds of an email send, all from the same data center IP range, is a reliable indicator of scanner activity rather than human engagement.

IP reputation scoring

Known scanner IP ranges are documented. Major security vendors publish their IP addresses for mail administrators; some of this data is consolidated in IP reputation databases. Filtering clicks from known corporate scanner IP ranges removes the most predictable source of inflation.

Headless browser detection

Some scanners execute JavaScript to check for dynamic redirects or malware payloads. These can be detected by looking for tells in the browser environment — missing browser APIs, unusual rendering characteristics, or the absence of expected browser behaviours.

Why the accuracy gap matters

The practical consequence of unfiltered bot data isn't just inflated numbers — it's wrong decisions. If you're:

  • Optimising email send times based on "peak click hours" that are actually scanner fire times
  • Comparing campaign performance across B2B and B2C segments without accounting for different scanner rates
  • Paying affiliates on a cost-per-click basis with unfiltered data
  • A/B testing link placements based on a dataset inflated by bot variance

…then your conclusions are built on noise. Bot filtering isn't a nice-to-have. It's the prerequisite for link analytics being useful.

What to look for in a link shortener's bot filtering

When evaluating a link shortener, ask specifically:

  • Does it show both raw click count and filtered click count, or only one of them?
  • Is the bot list updated regularly, or is it a static list from three years ago?
  • Does it filter email security scanners specifically (Proofpoint, Defender SafeLinks, Mimecast)?
  • Is filtering applied retroactively as new bots are identified, or only to new clicks?

Truthylink applies multi-layer bot filtering — user-agent matching, IP reputation checks, and click pattern analysis — and shows the filtered click count as the primary metric in your analytics dashboard. The raw total is also available if you need it for audit purposes. This is the number you should be using to make marketing decisions.

Free plan available

Start shortening links for free

50 links, QR codes, and bot-filtered analytics — no credit card needed.

Create free account →