Crawler Exclusion Lists

Recent IP / CIDR entries

Recent domain entries

Recent nameserver entries

iptables / ipset Example

If you want to use the IP/CIDR list at firewall level, the cleaner way is to load the list into ipset and then add one outbound iptables rule against that set.

sudo apt install -y ipset iptables-persistent netfilter-persistent

curl -fsS https://dushansavich.com/downloads/crawler-exclusion-ips.txt \
    -o /tmp/crawler-exclusion-ips.txt

sudo ipset create crawler_exclusion_ips hash:net family inet maxelem 200000 -exist
sudo ipset flush crawler_exclusion_ips

while read ip
do
    ip=$(echo "$ip" | sed 's/#.*//' | xargs)
    [ -z "$ip" ] && continue
    sudo ipset add crawler_exclusion_ips "$ip" -exist
done < /tmp/crawler-exclusion-ips.txt

sudo iptables -N CRAWLER_EXCLUSION_IPS 2>/dev/null || true
sudo iptables -F CRAWLER_EXCLUSION_IPS

sudo iptables -A CRAWLER_EXCLUSION_IPS \
    -m set --match-set crawler_exclusion_ips dst \
    -j REJECT --reject-with icmp-port-unreachable

sudo iptables -C OUTPUT -j CRAWLER_EXCLUSION_IPS 2>/dev/null || \
    sudo iptables -I OUTPUT 1 -j CRAWLER_EXCLUSION_IPS