Should publishers block AI bots from scraping their content?
Thu Jan 29 2026
Eric Shanfelt joins the Local Media Association to take an in-depth look at whether publishers should block AI bots from their websites and, if so, how to do it. He explains the tradeoffs between AI visibility and website traffic, then walks through simple, real-world defenses you can implement without overcomplicating your stack.
You will learn:
Traffic vs visibility: which matters more right nowHow to use robots.txt (and what not to block)How a firewall service like Cloudflare or Amazon Web Services can helpWhy local publishers should consider blocking international trafficThe “back door” way AI bots can still index your sites.We’d like to extend a huge thanks to the Local Media Association for hosting this session and for the work they do every day to support local publishers. Learn more and please consider joining LMA at https://www.localmedia.org/
Learn more at https://nearviewmedia.com/
More
Eric Shanfelt joins the Local Media Association to take an in-depth look at whether publishers should block AI bots from their websites and, if so, how to do it. He explains the tradeoffs between AI visibility and website traffic, then walks through simple, real-world defenses you can implement without overcomplicating your stack. You will learn: Traffic vs visibility: which matters more right nowHow to use robots.txt (and what not to block)How a firewall service like Cloudflare or Amazon Web Services can helpWhy local publishers should consider blocking international trafficThe “back door” way AI bots can still index your sites.We’d like to extend a huge thanks to the Local Media Association for hosting this session and for the work they do every day to support local publishers. Learn more and please consider joining LMA at https://www.localmedia.org/ Learn more at https://nearviewmedia.com/