{"id":210168,"date":"2026-02-04T22:31:00","date_gmt":"2026-02-05T03:31:00","guid":{"rendered":"https:\/\/news-you-need.com\/index.php\/2026\/02\/04\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\/"},"modified":"2026-02-05T01:15:07","modified_gmt":"2026-02-05T06:15:07","slug":"news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing","status":"publish","type":"post","link":"https:\/\/news-you-need.com\/index.php\/2026\/02\/04\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\/","title":{"rendered":"News sites are locking out the Internet Archive to stop AI crawling. Is the \u2018open web\u2019 closing?"},"content":{"rendered":"<p><a href=\"https:\/\/theconversation.com\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing-274968\">News sites are locking out the Internet Archive to stop AI crawling. Is the \u2018open web\u2019 closing?<\/a><\/p>\n<p><a href=\"https:\/\/theconversation.com\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing-274968\">https:\/\/theconversation.com\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing-274968<\/a><\/p>\n<p>Publish Date: <a href=\"publish_date]\">2026-02-04 22:31:00<\/a><\/p>\n<p>Source Domain: <a href=\"theconversation.com\">theconversation.com<\/a><\/p>\n<p>When the World Wide Web went live in the early 1990s, its founders hoped it would be a space for anyone to share information and collaborate. But today, the free and open web is shrinking.<\/p>\n<p>The Internet Archive has been recording the history of the internet and making it available to the public through its Wayback Machine since 1996. Now, some of the world\u2019s biggest news outlets are blocking the archive\u2019s access to their pages.<\/p>\n<p>Major publishers \u2013 including The Guardian, The New York Times, the Financial Times, and USA Today \u2013 have confirmed they\u2019re ending the Internet Archive\u2019s access to their content. <\/p>\n<p>While publishers say they support the archive\u2019s preservation mission, they argue unrestricted access creates unintended consequences, exposing journalism to AI crawlers and members of the public trying to skirt their paywalls.  <\/p>\n<p>Yet, publishers don\u2019t simply want to lock out AI crawlers. Rather, they want to sell their content to data-hungry tech companies. Their back catalogues of news, books and other media have become a hot commodity as data to train AI systems.<\/p>\n<h2>Robot readers<\/h2>\n<p>Generative AI systems such as ChatGPT, Copilot and Gemini require access to large archives of content (such as media content, books, art and academic research) for training and to answer user prompts. <\/p>\n<p>Publishers claim technology companies have accessed a lot of this content for free and without the consent of copyright owners. Some began taking tech companies to court, claiming they had stolen their intellectual property. High-profile examples include The New York Times\u2019 case against ChatGPT\u2019s parent company OpenAI and News Corp\u2019s lawsuit against Perplexity AI. <\/p>\n<p>              <span class=\"caption\">The New York Times has sued OpenAI for alleged copyright infringement.<\/span><br \/>\n              <span class=\"attribution\">Sarah Yenesel\/EPA<\/span><\/p>\n<h2>Old news, new money<\/h2>\n<p>In response, some tech companies have struck deals to pay for access to publishers\u2019 content&#8230;.<\/p>\n<p><a href=\"https:\/\/theconversation.com\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing-274968\">Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>News sites are locking out the Internet Archive to stop AI crawling. Is the \u2018open&#8230;<\/p>\n","protected":false},"author":1,"featured_media":210169,"comment_status":"closed","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/images.theconversation.com\/files\/716459\/original\/file-20260205-70-j8q5rl.png?ixlib=rb-4.1.0&rect=0%2C50%2C1600%2C800&q=45&auto=format&w=1356&h=668&fit=crop","fifu_image_alt":"","footnotes":""},"categories":[14],"tags":[19],"class_list":["post-210168","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-artificial-intelligence","tag-generative-ai"],"_links":{"self":[{"href":"https:\/\/news-you-need.com\/index.php\/wp-json\/wp\/v2\/posts\/210168"}],"collection":[{"href":"https:\/\/news-you-need.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/news-you-need.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/news-you-need.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/news-you-need.com\/index.php\/wp-json\/wp\/v2\/comments?post=210168"}],"version-history":[{"count":1,"href":"https:\/\/news-you-need.com\/index.php\/wp-json\/wp\/v2\/posts\/210168\/revisions"}],"predecessor-version":[{"id":210170,"href":"https:\/\/news-you-need.com\/index.php\/wp-json\/wp\/v2\/posts\/210168\/revisions\/210170"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/news-you-need.com\/index.php\/wp-json\/wp\/v2\/media\/210169"}],"wp:attachment":[{"href":"https:\/\/news-you-need.com\/index.php\/wp-json\/wp\/v2\/media?parent=210168"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/news-you-need.com\/index.php\/wp-json\/wp\/v2\/categories?post=210168"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/news-you-need.com\/index.php\/wp-json\/wp\/v2\/tags?post=210168"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}