Cloudflare has introduced a new setting called Disallow AI Training that gives website owners the ability to block their content from being used for AI model training while still allowing search engines to index their sites. The change addresses mixed-use crawlers, which now account for 36.6 percent of verified crawler traffic on Cloudflare's network. Apple, Google, and Microsoft have received an Accountable designation for meeting transparency requirements on their crawlers. Cloudflare CEO Matthew Becker said the response has been encouraging and that most people agree site owners should control how their content is used.
The new controls replace a single Block AI Bots switch with three separate settings covering search indexing, AI training, and AI agents. Cloudflare plans to let owners control content in AI search summaries by early 2027. The rollout is happening over the coming weeks, and the company is working with the AI research community to ensure legitimate research is not affected.
Beyond website-level controls, governments and institutions are also grappling with AI policy. The Riverside City Council formed a three-person committee to study AI over 13 months, focusing on ethics, cybersecurity, and community literacy. Pasadena's City Council is moving toward an AI policy that would protect residents' data and limit employee use of AI agents. At the federal level, Congress faces growing pressure to regulate AI development after OpenAI CEO Sam Altman testified before lawmakers, though consensus on whether regulation is even Congress's job remains elusive.
The business and investment world is also reacting to the rapid growth of AI. SpaceX is building an AI division focused on natural language processing and computer vision, which could help the company reach a $3 trillion valuation. Kakao AI aims for 6 trillion won in sales by 2030 through expansion in NLP, computer vision, and custom AI chips. Meanwhile, economists from Capital Economics warn that the Western economy may be in a late-stage AI bubble comparable to the dot-com boom, with the Federal Reserve facing a key decision on how to respond. Fortress co-CEO Jack Neumark cautioned that credit investors may not profit from AI private credit deals, advising short-term maturities and careful collateral evaluation.
Public sentiment on AI is mixed. A new poll shows Americans are increasingly worried about AI's environmental impact, particularly the energy demands of data centers. At the same time, concerns about existential AI risks have reached a new high, with companies like OpenAI and Anthropic at the center of those debates. On the cybersecurity front, CrowdStrike and Palo Alto Networks stocks have surged—CRWD up over 50 percent and PANW more than 30 percent in the past quarter—driven by strong demand for AI-powered security tools. AI is also reshaping consumer shopping, as assistants narrow product choices before shoppers ever reach a retailer, creating a new battleground for brands.
Key Takeaways
- Cloudflare launched a Disallow AI Training setting that lets website owners block AI training while keeping search engine access, replacing a single Block AI Bots switch with three separate controls.
- Mixed-use crawlers now make up 36.6 percent of verified crawler traffic on Cloudflare's network.
- Apple, Google, and Microsoft received an Accountable designation for meeting crawler transparency requirements.
- Cloudflare plans to let owners control content in AI search summaries by early 2027.
- Riverside City Council formed a three-person committee to study AI policy over 13 months, while Pasadena is moving toward its own AI policy focused on data protection and limiting AI agent use by employees.
- SpaceX is building an AI division in natural language processing and computer vision that could help it reach a $3 trillion valuation.
- Kakao AI set a target of 6 trillion won in sales by 2030, expanding into NLP, computer vision, and custom AI chips.
- Economists from Capital Economics warn the Western economy may be in a late-stage AI bubble similar to the dot-com boom, while Fortress co-CEO Jack Neumark cautioned that AI private credit deals may not pay off for debt investors.
- CrowdStrike gained over 50 percent and Palo Alto Networks rose more than 30 percent in the past quarter on strong demand for AI-powered cybersecurity tools.
- AI assistants are narrowing product choices before shoppers reach retailers, creating a new competition for brands to be recommended by AI discovery gateways.
Cloudflare adds setting to block AI training while keeping search
Cloudflare launched a new setting called Disallow AI Training. It lets website owners stop their content from being used to train AI models while still allowing search engines to index their sites. The change targets mixed-use crawlers that collect data for both search and AI training. These crawlers make up 36.6 percent of verified crawler traffic on Cloudflare's network. The new setting replaces the older Block AI Bots switch with three separate controls.
Cloudflare tool blocks AI scraping without hurting search results
Cloudflare introduced a Disallow AI Training setting to solve a problem caused by mixed-use crawlers. These crawlers collect content for both search indexes and AI training. Before this update, blocking AI training also blocked search access. Now website owners can refuse AI training while still allowing search crawlers to visit their sites. Cloudflare CEO Matthew Becker said the response has been encouraging and that most agree site owners should have control over their content.
Cloudflare launches controls to block AI training and protect sites
Cloudflare introduced new controls to block AI training on its network while keeping search access open. The controls are available to all Cloudflare customers. They aim to prevent malicious AI training that could be used for cyber attacks. Website owners can block AI training while still letting search engines crawl and index their content. Cloudflare is working with the AI research community to ensure legitimate research is not affected. The rollout will happen over the coming weeks.
Cloudflare separates search and AI training controls for websites
Cloudflare introduced separate controls for search indexing, AI training and AI agents. Website owners can reject AI model training without losing traditional search visibility. Apple, Google and Microsoft received an Accountable designation for meeting transparency requirements. The new settings replace a single Block AI Bots switch. Cloudflare said mixed-use crawlers account for 36.6 percent of verified crawler traffic on its network. The company plans to let owners control content in AI search summaries by early 2027.
Cloudflare rolls out AI content controls for website owners
Cloudflare introduced new AI access controls in September 2026 through its Accountable designation. The launch includes a Disallow AI Training option that lets publishers block content from AI model training. Cloudflare is adding clearer crawler labels and a Bot Preference Sync feature. AI agents from groups like Google, Microsoft and Apple must follow site settings. Cloudflare's research says the key proof point now is whether customers adopt these controls in upcoming quarters.
Cloudflare adds setting to block AI training while keeping search
Cloudflare launched a new setting called Disallow AI Training. It lets websites block AI training while staying visible in search results. Mixed-use crawlers now make up 36.6% of all verified crawler traffic on Cloudflare. Apple, Google, and Microsoft are the first companies to receive an Accountable designation for their crawlers.
Riverside Council creates committee to study AI policy
The Riverside City Council formed a temporary three-person committee to research artificial intelligence. The group includes Councilmembers Robillard, Falcone, and Hemenway. They will meet monthly for 13 months and focus on ethical AI use, innovation, cybersecurity, and community AI literacy. The committee will present policy recommendations and a strategic framework to the council.
Pasadena Council moves to consider AI policy
The Pasadena City Council will work on an artificial intelligence policy. Councilmember Justin Jones called for the policy to protect residents' data and limit AI agent use by employees. AI agents are autonomous systems that can reason, plan, and complete tasks with minimal human oversight. The policy item will go to either the EDTECH or Legislative Policy Committee.
SpaceX AI division could help company reach $3 trillion value
SpaceX is building an AI business that could help the company join the $3 trillion club. The AI division is developing products in natural language processing and computer vision. These technologies could impact industries like healthcare, finance, and education. A SpaceX spokesperson said the AI innovation is separate from Starship and Starlink.
Kakao AI aims for 6 trillion won in sales by 2030
Kakao AI, the AI division of South Korean company Kakao, set a goal to reach 6 trillion won in sales by 2030. The company plans to double its sales and grow its user base through AI integration. Kakao AI is working on natural language processing, computer vision, and its own AI chips. The company has partnerships in the United States, China, and other countries.
CRWD and PANW Stocks Rise on AI Security Demand
CrowdStrike (CRWD) and Palo Alto Networks (PANW) stocks have continued to rise due to strong demand for AI-powered cybersecurity tools. CRWD gained over 50% in the past quarter, while PANW rose more than 30%. The AI safety debate has increased focus on the need for strong security measures. Many companies are investing heavily in AI security solutions. This trend is seen as a major growth area for the cybersecurity industry.
Americans Worry More About AI's Environmental Impact
A new poll shows that Americans are increasingly worried about how AI affects the environment. The concern grows as AI systems require large amounts of energy and resources. Data centers that power AI tools are a major focus of these worries. The public debate now includes both safety and environmental issues. This shift in public opinion may influence future AI policies.
Economists Warn AI Bubble May Be Near Its Peak
Economists from Capital Economics say the Western economy may be in a late-stage AI bubble. They compare the current market to the dot-com boom before a major crash. The analysis states that many market factors are at levels seen before past stock market peaks. The US Federal Reserve faces a key decision on how to respond. Experts warn that the AI buildout is putting financial markets under heavy strain.
Existential AI Fears Reach New High
Concerns about the long-term risks of AI have reached a peak level. The debate has grown since before tools like ChatGPT and Claude became widely known. Companies such as OpenAI and Anthropic are central to these discussions. Leaders like Sam Altman and Dario Amodei have weighed in on AI risk. The article highlights growing fear that AI could pose serious threats to society.
Fortress Co-CEO Cautions on AI Private Credit Deals
Fortress Investment Group co-CEO Jack Neumark warned that credit investors may not profit from AI deals. He spoke at the Milken Institutes Canada Investment Summit about risks in funding AI infrastructure. He said that AI growth does not always make for a good debt investment. Other firms like Blackstone, Brookfield, KKR, and Blue Owl are also active in this space. Neumark advised short-term maturities and careful evaluation of collateral.
Congress faces pressure to regulate AI development
Congress is feeling pressure to regulate artificial intelligence development. OpenAI CEO Sam Altman testified before Congress earlier this year. Lawmakers are showing more urgency after high-profile events, but they lack consensus on whether regulation is even their job. Speaker of the House Mike Johnson has also weighed in on the issue. The path forward may not be a quick one.
AI narrows product choices before shoppers reach retailers
AI assistants are now narrowing product options to a shortlist before shoppers ever visit a retailer. This AI recommendation process is becoming a new competition for brands. Being recommended by AI could become as important as ranking on a search page. The technology helps consumers navigate complex decisions about products like laptops, TVs and running shoes. Companies that control AI discovery gateways can shape what consumers search for and which merchants receive traffic.
Sources
- Cloudflare lets sites block AI training but keep search
- Cloudflare Helps Website Owners Block AI Scraping Without Losing Search Visibility
- AIM — India's Leading AI & Data Science Media Platform
- Cloudflare separates search and AI training controls
- Cloudflare (NET) Rolls Out AI Content Controls For Website Owners
- Cloudflare lets sites block AI training but keep search
- Riverside Council Establishes Temporary Committee To Research Artificial Intelligence
- Council Will Consider Artificial Intelligence Policy
- Not Starship. Not Starlink. This Is the Part of SpaceX's Business That Could Lift It Into the $3 Trillion Club.
- Kakao AI Targets 6 Trillion Won Sales by 2030
- CRWD, PANW Stocks Extend AI-Fueled Rally As Cybersecurity Bets Outpace Software
- Americans increasingly concerned about environmental impact of AI, new poll finds
- Economists Concerned the AI Bubble Is About to Blow
- Why existential AI fears have hit a crescendo
- Fortress Co-CEO Warns Private Credit On AI Deals: 'You're Not Getting Paid For That Upside'
- Congress is under pressure to act on AI — here's what that could look like
- AI Picks the Shortlist Before Retailers Get the Click
Comments
Please log in to post a comment.