 |
|
|
|||||
We all know AI is eating the internet, with bots scraping sites for content and not giving anything in return. This, of course, is the impetus behind the many lawsuits that are playing out between media companies and the big AI labs, but in the here and now, the question remains what to do about those bots. Blocking them is an option, but how effective is it? And what types of content are most at risk of being scraped and substituted by AI answers? And can you actually get AI bots to pay up? A good place to start finding answers is the most recent State of the Bots report from AI startup TollBit. For publishers that are feeling the heat of AI, it attaches real numbers to the presence of AI in the media ecosystem and how quickly it’s growing. And while the rise of AI bots is a worrisome trend to those in the content business, it may also be an opportunity. {"blockType":"creator-network-promo","data":{"mediaUrl":"https:\/\/images.fastcompany.com\/image\/upload\/f_webp,q_auto,c_fit\/wp-cms-2\/2025\/03\/mediacopilot-logo-ss.png","headline":"Media CoPilot","description":"Want more about how AI is changing media? Never miss an update from Pete Pachal by signing up for Media CoPilot. To learn more visit mediacopilot.substack.com","substackDomain":"https:\/\/mediacopilot.substack.com\/","colorTheme":"blue","redirectUrl":""}} Bots in disguise In the interest of maximizing that opportunity, TollBit is doing more with this report than simply offering up charts and graphs. It’s also taking a stand, arguing that AI bots that crawl the internet should at the very least identify themselves to the sites they visit and scrape. The company is openly calling for regulation to force the issue, something CEO Toshit Panagrahi told me back in June after its previous State of the Bots report showed that certain bots from the likes of Perplexity, Meta, and Google were openly ignoring the Robots Exclusion Protocol, which websites use to manage bot traffic. There is some nuance to that. I wont rehash the entire thing here, but briefly: certain AI bots perform tasks on behalf of users (as opposed to training or search bots), and those are designated user agents. That affords them a certain status, at least according to AI companies: Because they are essentially human proxies, they believe sites should treat them as humans, not bots. So they don’t identify themselves as bots. What this does, at the very least, is make it very hard to tell what’s real human trafficthat is, a person navigating to a website and looking at a screenversus a robot doing the same thing. That’s going to make it very difficult to get accurate data about bot traffic, and TollBit predicts that the amount of “human” traffic will probably rebound once user agents become more common, but that’s only because trackers won’t be able to tell the difference between them and actual people. You can see the impetus to get bots to self-identify, but let’s assume that doesn’t happen, and a significant amount of traffic falls into this gray area: seemingly human but not behaving as such. Those ersatz humans won’t ever interact with advertising, and once that becomes evident, it will cheapen the value of advertising on the web overall. We may never technically reach Google Zero, but margins will be stripped so low that Google 30 might look more like Google 10. The content AI craves Something else the TollBit report reveals, though, is what kind of content appears to be of greatest interest to the AI crawlers, or rather, the people using AI engines for discovery. While the data isn’t definitive, it’s fair to conclude that if a particular category of content is being scraped more often, there are more people sending AI crawlers and user agents to get it. That, in turn, might help guide content strategy. By far the No. 1 category being scraped is B2B content, followed by parenting, sports, and consumer tech. Parenting, in fact, saw a big increase this past quarter, meaning more people are turning to AI portals for answers about parenting issues. If you produce content for parents (and this applies to any category that’s highly crawled by AI), you should consider a few things: Your content is at high risk of substitution by AI answers. That means it’s valuable to AI companies. You can point to the data as leverage in licensing negotiations (or a lawsuit). It sounds simple, but getting a major AI provider to license your content isn’t something that any site can do. OpenAI, by far the most prolific deal-maker, has signed only a few dozen agreements. And lawsuits are costly. If you’re a parenting site, you’re not just going to stop doing parenting content, so you have a choice: block the bots, or let them crawl to ensure your presence in AI answers. While the referral traffic remains negligible (we’re effectively already at “ChatGPT Zero”), there are intangibles, mostly brand presence, that being in an AI answer provide. You can’t build a business on intangibles, though, and that leaves the other option: blockingor rather, redirecting bots to a paywall. TollBit’s data does show that more bots than before are being successfully redirected to “forbidden” pages or hitting the company’s own paywalls. The illusion of control The key question, though, which the report doesn’t answer, is how many of those bots are actually paying up? The lack of answer suggests the number is quite low, and that’s because it’s simply too easy to access the content in another way. As the report describes, there are sophisticated ways for AI companies to use relays, third-party systems, and different species of bots to scrape content. And the “gray” status of consumer browser agents makes things even murkier. The number of ways to access blocked content are myriad. That’s ultimately why TollBit has taken its stance that bots should be required to self-identify, backed by legal teeth. It’s hard to imagine AI companies self-regulating in the interest of another industryin this case, the mediawithout some kind of regulatory pressure. Otherwise, we can look forward to something else: a lot more paywalls on parenting sites. {"blockType":"creator-network-promo","data":{"mediaUrl":"https:\/\/images.fastcompany.com\/image\/upload\/f_webp,q_auto,c_fit\/wp-cms-2\/2025\/03\/mediacopilot-logo-ss.png","hadline":"Media CoPilot","description":"Want more about how AI is changing media? Never miss an update from Pete Pachal by signing up for Media CoPilot. To learn more visit mediacopilot.substack.com","substackDomain":"https:\/\/mediacopilot.substack.com\/","colorTheme":"blue","redirectUrl":""}}
Category:
E-Commerce
Jim Ferrell is a best-selling author and thought leader whose work explores leadership, culture change, and human connection. As cofounder of the Arbinger Institute, he authored influential books like Leadership and Self-Deception, The Anatomy of Peace, and The Outward Mindset. He now leads Withiii Leadership, focusing on helping people apply relational approaches to leadership and organizational life, which he introduces in his newest book, You and We: A Relational Rethinking of Work, Life, and Leadership. With a background in economics, philosophy, and law, he is known for translating complex ideas into clear, transformative models that bridge divides and bring people together. Whats the big idea? You and We aims to help you see work and relationships in a whole new way. It details a practical framework, rooted in philosophy, for leading and running organizations. This approach is effective in business but also offers a powerful method for stitching the human family together in the face of our many threats. Below, Jim shares five key insights from his new book, You and We: A Relational Rethinking of Work, Life, and Leadership. Listen to the audio versionread by Jim himselfbelow, or in the Next Big Idea app. 1. Management of the individual is dead For most of modern history, weve treated individuals as the core unit of analysis in organizationsas if each person is a dot on a chart, and performance is about optimizing those dots. But heres the problem with this approach: The idea of a separate individual is a myth, and because its a myth, the strategies that mistake it as true generate systematically poor advice. Every individual you think you are seeing is relation in disguise. When you are seeing another person, you are the one who is doing the seeing. Since you are the one who is seeing, you are not seeing a person or world separate from yourself, but rather seeing your interaction with the world. This inherent relationality of observed reality is the most important scientific discovery of the last century. When we observe and measure the world, were not observing and measuring a world separate from ourselves; were observing and measuring our own interaction or relation with the world and the worlds interaction with us. As the great physicist Werner Heisenberg said, What we observe is not Nature itself, but Nature exposed to our method of questioning. Everything we see is relation. Coming back to the dot analogy, the real driver of performance is not in the dots on the org chart. Its in the space or relation between them. Its in the connectivity within and between teams and departments. Team-sport coaches know this. Listen to the winning coach after a game and you will often hear them say something like, We had great connectivity tonight. What the coach means is that they won not primarily because of individual talent, but because that talent moved and functioned as a fully synchronized whole. The leadership paradigm of the future is the measurement and management of relations. 2. The 5 levels of relation The most important part of any org chart is the space between the names and boxes on the chart. Thats where the action iswhere collaboration either lives or dies. This space between people isnt just a metaphor. Its a measurable, changeable reality. Collectively, it forms what you might think of as the relational field of your organization. This relational fieldthe levels of connectivity across your organizationis most predictive of organizational success. To see and measure this space, we first need a way to differentiate between levels of relation. I introduce five levels of relation: Division: People or teams that get in each others way are dividing. Subtraction: Those who resist or avoid others are subtracting. Addition: People or teams just focusing on their own work are adding. Multiplication: Those who are collaborating with others are multiplying. Compounding: People who care as much about others success as their own and integrate their work in deep ways to advance their collective success are compounding. With these levels of relation in mind, you can map team and organizational connectivity levels and, applying strategic priorities, decide which relational intersections across the system need to be improved. When you can see and track these levels of relation, you can start improving them intentionally and systematically. 3. Thinking in 4 dimensions To improve the connectivity levels in your organization, let me introduce a lens that I call the Four-Dimensional Playing Field. Every organization is, on the one hand, a collectivea thing, one unit. On the other hand, this collective is made up of many individuals. Both the organization and the individuals that comprise it have outsides (things you can see) and insides (things you cant see but can sense or feel). On the individual side are peoples behaviors (which you can see) and their attitudes (which you can sense or feel). Regarding the collective, you can see its structures, systems, and processes, but you can only sense or feel its culture or community. You build your playing field in such a way as to maximize connectivity. These two distinctionscollective vs. individual on the one hand and outsides vs. insides on the otherproduce a four-dimensional view of organizations. The two individual dimensions are individual behaviors and attitudes. The two collective dimensions are the groups structures and culture. Together, these four dimensions form the playing field of every organization. The realities within them are the levers you can pull to improve connectivity across a system. Elements that are dragging connectivity down can be replaced with features that provide connective lift. You build your playing field in such a way as to maximize connectivity. 4. Understanding connection I thought I had understood human connection. However, I had only really understood how to get myself and others to the multiplication level of relation. The highest level, compoundingboth as an idea and as a realityhad been beyond me. We assume that we are separate from others, that the world is divided between I and Other (or between Us and Them). But this is a mistake. Martin Buber shows there is no such thing as a separate-I, but only I-in-relation. The trouble is that despite being fundamentally connected, we live much of the time as if we were divided from others. We get ourselves stuck within our own heads. We generate our own separations by encountering others through the filters of our thoughts, assumptions, judgments, and concepts, which is just another way of encountering ourselves. When we learn how we do this, we also learn how to undo it. Getting stuck in our heads keeps us from connecting because connection happens in the space betwee people rather than in our minds. To connect, we have to learn how to escape the walls of our heads; drop the concepts, assumptions, and goals; and be present with others in the space between us. 5. The importance of difference When thinking about integration and unity, we often assume that this means agreement, giving in, complying, or becoming more similar. But this is incorrect. As an analogy, consider water. If two hydrogen atoms combine with a single oxygen atom, something completely beyond the capabilities of hydrogen or oxygen alone comes into being: water. This is an example of the law of progress. Whether talking about matter, life, or thought, vertical development arises from a three-part process: First, differences need to compress together. Then, if those differences open themselves to each other, they can overcome their apparent divides and converge. Out of this convergence of differences emerges something entirely new. For this transformation to happen, the elements need to retain their differences. Hydrogen atoms need to remain hydrogen, and oxygen atoms need to remain oxygen, for water to appear. Progress requires that we value and hold to our differences. If we stay only with our own kind, or only with our own thoughts, we will remain exactly as we already are. Progress requires connection with difference. Compress, converge, emerge. That is the arc of vertical progress. Learning to connect with difference is the path of growth. One way to incline oneself in the direction of this growth is to always be looking for what I call the next we. Maybe I hang out with or listen to people who are like me, but what about people who are unlike me? Or who dislike me? How about those who oppose me? Or even those who might hate me? Without realizing it, we draw lines in our minds that keep us cut off from others. Those who are currently outside whatever line you have drawn hold the key to your growth and transformation. The thing we need to progressdifferencelies on the other side of that line. Learning to connect with difference is the path of growth. And thats true not only of you, but also of your company and your community. The lines we draw are merely the places where differences meet. Your company and your community need people who are willing and able to bridge those divides. Only then will you be able to make water. This article originally appeared in Next Big Idea Club magazine and is reprinted with permission.
Category:
E-Commerce
In May 2025, the White House proposed reducing the budget of the National Institutes of Health by roughly 40%from about $48 billion to $27 billion. Such a move would return NIH funding to levels last seen in 2007. Since NIH budget records began in 1938, NIH has seen only one previous double-digit cut: a 12% reduction in 1952. Congress is now tasked with finalizing the budget ahead of the new fiscal year, which begins October 1. In July, the Senate rejected the White Houses proposed cuts and instead advanced a modest increase. And in early September, the House of Representatives also supported a budget that maintains the agencys current funding levels. However, talk of cutting NIH funding is not a new development. Such proposals tend to resurface from time to time, and the ongoing discussion has created uncertainty about the stability of research overall and prompted concern among scientists about the future of their work. As researchers studying complex health policy systemsand specifically, science funding policywe see the NIH as one node in an interconnected system that supports the discovery of new knowledge, trains the biomedical workforce, and makes possible medical and public health advances across the U.S. Our research shows that while cutting NIH funding may appear to save money in the short term, it can trigger a chain of effects that increase long-term healthcare costs and slow the development of new treatments and public health solutions over time. Seeing the bigger picture of NIH funding NIH funding does not just support the work of individual researchers and laboratories. It shapes the foundation of American science and healthcare by training scientists, supporting preventive health research, and creating the knowledge that biomedical companies can later build into new products. To understand how funding cuts may affect scientific progress, the training of new researchers, and the availability of new treatments, we took a broad look at existing evidence. We reviewed studies and data that connect NIH funding, or biomedical research more generally, to outcomes such as innovation, workforce development, and public health. In a study published in July 2025, we built a simple framework to show how changes in one part of the systemresearch grants, for examplecan lead to changes in others, like fewer training opportunities or slower development of new therapies. Eroding the basic research foundation The NIH funds early-stage research that lacks immediate commercial value but provides the building blocks for future innovations. This includes projects that map disease pathways, develop new laboratory methods, or collect large datasets that researchers use for decades. For example, NIH-supported research in the 1950s identified cholesterol and its role in disease pathways for heart disease, helping to lay the groundwork for the later discovery of statins used by millions of people to lower cholesterol levels. Cancer biology research in the 1960s led to the discovery of cisplatin, a chemotherapy prescribed to 10% to 20% of cancer patients. Basic research in the 1980s on how the kidneys handle sugar helped pave the way for a new class of drugs for type 2 diabetes, some of which are also used for weight management. Diabetes affects about 38 million Americans, and obesity affects more than 40% of the adults in the U.S. Cisplatin, a chemotherapy widely used today, was developed through NIH-supported cancer biology research. [Photo: FatCamera/Getty Images] Without this kind of public, taxpayer-funded investment, many foundational projects would never begin, because private firms rarely take on work with long timelines or unclear profits. Our study did not estimate dollar amounts, but the evidence we reviewed shows that when public research slows, downstream innovation and economic benefits are also delayed. That can mean fewer new treatments, slower adoption of cost-saving technologies, and reduced growth in industries that depend on scientific advances. Reducing the scientific workforce By providing grants that support students, postdoctoral researchers, and early-career investigators, along with the labs and facilities where they train, the NIH also plays a central role in preparing up-and-coming scientists. When funding is cut, fewer positions are available and some labs face closure. This can discourage young researchers from entering or staying in the field. The effect extends beyond academic research. Some NIH-trained scientists later move into biotechnology, medical device companies, and data science roles. A weaker training system today means fewer skilled professionals across the broader economy tomorrow. For example, NIH progams have produced not only academic researchers but also engineers and analysts who now work on immune therapies, brain-computer interfaces, diagnostics and AI-driven tools, as well as other technologies in startups and in more established biotech and pharmaceutical companies. If those training opportunities shrink, biotech and pharmaceutical industries may have less access to talent. A weakened NIH-supported workforce may also risk eroding U.S. global competitiveness, even in the private sector. Innovation shifts toward narrow markets Public and private investment serve different purposes. NIH funding often reduces scientific risk by advancing projects to a stage where companies can invest with greater confidence. Past examples include support for imaging physics that led to MRI and PET scans and early materials science research that enabled modern prosthetics. Our research highlights the fact that when public investment recedes, companies tend to focus on products with clearer near-term returns. That may tilt innovation toward specialty drugs or technologies with high launch prices and away from improvements that serve broader needs, such as more effective use of existing therapies or widely accessible diagnostics. Imaging technologies such as MRI were developed through NIH funding for basic research. [Photo: Tunvarat Pruksachat/Getty Images] Some cancer drugs, for instance, relied heavily on NIH-supported basic science discoveries in cell biology and clinical trial design. Independent studies have documented that without this early publicly supported work, development timelines lengthen and costs increase, which can translate into higher prices for patients and health systems. When public funding shrinks and companies shift toward expensive products instead of lower-cost improvements, overall health spending can rise. What looks like a budget saving in the near term can therefore have the opposite effect, with government programs such as Medicare and Medicaid ultimately shouldering higher costs. Prevention and public health are sidelined NIH is also a major funder of research aimed at promoting health and preventing disease. This includes studies on nutrition, chronic diseases, maternal health, and environmental exposures such as lead or air pollution. These projects often improve health long before disease becomes severe, but they rarely attract private investment because their benefits unfold gradually and do not translate into direct profits. Delaying or canceling prevention research can result in higher costs later, as more people require intensive treatment for conditions that could have been avoided or managed earlier. For example, decades of observation in the Framingham Heart Study shaped treatment guidelines for risk factors such as high blood pressure and heart rhythm disorders. Now this cornerstone of prevention helps to avert heart attacks and strokes, which are far more risky and costly to treat. A broader shift in direction? Beyond these specific areas, the larger issue is how the U.S. will choose to support science and medical research going forward. For decades, public investment has enabled researchers to take on difficult questions and conduct decades-long studies. This support has contributed to advances ranging from psychosocial therapies for depression to surgical methods for liver transplants that do not fit neatly into market priorities, unlike drugs or devices. If government support weakens, medical and health research may become more dependent on commercial markets and philanthropic donors. That can narrow the kinds of problems studied and limit flexibility to respond to urgent needs such as emerging infections or climate-related health risks. Countries that sustain public investment may also gain an edge by attracting top researchers and setting global standards for new technologies. On the other hand, once opportunities are lost and talent is dispersed, rebuilding takes far more time and resources. Mohammad S. Jalali is an associate professor of systems science and policy at Harvard University. Zeynep Hasgül is a research associate of data and systems science at Harvard University. This article is republished from The Conversation under a Creative Commons license. Read the original article.
Category:
E-Commerce
All news |
||||||||||||||||||
|
||||||||||||||||||