The Age of Agents: Managing “Shadow AI” in Procurement

In our personal lives, we can build an AI agent on our phone in an hour for under $50. But when that same “citizen developer” mindset enters the corporate world without guardrails, it creates a massive risk for the enterprise.

In this week’s jam session on Proc & Roll, Josh Earley joins Natasha Gurevich and Zachary Bachir for a deep dive into the critical, often overlooked topic of AI governance. They explore the tension between the speed of innovation and the need for control, and why “just because you can, doesn’t mean you should”.

Topics Discussed in This Episode:

• The “Shadow AI” Threat: Josh warns about the dangers of the “citizen developer.” The nightmare scenario? A business unit builds a critical workflow using an unapproved AI tool, the creator leaves the company, the workflow breaks, and procurement never even knew it existed. This creates a massive continuity risk that procurement must get ahead of.
• The 150-Question RFP Problem: Are suppliers using AI to answer your massive RFPs? “A thousand percent,” says Josh. The group discusses the rise of “AI Slop”—unverified, AI-generated responses to long questionnaires. This shift means procurement may need to start building “agent-ready” templates designed to be read by machines, rather than humans.
• Governance 101: Start Small: How do you build a governance framework without stifling innovation? Josh advises starting small by aligning just two key functions: Legal and Technology. You can’t rely on a grassroots effort; policy needs to be “bottom-up built, but top-down delivered” to be effective.
• Knowledge as a Commodity: In a profound “tombstone” moment, the group discusses a chilling realization from an AI conference: “Your knowledge is now a commodity”. If human expertise can be uploaded into a model and accessed by anyone, the true differentiator for professionals becomes the human element that cannot be digitized.
• The “Agent” Boss: The conversation concludes with a provocative look at the future of work. As agents become more autonomous, we may soon face a reality where employees are managed by “digital agents” rather than human bosses—a shift that requires entirely new frameworks for HR and management.

Watch now or read the transcript below.

Transcript: Proc-N-Roll | The Age of Agents: Managing “Shadow AI” in Procurement

Zach: Good morning, good afternoon, good evening, and welcome to Proc and Roll. Today, we are missing Conrad, who is buried under piles of procurement work, but replacing him is a very special guest: Josh Earley. Josh, welcome. We want to talk about AI, but specifically the governance aspect, which I think is being missed amidst all the shiny glitz. How do you see AI changing the way procurement governance works?

Josh: It depends on the size of the organization, but procurement is going to need to take an active role in regularly updating policies. Today, looking at a policy might be an annual process—you glance over it to see if there are major updates. But as AI moves forward and gets more active in everyday activities, the action of reviewing and updating policies to integrate AI thoughts needs to happen on a much more regular basis.

Zach: Do we have the right skills to do that today? Is it going to be a painful journey?

Josh: What a good question. It is a painful journey ahead. You’re crossing technology, procurement, risk, and compliance, and then true legal. All those parties are going to want a say, which will create pain points. When you start layering in AI modeling, usage, reasoning, and decision-making, it gets really complicated regarding who is accountable to do that work.

Natasha: The problem today is significantly more different. Anyone within the enterprise—distribution, manufacturing, etc.—can just create their own agreements or purchase orders because they are driven by “let’s speed things up.” How do we create guardrails that encourage modern tools but tell people not to take it “full speed and full length?”

Josh: We are digging into a hard one: the dichotomy between your personal life expectations and your professional life. I could build an AI agent on my phone right now for under $50. But I can’t go do that in the professional world because there are rules, governance, and policies. This creates friction between associates coming to work with an expectation of what AI can do versus what they are permitted to do. Just because you can doesn’t mean you should.

Natasha: Knowing that people still will—just like they use corporate cards for personal expenses despite the rules—how do we fix that? We usually do training or put limits in place. How does that translate to this current environment?

Josh: The risk is significantly higher here. Just because Co-Pilot can help anybody write a contract doesn’t mean they should. If someone writes a contract and procurement doesn’t know about it, we have a problem. The other dangerous scenario is the “citizen developer” atmosphere. What happens when a user in a business unit creates a critical workflow that no one knows about? Then they leave or it breaks, and now the business can no longer be done because we allowed users to build workflows without governance. That is dangerous.

Zach: How do we get ahead of that? Who do you bring into the room?

Josh: I think it’s really small to start: Legal and Technology. You need a policy that is “bottoms up built, but top down delivered.” It won’t work as a grassroots effort. Legal needs to handle the privacy, PII, and confidentiality aspect, and Technology needs to define our current capabilities and what we are willing to engage in. I’ve been intentionally slow in my role to put these things out because I needed that alignment first.

Zach: What about suppliers using AI? Are we going to see them using agents to answer our RFPs?

Josh: A thousand percent. And do I believe they are going to QA those answers? Not a chance. If you send a 150-question RFP, you’re going to get AI-generated responses because it’s just checking a box. We haven’t made the shift to operate our RFPs in a way that prevents this. Do I really need 150 questions to know who I should talk to? No.

Zach: Does that mean we should be developing “agent-ready” templates? Just like SEO works for search, are we creating a new discipline of building for the agent so they can find our product or service?

Josh: 100%. You’re seeing LLMs become the new SEOs. They are taking over.

Zach: We might need to caution suppliers that we won’t accept “AI slop.” But if everyone starts using LLMs backed by the same base models like OpenAI and Anthropic, everything will start to sound the same. The way to distinguish yourself will be to put a human there who can sound different to an AI and sound a bit more forward-thinking.

Josh: It’s fascinating. I was at an AI conference where a data scientist dropped a “tombstone” quote on the audience. He said, “Just remember, your knowledge is now a commodity.” I realized: if I can get into these models and dump everything I know into them, I’ve created my knowledge as a commodity to utilize.

Zach: That is a tombstone. We have to watch out; the future with agents is going to get very weird.

Josh: A provocative question to end on: What happens when you have enough agents and there is a swell to give them rights? Or when you have “managers of agents?” As an employee, how does it feel when your team—or even your boss—is a digital agent?

Zach: Thank you, Josh. This has been so interesting. I think this topic is going to become way more relevant very quickly. I hope to have another conversation in the future.

This transcript has been edited for clarity while maintaining all substantive content