Nicole Carignan is Senior Vice President, Security and AI Strategy, and Global Field CISO at Darktrace, a UK-based global cybersecurity company. An expert in the safe, secure, and responsible application of AI in cybersecurity, she works on product innovation and advisory and research, and provides technical and strategic guidance on cybersecurity, AI, and data science.
With more than 25 years of experience, Carignan has deep expertise in data science, machine learning, cybersecurity, threat intelligence, and cyber operations. She spent more than 20 years in and around the U.S. federal government, serving in technical and operational roles for the intelligence community and the U.S. Department of Defense, and consulting on large-scale data science efforts.
Her insights have been cited by global cybersecurity publications, and she is a frequent speaker at industry conferences including Women in Data Science, AUSA Cyberworld, Black Hat’s AI Summit, S4 ICS Security, and Nvidia GTC. Carignan spoke on a panel moderated by The Innovator’s editor-in-chief at the VivaTech conference in Paris in June. The Innovator caught up with her after the event to discuss how corporates should think about cybersecurity in the age of AI.
Q: What do you see as the biggest cybersecurity threats to corporates today?
NC: The biggest threat to organizations today is that most of them don’t have security programs built for the novelty and scale of attacks now coming at them. Those programs were built on a backbone of malicious classification — telling you ‘this is benign, this is malicious,’ based on previous attacks. That meant signatures, hashes, and IOCs [Indicators of Compromise] that had been seen before. But threat actor groups no longer reuse the same network script. They’re generating attacks on the fly and exploiting software faster, because they can find vulnerabilities more quickly, giving them an easier point of entry. They’re using AI to mine large data corpuses for identity compromises, then walking in the door as a compromised identity. Most security programs, built on historical approaches, aren’t designed to detect that, so security teams are having to adapt quickly to prepare for AI-powered, probabilistic, non-deterministic attacks.
Q: There’s a huge labor shortage in cybersecurity, so two questions related to what you just said about security teams: Does AI help solve the problem — can you substitute AI if you can’t find the right cybersecurity talent? And if you still need humans in the loop, do they need different skill sets? Should corporates consider hiring people with backgrounds in behavioral science, for example, in addition to people trained specifically in cybersecurity?
NC: Yes, and yes. We need AI to do massive data analytics that go beyond the scope of what humans can spot. We need to empower level-one and level-two SOC analysts [key members of a dedicated Security Operations Center team] to work at scale and faster, especially now that speed is critical. Offensive use of AI can run 24/7, constantly finding new ways into an organization or targeting vulnerabilities we don’t even know exist. Defensive use of AI enables immediate, autonomous response and containment, which buys the human SOC team time for more exhaustive analysis and remediation — tasks like resetting an identity or an endpoint. You want autonomous containment for surgical response actions, but you still want a human to perform the more exhaustive remediation.
There’s also a huge need today for integration engineering — making sure the products in your security stack are well integrated. It’s essential to test visibility and autonomous-response capabilities, and to confirm that the tools you’ve procured are functioning the way you think they should. I like to quote Nicole Perlroth [a venture capitalist who served on the advisory committee of the Department of Homeland Security’s cyber defense agency, CISA, and the Council on Foreign Relations’ Cyber Task Force] on this. She once said the major cyberattacks didn’t happen because organizations lacked tools — they happened because organizations failed to deploy and integrate them properly. I think that will remain a very human part of this: validating, deploying, testing, and confirming that the tools you’ve deployed are being used the way you intend.
On your second point, you’re right that we’ll probably see a skill-set shift in cybersecurity with AI adoption. I do think you should have a variety of backgrounds on your cybersecurity teams. When you’re talking about mass data analytics — and now using a lot of AI-powered tools to do that — having different perspectives, experiences, and backgrounds on your data science team helps mitigate the risk of bias in your data and analytics. I’d encourage organizations to bring in different backgrounds and perspectives, including psychology and behavioral analytics, and to keep a human core in the SOC to optimize the tools that protect the organization and reduce risk.
Q: We’ve talked about external risks. Can we talk about internal risks, such as shadow AI — the use of any unauthorized artificial intelligence product in a business setting?
NC: Shadow AI is a huge risk from a data-loss perspective, but there are others too. There’s a lot of experimentation inside organizations aimed at boosting productivity. We did a study over the winter and found that more than 90% of organizations had unregistered MCP [Model Context Protocol] server data flows. If it’s unregistered, it’s most likely someone inside the organization setting it up to gain a productivity edge — say, by adding a natural-language interface to a large data corpus — but that itself is a major data risk. There’s also a Resume Builder study that found nearly 30% of Gen Z employees are actively sabotaging AI within their organizations. Lastly, AI assets are non-deterministic and probabilistic by nature which means AI tools, particularly agents may start making decisions and taking actions that were unintended and potentially be quite risky. All of these risks need to be mitigated with AI adoption.
Q: What do you mean when you say employees are actively sabotaging AI? What are they doing?
NC: It’s almost a resistance stance: they don’t want AI to take their jobs, so they try to make sure it isn’t successful. There are a few ways people do this: poisoning the model or data set by feeding it wrong information; jailbreaking it, the way threat actors do, by giving it different instructions than the ones it was deployed with; and getting agents to take actions that violate policy, then placing the blame on the agent.
Q: How does a corporation police that kind of behavior?
NC: That’s the job of the security organization. They need to build insider-risk profiles and apply behavior-based security analytics and anomaly detection to understand how people in the organization are interacting with AI tools and assets — whether that’s an enterprise generative AI tool, a low-code agent, a high-code agent or autonomous agent. You need that visibility to see what normal interaction looks like, spot anomalous, high-risk activity you wouldn’t want on your infrastructure, and autonomously containment the threat.
Organizations also need full visibility into how AI is being used. For a generative AI license — interacting with a model or a low-code agent — you can get that through integrations with the enterprise license itself; ChatGPT, Copilot, Claude, and Gemini all support it. For autonomous agents, you need visibility into what they’re doing, what they’re interacting with, what actions they’re taking, what data they’re moving, and who they’re communicating with outside the organization. That requires a network or cloud tap; some people use an endpoint agent or browser extension. The point is to have that visibility and be able to apply behavior-based analytics.
I like a lot of the guidance coming out on this right now, including the Five Eyes [an intelligence and security partnership between the United Kingdom, United States, Canada, Australia, and New Zealand] guidance on careful adoption agentic AI services.
I’ve been impressed with how quickly this kind of guidance has emerged to help organizations get their heads around this. A lot of it matters to us because we’re ISO 42001 certified for AI management systems, which means we need to maintain a full risk registry of every AI system used across our enterprise — including everything we’re building in-house. But, also, HR, finance, go-to-market, sales are all using AI-embedded tools in some way. As an organization, we’re in the middle of a massive innovation and adoption cycle, and we’re taking on real risk. The question is how you mitigate that over time so you’re not accumulating what I call risk debt that keeps growing. A lot of these frameworks and resources help organizations build a defense-in-depth strategy so they can manage that risk instead of continually adding to it.
Q: It’s one thing to have a strategy, it’s another thing to have the right tools. The restrictions around who has access to Mythos, a powerful new frontier model that outperforms other AI models at finding and exploiting cybersecurity vulnerabilities, and the temporary ban on non-U.S. citizens’ use of Anthropic’s Fable 5 put companies and countries outside the U.S. in a precarious position. It’s up to the whim of private companies or the U.S. government about who gets access to the latest AI tools and when.
NC: I think that’s a concern for a lot of global organizations. You’re seeing organizations look to diversify their AI supply chain to consider availability, uptime, as well as associated risk. Organizations are also concerned about the differing government regulatory frameworks and compliance requirements that global organizations face. In Europe, we have the EU AI Act to comply with. How do we make sure we can fulfill that requirement with whatever tool provider we use? Often the real question becomes what additional controls, monitoring, and visibility need to be layered around these tools to meet the EU AI Act and other regulatory frameworks.
Q: If press reports are right that the U.S. government is taking a 5% stake in OpenAI, and other hyperscalers are pushed into similar arrangements, doesn’t that create security issues and risks?
NC: Generally, I think we’re already seeing AI tools follow a similar path to the cloud market where organizations evaluate and adopt tools from a range of vendors and sources to ensure resilience. I’m seeing organizations evaluating a variety of tools including open-source models because they offer more granularity and control and can deploy within environments that safeguard data residency and data privacy. That’s one of the biggest concerns with AI tools: what data is being used within these models, and can I control that? I also think we’ll see a lot more flexibility in deployment structures for organizations using frontier models. Look at Saudi Arabia: they’re using Microsoft, but with their own data centers, their own cloud, their own AI deployment. That let’s them mitigate the risk of sensitive data leaving the country.
Q: I wanted to go back to some of the newer technologies, and the dangers agentic AI introduces.
NC: One of the first questions to answer is: who owns that risk? When organizations adopt agentic systems, they’re going to take on that risk — and I think they have to. What’s interesting is that the security organization has historically been segmented from IT or the CTO’s organization, and you see a lot of rapid AI adoption happening in environments where security isn’t even aware of it. To me, that’s the riskiest decision an organization can make: if you’re rapidly adopting agentic systems and your security team isn’t aware of it, or isn’t wrapping security around it, you’re massively increasing your risk. I expect we’ll see some reorganization to fix this.
That’s not the only fix needed. Providers of identity control planes [a centralized framework that manages and governs all identity-related operations across an organization’s digital infrastructure] are going to have to offer much more granular access capabilities. Right now, a lot of organizations are giving autonomous agents the full permissions and access of a human — and that’s a huge risk, because my little low-code research agent doesn’t need access to my HR system. Think about the many ways that access could be abused. I expect the identity control plane will be fixed in the next six to 12 months or so, moving from human-like access for autonomous agents to much more granular, role-based, maybe just-in-time access. Ultimately, organizations need to treat autonomous agents for what they are: an insider-threat risk. Security needs to stay on top of it.
Q: What about industrial espionage and cybercrime from state-sponsored hackers, which is on the rise — the topic of our panel at VivaTech?
NC: If a state-sponsored actor gets its hands on an autonomous agent and moves laterally through an environment that hasn’t wrapped visibility and security around it, it just looks like a human identity moving around — a pliable, helpful insider they don’t even have to bribe.
We published a threat report on April 2 analyzing tactics across Chinese state-sponsored groups over the past three years. It found that 80% of observed attacks targeted critical national infrastructure rather than government directly — ICS [Industrial Control Systems], operational technology, power, water, energy, manufacturing: the systems countries and regions depend on. The report also distinguished between approaches used for intellectual property theft versus surveillance, which relies on low-and-slow staging for potential disruption. It was fascinating to see the different tools and approaches. With low-and-slow tactics, attackers simply don’t want to be detected — and in many cases, they aren’t. Detecting them comes down to full visibility and behavior analytics: you need to spot the small blips on the radar and piece them together across domains. An attacker might not beacon out regularly, but something that happens every 40-plus days, and you need to be able to see and detect that pattern.
The more worrying piece is operational technology. ICS environments are increasingly targeted because they run on legacy infrastructure that can’t be updated, creating vulnerabilities that let threat groups get in. It’s so interconnected with IT, and most OT engineers aren’t cybersecurity experts — they often don’t realize the risk they’re adding to the organization. Many organizations still run separate IT and OT security programs, even as OT and ICS environments adopt more AI for things like probabilistic modeling of power consumption — which expands the attack surface.
Q: What would you like The Innovator’s readers to take away from this interview?
NC: Invest in security as much as you’re investing in AI. At the board level, the conversation shouldn’t stop at ‘how much are we investing in AI?’ It has to include ‘how much are we investing in security to wrap around it?’ Security also needs visibility across the enterprise — AI can’t be adopted without their knowledge. There’s a debate now about whether to invest in humans or in AI; I’d recommend both. Corporations may not have the budget to substantially grow the human side of their security organizations, but they should focus on growing the skills of the people already there, so they can use AI-powered tools for security and integrate them across the enterprise. These conversations need to start at the top.
This article is content that would normally only be available to subscribers. Become a subscriber to see what you have been missing
