In this article, we will examine some of the emerging use cases for the inclusion of AI in Red Team operations. We will not attempt to examine all cases, but to hone in on a few which I feel are "just around the corner" and also the most immediately impactful to the security space. Namely, we will cover the use of AI for operational decision making, code generation, and runtime code synthesis. I'd like to include 'AI as a C2', but that should probably get it's own post (soon).

So let's examine the 3 use cases I've identified so far:

  • Operational Decision Making:

    • Based on specific intelligence about a system or systems, AI can be used to make determinations on how to best proceed as an offensive operator, with a focus on security evasion.

  • Code Generation:

    • One of the most popular uses of chatgpt, copilot, and other AI platforms. LLMs are increasingly being used for code generation by all sorts of developers. There is no reason offensive researchers cannot or will not also use it for this. I have wrote several papers and presentations already on the topic.

  • Runtime Code Synthesis:

    • AI can be used to generate code dynamically at runtime and subsequently load or integrate that generated code into the running program (implant). This new technique encompasses both the generation and loading aspects, emphasizing the synthesis of code at runtime.

Some of these use cases are already fairly popular, like code generation, but others, such as runtime code synthesis, are just emerging and perhaps yet to be seen "in the wild." Let's dig into each one a bit further.

Using AI for operational decision making sounds scary at first, but it also seems inevitable. Consider that AI is already being used for defensive operations. Most Anti-virus and EDR vendors are proud to be able to say that their product uses AI or Machine Learning to some degree. People think it helps. Well, why exactly would the same technology not be used by "the bad guys"? Of course, there is nothing preventing that, and it will be used for malicious purposes. Rather than attempting to somehow prevent that, through the use of regulation and politics (never going to work), it seems a better approach may be to understand what that means, so that you are prepared to properly defend yourself.

So what exactly will hackers gain by leveraging AI for operational decision making? What can it do? What does it make possible? These are the questions I feel we should be investigating, rather than trying desperately to "put the genie back in the bottle", as it were.

Operational decision making means simply, that instead of a human red teamer, manually pouring over troves of collected intel (ips, hostnames, user names, operating systems, vulns, cves, etc..) in order to discover vulnerabilities, you have AI do that in seconds with much greater capability. Now I know, people want to jump straight to automated pentesting and red teamers will either scoff at the idea of being replaced or be struck with fear. But, consider a middle ground, where current hackers are empowered greatly. We are not going to get rid of the enemy soldiers, but they just got upgraded from knives to guns. That's the immediate scenario.

So here's how it goes. Red team gets access to a target environment and runs basic enumeration, but instead of analyzing all that data by hand, it gets fed into an AI model designed to parse it and provide recommendations on things like "what process injection technique should be used for injecting shellcode into a remote process?" On top of that, the operators are no longer googling, but instead asking the AI for recommendations on how to proceed. Now this is a good segue into the next use case:

Code generation. Now consider that once the AI has analyzed the intel and determined the best actions to take next based on the environment, that it can also generate any additional code for carrying out the attack. Again, this seems inevitable, given how good the current LLM models are at code generation. I have already written several posts on the topic and given some presentations, and every day I am following researchers who are building programs using mostly code generated from an LLM model. There's no reason it will not see continued and increased use in the offensive security space. If you are interested in how it might be used to continue our scenario I began above, about writing a custom process injection, you can refer to another of my articles on this blog, titled "Coding with AI".

And the final step it would seem, tying these 3 use cases together, is of course, runtime code synthesis.

Runtime code synthesis is a step further in the application of AI to red teaming operations. This practice is quite novel and pertains to the generation of code at runtime based on the changing conditions and parameters of the system being attacked.

In essence, runtime code synthesis takes the potential of AI in operational decision making and code generation, and enhances it by adding a layer of dynamic responsiveness. The AI, in this case, has the ability to adapt its behavior based on new information it gathers, on-the-fly, while interacting with the target system.

In our ongoing scenario, imagine that the red team has successfully injected code into the target process as per the AI's recommendations and the code it generated. However, the conditions on the target system may change - perhaps due to the defensive actions of the blue team or automated security measures kicking in. The code the red team originally injected might not be suitable or effective anymore under these new circumstances. This is where runtime code synthesis comes into play.

With runtime code synthesis, the AI could potentially create new code to adapt to these changing conditions, right as they are happening. For instance, if a defensive measure in the system detects the injected process and tries to shut it down, the AI could generate code to move the injected process to a different, safer location within the system, or cloak it more effectively.

The advent of AI in red team operations is undeniably a double-edged sword. On one hand, it can enhance the capabilities of ethical hackers to identify vulnerabilities and strengthen security systems. On the other hand, it can also be used maliciously, potentially making cyber attacks more sophisticated and difficult to counter. This only underscores the importance of keeping abreast with the latest developments in this field, understanding their potential applications and implications, and using this knowledge to fortify defenses.

Things are about to get very interesting.

twitter: Kirk Trychel(@Teach2Breach)