AI-generated code in software development

Patrick Carey at the Synopsys Software Integrity Group explains how AI generated code can assist with rapid and secure innovation

It seems like we have been suddenly thrust into the age of Artificial Intelligence (AI).  With the rapid rise of commercial Large Language Models (LLMs) and Generative AI tools like OpenAI ChatGPT, Microsoft Copilot, and Google Gemini, everyone is now talking (and often worrying) about AI.  

But AI concepts and systems have actually been around us for decades. The term itself was first coined by John McCarthy in 1956 to mean “the science and engineering of making intelligent machines.”  For many software development teams AI is not new or particularly scary, but something they’ve been refining and integrating into their software for years. 

Machine Learning (ML), for example, is a category of AI that has been used for a long time in everything from fitness tracking apps to antivirus software. It focuses on enabling computers to perform tasks without explicit programming, with algorithms and models that enable programs to adapt and learn over time.

ML is good at identifying and acting on complex patterns in data, and is useful for tasks like spam and malware detection, predictive analysis, and computer-aided design. It’s so widely used that people often don’t even think of it as AI anymore.

But AI took a huge leap forward a couple of years ago with the launch of the first commercial LLMs and Generative AI tools.  These built on Machine Learning and a number of other AI technologies, including deep learning, neural networks, and natural language processing, as well as massive amounts of internet data to do something ML-enabled systems never did - think, or at least appear to.  Now you can simply “ask” or give an open-ended problem to a computer, and it will respond with a “new” answer.  This could be anything from a simple block of text to computer code to a work art.   

But reports quickly surfaced of Generative AI’s limitations: hallucinations, plagiarism, copyright infringement, and even very human-like mistakes. This leaves organisations, who see the tremendous potential of AI, looking for ways to get the benefits while minimising the risks. 

Gen AI will disrupt software development 

It’s not a question of if, but when.  Generative AI is already being used by many developers to assist in developing code. AI has the potential to dramatically improve developer productivity, helping them create and refine code faster than ever before. In fact, research suggests that generative AI can reduce the completion time for writing code by 35% to 45%.  

However, the way Generative AI uses the data/works it’s trained on to develop “new” works has raised the same intellectual property concerns for software as it has for written and visual art works. We’ve entered a phase of litigation against both the organisations developing these LLMs and those who have publicly claimed and shared ‘new’ AI-created works, like songs and images.

One such case is a class action lawsuit by developers against Microsoft, GitHub and OpenAI, who are being accused of violating copyright law by allowing Copilot to regurgitate licenced code snippets without providing proper attribution. Although this specific case has been thrown out, the legal wrangling over the issue of IP ownership is far from over.

Does AI-assisted coding make for better software?

As previously stated, the use of AI in software development will soon become common place. But as anyone who has used a tool like ChatGPT knows, the responses produced by AI can often be nonsensical, skewed, or flat-out incorrect. 

The same applies to AI-generated code.  LLMs are trained on vast amounts of publicly accessible code.  Much of it is good, but much of it is not. If developers have blind trust in the output, the results can be disastrous. AI can, and often will, generate buggy and non-secure code. 

So, is AI-generated code better or worse that that written by human developers?  Initial analysis of GitHub Copilot indicated that it is good at avoiding certain types of security weaknesses but not others.

In general, it is best at avoiding common syntax vulnerabilities. However, it’s not as good at avoiding those that stem from more complex interactions between applications and external systems and users. Bottom line?  Your mileage will vary depending on the code your team generates.

A not-so-novel approach to a new problem 

It stands to reason, then, that if software produced by AI is only as good as the human-developed code it was trained on, we need to take the same approach and precautions to reviewing and testing AI-generated code as we do when humans build it. This includes static analysis, which can identify both code quality and security defects. 

Most of the code used to train LLMs is open source - software that is freely available in the public domain under the terms of an open source license.  It’s not surprising that AI-generated code often includes “snippets” of open source components. Failure to identify and comply with their associated licenses can put teams at risk for IP litigation.

This is where software composition analysis (SCA), which many software teams already use to track the use of open-source software and license obligations in their applications, can help.  But this is only true if the SCA tool is capable of the fine-grained analysis needed to detect these snippets, so it’s important for teams to make sure their tool is up to the task. 

AI governance is essential

Generative AI will be as revolutionary to software development over the next two decades as open source has been over the last two. But as with open source, proper governance must be put in place to ensure teams can realise its full potential. This is a conversation that should happen across developer, security, legal and risk teams. 

First, it is imperative that organisations define their rules for AI usage in the workplace. The AI policies put in place must include IP protection as a top priority – both for the organisation’s IP as well as that of the open-source projects underpinning AI-generated code. 

Given the significant risks from poor quality or vulnerable code, organisations should also select (and vet) their AI-coding assistants carefully, using the same discipline they would for any 3rd party vendor. Teams should be wary of free tools (where is your data going?) and understand the data privacy and IP policies of any platforms they consider. Doing adequate due diligence can reduce risk significantly. 

Finally, don’t trust AI-generated code outright. Verify it, carefully. Static analysis should be used to catch code defects and vulnerabilities, and software composition analysis at the snippet level should be performed to also identify potential IP violations. 

A more productive, innovative and secure future

One thing is certain: AI-enabled software development is the new normal.  By taking a thoughtful approach to AI, leveraging the same practices used for human-developed software, organisations can indeed capitalise on its tremendous potential, while navigating its inherent risks, to deliver more secure, high-quality software, faster than ever before.

Patrick Carey is senior director at the Synopsys Software Integrity Group

Main image courtesy of iStockPhoto.com and monsitj