Daniel Ofosu's blog

The Advent of AI Coding

Originally published on Medium on May 16, 2022. (Read the original)

Exploring the new systems capable of generating code from natural language.

Automation is regarded as an area which will have major effects on our life in terms of labor markets. Many jobs have already transformed from being human-powered into being machine-powered with physical labor fields leading the way. Scientists have estimated that automation could replace even as much as 45% of the jobs people are currently doing, which means that a large part of our jobs will be transformed.

People often think that brain intensive workloads will be almost impossible to replace. These include areas such as programming, managing people and education. Yet, as we will later see, even some of these areas are not safe as AI has finally developed to a point where it can produce human-like code. This could lead to a situation where humans are only required for deploying the AI that then generates more code.

What is AI text generation?

One of the most exciting and hot research areas in AI is called Natural Language Processing (NLP). This is what powers systems such as Apple's Siri and Amazon's Alexa. Other areas include for example document reading where the AI is able to understand some context from a document and for example generate a shorter version of an article. Especially natural language understanding is an area of NLP where the advancements have led to huge discoveries, and I believe that through natural language understanding it is possible at some point to arrive at a point where AI can understand human language and all the cultural nuances as well as we humans can.

Previously NLP has been conducted through symbolic and statistical methods, but more recently deep learning neural networks (DLNNs) have been utilised in the task instead. Using DLNNs lowers development time as they are able to learn features on their own. Most of them work through predicting next words in sequences with the context in mind. This allows the network to also take into account other areas besides the project itself for learning purposes. For example, feeding data from all kinds of human text interactions can provide a huge amount of information regarding semantics and more culturally-bound things such as how memes work or what is sarcasm.

The Frontrunners: OpenAI Codex

Codex, is an artificial intelligence model developed by OpenAI, one of the most advanced research labs in the field at the moment. It is based on GPT-3 which is one of the best-known deep learning language models in the world, also produced by OpenAI. GPT-3 is often seen as a forerunner in utilising deep learning for various tasks related to reading and producing human text, and has in many cases already passed the popular Turing test. GPT-3 and thus Codex are one of the most interesting technological advancements ever as they can pave the way for AGI (Artificial General Intelligence) which is in many cases seen as the end goal of AI systems. However, it is to be noted that this doesn't come without risks, and even the developers themselves identified many areas in the paper they published where the system could be misused. These included things such as fraudulent essay writing, social engineering and spam/phishing.

OpenAI Codex has been trained on 159 gigabytes of Python code, and is able to complete 37% of requests. This means that it still is more of a complementary system for humans but definitely is leading the way in replacing us completely. It is currently in closed beta where the beta users can try it out in various tasks with limited capabilities. I myself was one of the lucky ones to get access and managed to test out many things before running out of free computing power on their platform. I can say that it was truly mesmerising how well it was able to do almost any coding tasks that I threw at it. For example, writing "Create a chess game in Python" produced a fully-fledged codebase of chess game with local multiplayer that I could then run on my system. The usability is still somewhat limited as many applications require dependencies and are never coded on a single file as the Codex did. However, one of the most interesting areas it is currently being used on is augmenting and pre-filling parts of code that a programmer is writing.

In an attempt to aid developers in creating code, OpenAI has developed together with GitHub a tool called GitHub Copilot which is based on Codex. Its main feature is generating suggestions for lines of code or functions inside a code editor. According to them it understands vast amounts of context and it is able to do tasks such as synthesising code from a comment. For example, writing a function for calculating how much time has passed since a point can be expressed as a comment in plain English and the Copilot will write the code part of the function. Other areas where it will save a lot of time from developers include autofilling repetitive code parts, automatically writing tests and showing alternative solutions for a given code. The possibilities are endless and through increasing adoption of Copilot the network will be able to learn even more from the users and thus improve the solutions and provide new novel areas of development.

Challengers to OpenAI's Codex

In addition to Microsoft with OpenAI Codex, Google is also working on a similar project through their DeepMind research organisation titled AlphaCode. The paper describing the project was released in March 2022, in which they demonstrate the capabilities of AlphaCode in competitive programming. Competitive programming is a popular field where different computer scientists battle in creating efficient solution on complex problems which are specifically created for the competitions. DeepMind claims claims that the performance of AlphaCode is even better than that of Codex and that it outperformed average programmers in contests hosted on Codeforces, which is a competitive coding platform with an Elo rating system. It is to be noted that this initial version of AlphaCode is only applicable in the world of competitive programming, however it will most certainly be applied later on into other areas.

Another large project in the field is PolyCoder. PolyCoder was not developed by a commercial entity like AlphaCode and Codex, but rather researchers at the Carnegie Mellon University. It does not match the breadth of the above discussed methods, but according to the researchers it surpasses them in the generation of C language code. Notable is that it is one of the only open-source code generators, as the amount of computing resources required to train a model are so vast that most of the organisations in the world cannot provide such resources. Other organisations are also exploring the field, for example Intel with ControlFlag that detects errors in code and Facebook with TransCoder, which converts code between programming languages.

In addition to DeepMind's AlphaCode, the researchers at Google have also recently come up with PaLM (Pathways Language Model) which a 540-billion parameter model trained with Google's own Pathways architecture. For those not familiar with ML development, having 540 billion parameters basically means there are 540 billion different pieces of information that the model is using to predict something. The Pathways architecture was created to mimic human brain more closely than previous methods and it is able to perform a multitude of things utilising a single model through specialising different neurons to do different things, as in a human brain. How this differs from current state of the art architectures is that it enables multimodal models which might for example utilize visual, auditory and language understanding at the same time instead of processing a single modality at a time. More projects like this are in the works in many different companies and we can expect to see a lot of new and improvised models in the recent years.

Possibilities and risks of AI generated code

As said before, the main application currently for AI-generated code is aiding programmers in mundane tasks and autofilling parts of their code thus speeding up their work process. It is obvious that it still cannot fully replace a programmer, and most often it requires a somewhat advanced programmer using the systems to understand the code that the AI generates. However, the process in the field is rapid and I am confident that within the next decade or two we will begin to see applications fully developed by AI. But even as a complementary tool it still provides great value.

Although the biggest problem is arguably the inability of the systems to generate more complex applications, there are also other concerns. Maybe the largest one of them being the security flaws, which are also partly the reason why researchers at PolyCoder and elsewhere are pushing for open-sourcing these products, contrary to what DeepMind and OpenAI are doing. This would allow security researchers to look and test the products better so that they can be more robust. As the models are trained from public data, people can for example hide malicious code in them which might for example show with a given keyword and executed somewhere where harm can be done.

A study found that up to 40% of the output of Copilot contained security vulnerabilities, which seems like a cause for concern. These challenges inevitably hinder the growth of the products and limit the usage to assisting programmers before they can be trusted to work on their own.

Another challenging area as mentioned before is the computational cost of training a model like this. Proper training can only be achieved by big players like Google, Amazon and Microsoft who host their own cloud services with great amounts of computing power available. This hinders the innovation as only a small part of the research community is enabled to building these models. There are already promising prototypes, but many of them can't be deployed due to a lack of resources. There might exist other options than open-sourcing everything, such as utilising blockchain technology to do decentralized model training with the use of resources from different groups.

Conclusion

Overall AI-generated code is something to definitely get excited about in case you are a programmer now as it most probably speed up your workflow in the coming years. However, this might not be the case in a few decades anymore and we might see the type of programming we know now slowly fade away as the AI gets better and better at creating software. To some people this might also sound scary, and it is why proper AI ethics handling is getting more and more important and we should start creating better societal structures to handle the transition to an AI-assisted labor market. As you can see, naturally the future of AI is an extremely controversial topic, and it is hard to say which kind of a future scenario we will have — will the work we humans do be obsolete thanks to cyber brains or will we be able to coexist and enhance our skills through AI. That is a question that remains to be seen and at least I am excited to see where all this goes.

References