Assessment 2 Blog

End of Project21 / 04 / 2022

I think my Project is in a state where I am happy to call it complete. Given the tools I have, I believe I've achieved what I set out to do, using GPT-3 to generate textures, materials, and environments for levels for a game. Of course there is always more to add, but I am happy with the fact that anyone can enter a prompt, get a level and think "Yeh, that makes sense" when they see what the game comes up with. It was an interesting experiment and I feel I made use of GPT-3 to be able to do what was previously impossible, getting real time information about an arbitrary prompt from the user and utilizing it in level design. The basic theming I added makes it feel less like a prototype, and while there are edge cases where the Project acts unexpectedly, it works quite well considering the entire process is being backed by an AI.

My main improvements were I to do this again would just be to try get a better classification system done sooner. The "Reverse Akinator" approach I took could definitely be refined with more research and testing, perhaps I'd be able to find other people who have tried to categorize all objects (A game like Scribblenauts comes to mind) and see how they completed this task. For now, I might spend a few days polishing the project and tweaking, but I'm content with where it's reached.

OpenAI Update19 / 04 / 2022

This doesn't affect the project much at this stage but in the recent weeks OpenAI has released a number of new technologies and changes. The first is that GPT-3 can now Insert and Edit text you send it as well as completing a prompt. My system works well enough but it would be interesting to see if I could've utilised the Edit functionality had it been available. The second much more major update is that they have opened the wait-list for DALL-E 2, which is able to generate images and art pieces from prompts, as well as performing advanced modifications like "age this person down". I have experimented with other similar technology and the main downside they had was the wait times from prompt to useable image, anywhere between 15 to 30 minutes. It's not clear what the time is for this one but the results are extremely accurate and very powerful.

Mostly it's just interesting to see that someone doing this project a year from now would have more tools than I do now, but the same could be said for myself, as a year before I started GPT-3 wasn't available publicly. It really just shows how fast this technology is moving and is a large part of why I wanted to do this project now.
Link to the Paper

Project Theming18 / 04 / 2022

When I came up with this project I had the idea in my head that almost everything you see should be computer generated from scratch. That immediately sort of went out the window when I introduced the third person character, but I still couldn't really see the game having any kind of theme or story, even in the menus, when the levels you enter are so abstract and wide ranging. However I was looking at the game LEGO Worlds recently, which uses the premise of you being a generic space explorer visiting planets which all have different NPCs, biomes and objects on them. The planets do all have a pre-made theme like Winter Wonderland or Volcano, which does make some planets very similar if them have the same theme, but its interesting that the game takes the fact of you the player exploring these random worlds and uses that as your characters theme too.

While my project isn't exploring full worlds, I realised I might be able to still add a bit of scenery to the menus and transitions so its at least slightly more interesting than you just entering text, waiting a few seconds and then suddenly appearing in a level. My current idea is having the Robot character just falling through a simple portal effect of some kind, similar to the Doctor Who opening (Though obviously far more basic). I could incorporate my branding of the black and green "Terminal style" to embrace the fact that everything is computer generated and give the UI an interesting theme. I think it might add a lot to the project to have even a basic thread of story to the player going through all these levels.

Change to the AI Categorization14 / 04 / 2022

After some testing I found there's a change I need to make to the AI, and that is to block prompts that are real peoples names. I realised that there's a big oversight on my end with making an AI that "Assigns a colour to a prompt". As far as I can tell OpenAI has very good filters and training for offensive prompts, which I am using, but given what the AI is doing, theres always a chance it could be unintentionally offensive if it's instructed to use data from a real person to make a level, especially as it isn't clear where exactly that data is coming from. For this reason, I've added a check for the AI to identify if the prompt entered is a real person, and block it if that is the case. It should still allow completely fictional cartoon characters for example, but I think banning the AI from pulling materials from real peoples lives is the right choice.

Implementing the Flowchart12 / 04 / 2022

It took a while but I believe I have the data system implemented to a good standard. As per the flowchart design, the game changes the skybox for places to match an appropriate time of day if there is one, and spawns walls with relevant textures as a kind of skybox. I still use the old system if the game believes one of the materials is a gas, or if there is land/liquid as part of the prompt. It also now has better knowledge of material properties, and can generate, for example, "geometric rough grey texture" for stone, or "noisy rough red texture" for fire. I improved the system somewhat to prevent black materials generating pure black textures, but I think maybe I can still iterate on it further. In the course of developing these systems, I had a few more Ideas on what to add to the project to improve it, so I think I may work on those ideas next.

Data Collection Flowchart4 / 04 / 2022

After a lot of work, I have designed a flowchart for the lines of questioning the Data Gatherer will use. It has a contingency for if it believes someone has tried to enter a nonsense prompt like "ifhsdiofsd", and can detect places, objects and living things, as well as if you enter a simple substance like water. I believe these categories should encompass enough of the world to make the game work well. It then can break it down further, asking the specific colours and features of these categories, so it asks if the living thing has fur, if the place has walls, and if the object is all one material. The goal is to get to a point where it has a list of substances which can then themselves be investigated to turn them into textures and materials.

Black and White Game Jam12 / 04 / 2022

Took another break over the weekend to do another Game Jam. This one was about making games with only Black and White. The theme was "Loop", so I had the idea of a game where you play as a loop and use momentum to swing yourself from one peg to another, climbing a peg wall to get to the end. This also ended up being a bit last minute, but given the restrictions it came out really well:

The Reverse Akinator Approach27 / 03 / 2022

After doing some thinking, I realised what I'm trying to do is effectively a Reverse Akinator System. Akinator is an online game where a genie uses a series of questions to guess the object, person or place you are thinking of:

I instead want to design a system which does the reverse, knowing what your object is called but not knowing the properties of the object. However, I can definitely take inspiration from Akinator to help me create a categorization system for prompts. My new plan is to categorize prompts in general terms like location, animal or object. GPT-3 is very good at sorting and classifying objects into groups, so this should work great. Then once I know what it is, I can design a system to interrogate each of these types. For example, if I know its a place, I can ask about the walls, the floor etc. This should allow me to make a unique level based on the prompt without the game having any information programmed into it by default. This is important as it should allow any prompt to work regardless of the colours, textures or materials associated with it. For now, my main task is designing a series of categories that encompass all objects in the world to a reasonable degree, while still being narrow enough that I can personalize the level generation for each one.

Material Problems19 / 03 / 2022

Now that I have a basis for the game, I decided to revisit some of the research I had previously done in preparation for this project. I looked at how I would go about breaking advanced prompts down to give me materials to build a level with. My first instinct is to just ask the AI if a prompt can be broken down into multiple materials. The problem is that this ends up being an ambiguous question a lot of the time. For example, should "Field" be broken up into Grass and Dirt? Should the Ocean be considered to be made of Hydrogen and Oxygen? I know the answer is "No" but I need to find the right way to word the prompt to the AI to let it know what is and isn't a material, and at what point it should stop breaking materials down further. I'll need to think about what the best way to do this is, as in theory it needs to be able to handle any prompt that could be entered no matter how complex or simple it is.

Playtesting AI Redundant18 / 03 / 2022

In my initial plan I was going to create a playtesting AI that can judge the difficulty of a level as its being created and assures that every level is beatable. However as I am developing the Level Generator, I realise there really isn't a need for a separate system. I can make the system check itself before placing each platform to make sure it is a suitable angle and distance that the player can jump to, and I can make sure it isn't blocking or blocked by any other platforms. This means that a playtesting AI system would really just increase generation times with no significant advantages. For this reason, I'm going to change my plan, integrating the testing as a part of the Level Generator itself.

Simple Texture Generation16 / 03 / 2022

Now that the Levels are generated, Today I implemented the data collection AI, so it can now generate textures instead of just colours, and then passes that information to the Level Generator. at the moment I'm still just trying to get a basic prototype, so I decided on having the AI work on basic materials like wood and stone for now, and later I can then have it break objects down to get these materials. My first attempt at this splits materials by state. For the simple textures, I have a large dataset of generated pattern textures that have been sorted into gas, liquid and solid textures, for now the AI will sort the material type into one of these categories, merges multiple of the dataset textures together, and colours it the appropriate colour for the material. Additionally, I made a few basic environments for the 3 types. Solids have a terrain generated and painted below them, liquids have a flatter plane of the appropriate colour, and gas causes a fog around the player that reduces visibility. I also change the skybox colour to match the material. The results are interesting but definitely far from what I want eventually. Another main issue is that if a material happens to be black, it cant have a texture, since the texture will be tinted black making it no longer visible. I have an idea on how I can tweak the texture creator to fix this, but for now I have a basic, but working demo.

Level Creator AI14 / 03 / 2022

With the player made, today I created a basic Level Generator AI. My first attempt was to simply start at a base platform the player is standing on, then pick a sensible direction and place a new platform. Then, starting from this platform, do it again, and repeat about 20 times to make a path of platforms. This sometimes worked but sometimes went back on itself and went way too close to previous platforms, allowing skipping chunks of the level. To fix this, I added a check before placing a new platform that it cannot be near a previous platform other than the one it is intended to be coming from. I also made it so the Level generator is aware of what direction the previous platform was placed from, and will only be able to deviate from that direction by up to 90 degrees. This means it is harder for the level to double back on itself, and will generally follow a single vague direction for the level.It works well enough to generate a simple challenge for the player, and I can always enhance it later.

GamesPlusJam13 / 03 / 2022

Took a short break over the weekend to do a game jam with a friend. It was fun to get back into doing game jams and the end idea we made was pretty fun. The theme was "Mix It Up" so we made a game where you swap out parts of a monster to give it different abilities and then try defeat it. initially the plan was that you'd beat the game by beating every part of each monster in some form or another, but I ran a bit short on time so we kept it as just a fun sandbox of combining and fighting creatures:


Overall it seems to have been well received, the monsters are a little janky but it's still a fun experience.

Choosing the Player11 / 03 / 2022

So with GPT-3 responding, I now had to start the Platformer side of things. The first thing I had to do was choose a model for the player. My initial plan was going to be to make the game first person, eliminating the need for a player model and having the players screen be entirely machine generated. However after some discussion I was advised that a third person player would let you see the scale of the level more, so I decided to have a player model visible and change the view to third person. For this I used Mixamo, which has a large variety of free characters with sets animations for all of them. However, my problem was that I didn't feel a highly detailed or human looking character would really fit, given that the world you're traversing is going to look quite clearly generated and abnormal, so in the end I chose to use a default- looking mannequin and recoloured it to be black and green, which both fits my general brand and should make the player able to be seen in both dark and bright environments: The player being a simple mannequin character should work well with any environment the game can generate, and should never look wildly out of place in one level compared to another. I downloaded a set of walking, running, jumping and landing animations and put together a player controller using these animations, meaning the game now has a functioning character for the player to control.

Communicating with GPT-310 / 03 / 2022

The first thing I had to do today was try get the GPT-3 AI hooked into Unity so the game can send and receive data. This turned out to be simple in theory, but I spent 2 hours trying to get Unity to send a request successfully before finding that Unity was forcing URL Encoding on all sent data. (replacing "/" with "%2F" etc.) This caused all the requests to GPT-3 to fail, and annoyingly the Unity documentation doesn't mention that it does this anywhere. Anyway, I eventually found a forum post that explained this behaviour, and they had also included a workaround to send requests normally.

So with all that finally sorted, I was able to send requests to GPT-3 and get responses. As a simple test I decided to ask the AI to respond with what colour is associated with a given word. The way GPT-3 works is by essentially being an extremely good "auto-complete" engine. so I send it a prompt as follows:

Sky : Blue
Grass : Green
Fire : Red
Stone : Grey
Honey :

The players prompt being the last line (In this case "Honey"). The AI has knowledge of a large segment of the information on the internet, and has the intelligence to use the context of previous lines to realise that the "completion" of this prompt involves telling you the colour of "Honey", so it responds "Yellow". When Unity receives this response, it has a handy function called "TryParseHTMLString" which is specifically for converting names of colours like "Yellow" or "Magenta" into an actual colour Unity can use, which is extremely handy, and cuts some work off me. With this done, I now have a simple test where the player can enter a prompt which is then sent to GPT-3, and will respond with the colour of the object, which I then colour the screen with:


It's simple, but its a good enough first test that I can send, receive, and act on data using a user prompt

Initial Plan09 / 03 / 2022

My plan for developing this project is to have 3 parts of the AI. First, I will need to create a data collection AI that can take the prompt and use the GPT-3 API to get information about it that can be used to create a level, such as the colours and properties associated with this prompt. it will then need to take this information and generate materials to be used by the second AI, the level generator. This AI will be in charge of creating a level, placing platforms, walls and any environmental objects, and applying relevant materials to them. This AI will work alongside the Playtesting AI, which will check each jump to the end is possible, making sure the generated level is beatable. It will take a lot of work to get these 3 systems working alongside each other, but I think I should be able to create a good final product.

FMP Overview 07 / 03 / 2022

For my final project I have chosen to use the latest advances in AI to create a 3D platformer that can generate entire levels from scratch based on a prompt. I have chosen this as it is a good opportunity to show off my main skills in programming, and involves something I am interested in, that being OpenAI's GPT-3. This technology is extremely recent, and will likely receive updates and additions during the course of this project's development, however their API is in a stable enough state that it shouldn't affect the project for the foreseeable future.

I have had the idea to do a project like this for a while, but with OpenAI recently granting me access to the GPT-3 API, I think this is a good opportunity for me to attempt it and see how far I can take the project in the time period I have. My goal is to see how much of the level I can have generated procedurally based off a single prompt. The ideal scenario is that if the player enters the prompt "castle" for example, the AI should know that a castle level should have stone walls, wooden floors, and a roof. It should then be able to generate a stone material, a wood material etc. and maybe even go so far as to change the lighting and environment appropriately, though I recognise that may be a stretch. This should then all come together to generate a themed level the player can play through.

Assessment 1 Reflection06 / 03 / 2022

I am keeping these blogs to keep track of the progress of my Final Project as well as my aims in the games industry.

After researching the industry and possible jobs, I feel I have got a better grasp of the industry and the way jobs are moving. I have found that over recent years people are beginning to be less tolerant of the crunch culture that is frequently an issue at AAA studios.

I think the pandemic has also changed the workplace due to proving that many jobs can be done remotely and don't require people to be in an office.

Is still want to begin my career working at a games studio, both for experience and as starting out solely making games of my own without having a stable income would be a large risk.