This project explores predetor and prey interection using NEAT algorithm. The aim for this project was to get better at programming in Java and get a hands on experience of building and employing a reinforcement learning algorithm, so any techniques that are outlined here were encoded from scratch, using only the most basic libraries.
PredatorsPrey.mp4
The environment I created for my actors is, fundamentally, a grid of cells: it is easier to start with something discrete. Despite that simplification, I wanted the generated world to look natural and have a similar terrain that one might find in grasslands: it has some bushes for prey to find food and, maybe, hide from predators, water reserves, a bit of sand. To achieve this I learnt how to create Perlin noise, a fascinating algorithm that made me revisit my Linear Algebra notes, and then transform it into the discrete intervals corresponding to the types of terrain and their proportions. However, the resultant world had very rigid and clear transitions between different terrains. That is not at all true about the real world. So I applied Gaussian fitler to smooth edges and create a nice transition. All that was left to do is to generate some berries that would serve as a food source for prey. The solution to that is just sampling points from another map of Perlin noise that generates the distribution of berries at each point corresponding to bushes in the original map.
NEAT stands for neuroevolution of augmenting topologies. The algorithm creates an optimal structure of a neural network for the given problem over the course of multiple iterations using evolutionary approach. Each actor in a simulation attempts to solve a given problem. With a certain objective set, actors are awarded or punished based on their perfomance. Once an iteration of the simulation ends, those actors who perfomed better than the rest are selected and reproduced to create offsprings, whose genome also undergoes a mutation. The simulation runs anew with the offsprings of previous generation. The steps are repeated for a certain period of time, after which, theoretically, we should get a structure of a genome that achieves the optimal score on the given problem. The benefit of the algorithm is that it finds the optimal neural network for a particular problem without an intervention of a human.
Now, in this simulation I am evolving two neural networks simultaneously: one for predators and another for preys. Structurely, those actors are encoded the same way: they both have a genome that corresponds to their neural network, they have the same activation functions, they have the same possible actions. The main difference is their objective. Predators are rewarded for catching prey, staying alive, mimimising distance to prey and penalised for idle behaviour. In contrast, preys are rewarded for avoiding predators and surviving as long as possible. Thus, a more complex dynamic arises where the problem that each actor is optimising for is constantly evolving.
So far, I implemented the backbone of the NEAT algorithm for predators keeping preys stationary to make the problem easier. Now, I'm working on the code to make predators more efficient. After I'm satisfied with the learning rate of predators, I'll make preys dynamic and look for any emergent behaviour. I want to see whether the dynamics of my simulation follow Lotka-Volterra equations. It would be interesting to add a more complex behaviour: allow preys to hide in bushes, make actors drown in the deep waters, or add a change of speed in different terrains (slowing down while running on the sand).