The Mathematical Journey at the Heart of Dreams4cars
17/05/2018
Bringing vehicles to life with Maths
When it comes to developing a car that can drive itself we have to remember, that cars start out having no instincts of their own whatsoever. Cars do not know what objects are and how to differentiate them and evaluate their course. All this has to be taught to them. And maths, with its equations, models and algorithms will be their teacher. This task of literally waking up a car through maths is the role of Sean Anderson from Sheffield University.
Sean is involved in two main areas. The first is mathematical modelling of vehicle dynamics, resulting in algorithms being developed that represent the behaviour spectrum of the car and other road users. The second is to develop algorithms that will allow the car to choose the best course of action. This process is called action-selection.
If maths is the life giver, data is what that life feeds on, huge amounts of data!
Sean and his team collect as much data as they can, in the form of speed, course and driver inputs, from real cars driven by real people. This data then allows them to develop mathematical models in the form of equations that then make up algorithms that describe behaviour. The more data the more accurate the model.
They also use the data to estimate parameters in the models and then they need to validate the models and check that the mathematical models do actually correspond to what they see in the car on the road. That is the process in a nutshell, but there is a lot more to it, including building simulation environments, which allow limitless evaluations to be carried out, in a safe, simulated world.
Simulation is the key to acquiring the huge amounts of data needed to get the car to respond appropriately
Significant developments within Dreams4cars are dependent on being able to simulate vehicle responses in a digital simulator, where all kinds of scenarios can be tried out in complete safety. This system is called Open DS (Open source Drive Simulator), and has been developed by DFKS, a project partner in Breman. The action-selection system is running in a simple two-lane scenario with an overtake manoeuvre.
Before the action-selection process is complete the team will be looking at action discovery, which is the process of the system, or agent, discovering what actions are available for it to choose and clearly a necessary prerequisite for having a fully developed and adequate action-selection system. Imagine you had a baby car that doesn’t even know how to overtake; it doesn’t even know yet that it can overtake a tractor going at 30kph in front of it. A simulation scenario has to be set up where the car tries out different kinds of legal manoeuvres which it can then simulate to its heart’s content to see what the outcomes would lead to, whether that be a successful and safe manoeuvre, a crash, near miss etc.
As the road scenarios become available for testing in Open DS they collect that batch of data - in terms of car velocities, whether they braked, crashed, etc. - log it and then go on and train the AI type algorithm off line with all that data. This, in essence, is the dreaming element of the project – to generate simulations and then update the algorithm to more appropriately respond on the road, based on what’s been learnt from those simulations. We must remember though, at this moment, we’re talking about a simulator, so anything goes and that’s the power of it. It can look at any situation, easily and figure out what’s going on.
As the agent becomes more accurate at modelling and feeling itself in the world around it through maths, this dreaming actually also involves it visualising what it expects to see in front of it in the imminent future, based on what it has experienced while driving. So as the maths gets sharper, so also the image sharpens of what the agent can literally generate, and this is quite literally breaking new ground. There is an extended discussion on this in an interview with the project co-ordinator, Professor Mauro Di Leo from Trenti University.
Road like simulation means fast-tracked data generation and therefore quicker model evolution
When you have an experimental physical environment you can only learn from what happened in the experiment, so near misses and problem scenarios are infrequent. When you build a mathematical representation of a world in a simulation environment, such as Open DS, the smart learning aspect is that you can try a vast quantity of solutions including the very infrequent problem, near-miss and accident scenarios. These scenarios produce enough critical data for the models and algorithms to be updated and so enable the agent to deal with them in the real world.
A significant part of what the project is about is validating and verifying that the learnt algorithms are correct and can operate safely before they are implemented on the road and that’s really essential, because it’s better to have a safe car that you know exactly what it’s going to do in any scenario, than one that’s slightly more optimal but also more risky. The only limit is computer processing power and technology is always increasing this. Simulation is a core link in the chain to give the maths enough data to breath critical awareness into the driving agent.
Once Open DS has been fully developed and the course of simulations is run to the point where the developed algorithms are producing correct responses they will be tested and verified on the independent Car Maker software at Fiat’s CRF establishment, which will invariably give it credibility.
Dreams4cars means that cars at night will dream through what they experienced in the day meaning more data and upgraded driving agents
It is worth pausing and glancing into the future. In the future when the algorithms have been exhaustively tested in virtual simulators and on test tracks, Dreams4cars vehicles will never stop improving, because every journey means more data, which means more analysis, and more accuracy. In theory, data from not just one car, but all Dreams4cars vehicles on the roads can be analysed or “dreamt about” over night in order to drive the very next day with an even smarter algorithm running it. It will be like a software update in some way as we experience it with our mobile devices, but self-taught.
Algorithms are developed through reward-based learning neural networks
It’s time to get back to the nitty-gritty. The current algorithms are a long way off coping with the full spectrum of road scenarios. They need to be improved. They “eat” data, discover how to react to best process that data, readjust and collect more and so on. But up close what is going on? How do they really improve?
The outcome of the simulations is to do with reward, or reinforcement based learning, the classic example being the rat maze with sweets placed where you want it to go. And the rat learns through rewards to complete the maze. In the simulation process when rewarding manoeuvres occur you keep the particular maths that led to the right response and fine-tune the control algorithm.
The problem is that traditional reinforcement algorithms and learning methods can only cope with a small number of manoeuvring states, like overtaking at constant speeds and linear braking. They work well within these rather narrow parameters, but this is clearly not adequate for the endless possible driving states that need to be simulated.
Enter deep reinforcement learning with neural networks - a powerful, reward-based network of interconnected nodes in layers that enable the selection of the best action of the car through a wide range of manouevering states. For deep neural networks to work though, they need huge amounts of data, exactly what Open DS can provide. So giving deep neural networks Open DS data is the perfect match because the neural network thrives on large quantities of data to be able to self adjust or learn to recognise shapes and create the algorithms needed to respond to road scenarios. This way you greatly increase your state-action space and correctly map any given state of the car to an action.
To distinguish the use of neural networks between Dreams4cars and other projects, the latter use them just to process the sensory data, e.g. to recognise other road users from video images, but Dreams4cars uses them for action-selection and manoeuvering.
Modelling starts with limited parameters and expands towards real driving conditions
It is important to remember that while successful fully autonomous cars are currently technology’s Holy Grail, no one has yet been able to produce the Holy Grail. According to the project partners and their careful observations of other commercial and academic projects, no one is even close. Any claim to have developed a fully functional autonomous system should be viewed with some suspicion for quite some years yet.
However, we can say where the search for the Holy Grail starts; by creating a predictable environment, allowing variables in one by one and then seeing what the model will do and how it will need to be modified. Cruise control was actually the very first step and a very satisfactory first step, because it could do what it claimed to do. Over long journeys on large motorways, not needing to use your foot at all is a very useful tool. It doesn’t change your world, but you appreciate the rest it offers you.
The real challenge is accurately modelling random unstructured events
From a robotics engineering point of view the key challenge is dealing with the random events. It’s about the structured and unstructured nature of the environment. Basically robots work well and have worked well for many years in factories in defined, determined, repeatable scenarios (structured environments) – such as cars in a factory moving along a production line. However, roads are an unstructured environment, where every moment of driving produces a new and unseen scenario, for instance, a pedestrian randomly walking out into the road. These are unpredictable events that robots don’t handle well. So to throw fully autonomous cars straight into a completely unstructured environment in the real world is premature at this point. We’re in a halfway house now where there’s a collaboration with a human driver and a car.
When it comes to modelling however, it is significant to note that there are different ways of handling uncertainty in a model. Mathematical models always contain errors – a well known quote amongst engineers is that, “all models are wrong, but some are useful.” - G. Box. There are areas like robust control, where you’re not just using the model to predict exactly what you think the car is going to do, you’re actually predicting a region of uncertainty along with the expected car behaviour. There are quite a few established algorithms for handling uncertainty and while Dreams4cars’ remit doesn’t extend to dealing with this problem fully, the project will definitely leave pointers for future research and development.
Modelling differences between vehicles will be necessary and more straightforward
Then there are other kinds of disturbances that our algorithms will need to work with such as a variable number of people in the car or adding a trailer or roof box. Typical feedback control systems are quite tolerant to those kinds of changes. You have to really change the fundamental behaviour of the car to severely adversely affect the control algorithms. If you think about handling changes as we humans experience them by mass changes, we know they aren’t that different.
What is often the case is that the form of the mathematical equation will be consistent across vehicles, but things like the mass of the car will change. For instance, the general mathematical model describing car acceleration is (based on Newton’s second law, F=ma),
Engine Force – Braking Force – Road Friction – Air Drag = Car Mass x Car Acceleration
This is the same equation typically used for all cars and even large trucks. However, a Fiat 500 will behave much differently compared to a jeep or SUV because they have different engines, different masses and different friction characteristics. In these cases the mathematical models will need to be specialised to the individual vehicle.
As a car ages the human driver would subtly learn to adjust to the changing dynamic behaviour and a machine would ideally do the same. If data were to be logged every day when the car is driven, overnight, when the car is doing it’s dreaming or learning phase, it could update the model. So some kind of nominal model representing the car dynamics can be used to start with, but then as you collect data, as the car is driving it could actually specialise that model and update it, giving you a responsive adaptable set of control algorithms for each specific vehicle.
Every manoeuvre has to be modelled mathematically, which means Sean and his team have to process what output actions such as steering, gas and brake will produce. They have made good progress with this and will turn their attention to expand this forward modelling, as it’s called, to dry and wet weather conditions where the car will behave differently. It is possible that different mathematical models for those scenarios will need to be derived.
Inverse modelling for better vehicle control
As already discussed, it is important to accurately model mathematically what the car will do when gas, brakes and steering is applied. However, for proper control the driving agent needs to know what to do to produce a certain manoeuvre and velocity. So the inputs become outputs and the outputs inputs, hence the name inverse modelling.
Inverse models are more difficult to construct than normal or forward models because forward models essentially obey physics whereas the inverse model is non-physical. Essentially, you ask the question, if the car followed this path, what would the steering wheel and gas pedal inputs need to be? In the later half of 2018 Sean and the team will develop and run their models in Open DS and then put it in the DFKI test car. Potentially they will see inverse models controlling the actual steering wheel of a car. That would be exciting and new as it has not been done before.
Putting the maths in a real car on a real road
Road testing commenced in summer 2018, with the key tests in September. The main test car is the CRF Fiat/Chrysler Jeep in Turin and the second a Mia electric vehicle, run by DFKI in Bremen. Certain scenarios will be tested in real time in the vehicles on the test track, which of course gives feedback to evaluate the simulation environment and mathematical models.
2019 will include action–discovery simulation, where the car agent becomes fully aware of all that it can do to control the car. This will involve cars implementing actions they’ve discovered in simulation, which they can then take back to the test track after further simulation in Open DS.
The biggest challenge is collecting vast quantities of varying data
What Sean and his team mainly do is data driven modelling. In any kind of AI or machine learning, it is driven by data and therefore the collection of data and good data is absolutely key. By ‘good’ data we mean data that exemplifies many different useful scenarios – there is no point having lots of data from just one scenario. If we think about the success of famous AI systems like Google’s Deep Mind, they are successful because they have lots of data but also a range of data. Google is about data and data recognition. Through Google you can train AI to recognise shapes, animals etc.
The problem in a car, which is not a static shape, but a far more complex assortment of behaviours with multiple parameters, is that you need lots of data examples of all the different possible behaviours of cars and only then will machine learning algorithms work well on them, because they only know about data. They can’t extrapolate the same way that humans do. It has to be all about the data. So collecting data and good data is the challenge.
The dream of cars learning by themselves
Dreams4cars is all about car agents simulating road scenarios or dreaming, and learning from their dreams how to better handle the road. This is indeed what inspires Sean the most. With an undergraduate background in control systems engineering, machine autonomy has always intrigued him, and in particular the notion that maths can design an algorithm which he can step back from and watch the system behave autonomously on its own to successfully perform what it was designed to do. It all comes down to maths in the end and maths is plotting a course to breath autonomy into our vehicles.