Tell me a little bit about your background since leaving college.
I started studying mathematics at the University of Belgrade back in 1988. At that time I wanted to study computer science, but it was not possible, it didn’t exist as such. So back then you had the choice of either mathematics or electrical engineering. With electrical engineering, it had a computer science direction, but it was more in terms of hardware rather than software so I went to pure maths because I enjoy mathematics.
We did finally move in the direction of computer science where we were mainly focused on databases and computer programming languages, but with no computer! There was absolutely no kind of computer at the university back then so when we had an exam in computer programming you would program exactly what they wanted you to program and the professor would type your program into the computer and if it crashed you failed!
If it didn’t crash they’d look at how well it was done, which functions you’d created, that kind of thing. This was a really stupid system, but it was good in one respect in that you learned to program well and learned not to make many mistakes.
In the first year we had 170 students on the course but only 12 made it through to the second year; it was extremely difficult. You’d do a whole year and then took an exam, which was very difficult.
I was in my third year when the riots against Slobodan Milošević started so I joined in the demonstrations. The problem at this time was the university would stay closed; they would say ‘we will open the university in a month’s time’, but then they would not, and this would continue for six or seven months. I realised then that there was no clear situation as to what would happen.
There were a lot of rumours about the war, which actually started in Croatia - they stopped the press so there were only rumours as to what was really going on. So, with this kind of climate - with no university because it was closed, and with the prospect of an upcoming war - my sister left for the Netherlands because she and her husband had got jobs there.
I decided to join them for the summer to see if I could continue my studies there, in 1992. But when I got there they said I’d have to learn Dutch first and I’d have to start my studies over from the beginning because there they did have computer science with computers!
So I enrolled in December 1992 and did my language course until December 1993, at which point I started my studies and finished in February 1997. I studied a five-year course (which included a Masters degree) in three and a half years.
I did well because back in Belgrade we learned how to store huge amounts of material in our heads whereas there they had many small courses, with lots of interim exams, so because of having the course delivered in smaller chunks, with so many exams throughout each year, it was not really a problem to learn quickly. I had this very high average grade, above 80 per cent, and had an extremely good Master of Science final thesis, for which I got above 90 per cent, which, I was told, was really quite rare.
I did my Master of Science work on the analysis of facial expression. They asked me if I wanted to stay to do a PhD in the same topic. What we looked at was the automatic recognition of pain and fear by examining body posture and movements.
We are currently working with hospitals and patients to automatically monitor the latter. We are looking at the management of emotions involved with pain and the related movements of the body. That’s one of the projects we are involved in, where we collaborate directly with physiotherapists. And now there is a professor from Leicester who specialises in pain.
What are you currently working on?
The biggest research project that we’re currently working on is with the European Research Council, who have supplied a Fellowship Grant. When I got it in 2008 it was only awarded to two per cent of the applicants, hence to 500 applications out of more than 12,000 applications, so it’s very prestigious. This is for any field of science in the whole of Europe. And that was a very large grant, almost 1.8 million Euros.
The main project of that grant is also the main topic of my current research, which is dealing with spontaneous human behaviour, so looking at naturalistic human behaviour. It’s looking at non-prescribed behaviour, not ‘acted’ behaviour - so if I was monitoring us now we’d just sit and talk. Our research asks ‘how can we monitor when there are no exaggerated expressions and no directions as to what participants need to do?’
The research we’ve done so far into automatic facial behaviour analysis had situations that were completely orchestrated. So you would have people in the lab and you’d say to them: ‘think when you are happy, what do you do?’ Or you’d ask them to express a disgusted face. If your subject couldn’t do it, you’d explain to them what the correct expression should look like so it was completely unrealistic.
It was very important to me that we move into this very naturalistic behaviour, because when you have a robot, a car, computers or games, whatever interaction you have with the technology, you would not do it in an orchestrated way - you’re not being directed as to what to show and how to show it.
What we really show in front of the robot or computer, we really have no clue. There are a lot of subtleties within people’s expressions, which are hard to interpret, where the movements are very small and short-lived, with small amplitudes, so our focus is on machine techniques to handle these kinds of problems.
So how are the nuts and bolts done? Do you have a camera trained on a subject and then you’d create a dot map of the face and study these points when the expression changes?
Yes. There are different ways, but that is one way of doing it. There’s the so-called feature-based approach where you monitor those and another one, which is an appearance-based approach, where you actually look into the changes of the texture or the motion of the skin so you don’t look into the shapes or facial components or points of the face.
The motion of the skin can be easily recorded using optics. We actually apply both methods, but we prefer the appearance-based approach for certain situations such as when you need to capture very subtle emotions. So, for example, if someone has a tic under their eye, you can catch that slight movement with the motion of the skin method. With a feature-based approach you can’t, since there is no feature just under the eye.
On the other side the features-based approach is much more robust as, if you can track the features in the first frame, you can track them with reasonable accuracy in the rest of the video. You know where they were in the previous frame and there is a certain assumption about where they can be because we cannot move arbitrarily and we cannot move very fast as every muscle on our face needs time to activate.
This is extremely important to take into account. There is a lot of anthropomorphic prior knowledge that we can include in our model because we know, when we talk about the face, how it should move. So that’s one part of the story.
The other part of the story is once you have this motion, official point or whatever, how do you know if a person is being ironic or if he or she is truly happy? That is very difficult. So one of the important factors that needs to be considered is the dynamics of behaviour. When we’re ironic we tend to have much faster movements than when you show a joyful smile, for example.
When you go to the airport and see a member of the check-in staff you get a big smile, but the moment you turn your back it’s gone. When people smile it’s usual asymmetric, they tend to smile more on one side of their face than the other, but if it’s a fake smile the movements are less dynamic, faster and don’t last as long as a natural joyful smile. So dynamics of behaviour seem to be extremely important for discerning the different types of behaviour. This is the same with anticipatory pain and actual pain.
Anticipatory pain is not true pain, so all the expressions come faster and stronger than if you have acute pain. When you have acute pain, you want to suppress it, so you also suppress the expression of the pain. So these are the aspects that we look for and model using specific machine techniques.
I think it’s pretty obvious that medical organisations would benefit from this research, but what other groups might also have use for this technology?
Well, medical definitely because we can deal with depression. We can determine if people are depressed, how depressed they are or, even better, how they’re dealing with their depression through having daily interviews with them and monitoring their facial expressions and behaviour.
We can also deal with people who’ve had plastic surgery. We can determine how much Botox there is, for example, because any kind of plastic surgery will effect the facial movements. It’s something we can account for, as a security application.
There is the monitoring of intensive care patients and checking their pain levels, checking whether they are awake or whether they want to get up - this kind of monitoring and surveillance we would be able to do. With facial paralysis the carers could be trained what to look out for - if the patient is moving their features in the right way - through monitoring and analysis.
A lot of this also relates to security; for example, facial expressions and facial behaviour, body behaviour, the way we laugh, how we laugh, how long for - all of these are biometric so people can be recognised by that.
There’s a research fellow in my group who’s working on those aspects. It’s called PK biometrics. The models we build, especially those based on appearance, are models where you learn what the appearance is of the object that you’re checking - you can learn the appearance of a face, but you can also learn the appearance of a car.
We’ve built models that can learn incrementally so they adopt the model that they track on a frame-by-frame basis. So even if the car turned or if you have shadows from a tree fall on the car, you can simply apply this tracking method into surveillance applications.
So would that help customs, for example?
Only to track. The detection software allows us to try to predict how atypical the behaviour is of a particular person. This may be due to nervousness or it may be due to an attempt to cover something up.
It’s very pretentious to say we will have vision-based deception detection software, but what we can show are the first signs of atypical or nervous behaviour. The human observer who is monitoring a person can see their scores and review their case. It’s more of an aid to the human observer rather than a clear-cut deception detector. That’s the whole security part.
There’s a lot of human / computer interaction involved, including ambient intelligence.
What is ambient intelligence?
The main thing with ambient intelligence is that you won’t have a computer in front of you; they will be integrated with the background. So your house will know when you come in, your fridge will know when there’s no coke and, when your child wants to play, something will be projected onto the wall. Identification may be included in all that, but it’s not RFID.
So it might provide an artificial playmate for your child or if you come home very tired, then the light may simply dim and your TV might ask you if you want to watch a nice drama or programme that you normally like to relax with.
Alternatively, if you come in early and you’re feeling happy it might ask you if you want to see some news. With ambient technology you need to be able to understand the state of mind of your human subjects, whether they are tired or playful. It needs to be able to take into account a number of factors including the weather, whether there was a plan to cook something that evening and so on.
There is another application for this technology in multimedia. We can use the behaviour to log multimedia. For example, if you watch a video on YouTube and you laugh, the video could be tagged as a funny video and, if many people laugh when they see the video, the confidence, which could be associated with that video, will grow. And this can be done in a very effective manner, in the sense that you don’t ‘do’ anything, you simply watch it and, if you laugh, then there is this funny label associated with it.
The system then logs it as being funny. No matter what the nationality of the person watching, if they laugh or comment in their own language that they find it amusing, then the system tags it as being so. Hence, statistically, the file will become more easily retrievable as it will gain many #tags.
You mentioned previously that you currently have a large grant, but who do you usually get your funding from for your research?
I had most of the funding from the European Commission. We have a number of European projects on the go and this is just one of them, albeit the largest one. I’ve had smaller ones, which are collaborative with different European universities. The reason why it probably worked out this way was because I was in the Netherlands and, when I came here,
I didn’t immediately have a UK network of collaborators and therefore I didn’t apply for a Engineering and Physical Sciences Research Council (EPSRC) grant. So I just went with European grants. I already had all these collaborators around with whom I wanted to work, and European collaborations are nice, so it felt right.
They may be difficult to get, especially recently; currently they’ve only been funding between 10 and 14 per cent of all applications. At the time of one project we got in 2009, grant funding was down to six per cent, which is really low. Our Semaine project, for example, is with University College London.
What have been the most important breakthroughs you’ve been able to make over the last four or five years?
There are many actually, but probably the most important is the analysis of images of facial muscle action, which are completely agnostic and this means you can integrate their behaviour, in whichever way you want to, in terms of emotional or mental state, and in terms of social signals like agreement or disagreement. That was initially in 1997 and then I continued that work and also worked on the dynamics of human behaviour and how you can model the dynamics of human behaviour.
Most recently we’ve started looking into machine learning techniques that could truly help us for our purposes. We didn’t do this before because I wanted to try what already existed first and the group was much smaller then.
On one side you need computer vision tools to analyse the images and video and then, once you have the per frame analysis, you then want to build this model to better understand it all. However, you have no idea if you will succeed or not with the current technique. So when we went into this area we realised that the existing technique didn’t give us the power to analyse this in a useful way.
Do you develop your own software?
Yes, some of it. Many parts of it are publically available as compiled versions, but not as open source. They are available online - you can download them and use them.
What sort of computer systems do you use - PCs, Macs?
It’s diverse. Many people use UNIX still and those can use Windows as well. I would say 70 per cent use Windows and 30 per cent use Macs and UNIX.
Who inspired you to move into computer science - do you have a role model?
I remember myself thinking that if I do pure maths I can only really be a teacher. I asked myself if I really wanted to be a teacher, which is a fine profession, but it didn’t really excite me. You end up teaching the kids more or less the same material each and every year and that was not particularly inspiring, though I liked the topic.
But with computers, it was something completely new; we just couldn’t predict where it would go. And we still don’t really know where it will go! At the time I started studying it was 1988 - it was the time before the internet - but I did like to play computer games and that was one of the reasons, for sure, that I looked into it.
What do you think have been the most significant breakthroughs within IT and computer science over the last few years?
Overall I think the arrival of the internet was unprecedented - it has completely changed the world. And it happened when I started to study in the Netherlands, which is why it had a massive impact on me. I actually felt how different the world actually became.
I mean, at one time, everything you did involved hard copies - books, papers, whatever... and then suddenly everything was electronic and this happened within the space of just a couple of years.
I think it absolutely changed the landscape of everything, especially technology, the economy and research. I think it made the world a much, much better place. I think it’s without a doubt the biggest invention of the last century, for sure, just from the knowledge-sharing angle.
What predictions do you think you could make about technology over the coming years, especially in your own field? For instance, do you think AI is a possibility and, if so, do you think, in future, we will have robot carers in our homes that can recognise when we’re in pain?
I think this is very possible and this is the path we are taking now. I’m not so sure about the robots, as such, but ambient intelligence within our homes is a definite possibility.
When it comes to robots I’m not sure if many humans will be able to get over their fear that we are, in effect, producing a new species, albeit a metal one with other types of artificial material, to coexist in our homes. I think only if this fear, as we see in films, can be overcome, can we go down the robot route. But the ambient intelligence route is more promising.
We all have mobile phones so why would you need a robot to physically help you; if a mobile phone or any other sort of technology can see what’s going on, a message can be sent to your neighbour, to your GP or carer, or a close friend or relative to come to your assistance. I’m not sure if robots are really necessary, but I may be wrong.
It seems the Japanese may disagree with you since they are investing a lot of money in robot technology. Probably because they believe, with their ageing population, there won’t be enough human carers available to help all the aged people.
I think it depends on how the population develops. If the trend is for less and less young people and for more people living for longer and longer we will probably all end up in halls where a robot moves around helping us. I really don’t know, it could be that technology will develop in unexpected ways, which enable us to be more self-sufficient for much longer.
What are your thoughts on professionalism in IT? Do you think the government needs to intervene or should industry be allowed to regulate itself?
I don’t really have an opinion on this. In general I’m against hard regulation because it involves getting the government to come up with a suitable policy, then needing people to police it and others to assess their peers to report their findings.
The question is how you will do it, when technology is progressing so fast you will have to keep changing these laws to accommodate all of these changes. So at the moment, with all this rapidly changing technology, I don’t think it’s a priority for government to get their own people to check up on everything that industry is trying to do. I think people’s individual skills dictate their own success - someone might not be so technically gifted, but might have excellent PR or interpersonal skills, which make them ideal for becoming a CEO or CIO.
I think industry knows what it wants and what they need to achieve and they need to be able to check that a person has the right skills. A company can ask for a person with a PhD in computer science if they want that level of academic achievement for a role, but the government doesn’t need to be involved with that. More and more companies, including the likes of Google, Microsoft and even banks, require people with specialised IT skills, but I don’t think having something that’s universally recognised would necessarily work as each organisation will have very different roles to fill.
At the moment only about 14 per cent of the people working in IT are women - what are your thoughts on this and how do you think the industry could improve this figure?
That’s very good actually! In the Netherlands, out of 300 staff working in my department, only two of us were women. So if you think you have it bad, think again!
So why do you think numbers are so low? Why do women stay away from working in the industry - is it because of its reputation of being peopled by geeks and nerds?
It’s also about where your friends go - you go. So, if your friends are going into another profession, why would you be the only one doing something else? For me it was completely different because when I arrived in the Netherlands I already knew what I wanted to do, so in that case being the only one in the class didn’t matter at all.
Later on, when I was doing some teaching, we had a lot more girls and some classes were 40:60 and even 50:50, which is really good. But back then it was more like 10 per cent. I think it is mainly because the friends don’t do it. I think here in the UK the situation is much better and I don’t think it’s at all problematic.
What are your thoughts on how IT is taught in schools? Do you think it needs to be taught in a more enthusiastic manner rather than just teaching them about spreadsheets?
I think IT needs to be taught appropriately. There are certain things everyone needs to know. Things like Microsoft Office, the internet, these are basic things and after that, going deeper, say into programming languages, I don’t think that’s for the basic schooling, but if we are talking about further education I think maybe it would be good to do something more advanced for pupils.
What everybody would need is to be able to do calculations, sometimes using some of the tools that Microsoft provide, and also social networking, from a PR angle, so looking at how to use viral to your advantage, and perhaps learn the key languages that underline the majority of IT applications and explain the advantages of going deeper into the programming side of things and the sorts of things that can be achieved. I don’t think everyone going into IT needs to know how to program.
If you go into biology, law or even literature you will still need a certain ability in IT so everyone should be taught the basics and more. I think learning how to use the internet efficiently is far more important for most people.
I think being internet-savvy is more important for most young people than the ability to program in HTML. However, if you then wanted to go into any sort of engineering discipline then I think programming and maths would be needed. And electrical engineers will need different IT training to computer scientists.
If you had one piece of advice to give to someone who was considering a career in computer science or IT what would it be - how would you sell the idea to them?
Because it’s the future! Just look at the use of the internet - most businesses wouldn’t now exist without the internet - so much business is done through it. We have lost most travel agencies - most travel providers are now online. If you just take that into account you cannot fail to realise the importance of IT. Technology is now critical for business.
In future a lot of manufacturers will build in devices to study people and their behaviours in order to improve their own businesses and also day-to-day equipment, like fridges, will have in-built technology to inform their owners what needs replacing and perhaps even connect to the internet to reorder milk etc. If you think about all of these things then computer science is the future.
Who, in your opinion, has been the most significant IT pioneer within the last 50 years?
Probably Sir Tim Berners-Lee, for his work on the internet. I think he has done something, and is doing something, that really matters. He’s provided the basics that everybody uses so I think this is impressive. When we consider the programming behind it now it’s relatively simple, but at the time it was revolutionary. I guess I’m looking more into the software part than the hardware.
