46

I've been attempting to develop a means of synthesizing human-like mouse movement in an application of mine for the past few weeks. At the start I used simple techniques like polynomial and spline interpolation, however even with a little noise the result still failed to appear sufficiently human-like.

In an effort to remedy this issue, I've been researching into ways of applying machine learning algorithms on real human mouse movement biometrics in order to synthesize mouse movements by learning from recorded real human ones. Users would be compiling a profile of recorded movements that would trainh= the program for synthesis purposes.

I've been searching for a few weeks and read several articles on application of inverse biometrics in generating mouse dynamics, such as Inverse Biometrics for Mouse Dynamics; they tend to focus, however, on generating realistic time from randomly-generated dynamics, while I was hoping to generate a path from specifically A to B. Plus, I still need to actually need to come up with a path, not just a few dynamics measured from one.

Does anyone have a few pointers to help a noob?

Currently, testing is done by recording movements and having I and several other developers watch the playback. Ideally the movement will be able to trick both an automatic biometric classifier, as well as a real, live, breathing Homo sapien, too.

obataku
  • 27,809
  • 3
  • 38
  • 50
  • 13
    how are you testing if it apears human-like ? – Yahia Aug 14 '11 at 18:41
  • Why exactly does something like a straight line not sufficient for your purpose? – toto2 Aug 14 '11 at 18:43
  • Testing is done by recording movement and playing it back to I and a few other developers. If we find the movements non-human-like, i.e. too smooth, too noisy, strange paths, etc., then so can anybody else viewing them. The end applications involve a game, but I don't think that's very relevant :P – obataku Aug 14 '11 at 18:44
  • 3
    your "detection algorithm" is hard to use and possibly error-prone... have you done a blind-test, i.e. recorded an artificial one and a real one, then gave it to someone to watch without telling which was artificial/real ? what were the results ? – Yahia Aug 14 '11 at 18:50
  • 1
    Interesting question. Would be curious to see feedback on this. This would be a complex task as the sole measure is mouse input and there is no indication of constraints (bone/muscle movement, positionning, friction...). Assume that the mouse could be moved effortlessly. You could attempt a polling approach to start with and set up a series of tasks (click a dot, drag an object...) as a person would seldom use a mouse without purpose. Spline interpolation would help produce continuous movement but this is usually used to smooth out trajectories like in 3d flybys and is by no means natural. – James P. Aug 14 '11 at 18:51
  • @Yahia: As of yet, no, Yahia; however, it can be done in the future. – obataku Aug 14 '11 at 18:54
  • 3
    you should do that FIRST before going any further with the machine learning since it is esential to your whole approach... after you sort the "detection algorithm" it will be much clearer what direction to take to achieve your goal! – Yahia Aug 14 '11 at 18:55
  • @James: Indeed, it's a fairly complex question :P Users would be prompted to play simple games using their mouse to help calibrate and populate the training sample, but that should be it. – obataku Aug 14 '11 at 18:56
  • Your generated curves appear non-human because of the shape of the path, or because of the speed of the mouse along the curve? – toto2 Aug 14 '11 at 18:58
  • @toto: The shape of the path, mostly. For example, using Bezier curves, we found the mouse path too smooth and "wavy", if you will. – obataku Aug 14 '11 at 18:59
  • 4
    I can suggest an algorithm for simulating a drunken user :) – Bozho Aug 14 '11 at 19:00
  • 3
    Machine learning might be overkill/time black hole here. Maybe you could try to understand why a path looks human. For example, maybe people always overshoot the target and slowly correct. (I have a track pad, so I can't quite tell myself.) – toto2 Aug 14 '11 at 19:03
  • 1
    @super_: I'd make sure that the mouse, surface and position are more or less the same. You could then record trajectories (coordinates+time) at high frequency on different exercices which you could a compare a model against. As explained above, beziers/quadratics won't really do for interpolation as the trajectory is weighted by the relative positionning of the arm and the limits imposed by the wrist. – James P. Aug 14 '11 at 19:13
  • @toto: Perhaps, perhaps... I'm just looking for a solution :P Applying Fitts' law (extended to two dimensions, e.g. screen space) would help calculate realistic timing and speed for movement, by modelling the human-like speed-accuracy trade-off, but actually generating a human-like path... well, that's proven more difficult. – obataku Aug 14 '11 at 19:16
  • 2
    To add to what @toto has said, it's true that people will sometimes overshoot and do a correction when going for a target. This is something I noticed a few years ago while comparing demos for a certain FPS game. Whereas the trajectory would hardly show any deviation with assisted aiming, similar movement carried out by an all-human player would appear more erratic at higher speed with a very sharp and rapid twitch before fire. – James P. Aug 14 '11 at 19:17
  • @James Poulson: Interesting indeed. An algorithm which emulates such a over-shoot [here](http://www.moparisthebest.com/downloads/simba/make_tools/plugin_builder/public-smart/src/smart/EventNazi.java) doesn't learn however uses a basic 2D rigid body simulation to find a realistic trajectory. – obataku Aug 14 '11 at 19:24
  • 1
    I find myself 'looking for the mouse' on a regular basis. That's possibly also because I'm used to using the keyboard (shortcuts) more than using the mouse for the same tasks. And maybe also caused by using 2 screens. 'Looking for the mouse' is typically some circular or repetitive movement that helps me find the mouse. – Wivani Aug 14 '11 at 19:46
  • 2
    You don't need it to look like a particular user, just some user, right? If your goal is to look humanlike, why don't you use some genetic programming and use a human judge to assign scores to various mouse movement algorithms. Algorithms generate movements, humans judge them and the genetic algorithm finds the method for you. – 101100 Aug 14 '11 at 22:34
  • Are you trying to build a convincing WoW-playing bot? lol – Vincent Aug 26 '11 at 22:59
  • Nah, those are not my intentions, Vincent. – obataku Aug 27 '11 at 14:51

7 Answers7

14

Fitt's law gives a very good estimation of the time needed to position the mouse pointer. In the derivation section there is a simple explanation I think you could use this as one of the basic building blocks of your app. Start with big movements, put some inacurracy both in the direction and the length of the movement, then do a smaller correction movement and so on...

Karoly Horvath
  • 88,860
  • 11
  • 107
  • 169
  • Thank you. You gave the scientific answer that IMHO was expected here. (+1) – AlexR Aug 15 '11 at 07:18
  • I am well-aware of Fitts' law. The paper I linked to uses a neural network which synthesizes realistic time estimates for different types of mouse movements which produces results that follow the law indeed. The only issue I have is that I have yet to derive a way to produce an actual route for the mouse movement. – obataku Aug 15 '11 at 23:16
11

First, i guess you record human mouse movements from A to B. Because otherwise, trying to synthesize a model for such movement does not seem possible to me.

Second, how about measuring the deviations from the "direct" path, maybe in relation to time. I actually suspect that movements look different for different angles, path lengths etc., but maybe you can try a normalized model first, that you just stretch (in space and time) and rotate like you need it.

Third, the learning. The easiest thing would be to just have a collection of real moves (in the form i discussed above), and sample from that collection. Evaluate how that looks like. If you really want a probabilistic model, then you have to evaluate what kind of models fit. is it enough to blurr the direct path with gaussian noise whose parameters you learn from your training set? Or some (sin-)wavy deviation? Or seperate models for "getting near the button" and "final corrections". Fitts law might be useful for evaluation.

Michael Kutschke
  • 221
  • 1
  • 2
  • This is the kind of response I'm looking for! Thanks. I've dealt with randomized noise added to a spline but once again the route appears too mathematical and not human-like. – obataku Aug 27 '11 at 15:10
2

This question reminded me of a website I knew about years ago, so I visited it and found this in-depth discussion on the topic.

The timing is so similar as to make me think this question is related in some way. In fact, someone in the thread linked to the same article you did. If it's not related, well, there's a link to a lot of people discussing exactly what you're thinking about.

Evan Cordell
  • 4,068
  • 2
  • 27
  • 47
  • 1
    But now I see there's a user named `super_` there as well. – Evan Cordell Sep 12 '11 at 20:29
  • I don't have an account. I drop by to see how client hacking is faring every couple of years or so. There're a lot of programming techniques used there that I just haven't really seen crop up in other places. – Evan Cordell Sep 13 '11 at 01:56
1

I don't think the problem is all that well defined. There is a important notion not mentioned so far, which is context. The mouse movement on my screen when Chrome has focus is massively different that the motion when Vim has focus.

carlosdc
  • 11,578
  • 3
  • 38
  • 60
  • The context is in regards to playing an MMORPG game. It ideally shouldn't matter, though, as the movements recorded and synthesized both target the same game and therefore context is consistent. – obataku Aug 16 '11 at 19:06
1

The way a mouse moves varies based on the type of the device, the type of action, the UI elements involved, familiarity with the UI, the speed at which the user is attempting to complete their task, the skill of the user, initial failure of the user (eg miss-clicks), the user's emotional state (as well as many other factors). Do you plan on creating several pathing strategies to correspond to different contexts? Also how well do you know the algorithm you are trying to fool? I assume not extensively or you would simply program directly against that algorithm.

If a human is looking at the pathing, they might be able to identify the state associated with a pathing strategy and may be more inclined to be fooled if they identify it as a human state (eg user is rushing, miss-clicks, quickly closes a resulting popup, tries again slower). UI comes into play with not just size and position. I often quickly point to a toolbar, then slide across the options until I get to my target. Another example is that I typically pause on menu items while I am scanning for my target or hover over text I am reading. Are you attempting to emulate human behavior or just their mouse movements (because I think they are joined at the hip)?

Adam Jones
  • 711
  • 3
  • 9
1

Are you wanting to simulate human-like mouse movement because you are doing real-time online training for your game? If your training sequences are static, just record your mouse movements and play a mouse clicking sound effect whenever you click the mouse button. No mouse movement is going to feel "real enough" to you more than your own.

Personally, I feel experts in software move their mice too quickly in training videos. I prefer an approach taken by screencast video software I've seen that always moves the mouse linearly from point A --> B. The trick was, every mouse move made in the video always took the same amount of time regardless of distance, say 3/4 of a second and then followed by a mouse click sound effect.

I believe they moved the mouse in this way because then the viewer could anticipate the landing area of the mouse by the direction and velocity the mouse moved at the start. In a training situation, I suppose that regular movements like this are gentler on the eye and perhaps easier to retain/recall.

DWoldrich
  • 3,385
  • 1
  • 15
  • 19
  • Not quite for testing a game... it is more tuned to fooling both biometric authentication as well as a real human observing such movements. – obataku Aug 27 '11 at 14:49
  • Sneaky. Best of luck simulating gold miners and bots, then?! ;) – DWoldrich Aug 27 '11 at 20:45
  • Simulating gold miners and bots? I mean it is for testing a game server but not the game itself. The idea is ultimately to see if there is anyway synthesized mouse movements can look perfectly human and bypass our current detection techniques which include a basic biometric classifier and humans watching recorded mouse movements. – obataku Aug 27 '11 at 21:42
0

Have you considered adding mouse tracking to your application so you essentially record how the user moves the mouse and then analyze the recordings?

I have not looked into this recently but I believe that a MouseListener in a Swing application get the information you need.

Thorbjørn Ravn Andersen
  • 68,906
  • 28
  • 171
  • 323
  • The way testing is currently done is by tracking mouse movement and recording it. Ideally every user keeps a profile of their recorded biometric mouse dynamics which is analyzed in *some* way to produce realistic mouse movements :) – obataku Aug 15 '11 at 23:13
  • So your problem is not recording but creating a model from your recording? – Thorbjørn Ravn Andersen Aug 16 '11 at 05:47
  • Indeed, creating a model that can synthesize similar mouse movements given two points. – obataku Aug 16 '11 at 19:03
  • tracking mouse is beyond trivial, processing the data is the hard one – bestsss Aug 22 '11 at 08:55
  • @super, I suggest you open a new question where you deliberately ask for help _modelling_ the mouse movements based on your empirical data. – Thorbjørn Ravn Andersen Aug 27 '11 at 18:58
  • Well the purpose of the model is to produce mouse movements... it is an integral part of a question asking how to learn to move a mouse based on human training samples. – obataku Aug 27 '11 at 20:03