Du kannst nicht mehr als 25 Themen auswählen Themen müssen mit entweder einem Buchstaben oder einer Ziffer beginnen. Sie können Bindestriche („-“) enthalten und bis zu 35 Zeichen lang sein.
sciion ab77bc9a48
added support for special characters in typing
vor 3 Jahren
assets added keyboard parser vor 4 Jahren
lib added support for special characters in typing vor 3 Jahren
project added support for special characters in typing vor 3 Jahren
scripts added support for special characters in typing vor 3 Jahren
states added support for states, added som states, expanded on the examples and removed the enter from API:type() vor 3 Jahren
README.md Update README.md vor 4 Jahren



“Scripting user activity is a hard thing to do” - me


This project aims at creating a system for scripting user activity inside computer systems using computer vision and peripheral devices. The technologies used are the following

  • OpenCV
  • Jython
  • Java
  • GraphLib

Tons of inspiration was taken from the Sikuli project and until I get on good terms with VirtualBox SDK this project can be seen as a inferior version of Sikuli. THe goal is to enable automization of guest systems in VirtualBox


All you have to do are the following esoteric tasks taken right out of M. C. Escher’s wildest nightmares:

  • Install java
  • Download OpenCV
  • Make sure the OpenCV library is present in your library path, on Windows this is your PATH varaible
  • Git clone this repo
  • Done


The primary way this project is meant to be used involves creating jython scripts that interacts with the function calls in the API.java file. Some example scripts are present in the project and can be executed using the following command

java -jar lib\jython.jar scripts\<filename>.py

More features are expected to be added as the project proceeds. The primary focus right now is to make the software somewhat useable.

Keep in mind that all files referenced in the scripts are relative to the current working dir

Keyboard interactions

This has proven to be the hardest nut to crack (apart from the entire “computer vision” thing). I searched for good tools but couldn’t find anything I liked. I decided to use what already was present in the java environment and this resulted in the java Robot class.

I’ve built a text parser to spare the user from scripting the robot by itself. The API can now take a string following the EMACS style of expressing key-combinations. That is, for example ctr+c and ctr+v being represented as the string:


and more complex stuff like typing “Hello World” either being done by:

Hello World

but also using shift modifier

S-hello S-world

this way has the added advantage of enabling more than the key codes that have dedicated keyboard-keys. For example the “!” character hiding under “1”


The full list of modifiers are the following:

S- Shift
W- Windows
C- Control 
M- Alt

When API.type() is called the call finishes with a ENTER.

As with everything the keyboard parser is a work in progress and does not support scoping more than one key to a modifier.