Nevar pievienot vairāk kā 25 tēmas Tēmai ir jāsākas ar burtu vai ciparu, tā var saturēt domu zīmes ('-') un var būt līdz 35 simboliem gara.
sciion ab77bc9a48
added support for special characters in typing
pirms 3 gadiem
assets added keyboard parser pirms 4 gadiem
lib added support for special characters in typing pirms 3 gadiem
project added support for special characters in typing pirms 3 gadiem
scripts added support for special characters in typing pirms 3 gadiem
states added support for states, added som states, expanded on the examples and removed the enter from API:type() pirms 3 gadiem Update pirms 4 gadiem


“Scripting user activity is a hard thing to do” - me


This project aims at creating a system for scripting user activity inside computer systems using computer vision and peripheral devices. The technologies used are the following

  • OpenCV
  • Jython
  • Java
  • GraphLib

Tons of inspiration was taken from the Sikuli project and until I get on good terms with VirtualBox SDK this project can be seen as a inferior version of Sikuli. THe goal is to enable automization of guest systems in VirtualBox


All you have to do are the following esoteric tasks taken right out of M. C. Escher’s wildest nightmares:

  • Install java
  • Download OpenCV
  • Make sure the OpenCV library is present in your library path, on Windows this is your PATH varaible
  • Git clone this repo
  • Done


The primary way this project is meant to be used involves creating jython scripts that interacts with the function calls in the file. Some example scripts are present in the project and can be executed using the following command

java -jar lib\jython.jar scripts\<filename>.py

More features are expected to be added as the project proceeds. The primary focus right now is to make the software somewhat useable.

Keep in mind that all files referenced in the scripts are relative to the current working dir

Keyboard interactions

This has proven to be the hardest nut to crack (apart from the entire “computer vision” thing). I searched for good tools but couldn’t find anything I liked. I decided to use what already was present in the java environment and this resulted in the java Robot class.

I’ve built a text parser to spare the user from scripting the robot by itself. The API can now take a string following the EMACS style of expressing key-combinations. That is, for example ctr+c and ctr+v being represented as the string:


and more complex stuff like typing “Hello World” either being done by:

Hello World

but also using shift modifier

S-hello S-world

this way has the added advantage of enabling more than the key codes that have dedicated keyboard-keys. For example the “!” character hiding under “1”


The full list of modifiers are the following:

S- Shift
W- Windows
C- Control 
M- Alt

When API.type() is called the call finishes with a ENTER.

As with everything the keyboard parser is a work in progress and does not support scoping more than one key to a modifier.