Interactive Devices - Unit Conclusion

The unit overall has taught me much, especially from the hardware side of things. It was very interesting to see how different pieces of software could connect in order to transmit hardware signals. In our case:

Arduino board -> Arduino program -> OpenFramworks program -> Windows

This course also helped to reinforce what I already suspected: hardware is not the side of computing that I am interested in. It's too messy and low-level for my liking. What I do like though is the idea of creating something from scratch in a team which works as a pipeline where team members specialised in hardware create an initial device/machine which I can then build on.

I personally found it challenging to write a research paper for a project which could be shown at a conference such as CHI. Finding the appropriate wording was not easy and it took a long time. In addition, all the extra parts a Work In Progress CHI paper needs, to be presented, augmented the time taken to write the report.

Finally, I think the unit would be beneficial to all students in Computer Science which have never tasted the adventure that is hardware. The reason is not to only encourage them to pick up products such as Arduino and make something creative/useful but also to see if hardware is something that they enjoy and never had the time to try, possibly changing their career path.

Interactive Devices: Final Lab - Rubikon Demo

On the morning of the demo, the group gathered to decide in which order the applications would be shown; this was critical for a good presentation flow.

Under my suggestion, we decided to start from authentication because usually a computer session starts from logging in. After authentication through our custom login screen, we would switch applications to go to the Desktop. Here we would use the folder navigation features to travel through the icons to select Google Chrome and launch it. Once in the browser, features such as: switching tabs, scrolling down/up and going back and forward in a page's history would be highlighted. These actions would all show how Rubikon works as an input device.

Even though we had a prototype, our final product would have been a personal device. This meant output would be required as well. To highlight this fact at the end of the presentation, a specific button would be pressed to play a Christmas song and light the LEDs on the cube, with different colours, according to the song's rhythm.

With the demo order finalised, we concluded the meeting by deciding what to say during the presentation to emphasise the potential of our device.

The feedback from the people that came to see and try out our device during the demo was very positive. In addition, everyone seemed to enjoy using the cube as an alternative means of input. However it was hard to convey how Rubikon would be, ideally, a personal device due to the early stages of the prototype. For this reason I brought to the demo a mini touch screen to give an idea of the feasibility of the final product.

Interactive Devices: Lab 5 Individual Post - Implementing the software features

From the last lab we had an Arduino program capable of transmitting rotary encoder inputs to our laptop. In order to make use of these inputs myself and Vlad, after much research, decided to use the OpenFrameworks library to implement the features we set out at the beginning of the project i.e. folder navigation, browser navigation and authentication.

We started by sending the device's rotations from Arduino to OpenFrameworks and subsequently created a Node JS server, listening for commands, in order to perform actions on the browser. As we implemented the new system we quickly remembered that the most important, if not all, computer commands are mapped to certain keyboard shortcuts, even better, these shortcuts are often consistent across programs. Luckily Windows provides a C++ function called keybd_event() which simulates key strokes.

Example:

keybd_event(VK_TAB, 0, 0, 0);                                  // Presses tab key
::Sleep(1000);
keybd_event(VK_TAB, 0, KEYEVENTF_KEYUP, 0);   // Releases tab key

For our authentication we decided to start from the Windows lock screen. The locking action on Windows has a shortcut, Windows Key + L, so we simulated this combination with our program. This first attempt didn't work so we decided to try other combinations to see where the problem lied. After a few successful attempts I quickly realised Windows had a security protocol in place which clearly disabled us from locking the computer programmatically.
As a solution we decided to make our own authentication screen which would record key presses in the background and on pressing enter, if the password was correct, it would let us perform the other previously mapped actions, basically locking down any cube functionality until access was granted.

Using a similar method, myself and Vlad then implemented the following applications for the demo: folder navigation, browser navigation and application switching.

Interactive Devices: Lab 4 Individual Post - Building the prototype

The day before the lab we received the electronic components required to start building our cube; seven rotary encoders. Each works as a rotary encoder, push button and LED. The degree of rotation is not restricted to 360 and it knows whether it's rotating clockwise or counter-clockwise while the button is a simple push down, i.e. it has a binary state. The LED is an RGB LED capable of displaying light of any RGB combination, this means vibration feedback was not needed anymore as we could just give feedback by lighting the LEDs.

With the components in our possession myself and Amir went to the electronic labs to solder each of the eight pins. I soldered the first potentiometer to its circuit board but after thinking about it, we
decided to do away with the boards because otherwise we would never be able to fit all the rotary encoders inside the cube.
When we got back to our lab the whole team started looking at how to connect the rotary encoder to the Arduino board by following a Bildr guide provided by the rotary encoder's page on Sparkfun.

After a fair bit of trial and error with Arduino coding we eventually managed to write a program which received movement data when rotating the encoder. Following from our success we quickly decided to try to get the button to work as well so we connected the push button wires to the Arduino board and edited our program. Soon enough our program could receive both rotation and push data.

Interactive Devices: Lab 3 Group Post - Research Papers Relating to the Project

In order to gather more information and possible ideas to include in our prototype, as a group we searched the Internet for, possibly recent, research papers which were related to our project. Unsurprisingly, there are quite a few academic papers on interacting with computers using alternative devices, which is our aim. Furthermore, many papers can be found relating to authentication, this being one of the primary features we plan to implement.

Below are the 10 best papers we have found:

1. Spontaneous Interaction with Everyday Devices Using a PDA
2. SideSight: multi-"touch" interaction around small devices
3. Reality-based interaction: a framework for post-WIMP interfaces
4. The cubic mouse: a new device for three-dimensional input
5. Real-time 3D interaction with ActiveCube
6. Sculpting: an interactive volumetric modelling technique
7. Reducing shoulder-surfing by using gaze-based password entry
8. An Association-Based Graphical Password Design Resistant to Shoulder-Surfing Attack
9. Novel Shoulder-Surfing Resistant Haptic-based Graphical Password
10. Weighted finite-state transducers in speech recognition

Interactive Devices: Lab 3 Individual Post - Interactive Rubik's Cube design

In this week's lab we started thinking about the design for our first prototype so that we could order all the parts we needed and start the building phase. After much research on the internet we found that in order to record rotations in the cube we needed some encoders, usually found in older trackball mice; six of them, one for each axis.

As we further developed the design we realised not only encoders might not be a good solution because of size and movement processing issues, but also that Arduino might have something similar.

Luckily there is a component called a Potentiometer for Arduino which is exactly what we needed. This switch resembles a light dimmer which changes value according to how much you turn it.

The next problem was that this Potentiometer would stop at 360 degrees and we needed something that could turn continuously. After more research on the web we found a version of the Potentiometer that behaved exactly like we wanted, so we decided to use an Arduino board as the core processing unit for our cube.

In our first prototype we also wanted to incorporate RGB LEDs, one for each cube, in order to give feedback to the user as to whether the rotation mapped successfully to a command (green) or not (red). While trying to figure out how to insert them and connect them we realised this might not be feasible because every time a user would rotate the cube, the LED's wires would also rotate and eventually snap.

Considering LED feedback is a minor, more aesthetic feature, it seemed pointless to design the whole cube around it, so I proposed to swap visual feedback for haptic feedback. This decision was well accepted by the group and so we will now start looking at how to include vibration.

Interactive Devices: Lab 2 Group Post - Final Idea

Following our discussion during the last workshop Amir went on a hunt to find an idea for our interactive device. After much research, and his passion of solving puzzles, he came up with the idea of creating an interactive Rubik’s cube.

The idea is to be able to control your computer using a Rubik’s cube. This includes simple tasks such as zooming in or scrolling down when reading a paper, unlocking your laptop using certain combinations, controlling a 3D simulation, etc.

Amir purchased couple of Rubik’s cubes and held a meeting with rest of us the next day. Everyone was on board and very excited about the idea. During the meeting he disassembled one of the cubes to explain to the group how we might go about designing the cube.
We then went to the HCI lab and pitched our idea to some of the research assistants and got positive feedback and techniques in which we can refine our device.

The uses of this device vary from accessing media player to scrolling pages on a web browser or even controlling a 3D simulation. The way it differs from keyboard or mouse is that first it’s wireless, second it can be personalised to suit users’ needs and third its just a fun way of interacting with your computer.

The device moves us away from WIMP interfaces and provides an opportunity to users to personalise the way they interact with their personal computers. The idea can further be extended such that the cube becomes more than just a controller; the cube itself can be a personal device where each of the 27 smaller cubes are screens. Then it can be used to do things like answering your calls, an alarm clock or even checking your emails.

This is the list of materials we are going to need for the completion of the project:

1. Encoders from Trackball Mice
2. Transparent Acrylics
3. RGB LEDs
4. Arduino Board
5. Buttons