Using Drawing by Emotions interface (November 2017 Update)

Current version

Current interface version

Algorithms and GUI Development

     In this stage of improvements I developed the algorithms and code structure to avoid some issues such as:

     Applying info to Photoshop while the port is busy. previously that was causing problems to me, because the code was bit simple so if the Python code send data to Photoshop while it is busy ( such as there is an open dialog box) the whole software will get stuck and send an error message. this development decreased the errors to a very low - zero in average. 

Also in this development I added two colour boxes in the top showing the saved colour and the current colour to be applied after changing the parameters by the two sensors.

UPDATES

The main updates in this version:

- Two colour boxes added to show both saved colour and it is new form after applying the new beat values without using Photoshop palette to estimate the colour change. 

- A beat simulator icon is added to make it easier to check if the software is picking heart beats already and it is switching from yellow to red with each beat.

- The system now is catching the decrease of heart rate under the saved value. this means that it is more accurate and represent my heat changes exactly.

- Saving heart value and colour value is separated now to be able to change the colour easily but to keep the saved heart rate.

Improving the GUI and the Software

this is a major update in apperance and function. following a list of the main points:

- Separate section for each option, now sections are more organised, all options are in groups to make using it easier. 

- Actual colour channels values for both saved colour and updated colour. at the top of The GUI, a list of colour chanels and theier current values, for both the oginally saved colour. also the processed colour that will be applied to photoshop once the artist click Auto apply or manual one time apply options.

- Effective smile slider is an option that gives the user the ability to increase or decrease smiles effect. The artist can chose a step number that smiles need to reach before it counts one smile effect, for instance if the user chose ten on the slider this will result that every ten smiles will be counted as a one smile and so on.

- affected channel selection. this feature provide the ability to chose the colour channel that will be affected by the changes in heart pulse or in smiles detector. the thing that will change colour identity, as if the changes occur to all channels of specific colour, it will result in keeping the same colour but change the brightness. But, if the changes occur equally to all colour channels, then the colour will change. 

Ready Interface

Now, the interface is ready to be used. the interface two main features are responding effectively. Turning on and off smile detector and heart rate detector features is possible now.

Smile Detector, the Interface new feature

     After successfully got my smile to be recognised , working now on adding this feature to the interface. The way I am hopping this feature to work is by being able to activate/deactivate this feature through the GUI, and to decide how to use the collected data in the artwork.

     The two main options I am thinking in are: ether to adjust a working parameter such as to affect the colours lines sizes of things, or the second option is to use the collected smiles to create the artwork depend on for example how often and the number in certain time. 

Smiles As Art

Can our smiles create an artwork?

Facial Expressions?

Emphasising the concept of reading artist emotions, the next interface feature will be facial expression scanner. facial expressions can tell much about our instant emotional condition. our smiles, eyebrows and eyes can tel much about us. As Drawing by Emotions interface used heart pulse rate before, it is the time to add facial expressions to the equation. scanning facial changes, analyse it and apply it to the artwork can support the concept of transferring emotions.

The New Developments

using the interface can point out issues, also it makes it easier to suggest solutions. what I notice using the current interface is that the processed colour manual apply is working fine, but sometimes I need the process to be automatic. the suggested development is to add auto apply button and a timer slide. the automatic function will start sub timer that can be set through the slider to the desired value, and the new values will be applied continuously every n time. in this case another button needs to be set to disable the auto function. 

this can offer a convenient way to apply the new values while the artist concentrate more on the artwork than the interface. 

GUI development 2

In this update, the most important additive is the automatic heart rate effect apply option. There is no need now to keep press a button to update Photoshop foreground colour. Automatic function can be selected and set to the required timer. And of course the user is still able to use the manual applying option if there is no need to the automatic process.

Another Development

       Adding the automatic apply feature created some interference with other functions, such as the other timer that collects heart pulses. also the new timer should not interfere with Photoshop as it can accidentally apply new values while painting, and this can interfere with the communicating port I am using. this issue costed me 3 weeks to figure it out, now every thing working fine and the main software is more organise and function much better than before.

Switching to GUI

    As I got very interesting results, I started to think about add and improve things that can enhance communication quality. the next development will be a GUI, a flexible interface that can make the process of experimenting with this interface more effective. automatic responses and pre-programmed effects can be  achieved much better using GUI side by side to Photoshop.

Promissing

   While using the interface I can feel that I am more attached to the digital environment. it is like having an extra hand that can function independently.

Using "Drawing by emotions", "butterfly"artwork

      following, links to two parts of a video showing me using "Drawing by Emotion" interface for the first time as a usable mean of production
part 1 

https://youtu.be/xn52G473ePI

part 2

https://youtu.be/mGwZK3PnEb0

Preparing a PC for "Drawing by emotion" project

   The project is involved different digital aspects, it is uses multi programming levels and different hardware components that needs to work together smoothly. I am using Windows as operating system. to make the project works, I need to install:

- Photoshop application as the main interacting art software.

- python, as the central processing language.

- win32.com protocol that connect Python with Photoshop

- visual Python (VPython) to create the visual interactions with Python.

- pySerial to communicate with the serial port that transfer Arduino Data 

- ِArduino IDE which is the software that control Arduino board, and enable the pc to upload Arduino code into the board

  these points represent the main preparing steps I need to do in order to create my connections. 

I have data flow from pulse reading system to the com port.

data contain several info so far it is :

- heart rate 

- pulse value

- save button signal

- apply button signal 

on the other end, python programming language is detecting these data by reading the port.

python code is able to separating all of these data, and use it. 

Develop the Interfacing Concept

      after developing the prototype board... it is time to develop the software and make it easier to use and more effective. the most tiny details can mean too much. the ability to reset colours, check values, monitor the changes before applying it .. and much more details that can make huge different.

First Organised Prototype

       This is the first organised prototype for this project. it is working OK now. it should be more stable in term of structure and data processing, will expect no missed data now on.

Better Prototype

Now, it is important to get more accurate and steady results by improving the prototype structure. initially I used breadboard to connect the electronic components, it is very easy way to experiment, but, heart rate sensor is vert sensitive to any minor changes in connections and will give inaccurate reads in case of there is any lose connection. So I think it is time to go to higher level of connecting, generic PCB board, wires and ironing solder work.
the image below shows the PCB iam going to use and some of the connections header and other components.

What I have for now

So far, I can access Photoshop, read specific data out of it, transfer it to the processing software I wrote in Python. in other direction, I can get readings from pulse sensor which represent the physical readings of the real world. Using the program I wrote in Python, I can combine these data, process it and send it back to Photoshop.
the project is going in the right direction now, little more steps and it will function as it is designed to.

Developing the Process

to solve some lagging issues, I did many developments to the way of accessing Photoshop sending and receiving process. Also, two physical buttons are there now to control colour saving and processed colour applying processes. pressing the save key will save the current foreground colour by storing RGB channels values in a special array designed for this purpose. the other button, the applying one, is used to activate colour processing procedure using the exact current heart rate average, and to apply the new colour to Photoshop foreground colour, channel by channel.

Closer Look

This image shows a closer look to the project breadboard, where all components are communicating and puling to each others. LEDs are blinking an fading just as my heart do. 

That's what it looks like now

wires, electronic parts, sensors and codes
that's how the interface looks so far.

Photoshop Open Now

     Photoshop application is designed with a Com port that allows programmers to control it's features and run pre-programmed tasks by communicating through this port. and as I need a way to collect the data that arrived to the computer to use it with Photoshop I used Python for those two tasks. through Python I accessed serial port to collect the sensor data and also I accessed Photoshop to get and transfer data from and to it.
     I managed to go through this successfully and now I can get foreground colour from Photoshop to process it and send it back to Photoshop. using the usual Photoshop colour palette to chose a colour, save it, process it and send it back to Photoshop.

Art Application to Use

     At this stage, and as I successfully got a live data from the sensor. I need to apply the collected data to the artwork in progress. my idea for now is to use Photoshop as the main art making application. within Photoshop art environment, I need to chose a specific parameter or action that can respond to the collected data, also can possibly indicate some information about the artist at this specific moment when the data will be applied. the element I chose within Photoshop is the foreground colour, the live selected colour that Photoshop tools use by default. I chose foreground colour to be linked to my heart rate status as I believe that the instant and continuous changes in the used colour can indicate much about artist body responses.

Got Data Successfully

So far.. the Arduino board is able to talk to the heart rate sensor and collect the required information successfully. as I have a live data out of my body, the plan is not to use these data to draw a an ECG like graphs, but to employ these data in a way that reflects these immediate information into an artistic actions.
the form of connection can be a direct controlling and processing of an active element in the process of artmaking, so this element will represent in it is existence my body responses.

New World

      I got the required components to start the project. The Arduino board and the Heart Rate Sensor and other wiring bits and boards .. the Arduino board is capable to communicate with the sensor and collect the required form of information through an internal software that needs to be written in Arduino programming language which is Java based programming language. the software will define the connection pins and set the multi timers that can provide the environment to allocate the collected heart beats on a timeline and this will provide the required readings.
also this software is controlling the flow of the processed information back to the main PC that running the other parts of the interface.
the other software that will collect Arduino collected data and analyse it with other collected information and communicate with Photoshop still unclear so far.

The new project TItle

the new suggested features that this project aiming to provide the working environment with, are mainly promoting the idea of adding new senses to the working canvas allowing the digital canvas to pentrate the physical environment and extract useful information and use it. in this sense this project will make the canvas a more ' Live Canvas'. 

Exploring New Horizons

Having an interesting ideas is not enough. the project I am willing to start requires a combination of electronics, computer programming and artistic tools development. the good thing is I worked before on several industrial projects including robotics and automation including both the mechanical and programming developments.
Arduino platform beside Raspberry pi are very common electronics developments platforms nowadays. they are flexible and able to provide a professional level of interaction an communication between different environments.

I started few experiments to explore this medium and it is possibilities. this will help me  figure out the design of my new interaction approach.

Art Concept

    One of things that made me think about the concept of using my body parameters as a source of information about me is an old story about (Ibn Sena) an ancient scientist . Ibn Sena was the first to read human thoughts and emotions, by listening carefully to heart reactions while asking the tested person a series of questions. the same principle is used nowadays in lie detecting machines. it is a human nature that human body is instantly responses to the sudden changes of our thoughts and feelings. the responses can be as heart rate changes, sweating, facial expressions and/or skin conductivity changes. 

   the current plan is to employ this principle for art making. For my art practice, the process of creating an artwork is very similar to a series of questions and answers that I go through that results in forming the artwork visual structure and develop the narrative gradually  

New Perspective

at this point I were able to notice a different perspective within the relation between me and digital environment, it is the digitizing process. although digitizing is a fundamental feature of digital environment and it is technically there all the time, but, me as a technology user was dealing with it as a natural communication process that have it is own forms of bonds its providing between the user and technology and these forms are unquestionable technological musts.

conceiving digital technology in this form provided me with the space to think about a possible use to this process, and to think again about the source, form and type of data to digitize alongside my usual digitized actions. 

developing plan

thorough the process of creating this interface, many questions came to my mind. such as, why developers tend to recreate digital form of existed techniques. without any doubt, there is loads of advantages doing this, but, my question is why not to think of other forms of communication that invest the massive technical abilities of today digital technology.

Finalizing the project

This project gave me a very clear insight of how to develop and design a semi traditional art technique with a digital flavor. this type of digital interfacing can be considered the most natural way to communicate with digital art environment, as it is mimicking what most artist used to do using non digital painting tools. 

First Try :)

Using the project to produce the first drawing, it was very effective way to interact with digital environment. it can be compared to traditional drawing, but of course with all benefits of digital environment and possibly some minor drawbacks, such as it lacks feel of fraction with and on different materials.

First result

This image shows the project first result. after a few position calibrations. every thing is working fine including pen pressure sensitivity.

Drawing Interface structure

The project is working fine now. I got, a working LCD panel that recognize stylus position and tip pressure. this electromagnetic field is providing pen location, it sense pen hoovering from about one inch distance, and supply it is X and Y coordinates to the system. following a list and structure shows the main and final components employed in this project:

- Rigid supporting panel isolated with aluminium foil coat to cut off signal interference.

- stylus detector board that sense pen location and it is equipped with a controller board that translate location into usable data, and also collect other pen actions such as tip pressure and the other keys and send it to the computer.

- LCD SXGA panel controller. this controller was also causing an electromagnetic interference at the top part of the panel as it was fixed behind the screen upper part. the solution was to detach it from LCD back and isolate it with aluminium isolating tape, it is working very good now.

- LCD back light as described in the previous post, was replaced with a white LED strip.

- LCD driver that connect the panel to the PC.

- Acrylic working surface for easy hand resting and pen movement and to protect the LCD panel fragile surface.

LCD back-light inverter

this probably was the most tricky part so far. So when I tried the digitizer in the very begin of this project it was working fine even when I put the LCD panel between the digitizer and the digital pen (LCD was off). the tricky part was when I tried it again with powered panel, strange thing happened, the curser was shifting wiggling on the screen. the only idea came to my mind was that there is a magnetic field was affecting the wireless signal when the screen is on. I stripped the LCD as much as I can, but again still too much interference there. the only thing that I was able to remove more was the LCD back light, it was a CCFL based lamp not an LED one. this solved the problem, so I replaced it with LED light strip and it did the job. 

Multi port LCD driver

This LCD Driver was very helpful in this experiment. it is flexible and accept a wide range of LCD panels, also it support multi port type as input connections. generic and very easy to setup. figure out the pin connection between the LCD and the Driver was bit tricky. and the 1.2 mm pitch connection was too tiny to solder wires to it. flashed the driver with the most compatible software to my panel properties, and it is all ready now. 

Phase One

the initial part of my experiments is to discover the technical side of different forms of interaction with digital environment.I will start with the most common artmaking interaction form, the stylus display. the reason to start analysing this interaction form is to identify the technical processes that can reveal the actual artist-artwork relation during the artmaking process using this form of interface. the analysis process can help develop new forms of conversation with different technical base that employed the power of digital environment.

parts used in this project:

- electromagnetic sensor that will works as a pen position digitiser. 

- LCD panel that fit or close to the digitiser size (in my experiment it is 14.1" screen with 16:9 ratio). 

- LCD driver, in order to run and connect the panel I got a generic LCD Driver that I managed to wire it to the panel. the LED driver is accessible through HDMI port that transfer PC data to the panel.

- wires and PCB to make the connections.

Starting Point

As a part of my PhD research, I am starting an art project discovers the artist-artwork relation within digital environment.