Virtual Projection Keyboard [using MATLAB]

Following project was created in order to achieve a working implementation of a Virtual Keyboard.

A Virtual Keyboard is one that provides an alternative for the bulky keyboards that are generally used.

Existing Virtual Keyboards that can be purchased today differ from the one implemented here in one key aspect – they are laser type keyboards. That is, the keyboard is projected using a laser device.This limits their scope and use.

[Review of one of such devices – https://www.youtube.com/watch?v=bKhbEz00ksM]

This project, on the other hand, attempts to create a Virtual Keyboard using easily found hardware – webcam, any normal projector and a red tape. Thus, it can be extended for other uses like a Projected Drawing Board, for playing games, for interactive charts etc.

The working and setup is as follows:

  • The projection module is used to project a keyboard on the wall or any plain surface. Not all keyboard models can be used, as it will require a change in mapping (function 3 in code).
  • The webcam takes in input of the projected keyboard, detects the keys and its centroids, saves in a text file and maps it to the actual key on the keyboard.
  • A red band attached on finger (or any red object) brought in the view of the webcam is detected, its centroids computed and compared with the centroids of the keys stored in the text file.

As can be understood, the project doesn’t permit movement of the setup once the initial stage of mapping is done as movement will change the coordinates and consequently centroids. However, it permits any change initially, that is, the projection can be made as per the need. It can be made on the wall, table, sheet etc. and the distance, alignment can be done as per requirement initially. This movement is not possible with the conventional laser keyboards.

Note: This project is inspired from the SixthSense Technology concept presented by Pranav Mistry (http://www.ted.com/talks/pranav_mistry_the_thrilling_potential_of_sixthsense_technology?language=en).

It is by no means a very well built or efficient program. There is much scope for improvement. One being to develop it further on other platforms using OpenCV.

Comments, suggestions, opinions – all are welcome.


Function 1. [Function 2 and function 3 are called subsequently]

%creating a "videoinput" object 'v'
v=videoinput('winvideo', 1, 'YUY2_640x480');
%Setting the "FramesPerTrigger" value to '1'
v.FramesPerTrigger=1;
%for previewing window of the "videoinput" object 'v'
preview(v);
%start capturing
start(v);
%for warming up of webcam
pause(2.0);

% Get image data from webcam
I=getdata(v);
%displaying the captured image
%writing/storing the captured image in the hard-disk of computer
imwrite(I,'C:\Users\-\-\-\keyboard.jpg'); %filepath

%% Convert to grayscale image
%I=imread('keyboard final.jpg'); 

Igray = rgb2gray(I);

%% Convert to binary image

BW = im2bw(Igray,0.567);

 imshow(BW);

%% Feature Extraction
st = regionprops(not(BW), 'BoundingBox', 'Area', 'Centroid' );  %notBW for detecting black boxes as bounding box typically detects white boxes. Inverting clr
disp(st);
disp(length(st)); % =86 out of which actual number of keys are 53
x=0;              % x is the serial number of centroid
fileID = fopen('centroid.txt','w');

for k = 1 : length(st)
    if(st(k).Area>500&&st(k).Area<15000)   %condition given to select only boxes with above 500pixels in the region and neglect smaller boxes
        thisBB = st(k).BoundingBox;
        rectangle('Position', [thisBB(1),thisBB(2),thisBB(3),thisBB(4)],...     % st(k).Bounding Box = X Y W H
            'EdgeColor','r','LineWidth',2 )
        centroids=st(k).Centroid;
        hold on
        plot(centroids(:,1),centroids(:,2), 'b*') %  blue color
        x=x+1;
        fprintf('Centroid number= %g ',k)
        fprintf('Sno= %g',x);
        disp(st(k).Centroid);   % 53 centroids, X Y

        x_centroid(k) = st(k).Centroid(1);  %extracting coordiantes and saving in a text file
        y_centroid(k) = st(k).Centroid(2);
        fprintf(fileID,'%d %d %8.4f %8.4f\r\n',x,k,x_centroid(k),y_centroid(k));
        hold off
    end
end
fclose(fileID);

delete(v); %deleting the "videoinput" object to avoid filling up of memory

prompt = 'Proceed to Step 2? [Y/N] Check the output. Number of detected centroids should be 53 \n';
str = input(prompt,'s');
if (str=='Y')
    secondpart()
end

Function 2 – secondpart().m

function [] = secondpart()

%% Initialization
redThresh = 0.2; % Threshold for red detection
vidDevice = imaq.VideoDevice('winvideo', 1, 'YUY2_640x480', ... % Acquire input video stream
                    'ROI', [1 1 640 480], ...         %ROI is region of interest specified in [x y width height], width height being the resolution of cam
                    'ReturnedColorSpace', 'rgb');               % rgb colour format required, by default it is Ycbr
vidInfo = imaqhwinfo(vidDevice); % Acquire input video property
%vidInfo.FramesPerTrigger = 40*45;

%Setting properties
hblob = vision.BlobAnalysis('AreaOutputPort', false, ... %  Returns blob of area if set default. we don't require it
                                'CentroidOutputPort', true, ...    % returns centroids of blobs
                                'BoundingBoxOutputPort', true', ...  %returns coordinates of bounding box
                                'MinimumBlobArea', 800, ...
                                'MaximumBlobArea', 3000, ...
                                'MaximumCount', 10);
hshapeinsRedBox = vision.ShapeInserter('BorderColor', 'Custom', ... % Set Red box handling
                                        'CustomBorderColor', [1 0 0], ...
                                        'Fill', true, ...
                                        'FillColor', 'Custom', ...
                                        'CustomFillColor', [1 0 0], ...
                                        'Opacity', 0.4);
htextins = vision.TextInserter('Text', 'Number of Red Object: %2d', ... % Set text for number of blobs
                                    'Location',  [7 2], ...
                                    'Color', [1 0 0], ... // red color
                                    'FontSize', 12);
htextinsCent = vision.TextInserter('Text', '+      X:%4d, Y:%4d', ... % set text for centroid
                                    'LocationSource', 'Input port', ...
                                    'Color', [1 1 0], ... // yellow color
                                    'FontSize', 14);
hVideoIn = vision.VideoPlayer('Name', 'Final Video', ... % Output video player
                                'Position', [100 100 vidInfo.MaxWidth+20 vidInfo.MaxHeight+30]);
nFrame = 0; % Frame number initialization

filename = 'centroid.txt';
delimiterIn = ' ';
format long g;  %to get decimal points as in the text file, otherwise MATLAB formats it
A = importdata(filename,delimiterIn);

%% Processing Loop
while(nFrame < 300)
    pause(0.5);
    rgbFrame = step(vidDevice); % Acquire single frame

  %  rgbFrame = flipdim(rgbFrame,2); % obtain the mirror image for displaying
    diffFrame = imsubtract(rgbFrame(:,:,1), rgb2gray(rgbFrame)); % Get red component of the image
    diffFrame = medfilt2(diffFrame, [3 3]); % Filter out the noise by using median filter
    binFrame = im2bw(diffFrame, redThresh); % Convert the image into binary image with the red objects as white
    [centroid, bbox] = step(hblob, binFrame); % Get the centroids and bounding boxes of the blobs
    centroid = (centroid); % Convert the centroids into Integer for further steps
    rgbFrame(1:20,1:165,:) = 0; % put a black region on the output stream, size of the black region
    vidIn = step(hshapeinsRedBox, rgbFrame, bbox); % Instert the red box
    for object = 1:1:length(bbox(:,1)) % Write the corresponding centroids
        centX = centroid(object,1);
        centY = centroid(object,2);
        disp(centX);
        disp(centY);
        vidIn = step(htextinsCent, vidIn, [uint16(centX) uint16(centY)], [uint16(centX-6) uint16(centY-9)]);  %required to convert in this format for next step
        compare(centX,centY,A)
    end
    vidIn = step(htextins, vidIn, uint8(length(bbox(:,1)))); % Count the number of blobs
    step(hVideoIn, vidIn); % Output video stream
    pause(0.25);

    nFrame = nFrame+1;
end
%% Clearing Memory
release(hVideoIn); % Release all memory and buffer used
release(vidDevice);
% clear all;
%clc;
end

Function 3 – compare().m

function [] = compare(centX,centY,A)

%% Calculating Euclidean Distance of detected point from centroids of Keys
x2=centX;
y2=centY;
dis=zeros(53,1);  %pre-allocating distance array to save time and space. (Suggested by MATLAB HELP)

for p=1:53
    x1=(A(p,3)); % display all rows and their 3rd 4th column (centroids) using a loop
    y1=(A(p,4));
    fprintf('x1=%8.4f y1=%8.4f x2=%8.4f y2=%8.4f',x1,y1,x2,y2)
    dis(p)=(sqrt((x2-x1).^2+(y2-y1).^2));
    %distance(p)=dis.^(1/2);

end

disp(dis);  %storing distance in array 'distance'

%% Search to find shortest distance, Linear Search

num=0;
short=dis(1);  %assuming
for i=1:53
    if(dis(i)<short)
        short=dis(i);
        num=i;
    end
end

fprintf('Serial Number=%g Centroid number= %g Shortest Distance= %g',A(num,1),A(num,2),short); 

%% Compare the key pressed and show it
       % SNo. KeyValue
keyMap ={1,'Tab';
        2,'CapsLock';     %we are using cell ie, {} instead of char array [] to put strings in this array
        3,'Shift';        % char array[] allows only for identical sized matrix.
        4,'1';
        5,'Q';
        6,'2';
        7,'A';
        8,'Z';
        9,'W';
        10,'S';
        11,'3';
        12,'X';
        13,'E';
        14,'D';
        15,'4';
        16,'C';
        17,' ';
        18,'R';
        19,'F';
        20,'5';
        21,'V';
        22,'T';
        23,'G';
        24,'6';
        25,'B';
        26,'Y';
        27,'H';
        28,'7';
        29,'N';
        30,'U';
        31,'J';
        32,'8';
        33,'M';
        34,'I';
        35,'K';
        36,'9';
        37,'<';         38,'O';         39,'L';         40,'0';         41,'>';
        42,'P';
        43,';';
        44,'-';
        45,'?';
        46,'[';
        47,'@';
        48,'+';
        49,'Shift' ;
        50,']';
        51,'#';
        52,'Backspace' ;
        53,'\n'};

 keypress=keyMap(num,2);  %this returns a cell that is the Key Name as specified aboce
 fprintf('\nKey Pressed is');
 fileID = fopen('output.txt','a');
 fprintf(fileID,'%s',keypress{1});  %keypress{1} to access contents of the 1-D cell
 disp(keypress);
end

matching-key-with-centroid

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s